๐๐ ๐๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐ถ๐ป๐ด ๐๐ต๐ฎ๐ฝ๐๐ฒ๐ฟ ๐ฏ: ๐ช๐ต๐ ๐๐๐ฎ๐น๐๐ฎ๐๐ถ๐ป๐ด ๐๐ ๐๐ ๐๐ฎ๐ฟ๐ฑ๐ฒ๐ฟ ๐ง๐ต๐ฎ๐ป ๐๐ ๐๐ผ๐ผ๐ธ๐
One of the biggest takeaways from Chapter 3 of AI Engineering was realizing that building an AI model is only part of the challenge. Figuring out ๐ต๐ผ๐ ๐๐ผ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ฒ ๐ถ๐ ๐ณ๐ฎ๐ถ๐ฟ๐น๐ ๐ฎ๐ป๐ฑ ๐ฎ๐ฐ๐ฐ๐๐ฟ๐ฎ๐๐ฒ๐น๐ can be just as difficult. With traditional software, it's usually easy to tell whether something works. If a calculation is wrong or a test fails, you know there's a bug. But AI doesn't always work that way. A model can generate multiple reasonable answers to the same question, making it much harder to determine which one is actually better. That made me think: ๐๐ผ๐ ๐ฑ๐ผ ๐๐ฒ ๐ธ๐ป๐ผ๐ ๐ถ๐ณ ๐ฎ๐ป ๐๐ ๐บ๐ผ๐ฑ๐ฒ๐น ๐ถ๐ ๐ฎ๐ฐ๐๐๐ฎ๐น๐น๐ ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ถ๐ป๐ด? ๐๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐ ๐ก๐ฒ๐ฒ๐ฑ ๐๐ผ ๐๐ฒ๐ฒ๐ฝ ๐๐๐ผ๐น๐๐ถ๐ป๐ด Reading this section made me realize how difficult it is for evaluation benchmarks to keep up with the pace of AI development. The chapter explains that GLUE (General Language Understanding Evaluation) was introduced in 2018 to measure how well language models performed on common natural language tasks. But within about a year, models had already become so good at it that researchers introduced SuperGLUE in 2019 as a more difficult benchmark. GLUE evaluates tasks such as: Question answering Sentiment analysis Sentence similarity Text classification The chapter also mentions newer benchmarks like: SuperGLUE MMLU (Massive Multitask Language Understanding) MMLU-Pro Each one was introduced because the previous benchmark was no longer challenging enough. What I found interesting is that a model getting a higher benchmark score doesn't always mean it understands language better. Sometimes it simply means the model has become very good at solving that particular benchmark. ๐จ๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ๐ถ๐ป๐ด ๐๐ป๐๐ฟ๐ผ๐ฝ๐ ๐ฎ๐ป๐ฑ ๐ฃ๐ฒ๐ฟ๐ฝ๐น๐ฒ๐ ๐ถ๐๐ Another section I really enjoyed was the explanation of entropy and perplexity. The chapter explains entropy as a measure of how much information a token carries and how difficult it is to predict the next token in a sequence. Perplexity measures uncertainty. If a model is very uncertain about what comes next, its perplexity will be higher. If