Software Engineering
Why code by hand when AI is faster?
- It’s fun and rewarding, especially when speed isn’t the most important factor.
- When you write code, there’s a creative aspect that you lose with coding tools (especially when you don’t read the code). You iterate and define while you’re building it.
- By doing things by hand, you learn things (sometimes new topics or ideas) you need to do the work.
- There is an obsession with speed in software development. How much you build and how fast you build it doesn’t necessarily corresponding to quality, correctness, or what can be sold.
- AI-built stuff is disposable because you didn’t “sweat” over it.
- Write critical code by hand so that you understand it more deeply. AI doesn’t think: It emits a plausible solution to the given goal with the information it has.
- Treat coding by hand as a way to be curious, try things, learn, and reconnect with what got you excited about software development in the first place.
- AI is pitched as a revolution (how we work and learn), but this frames learning as something you can shortcut.
- TL;DW — every time AI is used as a shortcut for learning, it wasn’t helpful. When AI is used to create friction to help people learn, the results were better.
- We learn through friction. The struggle isn’t what’s getting in the way; the struggle is the learning.
- Understanding something isn’t the same as being able to do it. See “Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom” (Deslauriers et al. PNAS 2019). Course material delivered by lecture vs having to work through it — the students that had to actively work through it feel like they learn less but actually learned more. When something is explained smoothly, it feels like learning (Dunning-Kruger effect).
- Another study split 1,000 students into three groups: GPT, tutor that gave hints, textbook + notes. The GPT group got 48% more problems right compared to the control, and the tutor group got 127% more right. But when each group took the exam with no AI, the GPT group did 17% worse, and the tutor group was about the same as control. The GPT group didn’t think they did any worse.
- Anthropic’s How AI assistance impacts the formation of coding skills. 52 Python devs that hadn’t used a particular library, half used AI and half didn’t. The AI group averaged 50% on a quiz compared to 67% without AI. When it came to debugging, the AI group did worse. Although the AI group finished quicker, the results were not statistically significant (read: AI = not that much faster and devs are worse at understanding). The ones who did well when using AI asked more conceptual questions — e.g,. why does this work?
- Developers have had tools like Stack Overflow, but there was friction. What do I search for? Is this a good fit for my problem? How do I adapt this code to my code?
- You can have AI make flashcards for you to help you practice retrieval, but making the flashcards yourself is an extra layer of learning (what to make cards for, what goes on the cards).
- Harvard’s study “AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting” showed that an AI tutor (not giving answers) helped students learn more and in less time. World Bank’s “From chalkboards to chatbots: Transforming learning in Nigeria, one prompt at a time” showed students gained two years of learning (this had teachers guiding students).
- AI isn’t always right (unlike a calculator) and can be confidently wrong. You need engineering skills to handle the situation where the AI code doesn’t work. Anthropic’s article was worried about this, because if you’re supervising AI, you need the skills to catch the mistakes.
- Microsoft’s “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers”. The more people trust AI the less they thought about the results it gave them.
- The less you know, the less useful AI is because you can’t distinguish good from bad. It just sounds right, so it must be right.