You'll never think of it! Repeating this simple command makes AI generate code 100 times faster

Recently, Max Woolf, senior data scientist at BuzzFeed, conducted an experiment to explore the effect of repeatedly requesting AI to improve code. In the experiment, he used the Claude3.5 language model and gave a classic programming challenge: write Python code to find the maximum and minimum value among one million random numbers whose sum is 30. difference.

In the initial version, the code generated by Claude ran in 657 milliseconds. However, as Woolf kept typing the simple command "write better code," the resulting code was reduced to just 6 milliseconds, a 100-fold improvement in performance. Not only is this result impressive, AI also shows unexpected changes in the process of defining "better code."

On his fourth request to "write better code," Claude unexpectedly transformed the code into an enterprise application-like structure, adding some typical enterprise features that Woolf had not asked for. This suggests that the AI ​​may be tying “better code” to “enterprise-grade software,” reflecting the knowledge it absorbs during its training.

Developer Simon Willison analyzed this phenomenon of iterative improvement and believed that the language model looks at the code from a new perspective with each new request. Although each request contains the context of the previous conversation, Claude analyzes the code as if it were seeing it for the first time, allowing it to continually improve.

However, in her attempts to make more specific requests, Woolf found that while this resulted in better results faster, there were still some subtle bugs in the code that needed humans to fix. Therefore, he stresses, precise cue engineering remains crucial. While simple follow-up questions can initially improve code quality, targeted hint engineering can lead to significant performance improvements, albeit with a corresponding increase in risk.

Notably, in this experiment, Claude skipped some optimization steps that human developers take for granted, such as removing duplicates or sorting numbers first. In addition, subtle changes in the way questions are asked can significantly affect Claude's output.

While these performance improvements are impressive, Woolf reminds us that human developers are still essential in validating solutions and troubleshooting. He pointed out that although the code generated by AI cannot be used directly, its ability to suggest ideas and tools deserves attention.