Amazon congratulates itself for AI code that mostly works

Amazon Web Services on Tuesday took a moment to pat itself on the back for being thought of inside the box, specifically, the upper right-hand square that's part of Gartner's trademarked Magic Quadrant.

This particular set of boxes maps the IT consultancy's view of AI code assistants. AWS is understandably chuffed to land a spot in the "leaders" quadrangle for its Q Developer service, alongside GitHub Copilot, GitLab Duo, and Google Cloud's Gemini Code Assist.

The other three boxes are reserved for "visionaries" (less ability to execute compared to leaders), "challengers" (executing just fine, but vision impaired), and "niche players" (lagging in execution and vision).

As with the security industry, where companies have filed legal complaints after their software is labeled spyware or malware, IT firms placed in less desirable boxes have challenged the designation in court.

NetScout Systems, for example, filed a lawsuit against Gartner in 2014 over its placement in the "challenger" box. The case was dismissed in 2017, with Gartner emphasizing that its rankings are not for sale. Another firm, ZL Technologies, sued Gartner in 2009 over its "niche player" designation, only to lose in court and on appeal.

So it's perhaps understandable that AWS would crow about recognition from Gartner, as it has been doing on a regular basis for years.

"We believe this Leader placement reflects our rapid pace of innovation, which makes the whole software development lifecycle easier and increases developer productivity with enterprise-grade access controls and security," said Channy Yun, principal developer advocate for AWS.

However, "leader" doesn't necessarily refer to customer count for Q Developer. An Amazon spokesperson said the company does not disclose specific adoption numbers, which generally means they're less than the competition. Microsoft's GitHub, on the other hand, happily brags about Copilot usage. As of February, there were "over 1.3 million paid GitHub Copilot subscribers."

• White House thinks it's time to fix the insecure glue of the internet: Yup, BGP
• Copilot for Microsoft 365 might boost productivity if you survive the compliance minefield
• Microsoft's Inflection acquihire is too small to matter, say UK regulators
• Admins wonder if the cloud was such a good idea after all

Amazon CEO Andy Jassy last month offered a different way to assess the success of Q Developer. He said Amazon had used the generative AI coding service to modernize its production Java systems by updating the code to Java 17.

"The benefits go beyond how much effort we've saved developers," Jassy said. "The upgrades have enhanced security and reduced infrastructure costs, providing an estimated $260 million in annualized efficiency gains."

Amazon claims that Q Developer's agents for code transformation helped Amazon migrate 30,000 production applications from Java 8 to Java 17, saving over 4,500 years of development work, in addition to the $260 million in performance improvements cited by Jassy.

Jassy did not mention how often AI code suggestions need correction or the costs associated with these fixes. Last year, researchers evaluated ChatGPT, GitHub Copilot, and Amazon CodeWhisperer (now Q Developer), and found that the AI helpers generated correct code 65.2 percent, 46.3 percent, and 31.1 percent of the time, respectively.

Asked about this, an Amazon spokesperson said: "Amazon Q Developer has the highest reported code acceptance rates in the industry for assistants that perform multi-line code suggestions – with BT Group reporting they accepted 37 percent of Q's code suggestions and National Australia Bank reporting a 50 percent acceptance rate."

A test suite for small AI models called CanAiCode doesn't mention Q Developer and ranks Anthropic's Claude 3 Opus at the top of the models evaluated.

Gartner, in its report on AI code assistants, projects that by 2028, 90 percent of enterprise software engineers will use the technology, up from less than 14 percent as of early 2024.

Nonetheless, academics argue that AI will not render software engineering obsolete. In a recent preprint paper, Carnegie Mellon computer science professors Eunsuk Kang and Mary Shaw conclude: "Generative AI is now eagerly inflating our aspirations, but its capability is not yet trustworthy and robust enough to be part of the stable core of [software engineering] methods. AI is already demonstrably useful under careful supervision, and we can expect its utility for routine programming tasks to improve quickly." ®