Your Best AI Engineer Might Have the Worst Metrics | Sonar CTO Andrea Malagodi

Evaluation and Summary

8.4/10

Andrea Malagodi, CTO at Sonar, joins Conor Bronsdon to explore why simple metrics for measuring AI engineer productivity—like cost per pull request—can be dangerously misleading. The episode opens with a concrete case study: Malagodi's finance team flagged two engineers with opposite metrics. One had 500+ PRs at low cost per PR but was actually running a personal agent factory that produced mostly tests and validation (two-thirds of output). The other had high spend and few PRs but was tackling genuinely hard problems requiring long, multi-turn AI reasoning sessions. Judged on cost alone, the first looks efficient; in reality, the second was doing more valuable work. The conversation traces a history of bad productivity proxies in software—lines of code, story points, token leaderboards—and argues that the people who *removed* code were often most important. Malagodi outlines Sonar's "guide, verify, solve" agent loop: Vortex for codebase context, algorithmic plus LLM-based PR analysis, a remediation agent for legacy debt, and Hunter for security vulnerabilities. Key technical insights include why breaking agent work into small pieces matters (a five-day session producing 400,000 lines leaves you blind to what's in your codebase), how to set up disagreement between agents using orchestrators with personas for engineering, product, and quality, and why mixing model families beats picking one. The episode covers enterprise rollout challenges: unified cost views for finance, 30% savings from cutting repeated context reads, and why tiny budgets ($50/week) are operationally hard. Practical governance topics include rotating triage duty to protect deep work, building organizational skills to sort hundreds of CVEs (many false positives), and defense-in-depth for AI-written code. The episode closes with personalized software for teams and individuals. Throughout, Malagodi grounds abstract principles in real engineering constraints: what works at scale, how to avoid false economies, and why human judgment remains essential when metrics diverge from actual value creation.

Verdict: A sharp, practically grounded conversation that dismantles a dangerous metric and offers real alternatives; essential for engineering leaders deploying AI at scale.

Technology rubric

Technical accuracy 4/5 20%

Concrete case studies and product details (Vortex, Hunter agent, guide-verify-solve loop) are grounded; no obvious errors, though depth is limited by show notes alone
Insight beyond the headlines 5/5 20%

Core insight—that cost-per-PR metrics hide real value and can incentivize the wrong behavior—is original and forward-looking; the history of failed proxies and why removals matter adds depth
Clarity for non-experts 4/5 15%

The 500-PR vs. high-spend engineer framing is immediately clear; agent terminology and enterprise concepts are accessible without dumbing down
Substance 4/5 15%

Dense with specific examples, technical architecture (guide-verify-solve, agent personas, model mixing), and operational lessons; covers governance, cost, and quality without filler
Delivers on its title 5/5 15%

Title promises that best engineers may have worst metrics; episode delivers exactly that thesis with detailed case study, explores why, and offers frameworks to measure better
Value for time 4/5 15%

57 minutes covering metrics, agent architecture, enterprise rollout, and governance; well-chaptered and substantive, though some sections (e.g., model mixing, CVE triage) could be tighter

How we score Technology

Worth it if you lead engineering teams, manage AI spend, or care about measuring developer productivity beyond vanity metrics.

Content flags

Ad-heavy Three sponsor reads (Svix, Walrus, Inngest, G2i) take up meaningful time in a 57-minute episode

#ai-agents #engineering-metrics #code-quality #enterprise-ai #productivity #governance

Based on the episode description. AI-generated; it can be wrong.

Show notes

Andrea Malagodi is CTO at Sonar, which builds AI code verification and governance tools. When finance asked about the size of his team's AI bills, he calculated cost per PR. At one extreme was an engineer with more than 500 PRs in a short period and a low cost per PR. At the other was an engineer with very high spend and very few PRs. Judged on cost alone, the first engineer wins and the second looks like a problem. Malagodi looked closer. The first had built a personal agent factory, with specification, design review, documentation records, coding, verification and functional testing, and about two-thirds of the output was tests and validations. The second was working on a hard problem that needed the AI to reason through long, multi-turn sessions and didn't reduce to a line count. Numbers alone tell you something, but not which engineer was doing the more valuable work. We cover: Why cost per PR is becoming the industry's standard AI spend metric, and how Malagodi's 500-PR engineer and high-spend engineer broke it The history of bad proxies, from lines of code to story points to token leaderboards, and why Malagodi says the people who removed code were often the most important Sonar's guide, verify, solve loop: Vortex for codebase context, algorithmic plus LLM-based PR analysis, a remediation agent for legacy debt, and the Hunter agent for security vulnerabilities Why Malagodi breaks agent work into small pieces, and why a five-day session that produces 400,000 lines of code leaves you with no idea what is in your codebase Setting up disagreement between agents with an orchestrator and personas for engineering, product management and quality, plus Conor Bronsdon's cross-model-family review setup Rolling AI tools out at enterprise scale: one cost view for finance, up to 30% savings from cutting repeated context reads, and why budgets as small as $50 a week are hard to work with Rotating a triage duty to protect deep work, including how Sonar's teams built skills to sort through hundreds of reported CVEs, a number of them likely false positives New episodes, the ideas behind them and Conor's essays land in the Chain of Thought newsletter first. Subscribe: https://newsletter.chainofthought.show/ Chapters: (0:00) Human reviewers rubber-stamp big AI changes (2:43) Learning the craft alongside AI tools (9:09) From lines of code to PR counts (12:17) Spreading top engineers' setups org-wide (16:50) The guide, verify, solve agent loop (22:51) Specialized agent lanes and multitasking limits (26:51) Clear asks and guardrails for agent coordination (30:29) Building disagreement between agents (34:59) Mixing models instead of picking one (39:46) Rolling out AI tooling at enterprise scale (46:19) Defense in depth for AI-written code (50:52) Protecting time for incident investigations (54:08) Personalized software for teams and individuals Links from the episode: Sonar Vortex research: https://www.sonarsource.com/blog/cut-your-coding-agents-cost-with-sonar-semantic-code-navigation/ SonarQube Hunter Agent: https://www.sonarsource.com/products/sonarqube/hunter-agent/ Connect with Andrea Malagodi: LinkedIn: https://www.linkedin.com/in/malagodia/ Sonar: https://www.sonar.com/ Connect with Chain of Thought host Conor Bronsdon: Newsletter: https://newsletter.chainofthought.show/ Twitter/X: https://x.com/ConorBronsdon LinkedIn: https://www.linkedin.com/in/conorbronsdon/ YouTube: https://www.youtube.com/@ConorBronsdon More episodes: https://chainofthought.show Thanks to Svix, presenting sponsor of season four of Chain of Thought. Svix delivers billions of reliable webhooks for startups and the Fortune 500. Get started at https://link.svix.com/cot . Qualified startups get $12,000 in credits, and YC companies get $50,000. Thanks to Walrus, presenting sponsor of season four of Chain of Thought. Walrus Memory gives AI agents portable, verifiable memory that carries context across apps, sessions, and other agents. Get started at https://walrus.xyz/cot Thanks to Inngest, presenting sponsor of season four of Chain of Thought. Agents in production run long - they call models and wait on APIs and people. But the longer agents run, the more they break. Inngest handles that with durable execution. You build your agent as steps in TypeScript, Python, or Go. When a step fails, Inngest retries it with exponential backoff, and completed steps are saved and skipped. Try it out: https://inngest.link/cot-pod Thanks to G2i for sponsoring this episode - for over a decade, they vetted and placed engineers at other companies, from startups to FAANG. Two years ago, they turned that same judgment inward, building their own bench to review RL environments, evals, and training data that models are trained on. Get access: https://fandf.co/3SFxVm6

Episode page

More from Chain of Thought | AI Agents, Infrastructure & Engineering

Chain of Thought | AI Agents, Infrastructure & Engineering ·1w ago ·38 min

Switching Models Shouldn't Mean Starting Over | Walrus's Kimberly Logan

Chain of Thought host Conor Bronsdon interviews Kimberly Logan, Head of Product at the Walrus Foundation, about portable and verifiable memory for AI agents. The core problem: if an agent's memory lives inside one model provider's infrastructure, switching models or harnesses means losing that context and starting…

Worth it if you build or deploy AI agents, care about vendor lock-in, or want to understand why decentralized storage matters for agent infrastructure.