AI coding agent evaluation should measure whether a change is correct, reviewable and safe to ship. Count accepted outcomes and the effort required to reach them, not just generated code or fast demonstrations. Recent research offers a useful reason to evaluate tasks separately rather than assuming one productivity result applies everywhere.

What does the latest research actually say?

OpenAI published Research acceleration: The jagged frontier on 18 November 2026. It examines internal research activity and describes uneven benefits and remaining agent limitations. Its results concern a particular organisation and research setting; they are not a universal productivity forecast for a small development team.

The useful business question is narrower: which tasks can your team delegate successfully, under its own constraints? A tool that handles an isolated bug well may struggle when an undocumented integration, a migration or an ambiguous request changes the problem.

This article focuses on evaluating that boundary. For operational logging after rollout, use the separate AI agent monitoring guide.

Define completion before the agent starts

Write down what a maintainer would need to accept the work. Include the expected behaviour, constraints and evidence. If the task affects an enquiry form, the requirement might include successful delivery, accessible errors and preservation of the existing destination. Use the downloadable AI agent evaluation record to keep task, reviewer and outcome evidence together.

Keep the acceptance conditions available to both the person assigning work and the reviewer. Otherwise, a demonstration can look impressive while solving a different problem from the one the business needed.

An illustrative brief could be: “Repair duplicate CRM records from a repeated submission. Preserve the first source, retain the audit history, and prove that a normal new enquiry still creates one record.” That defines an outcome without dictating the implementation.

Select tasks that expose different weaknesses

Choose representative work from your own backlog or synthetic fixtures. Do not give an agent uncontrolled production access simply to make an evaluation feel realistic.

Task family What it reveals Example acceptance evidence
Local bug fix Understanding a bounded failure Reproduction fails before and passes after
Integration repair Handling external assumptions Controlled success, timeout and retry cases
Refactor Preserving callers and behaviour Existing contract remains intact
UI change Matching user-visible requirements Browser verification at required breakpoints
Ambiguous request Recognising missing decisions Useful clarification before risky work

Include cases where the correct action is to stop, ask a question or report a blocker. Finishing quickly is not useful when the agent silently invents a missing requirement.

Compare human review effort, not just execution time

Track the full path from assignment to acceptance. Record agent time, reviewer time, repair rounds and whether a separate maintainer could understand the final change. Keep repeated attempts in the record instead of reporting only the best run.

For example, an agent may prepare a patch quickly but leave a long review burden because it changed unrelated files. Another may take longer while producing a small, well-explained repair. Which is better depends on your team's bottleneck and the work's consequences.

Avoid turning this comparison into a claim that your team has replicated a vendor's benchmark. The test set, permissions, environment and reviewer criteria should be reported as part of your own evaluation.

Make tool permissions part of the test

Evaluation should include how an agent uses its access. Can it read the required files without collecting unrelated information? Does it request an appropriate decision before an irreversible action? Does it treat material inside a repository or web page as data rather than authority?

The prompt-injection guide explains why external instructions are a separate risk from ordinary coding mistakes. A model can pass functional tests while making an unacceptable access decision.

Use a disposable environment for cases involving writes or external systems. Provide fake credentials and synthetic records where possible. Prevent test messages, charges or scheduled jobs from reaching real customers.

Keep a holdout and review what changes

Reserve some tasks that are not used while adjusting prompts or workflow instructions. Recheck them when the model, tool configuration or environment changes. Otherwise, the team can improve performance on a familiar demonstration without learning whether the process generalises.

Keep a dated evaluation record with the model, task, configuration, result and reviewer decision. Link any deployment approval to that evidence. A release should not rely on a successful experiment from a materially different setup.

Frequently asked questions

Is passing the test suite sufficient?

Only if the suite covers every relevant requirement, which is uncommon. Review the change and verify the behaviour that matters to users and operators.

Should we rank agents by lines of code written?

No. Code volume does not establish correctness or maintainability. Measure accepted work and the total effort required to achieve it.

Can a research productivity result predict our savings?

Treat it as a reason to investigate, not a forecast. Your tasks, tooling, reviewers and operating constraints determine the result you need to measure.

Put the evaluation beside the work

Start with a bounded backlog and a clear acceptance record. RKWS can help scope software development and automation around observable outcomes. The goal is a process your team can inspect and improve, with evidence kept alongside each change.