Insights5 May 2026Updated 26 October 20265 min read

AI-Augmented Engineering: Amnesia, Invented APIs and Team Knowledge Silos

Three abstract glowing clusters representing amnesia, hallucinated endpoints, and tribal silos in AI-augmented engineering.

In METR's July 2025 randomised study, allowing AI increased task completion time by 19% for the experienced open-source developers studied. That is a reason to measure the work, not a verdict that AI makes every engineering team slower. The practical question is where to look when promising suggestions are not translating into completed changes.

We recommend checking three places: what survives between sessions, whether generated code uses real interfaces, and whether the relevant team knowledge is accessible. These are failure modes to investigate, not three proven causes of METR's result or a diagnosis of your team.

Start with the limits of the evidence

METR recruited 16 developers and randomised 246 real issues to allow or disallow AI assistance, with AI use primarily involving Cursor Pro and Claude 3.5/3.7 Sonnet. Participants expected a 24% speed-up beforehand and still estimated a 20% speed-up afterwards, despite the measured slowdown in that setting. The authors explicitly warned against generalising to most development work or future systems.

The picture was already changing by METR's February 2026 follow-up, which reported signs of speed-up but said selection effects and time-measurement problems made the new experiment unreliable for estimating the current effect. Do not turn either study into a prediction for a different team, task mix or toolchain. Instead, use the three checks below to frame an investigation and test any proposed improvement locally.

Failure mode 1: session amnesia

A row of glowing processors linked by arcs of light, the last one dark and unlinked

Treat session amnesia as a handover test: can a fresh session recover the current objective, decisions, unresolved questions and next verification step without a colleague reconstructing the conversation? This is not a claim that every assistant forgets everything between chats. MemGPT's 2023 research describes a system that moves information between limited working context and external memory, including experiments with multi-session conversations. It is evidence of an existing memory design, not evidence that a coding team will ship faster.

For a repository, we recommend a small restart record tied to the task and code revision rather than an indiscriminate transcript dump. Record what changed, which tests actually ran, what remains unverified and where the supporting files or decisions can be found. Separate an agreed decision from a suggestion, and an observed test result from an agent's intention to run it. Keep credentials and material the next reader is not authorised to see out of the record.

Choose checkpoint frequency according to the work's recovery needs, then test interruption and restart rather than assuming a session-end hook is sufficient. Ask the fresh session to state its next action and cite the record behind it before allowing further edits. If it repeats a rejected approach or treats an unrun test as passed, repair the handover format before adding more memory. Our proposed success measure is less repeated orientation without an increase in incorrect carry-forwards.

Failure mode 2: invented APIs

Treat an endpoint, table column or imported function in generated code as a claim to verify, not evidence that the interface exists. Research by Zhang and colleagues distinguishes API knowledge conflicts from project-context conflicts, including dependencies on undefined project entities. That supports investigating interface mistakes, but it does not establish their frequency in your repository or prove that an index eliminates them.

Before implementation, require the agent to locate the route definition, schema, exported symbol or versioned dependency documentation that supports each integration it proposes. For an HTTP call, ask for the method, path, authentication expectation and relevant request and response shapes. Where the task genuinely needs a new interface, label it as proposed and get the contract agreed rather than presenting it as existing behaviour.

A mechanically generated repository inventory is one possible starting point, not a replacement for inspecting the code. Give it a revision or refresh timestamp, and state which languages, repositories and generated surfaces it covers. Treat an empty search as unresolved until the relevant scope has been checked, rather than as proof that something does not exist. Then run the applicable type checks, contract tests and integration tests against the intended version.

The review question should be concrete: which source establishes this interface, and which executed check exercises the call? Keep a separate record of integration defects caught during review so that a tidier prompt is not mistaken for a verified improvement.

Failure mode 3: inaccessible team knowledge

Sealed glass towers each holding light, with a small node outside unable to reach any of them

Use a hypothetical notification service to test the third failure mode: the code shows a retry loop, but only its maintainer knows why a particular customer-facing action must not be retried. Ask whether another engineer or an authorised agent could find that constraint and its rationale before changing the loop. The issue to investigate is not missing conversation history alone, but a missing or inaccessible explanation of a decision.

In chapter 2 of Continuous Architecture in Practice, Erder, Pureur and Woods recommend making architectural decisions explicit artefacts and recording constraints, alternatives, rationale and accountability. We recommend applying that discipline to the decisions an agent needs, with a named owner, status, date and links to the relevant implementation or incident record. Ask the maintainer to review the account before treating a generated summary as team guidance. Retain the underlying source so a reviewer can distinguish an agreed constraint from an assistant's interpretation.

Make access part of the test: a document being searchable by its author does not demonstrate that the intended reader can retrieve it. Use an authorised reader to check discovery, permissions and whether superseded guidance is clearly marked. Do not promise that a growing wiki removes dependence on a specialist or becomes authoritative after a fixed number of months.

Improve one boundary, then measure

Choose one recurring class of work and establish a baseline before changing its context or review process. Track time through implementation and review, along with rework and defects, using definitions agreed before the trial. Record the extra effort spent maintaining memory, checking contracts and curating decisions as part of the intervention's cost. Keep task mix and tool versions visible, and distinguish a useful local comparison from a causal experiment.

Our related engineering account of team-context infrastructure explores the implementation approach, not independent proof that it will improve another team's delivery. The next step is a bounded trial with an acceptance test at each boundary: restart correctly, call a verified interface and retrieve the decision that constrains the change. If you want help designing that trial, let's have a chat.

Sources

Want this in your team?

Tell us what you're building and where you want to get to, and we'll think it through with you.

Let's have a chat Follow by RSS →

All insights →