Three gaps separate a convincing answer from an experience worth learning from.
A company's online checkout stops working. An AI coding assistant identifies a possible bug, writes a patch, and explains why it should work. Another AI reviews the code and agrees. The automated tests pass.
The assistant announces: “The checkout problem is fixed.”
But customers still cannot complete purchases. The test suite never exercised a particular failure in the connection to an external payment service.
The code change may be reasonable. The review may be competent. The tests may genuinely have passed. Yet none of those facts establishes that customers can now complete the task they came to the site to perform.
A green test should be recorded as a green test. It should not silently become a record of a successful customer purchase.
And suppose purchases do start succeeding after the patch. What should the AI learn? Perhaps its patch solved the problem. Perhaps the payment service recovered independently. Perhaps another engineer changed something at the same time. Even a genuine improvement in the world does not, on its own, identify the action that caused it.
For a continuing AI system, moving from an action to a useful lesson involves three distinct gaps: from a claim to an observation, from an observation to a cause, and from a cause to the conditions under which the lesson remains valid.
First gap: Did the claimed result actually occur?
A checkout repair produces at least four kinds of evidence. The assistant proposes a solution. A reviewer provides an assessment. An automated test yields a test result. Customers attempting payment generate evidence about the actual task outcome.
All four matter, but they answer different questions.
An AI reviewer can detect mistakes in code or reasoning before deployment. It cannot establish what happened in production merely by agreeing that a patch looks correct. Two reviewers may share the same incomplete assumptions. Ten similar explanations do not become ten independent observations.
An executable test is stronger evidence of a tested behavior than an approving paragraph. But its result is bounded by the conditions it exercises. A passing unit test can coexist with a failed end-to-end purchase; conversely, a purchase can succeed even while a particular test fails for an unrelated reason.
Existing research already takes evaluation seriously. SWE-bench, introduced by Jimenez and colleagues, evaluates proposed changes against executable tests for software issues drawn from real repositories. Lightman and colleagues investigated process supervision in mathematical reasoning, showing why feedback on intermediate steps can matter. Yet neither a process assessment nor a fixed test suite automatically establishes that a customer's task succeeded under its actual conditions.
In the checkout example, the business outcome might be expressed as the proportion of eligible checkout attempts that complete payment successfully, measured with a defined denominator and observation window. The metric is still not identical to every customer's experience, but it is closer to the task than an assertion that the code looks right or that a test finished green.
The point is not that AI assessment is worthless. It is that a statement about an effect and evidence that the effect occurred must retain different evidential standing.
The previous essay asked whether an AI can recognize missing information before making a decision. Here the question comes after action: what does the arriving evidence actually support?
Second gap: Did our action cause the observed result?
Imagine that the checkout completion rate rises after the patch. The improvement is real. Does it establish that the patch caused it?
No. The payment provider could have restored service during the same period. A second engineer could have deployed a separate fix. Or the mix of users and payment methods could have changed. A rising rate may reflect one of those changes, several together, or the patch itself.
Now reverse the situation. The completion rate does not rise. That does not prove the patch was ineffective. It could have repaired one genuine defect while an independent outage continued to block payments. Observed improvement does not automatically prove a particular fix worked; lack of aggregate improvement does not automatically prove it failed.
What happened is one question. Why it happened is another.
This problem has an established name in reinforcement learning: credit assignment—determining which actions or earlier states deserve credit for rewards or other later outcomes. Sutton and Barto's textbook develops learning from delayed consequences as part of a mature research field. The checkout example also raises a broader causal-inference problem: an observed association between a deployment and a subsequent change does not, without suitable evidence, identify the effect of that deployment. The two problems overlap but are not identical.
Engineering has well-known ways to strengthen causal evidence: controlled A/B comparisons where appropriate, staged rollouts, reversible changes, comparisons of relevant failure modes, and careful examination of what else changed. Kohavi, Tang, and Xu's work on trustworthy online experiments documents why experimental design, measurement quality, and interpretation matter. Such controls have their own limitations; a rollback, for example, may establish very little if another service has changed in the meantime.
The practical challenge for a long-lived AI is often that evidence arrives late, from several sources, and mixed with other events. A code repository reports one thing, tests another, payment logs another, and customers may report outcomes only after a delay. The system should not treat the most recent favorable event as automatic credit for its last action.
This is also why a numerical success metric is not a substitute for interpretation. Suppose a support assistant is rewarded for closing tickets. If it learns to mark unresolved requests as closed, its measured score improves while customers remain dissatisfied. This illustrates specification gaming, discussed by Krakovna and colleagues: optimization can exploit a measure that does not faithfully represent the intended result.
The link is important: if a system misreads a proxy as proof of success, optimization pressure can reinforce the mistake. The wrong conclusion is no longer just a bad report of the past; it becomes a reason to repeat the wrong behavior.
Third gap: How far does the lesson apply?
Suppose careful testing now establishes that the patch really corrected the fault for one payment provider in one operating configuration. The AI has good evidence about a real effect.
Should it apply the same patch to every checkout failure?
Not necessarily. A second provider may use a different interface. Another region may have different authentication rules. A failure that looks similar in the user interface may arise from a database timeout rather than the payment connection. The same fix might become inappropriate after a software upgrade.
A justified causal conclusion is not yet a universal rule.
To learn usefully, a system needs to preserve the conditions under which the evidence was obtained, not just a simplified slogan such as “this patch fixes checkout failures.” Those conditions might include the service version, the payment path, the observed failure signature, the tests performed, and known exceptions.
There is a difficulty that only becomes visible later: when recording an experience, we may not know which seemingly minor conditions will matter next time. A future failure can expose a dependency that nobody thought important enough to mention. The previous essay asked whether an AI can notice an unmentioned variable before a decision. Here the same problem returns across time: can the system notice that an earlier lesson depended on a condition it did not originally recognize?
The aim is not to treat every detail as equally important or to claim that any record can capture the entire environment. It is to avoid extending a successful intervention beyond what its evidence supports—and to remain open to revising that boundary when new evidence arrives.
This question connects to established research on generalization and distribution shift. A model may succeed on one distribution of problems and perform poorly when relevant conditions change; Zhou and colleagues' survey of domain generalization reviews this broader challenge. In causal inference, the related problem of external validity or transportability asks when a causal result can be extended to another setting; Pearl and Bareinboim (2014) develop a formal approach to the latter problem. These are established research areas, not new names for an idea invented here.
For a continuing AI, the test is practical. When the next checkout error arrives, can it recognize what matches the earlier case, what differs, and what still requires investigation? Can it use an earlier result without turning one verified experience into an unrestricted rule?
Improvement should ultimately be demonstrated through better behavior on later, appropriately related problems, not merely through a fluent account of what happened last time. A system may store the right explanation and still fail to act on it when it matters.
Testing the three gaps without exposing real customers
The checkout story can be turned into a controlled evaluation. A safe test environment can reproduce payment success and failure without putting real transactions or users at risk. Learning from consequences does not require causing harmful production failures.
First, keep the proposed patch and the original test results the same, but independently vary the simulated end-to-end checkout outcome. Does the system distinguish a passed test from a customer task that succeeded or failed?
Next, introduce competing causes. One scenario contains an effective patch. Another has an ineffective patch but an independent payment-service recovery. A third contains a useful patch whose effect is masked by a continuing separate outage. In a stronger variant, the competing service event appears in ordinary timestamped logs without a label announcing that it is the confounder. Does the system notice and investigate it, rather than simply repeat a supplied causal verdict?
A useful evaluation should not demand certainty from evidence that cannot identify a cause. In some cases, the justified conclusion will be that the cause remains unresolved. Where safe and feasible, controlled intervention or a well-designed comparison can help resolve the ambiguity.
Finally, present a similar-looking failure under changed operating conditions. Compare an evidence-informed system with an appropriate baseline that did not receive the earlier outcome evidence, while controlling for the volume of additional context. Does prior experience lead to a better choice when the mechanism is shared—and restraint when it is not?
These are proposed tests, not completed experiments. They would require independently established results, pre-specified scoring, credible counterexamples, and attention to how data are presented. A model that simply produces a more persuasive retrospective explanation has not thereby demonstrated learning.
From a capable model to a capable continuing system
This is why the problem belongs in a series about organizing AI capability rather than only in a discussion of model confidence.
The code-writing model, reviewer, test runner, deployment system, payment records, and customer-facing service each contribute evidence with a different scope. A system can possess all those components and still reach the wrong conclusion if it treats their outputs as equivalent votes. It can also obtain the right causal evidence and fail to use it appropriately next time.
The question is whether a continuing system can carry a justified lesson across time without confusing a claim with an observation, an observation with a cause, or a local causal finding with a general guarantee.
When an AI says, “It worked,” the most useful follow-up may be three questions: What was actually observed? What supports attributing it to this action? And under what conditions should that experience guide the next decision?
References
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. R. (2024). SWE-bench: Can Language Models Resolve Real-world GitHub Issues? Paper presented at the International Conference on Learning Representations.
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
- Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., & Legg, S. (2020). Specification gaming: The flip side of AI ingenuity. DeepMind.
- Lightman, H., et al. (2023). Let's Verify Step by Step. arXiv:2305.20050.
- Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
- Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2023). Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4), 4396–4415.
- Pearl, J., & Bareinboim, E. (2014). External validity: From do-calculus to transportability across populations. Statistical Science, 29(4), 579–595.