How Agents Learn

Section 1

The economic frame: from prediction machines to decision machines

Ajay Agrawal, Joshua Gans, and Avi Goldfarb framed the economics of the last decade of AI cleanly: AI makes prediction cheap. Their AI Canvas decomposes any task into Prediction, Judgment, Action, Outcome (top row) and Input, Training, Feedback (bottom row). The economic argument is that cheap prediction increases the value of its complements, data, judgment, and the action that turns a prediction into a decision.

The framing has held into 2024. Their HBR piece "Generative AI Is Still Just a Prediction Machine" doubled down: even generative AI's writing, drawing, coding, and summarising are reframed prediction tasks. The judgment and action stay with humans. The 2022 book Power and Prediction extended the original argument to systemic transformation, but kept the same boundary: AI predicts; humans decide.

Autonomy starts where Agrawal stops. In industrial supply chains, the substrate now makes decisions cheap, not just predictions. A calibrated, traceable, inspectable, certifiable decision is no longer a high-cost artefact requiring scarce human judgment for every emission. It is something the substrate can produce at the cadence the operation demands, with the audit surface the regulator, the auditor, and the insurer require. Cheap decisions become valuable when the complements to decisions become valuable: the verifiable ledger (Decision Trace), the elicited heuristics that shape them (Operating Knowledge), and the compounding policy that learns from running them (Learned Judgment).

Cheap prediction made data the moat. Cheap decisions make the audit trail of those decisions, and the policy compiled from them, the moat. Neither can be bought; both compound only through operation.

The architectural consequence is what the rest of this whitepaper covers. Building a decision machine for industrial operations requires a different substrate than wrapping an LLM in a governance harness. It requires a trained policy that runs inside a calibrated uncertainty layer, a causal layer that estimates the counterfactuals you cannot run live, an elicitation substrate for the heuristics experienced operators carry, and a per-period rollup that makes the substrate's learning legible.

Section 3

The cognitive cycle: OODA, ORPA, and where Autonomy puts learning

Industrial operators run a cognitive cycle on every consequential decision. John Boyd named it OODA: Observe, Orient, Decide, Act. The cycle is the canonical decomposition of operator cognition in aviation, defence, control rooms, and industrial operations. Boyd's central insight was that Orient is the most important step, because that is where the observer's internal model gets updated by the observation. OODA is not just situational assessment; it is the loop where learning and decision-making happen together.

ORPA: Boyd's cycle, reformulated for agents

XMPro's MAGS architecture reformulates OODA for industrial AI agents as ORPA: Observe, Reflect, Plan, Act. The reformulation splits Boyd's Orient step into a dedicated Reflect slot because in an agent architecture the sense-making computation deserves its own structural location. XMPro's Reflect step draws on Park et al.'s 2023 Stanford paper, Generative Agents: Interactive Simulacra of Human Behavior, where the agent's memory stream stores observations and reflections, and reflections write back into the stream for next-cycle retrieval.

The two cycles are almost the same, but the architectural intent differs subtly. ORPA isolates Reflect so that an LLM can be slotted in as the reflection engine, producing both the next decision and an in-loop learning update. The memory stream is what makes the agent feel like it has continuity: next cycle reads back what last cycle wrote. The agent learns by reflecting.

Autonomy: the cycle stays, the learning leaves

Autonomy's agents follow a structurally similar cycle. Each agent observes canonical state, evaluates that state against its trained policy and the conformal calibration layer, plans an action within its envelope, and acts. The cycle is tight and fast: an NN forward pass takes milliseconds, the policy and the calibration are pre-trained and loaded, the action emits a Decision Trace row and writes canonical state. The cognitive cycle is structurally identical to ORPA.

The architectural distinction is where the learning lives. OODA and ORPA both bake learning into the inner loop: Boyd's Orient updates the model; ORPA's Reflect writes to the memory stream. Each cycle does decision-making and learning in the same breath. Autonomy deliberately lifts learning out of the cycle.

Why out-of-loop learning

In-loop learning has two costs. First, the learning step competes for the decision's compute and time budget. An LLM reflecting on the memory stream during a control loop is doing language-model inference where engineered policy evaluation would do. Second, the learning gets baked into whatever the model's reasoning happened to be in that single moment, with no cross-event aggregation, no calibration against realised outcomes, and no audit boundary between situations where the model was right and situations where it merely sounded right.

Out-of-loop learning gets the cross-event time and aggregation it needs to be calibrated. The RL trainer sees thousands of trajectories before updating policy weights. The conformal recalibration layer sees enough realised outcomes to compute coverage drift before adjusting intervals. The causal layer accumulates paired counterfactuals before updating estimands. None of these run during the decision; all of them run on their own cadence, supervised by the realised outcome rather than by the model's confidence in itself.

In-loop learning competes for compute with the decision itself and bakes the update into whatever the model's reasoning happened to be in that moment. Out-of-loop learning gets the cross-event time and aggregation it needs to be calibrated.

AI·IO·ML sits around the cycle, not inside it

Autonomy's operating model, AI·IO·ML, is one model in three couplets: the agent acts (Automate, Inform), the human engages (Inspect, Override), and the system improves (Measure, Learn). It is not a replacement for OODA or ORPA. The agent still runs its own cognitive cycle on every decision. AI·IO·ML is the contract that governs the interaction between the substrate and the operator: when the substrate brings a human in (Inform), when a human can pull on the substrate to ask why (Inspect), and when a human supersedes (Override), with every outcome measured and fed back as training signal. The agent's cognitive cycle is its private business; AI·IO·ML is what the operator sees and acts on, and what the system learns from.

The same AI·IO·ML contract also governs the learning loop. Parametric learning events (RL retraining, conformal recalibration, causal estimand updates) flow into the same Decision Stream surface that operational decisions use, with the same row template and the same Inspect contract. Most parametric updates Automate; high- magnitude shifts Inform; LLM-proposed CANDIDATEs always Inspect until the confidence head is calibrated. One governance model covers both decision events and learning events.

Section 5

Calibrated learning: RL, conformal, causal

Out-of-loop learning is only useful if the mechanism is calibrated. An uncalibrated learner produces a policy that drifts in directions the substrate cannot verify against realised outcomes. Three calibrated mechanisms run on independent cadences and feed Learned Judgment.

RL on the digital twin

The reinforcement-learning trainer plays trajectories on the digital twin against the BSC reward function. Conservative offline RL keeps the trained policy close to the behavioural distribution of the warm-started corpus and the deterministic teacher, with explicit penalties for actions outside the visited region. The training is supervised by the realised BSC outcome, not by the model's own confidence. Regression guard gates every candidate checkpoint: if the new model regresses on BSC utility against held-out trajectories, it is discarded; only checkpoints that match or improve get promoted.

Conformal calibration

Every prediction the substrate emits carries a calibrated P10 / P50 / P90 interval. The conformal layer holds out a calibration set of realised outcomes and computes the empirical coverage. When the observed coverage drifts more than a configured tolerance from the promised coverage (the predictor promised 90 percent intervals; realised coverage measured 78 percent), the calibration set is updated and the intervals are recomputed. The upstream policy is also flagged for retraining if coverage drift persists. The substrate does not Monte-Carlo plans by re-running the twin under noise; uncertainty is quantified by conformal intervals at inference time. This is an architectural invariant: the twin's PLAN_PRODUCTION mode raises an error if any stochasticity knob is left on during plan generation.

Causal posterior updates

Override effectiveness is the cleanest causal signal the substrate produces. When a planner overrides an agent's recommendation, the substrate captures the override and the realised outcome. A propensity-score matching service constructs counterfactual pairs, and the Bayesian posterior over override effectiveness is updated daily. The posterior is what tells the substrate whether the planner's override pattern consistently improves outcomes (the override-effectiveness estimand is positive and tightening), whether it is neutral (the estimand is centred on zero and the credible interval is wide), or whether it consistently degrades outcomes (the estimand is negative). Per-planner posteriors feed governance dashboards; per-decision-class posteriors feed envelope adjustments.

A first structural counterfactual is live in narrow scope: inventory-buffer overrides in the SCP plane are replayed forward through the digital twin with the agent's recommended buffer installed at the swap point, and the resulting service-level / inventory trajectory is what the override is scored against. Most overrides do not need the twin: analytical substitution and propensity-matched control pairs estimate the counterfactual directly from observed history. The structural twin-replay is reserved for the heavily-confounded decision classes where observational identification is weak, and even there a structural time-series counterfactual does the work where a full twin re-run would be too costly. Completing this observability-tiered coverage across decision classes is on the near-term roadmap and is one of Autonomy's four pillars.

Behavioural drift detector

A fourth mechanism watches the output distribution of each agent for distributional drift independent of accuracy. A z-score for continuous outputs, a total-variation distance for categorical outputs, evaluated every two hours. Severity levels (ADVISORY, RECOMMENDED, CRITICAL) drive the AI·IO·ML mode assignment: ADVISORY Automates as a log entry, RECOMMENDED Informs the operator, CRITICAL Inspects and triggers the guardrail escalation path. The drift detector does not retrain the policy; it surfaces evidence that the calibrated mechanisms above should be re-run.

The four mechanisms compose

The four mechanisms answer different questions. RL on the twin updates the policy itself. Conformal recalibration keeps the uncertainty layer honest. Causal posterior updates score the human overrides against the substrate's own recommendations. Behavioural drift detection surfaces the question. Together they give the substrate a calibrated learning loop where every update is supervised by a realised outcome and every artefact (policy weights, calibration set, estimands, drift events) is versioned, inspectable, and auditable.

Section 7

The Learning Digest: what we learned this period

Learned Judgment compounds invisibly without a periodic surface. The conformal layer recalibrates intervals on every realised outcome. The causal layer updates estimands daily. The drift detector flags events every two hours. The RL trainer updates policy weights on cadence. Each mechanism is calibrated and audited, but none of them by themselves answers the question an operator, a CFO, or an auditor eventually asks: what did the substrate learn this week, this month, this quarter?

The Learning Digest is the period-readable form of Learned Judgment. Each digest aggregates the period's learning events, characterises their magnitude and direction, and produces a narrative paragraph that a non-engineer can read. The structured rollup is the audit substrate; the narrative is the customer-facing legibility.

What a digest contains

Cadence

Three digests run on cadence per tenant: a daily digest at 04:00 (after the daily causal matching at 02:40 and the NN orchestration at 03:00 have completed), a weekly digest on Monday 05:00 (after the S&OP cycle), and a monthly digest on the first of the month at 05:30. Each digest is idempotent: re-running for the same period upserts the narrative on the existing row, preserves the original structured rollup, and keeps the hash-chain position. The chain itself is per tenant, per config, per period kind; verifiable end-to-end the same way the Decision Trace ledger is.

The Inspect contract for learning events

Every learning event that lands on the Decision Stream carries the four Inspect-contract fields, adapted from their decision-event meaning. Prompt: what evidence triggered the event (the drift signal, the override pattern, the planner interview transcript). Decision: what the substrate updated (which policy weight, calibration interval, estimand, prior). Expected: the behaviour change the substrate predicts as a result. Likelihood: the calibrated probability the predicted change holds, with the source tagged (conformal probability for parametric updates, LLM self-report confidence for CANDIDATEs, drift z-score normalised to [0, 1] for drift events). The same Inspect surface that explains why an agent ordered three thousand cases of product X explains why the substrate recalibrated the lead-time interval on lane LAX → ATL.

Section 9

References

• Agrawal, A.; Gans, J.; Goldfarb, A. (2018). Prediction Machines: The Simple Economics of Artificial Intelligence. Harvard Business Review Press.
• Agrawal, A.; Gans, J.; Goldfarb, A. (2022). Power and Prediction: The Disruptive Economics of Artificial Intelligence. Harvard Business Review Press.
• Agrawal, A.; Gans, J.; Goldfarb, A. (2024). "Generative AI Is Still Just a Prediction Machine." Harvard Business Review, November 18, 2024.
• Alicke, K. (2026). "The Planner Was the System." Substack.
• Boyd, J. R. (1976). "Destruction and Creation." Unpublished essay. The canonical reference for the OODA cycle.
• Brynjolfsson, E.; McAfee, A. (2014). The Second Machine Age. W. W. Norton. The "bounty" framing.
• Goodness, E. (2026). Gartner. The "bonded autonomy" term as a refinement of bounded autonomy.
• Park, J. S.; O'Brien, J. C.; Cai, C. J.; Morris, M. R.; Liang, P.; Bernstein, M. S. (2023). "Generative Agents: Interactive Simulacra of Human Behavior." Stanford. The Smallville paper that introduces the memory-stream-and-reflection pattern XMPro's MAGS adopts.
• Silver, D.; Hubert, T.; Schrittwieser, J.; et al. (2018). "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play." Science 362.6419.
• van Schalkwyk, P. (2026, May 19). "Why Industrial AI Agents Don't Need to 'Reason' Like ChatGPT." LinkedIn / The Digital Engineer. ORPA cycle, Cognitive Decision Loop.
• van Schalkwyk, P. (2026, February 28). "The Industrial AI Moonshot Nobody Is Talking About." LinkedIn. Bonded autonomy, Decision Trace, certification flywheel.

Internal architectural rules referenced

• · aiio-model.md: the per-decision sequence (Automate, Inform, Inspect, Override), the mode-assignment formula, the extension to learning events.
• · llm-usage-discipline.md: LLM never on the decision axis; LLM may propose CANDIDATEs, never commit them; workload routing.
• · data-lineage.md: the ProvenanceMixin contract for every data-bearing entity at a producer boundary.
• · four-pillars.md: AI agents, conformal prediction, digital twin, causal AI. The 90/10 ratio of decision substrate to LLM narration.
• · plane-module-invariant.md: one plane per functional domain; no transient code copies.
• · guardrail-escalation.md: the deontic-vocabulary contract (O / P / F, with contrary-to-duty escalation targets) for substrate guardrails.