Agent monitoring and the shape of oversight

Here is what that night looks like in a system I run. At 2 a.m., an AI call inside a flight-feasibility engine fails. A vendor timeout, nothing exotic. The system does not crash the trip, and it does not silently pass the check either. It converts the failure into a status the dispatcher cannot miss the next morning: “Manual Review Required,” confidence zero. The frontend has a code path keyed on that literal string, built to render it louder than a pass. The design rule underneath it: not knowing must always be visible, never absorbed. Nobody was awake when that decision got made, because it had been made months earlier, in architecture. This whole piece is about the family of design decisions that one belongs to, and about the two very different moves a leader can make when the system ships.

The syllabus splits these across two sessions for a reason. The control system is the technical expert leader's design problem. The supervisors are the domain expert leader's development problem. But they are one machine, and the place they meet, the exception queue where the system's output becomes a person's decision, is where this piece will keep returning.

Notice the deliberate overlap at layer five. The exception surface belongs to both brackets, because it is the seam where the machine's output becomes a person's queue. That maps word for word onto how the Session 8 syllabus splits the job: technical surveillance on one side, human oversight on the other, meeting at the exceptions.

The dispatcher's morning is the top of this stack. She opens a Priority Flags queue that shows only the upcoming legs with Incomplete or Warning checks, sorted by nearest departure, tagged with which module has the most issues. She does not review seventeen checks times every trip on the board. The stack reviews them and surfaces the exceptions. Hold that image. Half one explains how the stack is built. Half two asks what happens to her.

Here is the part that surprises people: the most load-bearing layers are the least intelligent. In the flight system, the one life-safety-adjacent call, whether an aircraft's pavement load rating exceeds every runway's rated strength, is pure arithmetic. When the math says no runway can support the aircraft, the code returns RED_FLAG at full confidence and bypasses the LLM entirely. The agent never gets a vote. For the same check on ordinary days, the agent is handed pre-filled PASS, FAIL, and MARGINAL results and asked to validate and add context, not to compute from scratch, and its own prompt opens with “Safety is paramount — when in doubt, flag for manual review.” The design principle for a technical expert leader: decide up front which facts are too safety-critical to be adjudicated by a probabilistic model, and give the deterministic engine veto power the agent cannot override. The LLM narrates. It doesn't decide.

The same discipline shows up in the patient-intake system I run for a physio clinic. A deterministic guardrail engine gates every AI response before a patient sees it. If a response violates a rule, the model gets exactly one capped “repair” attempt, and if the repair also fails, the deterministic fallback ships instead. The deterministic engine is always authoritative; the LLM repair is a bounded appeal. And in the overnight coaching pipeline, a verbatim-quote verifier runs on 100% of output, not a sample: every AI-extracted claim must cite a quote that appears word for word in the session transcript, or it is stripped and surfaced as a question.

One more piece of vocabulary worth a leader's time, because the two postures get conflated. Some layers log and continue: they write the audit trail and let the work proceed, a diagnostic record for later. Others tripwire and halt: OpenAI's term is exact, a guardrail that raises and stops execution the moment it fires. The automatic stop-work order, not a note for the file. A regime built entirely of the first kind knows everything and prevents nothing.

Row six deserves more than a card, because it answers a question leaders actually face: why should different agent workflows get different monitoring? William Ouchi's answer, from 1979, needs no translation. Monitor the process when you understand the transformation well enough to watch it. Monitor the output when you can measure results cleanly. And when you can do neither, you are left with what he called clan control: shared norms and trust, “we hired the right person and we trust them.” For an agent, clan control means “we trust whoever owns the agent,” which is exactly where unmonitored deployments quietly end up. Not by decision. By default.

And row seven, unpacked, because the five high-reliability principles map one-to-one onto mechanisms this piece has already shown. Preoccupation with failure: treat small anomalies as symptoms; NIST's MANAGE 4.3 formalizes it as a maintained register of errors and near-misses. Reluctance to simplify: read transcripts, not just scores; Anthropic's own rule is “we do not take eval scores at face value until someone digs into the details and reads some transcripts.” Sensitivity to operations: the flight system's live progress hub streaming check status to the dispatcher's screen, not an after-the-fact report. Commitment to resilience: graceful degradation, the failure that becomes a loud to-do instead of a crash. Deference to expertise: escalation that routes to whoever can actually judge the case, like the clinic's claim-to-act alerts, not to whoever ranks highest.

The technical expert leader is not starting from zero. The CFO's control vocabulary transfers almost whole. What is genuinely new is the speed, the volume, and one property no human auditee ever had. That one is held for half two: this auditee argues back.

First, the judge needs auditing. LLM judges carry position, verbosity, and self-preference biases, and the current mitigations, judging from a different model family, running ensembles for high-stakes calls, are practitioner consensus, not settled science. NIST's MEASURE 1.2 generalizes the point into governance language: assess the effectiveness of your metrics and controls themselves, on a regular basis, across the lifecycle. The auditor gets audited. Internal audit reports to the board for the same reason.

Second, the alarm nobody wired. In the clinic system, the shadow harness's data-retention cron, written, authenticated with a timing-safe secret, code-reviewed, merged, has never had a scheduler entry in any deployment. It is complete, correct, and inert, because nothing ever calls it. Separately, a shadow call that times out never pages anyone; only genuine failures log a warning, so a spike in timeouts is visible only to someone who thinks to query the table. Building a control and operating it are different jobs. Monitoring infrastructure needs its own owner and its own monitoring, and this is a class of gap no output-watching domain expert would ever catch from a dashboard. It takes someone auditing the architecture.

Third, the adversarial layer earns its keep. On the hiring platform, a structured adversarial review pass, a second model explicitly instructed to attack tenancy, release-visibility, and data-minimization, caught two bugs the 600-plus-test suite had missed: degraded scores silently polluting the hiring-manager-facing cohort rankings, in direct violation of the platform's own partial-scoring policy, and candidates able to see the employer's private interview notes about themselves. Both were fixed and verified with live end-to-end controls. Now scale that up. McKinsey's internal AI platform Lilli, in production since mid-2023 and used by 72% of the firm, was breached in about two hours by an autonomous third-party red-team agent that found 22 unauthenticated endpoints and chained a SQL injection to full database access. The syllabus's “evaluator agents” are not a hypothetical. They are already probing production systems, and the only question is whether you run them on yourself before someone else does.

The EU AI Act's Article 12 turns the audit log into statute: high-risk systems “shall technically allow for the automatic recording of events (logs) over the lifetime of the system,” with the high-risk obligations biting on August 2, 2026, mid-semester for this course. Automatic is doing real work in that sentence. Manual documentation does not satisfy it. A diagnostic control has become law.

Meanwhile in the US, the most rigorous model-governance regime in finance just did something stranger. In April 2026 the Federal Reserve, FDIC, and OCC issued SR 26-2, the successor to SR 11-7, banking's foundational model-risk guidance, and it explicitly carves generative and agentic AI out of scope as “novel and rapidly evolving.” Ten weeks before this course's term, the regulators told banks, and by extension their portfolio companies, that on the exact technology of this session they are on their own. That is precisely the gap the technical expert leader's monitoring architecture has to fill.

And the private-equity angle, for a session whose speaker runs value creation across a portfolio: HBS working-paper evidence shows PE ownership sharply raises monitoring demand at portfolio companies, and that the digital-adoption effect is driven specifically by investors with technology board experience. Governance pressure alone does not build monitoring. Technical expert leaders convert it, which is why an “AI Operating Partner” role is now emerging in the industry's own hiring commentary.

Return to the dispatcher's morning. The Priority Flags queue is triaged. One leg carries a RED_FLAG runway verdict the arithmetic issued without asking the model. One check reads “Manual Review Required” because a vendor timed out at 2 a.m. One NOTAM sits classified at LOW confidence, below the band the manual says to trust. Every layer of half one exists so that this queue is short and honest.

But notice what her job has become. She no longer performs the feasibility checks. She supervises the thing that does. The syllabus calls this “a fundamental shift from manual engagement with work to abstract, information-mediated oversight,” and that sentence deserves to be taken literally: her contact with the work now arrives entirely through the monitoring surface that half one built. Half two is about what that shift does to her judgment, her trust, and her skills, and what a leader has to build so the supervision stays real.

Half one built the control room. Half two is about the controller.

Bainbridge's ironies, compressed: automate the routine parts of a job and you leave the human the hardest residue, under time pressure, exactly when the informative, easy-to-practice parts of the job are gone. Because the automation handles routine operation, the operator never gets the reps that built the skill they will need when it fails. And the job converts from doer to monitor, which is a harder cognitive task, because humans are poor at sustained passive vigilance for rare events. The net irony: automation was meant to reduce the importance of the operator, and instead it increases the skill required of them, at precisely the moments they are least practiced.

Mica Endsley supplied the mechanism. Her out-of-the-loop research found that passive monitors lose situation awareness specifically at Level 2, comprehension, not Level 1, perception. You still see the alert. What you have lost is the model of what it means. And her studies found the degree of control moderates the effect, which is a design instruction, not just a diagnosis: build supervision with real decisions in it, not read-only dashboards.

Make it concrete with the dispatcher. Before, she cross-checked a dozen systems by hand, and when a runway was marginal she knew why, because she had done the arithmetic herself. After, she confirms a pre-computed verdict. The 17-check engine is exactly the kind of automation Bainbridge warned about, unless the system is deliberately designed to keep her judgment engaged. The rest of half two inventories what that design looks like.

Overtrust is measurable, not just nameable. My hiring platform computes a blind-accept signal: a candidate counts as blind-accepting only when they took AI text essentially unedited and that text contained a checkable claim, a figure, a citation, a named entity. Accepting harmless prose does not count. Employers see a cohort-level blind-accept-rate panel: how much unverified AI output moved through this team. It is deterministic, cheap, and always on, the supervisory metric most organizations do not know they can have.

The professionals' heuristics confirm the need for it. A 2026 interview study of 17 experienced developers overseeing coding agents found real oversight work collapsing into four shortcuts: plan-as-proxy, treating the agent's stated plan as a faithful account of what it did; test-as-guarantee; eyeballing; and trust-by-necessity, deferring precisely in the domains they knew least. One participant, verbatim: “I don't care what it does in between… I don't go through it line by line.” Meanwhile the agents' self-reported reasoning was wrong about 10% of the time, delivered with unwarranted confidence. Read that as a design finding, not a competence failure: review workload pushed skilled people into automation bias.

Undertrust is real too. The algorithm-aversion studies found people abandon an algorithm after seeing it err once, faster than they abandon a human making the same mistake, even when the algorithm demonstrably outperforms. And the follow-up work found aversion drops sharply when users can adjust the output even slightly. That is a direct design implication: give supervisors a modification lever. The flight system's corrections, coming up next section, are exactly that lever.

Here is the artifact. When the dispatcher corrects an agent, say, “this passport IS eligible despite what the agent said,” the correction persists: author, date, notes, soft-deleted but never destroyed. And it automatically reapplies to future matching trips. Immigration corrections match by passport-country and destination-country pair. Customs, aerodrome, and noise corrections match by airport. One dispatcher's judgment becomes everyone's default, without retraining anything.

Two brakes are built in. First, an explicit scope flag, IsFlightLegSpecific: “just this leg” versus “generalize this,” a decision the human makes explicitly on every correction. Second, a guardrail inside the guardrail: the code refuses to apply a customs-hours correction that has no date, weekday, or range scope at all, with a comment that reads like a management principle: do not apply a correction broadly without any scope selector. And every applied correction surfaces back in the interface with who, when, and why, so the next dispatcher always sees that a human intervened, never a silently altered number.

Name the pattern: a living exception log that becomes policy. Each correction is a bet that the pattern recurs, which is exploitation. The scope flag is the human deciding whether a judgment generalizes or stays local, the exploit-explore choice made concrete, dozens of times a week, by a domain expert rather than annually by a committee. And corrections run in both directions, dismissing the agent's false positives and adding events the agent missed. Supervision here is bidirectional teaching, not just veto.

The same tension has a dollar version. In the coaching pipeline I ran a model-downgrade trial with a pre-registered rule set before seeing any data: any dropped real client commitment means the cheaper model does not ship. Haiku dropped six real commitments across nine runs. Verdict: the roughly two-dollars-per-coach-per-month premium for Sonnet is trivial against a single client commitment silently vanishing. And it has a safety version. The clinic's Haiku A/B caught a 25% hard-stuck session rate and, sharpest of all, a silently dropped cauda-equina red-flag screen for a sciatica patient, before any real patient saw it. The designed auto-escalation cascade did not fire during those stuck loops, and the team ticketed that gap instead of papering over it. The lesson across all three: explore deliberately, with pre-registered promotion rules and a domain expert reading the transcripts, because it took a clinician to recognize what the cauda-equina miss meant.

The industry has its own words for trust that loosens by design. OpenAI's guide, verbatim: high-risk actions “should trigger human oversight until confidence in the agent's reliability grows.” Trust calibration as a designed, time-varying system. The correction ledger is what that sentence looks like when it ships.

Beane found that the traditional apprenticeship, learning by approved, increasing participation in real procedures, broke when the robot arrived. Residents got ten to twenty times less practice. The ones who built real skill did it through what he calls shadow learning, norm-challenging, off-book routes the formal system never sanctioned. His later work names the three conditions skill formation needs, the three C's: challenge, real stakes and real difficulty; complexity, enough variation to generalize from; connection, proximity to someone more expert. Machines that absorb the challenging parts of the work remove those conditions, even with a human nominally in the loop.

Translate to the dispatch floor. Today's dispatcher catches the agent's errors because she spent years doing feasibility by hand. The dispatcher hired next year starts above the work, never in it. And if leaders only build formal oversight roles, Beane's finding says the capable future supervisors will grow their competence off the books and invisibly, or not at all.

And you can train the skill on purpose, because one of my systems does exactly that. The hiring platform's drift-recovery task deliberately makes its own AI misbehave. From turn two, the “AI” is a deterministic script planted with calibrated drift, constraint drift, numeric drift, or tone drift, subtle or overt, tunable by an admin. Every injection is logged as an idempotent audit event so the grader can join what was planted against what the candidate caught. A flight simulator for supervision. Its sibling is the planted-fabrication case: a fake citation, the “Halberton Retail Analytics 2024 Grocery Reinvention Index,” that exists in the fixtures solely so catching fabrication is a measurable skill. The apprenticeship gap is a design problem, not fate.

Element one deserves a word, because it is a design choice most teams never consider. The coaching pipeline contains no numeric confidence score anywhere. Instead, the verifier converts every unverifiable claim into a discrete gap question the coach resolves in one click, keep, fix, or drop, and unresolved gaps block approval outright. For a non-technical supervisor, an answerable question beats a 0-to-100 score they would have to calibrate a cutoff for. Element six is the arc the syllabus names: the verifier's failure signal, both models manufacturing quotes by stitching two speaker turns with an ellipsis, produced a one-line prompt rule, one contiguous span from one speaker turn, never joined quotes. On re-measure, Sonnet's dropped-commitment rate went four to zero and Haiku's six to one. Human judgment improved the agent. The improved agent changed what the human watches for. That loop, “calibrating the co-evolution of human supervisory judgment and agent capabilities over time,” is the Session 9 sentence, running in production.

None of this is a dashboard. A dashboard is element zero. The learning system is the set of decisions the humans keep making, and the design question for a domain expert leader is whether those decisions are engineered to keep their judgment alive.

And beneath the quad, the short list of things that do not move as trust grows. The non-delegables.

Accountability. In the MIT SMR/BCG expert panel, the experts quoted agreed that people and organizations, not AI systems, bear ultimate responsibility for deployed agents, and 69% said holding agentic AI accountable requires new management approaches. The Air Canada tribunal is the same sentence with legal force: the company argued its chatbot was a separate entity responsible for its own actions, and the tribunal held that the chatbot's word is the company's word. With no owner designed in, the court appointed one.

The authority to pull the plug. NIST MANAGE 2.4, almost verbatim: mechanisms in place, responsibilities assigned and understood, to supersede, disengage, or deactivate. Someone specific, named before the bad day.

The generalization decision. When one correction becomes policy. The scope toggle stays under a human thumb.

Reading the anomaly that fits no category. High-reliability organizations call it reluctance to simplify. It took a clinician to recognize what a missed cauda-equina screen meant, because no dashboard category existed for it.

Deciding what “good” means. Anthropic, verbatim: “the people closest to product requirements and users are best positioned to define success.” Eval-writing is a domain-expert skill to build, not an engineering task to outsource.