15.S50 · Sessions 8 & 9 · Monitoring Agents

Running the sessions

Working notes for Sessions 8 and 9: draft assignment questions for each session and one exercise idea. The real teaching window is 80 minutes, Sloan classes start five late and end five early.

01 · Pre-class assignment drafts

Assignment questions, drafted

Three per session. The bracketed ones sharpen once the cases are drafted.

S8 · Q1

Take an agentic system you know, from work or from the news. Place it on Perrow's two axes, how tightly coupled its actions are to other systems, and how non-deterministic its behavior is run to run. What does that position demand of its monitoring, concretely?

S8 · Q2

[From the case, once drafted: which monitoring layer failed first, and was it a technology failure or an ownership failure?]

S8 · Q3

Pick one public incident from the reading, Air Canada's chatbot ruling, Replit's deleted database, or Cursor's invented policy. Name the missing monitoring layer, then name the cheapest mechanism that would have caught it. Defend the word cheapest.

S9 · Q1

Describe a time you supervised work you used to do yourself. What did you check at first, what did you stop checking, and how did you decide? What did you lose by stopping?

S9 · Q2

[From the case, once drafted: where should the authority line sit for this company today, and what evidence should move it?]

S9 · Q3

An agent's override log shows overrides falling month over month. Give one reading of that log that should increase your confidence in the system, and one that should alarm you. What single piece of additional evidence separates the two?

02 · Exercise idea, Session 9

Design the supervision regime

Small groups, the same credit-limit-increase call from Session 3, now live in production.

  • same workflow as Session 3's exercise (credit-limit-increase call), now live in production -> the group owns supervising it
  • each group produces four things: what gets logged (and is the log independent of the agent's own narration) -> what an evaluator checks, on what sample -> what escalates to a person, on what trigger -> who owns the agent's mistakes, by name
  • groups that remember their Session 3 topology choice will find it constrains their monitoring options