Incident Retrospective — INC-4471
PARK — engineering manager. Facilitating. Watching the clock.
DELACROIX — on-call for the weekend. Presenting.
HOYT — tech lead. Owns the triage router.
RHEE — analytics. Has the dashboard.
VOSS — engineer. Says things.
PARK: Okay. INC-4471. We've got the room till the half hour. Delacroix, walk it, keep it tight, we don't need the full timeline.
DELACROIX: Saturday, 2:10 a.m., the triage router starts double-routing. Same ticket into billing and into abuse, both queues, same second. By four the retry loop's compounding, queue depth climbs, support's SLA board goes red end to end. It stayed red fourteen hours.
PARK: Root cause.
DELACROIX: A batch of tickets the router couldn't split. Confidence came back near fifty-fifty between two intents. Downstream retries on low confidence. The retry re-submits, the router splits it again, same fifty-fifty. Loop.
HOYT: So — edge case. Weird input, model does something weird, retry amplifies. We put a floor on the confidence, low-confidence goes to a human, done. I can have that up today.
PARK: Good. That's the action item, let's—
DELACROIX: It wasn't weird input.
(a beat)
HOYT: It routed one ticket to two queues, Delacroix. That's not the happy path.
DELACROIX: No. But it wasn't noise. It was one phrasing. The same phrasing, thousands of times, all weekend. It's the template.
RHEE: I pulled the numbers before this. The affected sessions are zero-point-oh-two percent of Saturday volume. Two-hundredths of a percent. It's in the deck.
HOYT: Right. So — long tail. Rare input, high blast radius because of the retry, but rare. That's a low-frequency edge case with a nasty amplifier. Fix the amplifier.
DELACROIX: It's not rare. It's the onboarding template. The one we email new enterprise accounts. "Copy this and describe your issue." They copy it. The header line reads as billing, the body reads as abuse, and the model can't pick, because the input genuinely is both. Every account that used the template hit the same wall. It's zero-point-oh-two percent of sessions because most people don't file tickets on a Saturday. It's a hundred percent of that template.
RHEE: (checking) ...The template's in the standard onboarding kit, yeah.
HOYT: Okay, but that's still a specific input. Non-representative. Real users don't write like the template, they write like people. We're building for the general case, not for one canned string that happens to straddle two intents.
VOSS: The canned string is the general case. It's the one input we ship. Everything else is what people write. That's the one we hand them.
HOYT: Sure. So we harden it. Add the template to a preprocessing rule, strip the header, route on the body. Clean.
VOSS: Then the next template straddles a different pair. You've moved the wall, you haven't removed it. The input's degenerate — the two intents collapse onto the same point, the model can't separate what isn't separate. A floor doesn't fix that. A floor just says "I can't tell" faster.
HOYT: Right, if the input's degenerate we filter it. Malformed in, garbage out. I'll add it to the denylist with the rest of the junk.
(That was not what VOSS meant. Nobody notices. VOSS tries again, plainer.)
VOSS: The form asks for two things in one box. The router's being punished for noticing.
HOYT: (already typing) Yeah, so we stop it noticing. Default it. Moving on.
(VOSS lets it go.)
PARK: I want to keep us moving. Where does this land — is it a fix-now or a backlog?
HOYT: Backlog. The confidence floor is the now-fix, that stops the bleeding today. The template thing is a follow-up. We can add a guard, but honestly it's working as designed most of the time — the router's fine on the happy path, this is one corner.
DELACROIX: It's going to come back.
HOYT: With the floor in? It can't loop with the floor in.
DELACROIX: The loop can't. The mis-split can. Fourteen hours of tickets went to the wrong team and got worked by the wrong team. The floor doesn't tell them apart, it just stops asking twice. Next enterprise cohort onboards, same template, and it quietly routes wrong again, no incident, no page, because we took out the alarm and left the thing the alarm was for.
HOYT: That's — okay, that's a real point, but that's a different ticket. That's a routing-quality thing, that's not INC-4471. 4471 is the outage. The outage is the loop. We fixed the loop.
PARK: I think that's fair. Let's scope this retro to the incident. Delacroix, the routing-quality question is good, write it up separately, we'll groom it.
DELACROIX: It'll get a P3 and sit.
PARK: It'll get triaged like everything else.
RHEE: For the writeup — do we call 4471 resolved? I need a status for the exec summary.
HOYT: Resolved, root cause identified, mitigation deployed. Contributing factor: known limitation in intent disambiguation on ambiguous inputs. Low frequency. Acceptable residual risk with the floor in place.
RHEE: "Known limitation, low frequency, acceptable residual." Got it.
VOSS: It's not a limitation. It's the shape of the thing. You built a router that picks one, and you feed it inputs that are two. It'll do exactly this every time the input is two. That's not the model failing. That's the model telling you the input doesn't have one answer.
HOYT: Voss, they always have one answer. Someone has to work the ticket. It goes to a queue. Billing or abuse. We pick. That's the job.
VOSS: I know. That's why it'll keep happening. You need one and it has two.
(HOYT nods, not hearing it as disagreement, hearing it as agreement.)
HOYT: Exactly. So we pick a default, route ambiguous-both to billing, they escalate if it's abuse. I'll add that too. Good call.
VOSS: (flat) Sure.
(VOSS does not say anything else. VOSS is not sulking. VOSS has simply run out of ways to say it in words this room uses.)
PARK: Okay. Action items. One: confidence floor, Hoyt, today. Two: default-route on ambiguous-both, Hoyt, this sprint. Three: Delacroix files the routing-quality follow-up, we groom it next planning. 4471 marked resolved. Anything else for the incident?
DELACROIX: No.
PARK: Good retro, everyone. We got out early. Nice work this weekend, Delacroix, sorry about your Saturday.
DELACROIX: Yeah.
(People start dropping off the call. The shared doc updates: INC-4471 — RESOLVED. Under "Contributing Factors," a new line: Known limitation — intent disambiguation — low frequency — acceptable.)
(On a dashboard nobody has open, a small counter — "onboarding-template-v3, tickets 24h" — ticks from 1,208 to 1,209. Then 1,210.)
(The next enterprise cohort onboards Monday.)
— END —