Engineering Lab
Production Is On Fire
Choose incident actions and watch downtime, latency, cost, and users affected change.
Al Beltran Engineering Lab
Loading incident…
How the challenge works
- You have a short clock. Each action moves downtime, users, cost, latency, and reliability.
- Helpful actions are the ones a sane on-call would take. The score is how healthy the system is when time runs out.
Key concepts
Mitigate, then investigate
Rollback, flag-off, fail open, and rate limits are mitigations. Logs confirm you picked the right one.
Do not scale a broken artifact
More replicas of a bad deploy or a blocked thread pool make the bill worse.
Common mistakes
- Rolling back when the signal is traffic or a vendor.
- Touching the database because it is the most expensive knob.
Interview tips
- State the user impact in one sentence, then the first mitigation.
- Say what you would stop doing, not only what you would add.
Related challenges
Built by Al Beltran