Failure Types
| Failure type | Example | Action |
|---|---|---|
| Temporary | DB timeout, network glitch | retry |
| Permanent | invalid JSON schema, unknown required field | DLQ / reject |
| External slow dependency | payment provider lag | retry with delay |
args := amqp.Table{
"x-dead-letter-exchange": "",
"x-dead-letter-routing-key": "orders.dlq",
}
_, err := ch.QueueDeclare("orders.main", true, false, false, false, args)
if err != nil {
log.Fatal(err)
}Incident Thinking
Situation: āĻāĻāĻāĻŋ payment consumer downstream gateway timeout-āĻāϰ āĻāĻžāϰāĻŖā§ fail āĻāϰāĻā§āĨ¤ Message requeue āĻāϰāĻž āĻšāĻā§āĻā§, āĻāĻŋāύā§āϤ⧠queue hot loop-āĻ āĻāϞ⧠āĻā§āĻā§āĨ¤
Goal: Safer redesign āĻāĻžāĻŦāĻžāĨ¤
What to think about:
- immediate requeue same failure āĻĻā§āϰā§āϤ repeat āĻāϰāĻžāĻŦā§
- delay āĻĻāϰāĻāĻžāϰ
- max retry limit āĻĻāϰāĻāĻžāϰ
āϏāĻŽā§āĻāĻžāĻŦā§āϝ āϏāĻŽāĻžāϧāĻžāύ āĻĻā§āĻā§āύ
Retry queue + TTL + DLQ strategy betterāĨ¤ Temporary failure message āĻāĻŋāĻā§ delay āĻĒāϰ⧠main queue-āϤ⧠āĻĢāĻŋāϰ⧠āĻāϏāĻŦā§āĨ¤ āĻŦāĻžāϰāĻŦāĻžāϰ fail āĻšāϞ⧠DLQ-āϤ⧠āϝāĻžāĻŦā§āĨ¤ āĻāϤ⧠hot loop āĻāĻŽāĻŦā§āĨ¤
Poison message āĻŦāϞāϤ⧠āĻā§ āĻŦā§āĻāĻžā§?
Checkpoint
āύāĻŋāĻā§ explain āĻāϰā§: retry queue āĻāϰ DLQ āĻāĻāĻ āĻāĻŋāύāĻŋāϏ āύāĻžāĨ¤
- Retry queue = āĻāĻŦāĻžāϰ āĻā§āώā§āĻāĻž āĻāϰāĻžāϰ staging area
- DLQ = normal flow āĻĨā§āĻā§ āĻāϞāĻžāĻĻāĻž inspection area
- Retry loop bounded āύāĻž āĻšāϞ⧠system unhealthy āĻšā§