Skip to Content
Go Realm v1 is released 🎉
RabbitMQDay 8: Retry and Dead Letter Queue

Day 8

Retry Queues, Dead-Letter Queues, and Failure Flow

āφāϜāϕ⧇āϰ lesson production survival-āĻāϰ āϜāĻ¨ā§āϝāĨ¤ āϏāĻŦ failure same āύāĻž; āϤāĻžāχ retry strategy āφāϰ poison-message handling clean āĻšāĻ“ā§ŸāĻž āĻĻāϰāĻ•āĻžāϰāĨ¤

Retry and DLQ

Failure Types

Failure typeExampleAction
TemporaryDB timeout, network glitchretry
Permanentinvalid JSON schema, unknown required fieldDLQ / reject
External slow dependencypayment provider lagretry with delay
args := amqp.Table{ "x-dead-letter-exchange": "", "x-dead-letter-routing-key": "orders.dlq", } _, err := ch.QueueDeclare("orders.main", true, false, false, false, args) if err != nil { log.Fatal(err) }

Incident Thinking

Situation: āĻāĻ•āϟāĻŋ payment consumer downstream gateway timeout-āĻāϰ āĻ•āĻžāϰāϪ⧇ fail āĻ•āϰāϛ⧇āĨ¤ Message requeue āĻ•āϰāĻž āĻšāĻšā§āϛ⧇, āĻ•āĻŋāĻ¨ā§āϤ⧁ queue hot loop-āĻ āϚāϞ⧇ āϗ⧇āϛ⧇āĨ¤

Goal: Safer redesign āĻ­āĻžāĻŦāĻžāĨ¤

What to think about:

  • immediate requeue same failure āĻĻā§āϰ⧁āϤ repeat āĻ•āϰāĻžāĻŦ⧇
  • delay āĻĻāϰāĻ•āĻžāϰ
  • max retry limit āĻĻāϰāĻ•āĻžāϰ
āϏāĻŽā§āĻ­āĻžāĻŦā§āϝ āϏāĻŽāĻžāϧāĻžāύ āĻĻ⧇āϖ⧁āύ

Retry queue + TTL + DLQ strategy betterāĨ¤ Temporary failure message āĻ•āĻŋāϛ⧁ delay āĻĒāϰ⧇ main queue-āϤ⧇ āĻĢāĻŋāϰ⧇ āφāϏāĻŦ⧇āĨ¤ āĻŦāĻžāϰāĻŦāĻžāϰ fail āĻšāϞ⧇ DLQ-āϤ⧇ āϝāĻžāĻŦ⧇āĨ¤ āĻāϤ⧇ hot loop āĻ•āĻŽāĻŦ⧇āĨ¤

Poison message āĻŦāϞāϤ⧇ āϕ⧀ āĻŦā§‹āĻāĻžā§Ÿ?

Checkpoint

āύāĻŋāĻœā§‡ explain āĻ•āϰ⧋: retry queue āφāϰ DLQ āĻāĻ•āχ āϜāĻŋāύāĻŋāϏ āύāĻžāĨ¤

  • Retry queue = āφāĻŦāĻžāϰ āĻšā§‡āĻˇā§āϟāĻž āĻ•āϰāĻžāϰ staging area
  • DLQ = normal flow āĻĨ⧇āϕ⧇ āφāϞāĻžāĻĻāĻž inspection area
  • Retry loop bounded āύāĻž āĻšāϞ⧇ system unhealthy āĻšā§Ÿ