Broker OutageP0 CRITICAL
rb-broker-outage

TRIGGER

Broker heartbeat fails, connection drops, or API returns 5xx for >30s

CUSTOMER / FINANCIAL IMPACT

Live positions unmonitored, new entries blocked, reconciliation halted

SAFETY ACTION

BLOCK NEW ENTRIES — preserve last-known state — mark data STALE

FIRST 5 MINUTES

1Confirm outage is broker-side (not AXIS network)
2Block all new live entries immediately
3Preserve last-known account state with STALE timestamp
4Notify on-call commander
5Check broker status page and official channels

DIAGNOSTICS

→Check broker API status endpoint
→Verify AXIS network connectivity
→Check broker session state
→Review recent error logs for pattern
→Check rate limit counters

MITIGATION

⚡Maintain local evidence of last-known state
⚡Do not fabricate live account truth
⚡Attempt reconnect with exponential backoff
⚡Query broker on reconnect before resuming

RECOVERY

1Reconnect to broker
2Query account state
3Query open positions
4Query pending orders
5Query recent history
6Full reconciliation before any new entries

VALIDATION

Reconciliation shows MATCHED state
Account snapshot freshness confirmed
No unknown orders remain
Risk state reflects current broker reality

ESCALATION

P0 → Incident Commander → Risk Team → Broker support contact

EVIDENCE TO PRESERVE

📎Last-known account snapshot
📎Connection logs
📎Error codes
📎Timestamps
📎AxisTraceIds

POST-INCIDENT REQUIREMENTS

Postmortem requiredBroker SLA reviewReconnect procedure review