Resolved -
The incident has been resolved.
Dec 3, 02:00 UTC
Update -
We are seeing full recovery of API latency across all services. Benefit verification APIs are operating as intended, but we are still working through a backlog of queued BV jobs from earlier in the incident. As we continue to drain this queue and tune our schedulers, we do not expect additional impact to real-time BV requests. We’ll provide another update if we observe any regression or once clean-up work is complete.
Dec 2, 21:07 UTC
Monitoring -
We deployed a fix and are seeing improved API latency across all services. Instant benefit verification results and AI answering remain degraded while we continue to drain the job queue and stabilize our workers. We are closely monitoring performance and will provide another update as these pathways recover.
Dec 2, 20:56 UTC
Identified -
We have identified the cause of elevated errors and timeouts impacting benefit verifications and related API traffic. A large backlog of scheduled benefit verification jobs accumulated over the holiday period, all firing around 4–5am, which overloaded our Redis-backed job queue and caused a spike in latency and BV failures.
We have scaled up Redis capacity and are actively processing the backlog while adjusting our scheduling and rate-limiting behavior to prevent similar traffic spikes in the future. We’ll provide another update once performance has stabilized and long-term fixes are in place.
Dec 2, 15:53 UTC