Virtuous Payments

Major incident Donation Page
2022-08-31 14:53 PST · 4 minutes

Updates

Post-mortem

We received the root cause analysis from our provider today. Here is the summary of the RCA.

The Core API started having issues at 21:41:50 UTC August 31, 2022 until around 22:28:00 UTC when an error occurred that brought down ALL transactions for 46 minutes.

After some investigation, an engineer determined there were trailing commas after functions in the code that were breaking the production system. The engineer determined that the reason the trailing commas did not break QA or local builds and unit tests is that the versions of the framework on local and QA builds were different and be behaved differently for trailing commas.

We are confident our provider has proper checks and balances in place to prevent outages similar to this from happening again in the future.

September 2, 2022 · 22:59 PST
Issue

Our payment provider experienced an outage. We quickly notified our payment provider and they resolved the issue. We are waiting for RCA.

August 31, 2022 · 16:10 PST

← Back