Virtuous Payments

We received the root cause analysis from our provider today. Here is the summary of the RCA.

The Core API started having issues at 21:41:50 UTC August 31, 2022
until around 22:28:00 UTC when an error occurred that brought down ALL
transactions for 46 minutes.

After some investigation, an engineer determined there were trailing
commas after functions in the code that were breaking the production
system. The engineer determined that the reason the trailing commas
did not break QA or local builds and unit tests is that the versions
of the framework on local and QA builds were different and be behaved
differently for trailing commas.

We are confident our provider has proper checks and balances in place to prevent outages similar to this from happening again in the future.

Our payment provider experienced an outage. We quickly notified our payment provider and they resolved the issue. We are waiting for RCA.