RCA for Validator Outage August 11, 2022
Summary:
On August 11, Gelotto’s Juno and Stars validators experienced a brief period of downtime and missed approximately 142 blocks due to an issue associated with networking equipment at the datacenter that houses Gelotto’s servers. This issue caused a network outage to some parts of the datacenter.
Gelotto received an alert within 3 minutes of the incident and responded to the issue within 5 minutes of receiving the alert.
The issue was corrected in approximately 15 minutes.
The downtime was brief, and a very small number of blocks was missed, so no slashing by the Juno network occurred.
What went well:
Gelotto received an alert, responded to it, and resolved the issue extremely quickly.
Action Items:
Gelotto will expedite its existing plans to use additional datacenters to add redundancy and durability to its validators.
Gelotto will begin work to create its own datacenter locally to further improve response times and reliability.
