Incident description
ProxySQL keeps crashing and reloading every 5 minutes and client connections keep being dropped continuously.
Incident Severity: CRITICAL
Data Loss: YES
Timeline
| Time (UTC) | |
|---|---|
| 10 July 2021 7:13 | an upgrade from version 2.0.7 to version 2.1.1 was triggered. |
| 10 July 2018 11:00 | we started receiving notifications on Telegram about CAMS not being available. |
| 10 July 2018 17:00 | Massimiliano Adamo identified an issue with ProxySQL. PRoxySQL was reloaded and logging was increased. |
| 12 July 2018 10:30 | Bjarke Madsen and Robert Latta sent a message to Massimiliano Adamo , to notify an issue with the Web UI of BRIAN. |
| 12 July 2018 13:30 | ProxySQL was pinned to an older version |
Total Downtime: it was an intermittent downtime of ~ 48 hours
Details of Solution
A parameter in the Puppet module defaults the version number to 'latest': https://gitlab.com/maxadamo/galera_proxysql/-/blob/master/manifests/params.pp#L54
I have enabled version and version pinning in puppet to avoid future updates.
Future mitigation
In general, it's a good practice to always pin critical applications.