4 hours of downtime, a failover system, and radical transparency

7/11/2026 / 4 minutes to read / Tags: infrastruttura, resilienza, self-hosted, trasparenza, DNS failover

Last night lake8.dev was offline for about 5 hours.

It wasn’t an attack. It wasn’t a bug. It was an electrical surge that tripped the circuit breaker — one of those things that happen at night, cutting power while you sleep and there’s nothing you can do until you wake up.

We write this because radical transparency is one of our core values. The traffic data of lake8.dev is public on stats.lake8.dev. The downtime is visible in the logs. It makes no sense not to talk about it.

What happened

Nighttime electrical surge — the circuit breaker tripped. The infrastructure of lake8.dev runs on physical hardware in San Pietro in Casale — a Raspberry Pi 4B, an Orange Pi Zero 3, an Orange Pi Lite. Without power, everything shuts down.

We have an Eaton 5E900UD UPS installed, but for 5 hours without power, the battery is not sufficient. The site remained offline until the morning, when we found the tripped circuit breaker, restored power, and rebooted the systems automatically.

No data lost. No damage to the infrastructure. Just 4 hours of unreachability.

What we did this morning

As soon as we woke up, we implemented an automatic DNS failover system.

How it works:

A monitor runs on the Orange Pi Zero 3 every 60 seconds. It tests connectivity to the outside. If it detects that Eolo (our ISP) is unreachable, it automatically executes this sequence:

No Eolo connectivity
    ↓
Automatic switch to 4G backup connection (via USB ethernet)
    ↓
GoDaddy API call → DNS lake8.dev changed to GitHub Pages
    ↓
lake8.dev → maintenance page HTTPS
    ↓
When Eolo returns → DNS restored → site online

The maintenance page is already live, multilingual (IT/EN/DE/ES), accessible via HTTPS with a valid certificate.

Failover time verified in test: ~90 seconds for the DNS switch + ~10 minutes TTL propagation (600s). Total time before visitors see the maintenance page: ~12 minutes.

Recovery time verified in test: ~60 seconds for the DNS switch + ~10 minutes TTL propagation (600s). Total time before visitors see the site back online: ~12 minutes.

Why we tell this

Because lake8.dev is the zero customer of its own products and philosophies.

One of the pillars of Lagotto BI and the lake8.dev ecosystem is radical transparency — real, public, verifiable data, without filters. If we hide a nighttime outage, we are applying that philosophy only when it is convenient.

The downtime happened. The failover system didn’t exist. Now it exists. The data on stats.lake8.dev show the traffic gap. All verifiable.

What’s left to do

The DNS failover system is active and tested. But full resilience still requires:

• NUT configuration — controlled shutdown when the UPS battery falls below critical threshold
• Watchdog script — automatic detection and reset in case of system freeze
• Complete test sequence — end-to-end blackout simulation

We will implement them in the coming weeks. When they are ready, we will write about it.

lake8.dev — San Pietro in Casale, Bologna, July 11, 2026 All cited data is verifiable in real-time on stats.lake8.dev

Sources

Author: Giantommaso Fogli
Publication Date: 2026-07-11

Rights and attributions

Images, logos, and photographs are the property of their rightful owners. Used for commentary purposes.

Sources

Author: Giantommaso Fogli
Publication Date: 2026-07-11

Rights and Attribution

Images, logos, and photographs are the property of their respective owners. Used for commentary purposes.