Info

  • Hardware
  • Giantommaso Fogli

Table of contents

  1. 4 hours of downtime, a failover system, and radical transparency
  2. What happened
  3. What we did this morning
  4. Why we tell this
  5. What’s left to do
  6. Sources
  7. Sources
4 hours of downtime, a failover system, and radical transparency4 hours of downtime, a failover system, and radical transparency

4 hours of downtime, a failover system, and radical transparency

11/07/2026 / 5 minutes to read / Tags: infrastruttura, resilienza, self-hosted, trasparenza, DNS failover

Last night lake8.dev was offline for about 5 hours.

It wasn’t an attack. It wasn’t a bug. It was an electrical surge that tripped the circuit breaker — one of those things that happen at night, cutting power while you sleep and there’s nothing you can do until you wake up.

We write this because radical transparency is one of our core values. The traffic data of lake8.dev is public on stats.lake8.dev. The downtime is visible in the logs. It makes no sense not to talk about it.

What happened

Nighttime electrical surge — the circuit breaker tripped. The infrastructure of lake8.dev runs on physical hardware in San Pietro in Casale — a Raspberry Pi 4B, an Orange Pi Zero 3, an Orange Pi Lite. Without power, everything shuts down.

We have an Eaton 5E900UD UPS installed, but for 5 hours without power, the battery is not sufficient. The site remained offline until the morning, when we found the tripped circuit breaker, restored power, and rebooted the systems automatically.

No data lost. No damage to the infrastructure. Just 4 hours of unreachability.

What we did this morning

As soon as we woke up, we implemented an automatic DNS failover system.

How it works:

A monitor runs on the Orange Pi Zero 3 every 60 seconds. It tests connectivity to the outside. If it detects that Eolo (our ISP) is unreachable, it automatically executes this sequence:

No Eolo connectivity
    ↓
Automatic switch to 4G backup connection (via USB ethernet)
    ↓
GoDaddy API call → DNS lake8.dev changed to GitHub Pages
    ↓
lake8.dev → maintenance page HTTPS
    ↓
When Eolo returns → DNS restored → site online

The maintenance page is already live, multilingual (IT/EN/DE/ES), accessible via HTTPS with a valid certificate.

⚠️ Errata corrige — 12 Jul 2026
A previous version of this article stated failover time as ~90 seconds and recovery time as ~60 seconds. Those values referred to the technical DNS switch, not the time perceived by visitors. The TTL propagation (600s, ~10 minutes) was missing. Correct times are shown below.

Failover time verified in test: ~90 seconds for the DNS switch + ~10 minutes TTL propagation (600s). Total time before visitors see the maintenance page: ~12 minutes.

Recovery time verified in test: ~60 seconds for the DNS switch + ~10 minutes TTL propagation (600s). Total time before visitors see the site back online: ~12 minutes.

Why we tell this

Because lake8.dev is the zero customer of its own products and philosophies.

One of the pillars of Lagotto BI and the lake8.dev ecosystem is radical transparency — real, public, verifiable data, without filters. If we hide a nighttime outage, we are applying that philosophy only when it is convenient.

The downtime happened. The failover system didn’t exist. Now it exists. The data on stats.lake8.dev show the traffic gap. All verifiable.

What’s left to do

The DNS failover system is active and tested. But full resilience still requires:

  • NUT configuration — controlled shutdown when the UPS battery falls below critical threshold
  • Watchdog script — automatic detection and reset in case of system freeze
  • Complete test sequence — end-to-end blackout simulation

We will implement them in the coming weeks. When they are ready, we will write about it.

Update — July 2026

The failover system described in this article has been updated.

The dedicated 4G phone that acted as the backup control and routing system has been decommissioned. The reason is practical: summer thunderstorms in Emilia-Romagna knock out mobile carriers too. A control system that depends on the cellular network is unreliable precisely when it is needed most.

Control has been moved to a Cloudflare Worker that runs a TCP probe on port 443 every minute. If the main server does not respond, the Worker automatically updates the DNS records via the GoDaddy API and activates the maintenance page on GitHub Pages (HTTPS, dedicated CNAME).

Effective switch time: ~90 seconds for the DNS change, plus about 10 minutes of TTL propagation. Tested end-to-end in production.

The system is now independent of the mobile network and of any local physical hardware — control lives on Cloudflare’s edge, outside the rack.

lake8.dev — San Pietro in Casale, Bologna, July 11, 2026 All cited data is verifiable in real-time on stats.lake8.dev

Sources

Author: Giantommaso Fogli
Publication Date: 2026-07-11

Rights and attributions

Images, logos, and photographs are the property of their rightful owners. Used for commentary purposes.

Sources

Author: Giantommaso Fogli
Publication Date: 2026-07-11

Rights and Attribution

Images, logos, and photographs are the property of their respective owners. Used for commentary purposes.

← Back to blog