I host a server out at my parents’ lake house. It’s been great having an off-site backup location, but sometimes nobody is there for weeks at a time and, as technology tends to do, it breaks. Recently our biggest challenge is that the router or modem gets stuck. It requires the age-old solution of turning it off and back on again. I tried using the Unifi API to get the router to reboot, but when the router itself is non-responsive we really need the ability to cut power fully and turn itb ack on. Easier said than done when the server is 1,000+ miles away. So I got to work on a DIY smart outlet that can fix the problem itself.
Okay, it’s not very pretty… But it works!
Planning
The idea was pretty simple. Plug the router and modem into a socket that is controllable by an ESP32. The ESP32 can periodically check whether the internet is reachable, and do a power cycle if certain conditions are met. I wanted the checks to happen on the device itself rather than in something like Home Assistant because if the internet or local network is the thing that is broken, relying on multi-device communication to tell the outlet to reboot would kind of defeat the purpose. Home Assistant is still useful for seeing the current state of the device and controlling it manually, but the rescue behavior needed to be on-board and autonomous.
I wrote a script so the device pings 8.8.8.8, 1.1.1.1, and the router every minute. Checking all three makes me feel like we’re making a slightly more informed decision than “one ping failed, everybody panic!” The two dns addresses are the actual test for whether the internet is down. If either one answers, the failure counters reset. When both fail, the router’s available just helps decide how urgently to react. If the router is also unreachable, the router is probably borked and gets rebooted after five failed checks. If the router still answers, it might just be an ISP outage that rebooting won’t fix, so the watchdog waits fifteen checks before power cycling.
I baked in a simple backoff too. After a reboot, it waits at least ten minutes for everything to come back. If the internet is still broken, that doubles to twenty, then forty, and finally caps at sixty minutes. A successful check resets it back to ten. Hopefully that will ensure we aren’t putting a bunch of strain on the router/modem.
Putting it together
The parts list was pretty short: an ESP32, a relay switch, and a one-foot power cable that I cut in half. The ESP32 controls the relay, the relay controls power to the outlet, and ESPHome handles everything else.
I wired through the relay’s normally-closed contact which means when it is in its relaxed state, the router still has power. To turn it off, the ESP32 has to energize the relay. That means if the ESP32 crashes, loses power, or otherwise fails, it doesn’t take the router down with it. I used ESPHome for the software side of things because it handles wifi, over-the-air updates, Home Assistant integration, prometheus metrics, and all the other plumbing I’ll ever need.
switch:
- platform: gpio
pin: 2
id: router_relay
inverted: false
restore_mode: ALWAYS_OFF
script:
- id: cycle_router
mode: single
then:
- switch.turn_on: router_relay
- delay: 10s
- switch.turn_off: router_relayDisclaimer, I am not an electrician, and this project involves mains voltage. Don’t take this as wiring advice. I probably did this wrong. Use properly rated components. Keep everything enclosed, and don’t build one unless you know how to work with mains power safely. Hopefully I don’t cause a fire 😬
The Enclosure
I found a couple of existing 3D models for holding the relay and the ESP32, but the dimensions were very off… After enough flexing and clipping off pieces that were in the way, I had an enclosure. It is not going to win any awards, but it keeps everything contained and it works fine enough for our use case.
Testing the Watchdog
I didn’t want the first real test to happen while nobody was at the lake house, so I used ESPHome to exercise the relay and simulate different kinds of failures. A brief internet hiccup should do nothing. A sustained outage should cut power, wait long enough for everything to fully shut down, restore power, and then give the router and modem enough time to boot before judging them again. The important part is avoiding a device that reboots too often.
The Prometheus integration provides data points like packet loss, wifi signal, CPU temperature, failure counts, backoff time, and lifetime power cycles. If the heartbeat stops moving, I’ll know that I need to make a watchdog for the watchdog. For a little extra paranoia, the ESP32 safely reboots itself every night at 3 AM because I’ve had some issues with them getting stuck while running unattended as well.
Home Assistant
I added the outlet to Home Assistant through the ESPHome integration like I did with my light bars, and it showed up like any other smart device. I can manually cycle the router, see all the diagnostics, and disable the watchdog while doing maintenance so it doesn’t reboot things out from under me. It’s nice to use Home Assistant as an extra control surface without depending on it. The outlet still does its one job on its own.
It’s so fun working on these little projects. It keeps blowing my mind how many of the smart devices we all own are capable of this stuff. They just don’t expose the software layer to us. I’d have much preferred a commercial wifi outlet product that I could flash a custom firmware on instead of depending on my sketchy skills. Maybe by making these side projects we can show demand for having more control available to us.
The Full Configuration
This is the complete configuration I’m running just to show off how simple ESPHome makes this stuff.
esphome:
name: router-power-watchdog
esp32:
board: esp32-c6-devkitc-1
framework:
type: esp-idf
logger:
api:
reboot_timeout: 0s
encryption:
key: !secret api_encryption_key
ota:
- platform: esphome
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
ap:
ssid: "Router-Power-Watchdog"
password: !secret fallback_ap_password
captive_portal:
web_server:
port: 80
prometheus:
include_internal: false
external_components:
- source:
type: git
url: https://github.com/trombik/esphome-component-ping
ref: main # TODO: Pin to a commit SHA before a future OTA build.
switch:
- platform: gpio
pin: 2
name: "Router Power Relay"
id: router_relay
inverted: false
restore_mode: ALWAYS_OFF
- platform: template
name: "Watchdog Enabled"
id: watchdog_enabled
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
button:
- platform: template
name: "Manual Router Power Cycle"
on_press:
- script.execute: cycle_router
script:
- id: cycle_router
mode: single
then:
- switch.turn_on: router_relay
- delay: 10s
- switch.turn_off: router_relay
globals:
- id: consecutive_failures
type: int
restore_value: no
initial_value: "0"
- id: current_backoff_minutes
type: int
restore_value: yes
initial_value: "10"
- id: total_power_cycles
type: int
restore_value: yes
initial_value: "0"
- id: upstream_failures
type: int
restore_value: no
initial_value: "0"
- id: upstream_outages
type: int
restore_value: yes
initial_value: "0"
- id: watchdog_heartbeat
type: int
restore_value: no
initial_value: "0"
- id: cooldown_remaining_checks
type: int
restore_value: no
initial_value: "10"
- id: is_cooldown_active
type: bool
restore_value: no
initial_value: "true"
sensor:
- platform: ping
ip_address: 8.8.8.8
num_attempts: 2
timeout: 2s
update_interval: 60s
id: ping_google
loss:
name: "Google Packet Loss"
id: google_loss
- platform: ping
ip_address: 192.168.1.1
num_attempts: 2
timeout: 2s
update_interval: 60s
id: ping_gateway
loss:
name: "Gateway Packet Loss"
id: gateway_loss
- platform: ping
ip_address: 1.1.1.1
num_attempts: 2
timeout: 2s
update_interval: 60s
id: ping_cloudflare
loss:
name: "Cloudflare Packet Loss"
id: cloudflare_loss
on_value:
then:
- lambda: |-
id(watchdog_heartbeat)++;
if (!id(watchdog_enabled).state) {
ESP_LOGD("watchdog", "Watchdog disabled. Skipping.");
return;
}
if (id(is_cooldown_active)) {
if (id(cooldown_remaining_checks) > 0) {
ESP_LOGI("watchdog", "Cooldown: %d minutes remaining.", id(cooldown_remaining_checks));
id(cooldown_remaining_checks)--;
return;
} else {
id(is_cooldown_active) = false;
ESP_LOGI("watchdog", "Cooldown expired. Monitoring resumed.");
}
}
bool google_failed = (id(google_loss).state > 99.0);
bool cloudflare_failed = (x > 99.0);
bool gateway_failed = (id(gateway_loss).state > 99.0);
bool internet_down = google_failed && cloudflare_failed;
const int WEDGE_THRESHOLD = 5;
const int UPSTREAM_THRESHOLD = 15;
if (internet_down) {
if (gateway_failed) {
id(consecutive_failures)++;
ESP_LOGW("watchdog", "Router wedged: %d/%d", id(consecutive_failures), WEDGE_THRESHOLD);
} else {
id(upstream_failures)++;
ESP_LOGW("watchdog", "Internet down, gateway healthy: %d/%d", id(upstream_failures), UPSTREAM_THRESHOLD);
}
} else {
id(consecutive_failures) = 0;
id(upstream_failures) = 0;
id(current_backoff_minutes) = 10;
}
bool wedged = id(consecutive_failures) >= WEDGE_THRESHOLD;
bool upstream_stuck = id(upstream_failures) >= UPSTREAM_THRESHOLD;
if (wedged || upstream_stuck) {
if (upstream_stuck) {
id(upstream_outages)++;
}
id(total_power_cycles)++;
id(cycle_router).execute();
int next_cooldown = id(current_backoff_minutes);
id(cooldown_remaining_checks) = next_cooldown;
id(is_cooldown_active) = true;
id(consecutive_failures) = 0;
id(upstream_failures) = 0;
int doubled = next_cooldown * 2;
id(current_backoff_minutes) = doubled > 60 ? 60 : doubled;
}
- platform: wifi_signal
name: "Watchdog WiFi Signal"
id: wifi_signal_db
update_interval: 30s
- platform: internal_temperature
name: "Watchdog CPU Temperature"
id: cpu_temp
update_interval: 30s
- platform: template
name: "Watchdog Ping Failures"
unit_of_measurement: "drops"
accuracy_decimals: 0
lambda: return id(consecutive_failures);
update_interval: 30s
- platform: template
name: "Watchdog Backoff Minutes"
unit_of_measurement: "min"
accuracy_decimals: 0
lambda: return id(current_backoff_minutes);
update_interval: 30s
- platform: template
name: "Watchdog Lifetime Power Cycles"
unit_of_measurement: "cycles"
accuracy_decimals: 0
lambda: return id(total_power_cycles);
update_interval: 30s
- platform: template
name: "Watchdog Upstream Outages"
unit_of_measurement: "outages"
accuracy_decimals: 0
lambda: return id(upstream_outages);
update_interval: 30s
- platform: template
name: "Watchdog Heartbeat"
unit_of_measurement: "checks"
accuracy_decimals: 0
lambda: return id(watchdog_heartbeat);
update_interval: 30s
time:
- platform: sntp
id: sntp_time
on_time:
- seconds: 0
minutes: 0
hours: 3
then:
- lambda: 'ESP_LOGI("maintenance", "Executing daily maintenance reboot...");'
- switch.turn_off: router_relay
- delay: 500ms
- lambda: 'App.safe_reboot();'
Comments
Powered by Giscus (may need to disable adblockers)