My Homelab Monitoring Stack: Observability on a Budget

My homelab runs 14 services — reverse proxy, Git server, media stack, backup daemon, a few experiments — on a used mini-PC with 16 GB of RAM and a 1 TB SSD. For a year I ran it with no monitoring at all: services either worked or didn't, and I found out when something stopped responding. Then I spent a weekend building a real observability stack, and the difference isn't just "I get alerts" — it's that I now know what my machines are doing before they break. This is that stack: the pieces, the configs, and the honest lessons from a year of running it.

The stack

Four components, all free, all self-hosted, all running as containers on the same box:

Prometheus — scrapes metrics from everything, stores time series.
node_exporter — exposes OS metrics (CPU, memory, disk, network).
Grafana — dashboards over Prometheus data.
Loki + Promtail — logs, collected and searchable without a heavy Elastic stack.
Alertmanager — routes Prometheus alerts to your phone (I use ntfy.sh; no account needed, push notifications for free).

The design goal: metrics answer "is it healthy now?", logs answer "what happened then?", and alerts answer "should I be awake?" — three questions, three tools, one compose file.

The compose file

Everything lives in one docker-compose.yml, because the homelab virtue is explicable:

1
services:
2
  prometheus:
3
    image: prom/prometheus:latest
4
    volumes:
5
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
6
      - prom-data:/prometheus
7
    ports: ["9090:9090"]
8
    restart: unless-stopped
9

10
  node-exporter:
11
    image: prom/node-exporter:latest
12
    network_mode: host
13
    restart: unless-stopped
14

15
  grafana:
16
    image: grafana/grafana:latest
17
    environment:
18
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
19
    volumes:
20
      - grafana-data:/var/lib/grafana
21
    ports: ["3000:3000"]
22
    restart: unless-stopped
23

24
  loki:
25
    image: grafana/loki:latest
26
    ports: ["3100:3100"]
27
    restart: unless-stopped
28

29
volumes:
30
  prom-data:
31
  grafana-data:

The one-line secrets note: ${GRAFANA_PASSWORD} comes from a .env file next to the compose file — never commit credentials, even for a homelab. The whole stack idles around 1.5 GB of RAM, which is the entire budget allocation and the reason I chose Loki over Elasticsearch.

What to scrape: the 80/20

You don't need to instrument everything on day one. The 80/20 scrape targets that catch the important failures:

1
# prometheus.yml — the essentials
2
scrape_configs:
3
  - job_name: "node"
4
    static_configs:
5
      - targets: ["localhost:9100"]
6

7
  - job_name: "docker"
8
    static_configs:
9
      - targets: ["localhost:9323"]  # Docker engine metrics
10

11
  - job_name: "blackbox"
12
    metrics_path: /probe
13
    params:
14
      module: [http_2xx]
15
    static_configs:
16
      - targets:
17
          - "https://git.example.com"      # is my Git server up?
18
          - "https://media.example.com"    # is the media stack up?
19
          - "https://backup.example.com"   # is the dashboard up?
20
    relabel_configs:
21
      - source_labels: [__address__]
22
        target_label: __param_target
23
      - source_labels: [__param_target]
24
        target_label: instance

The blackbox exporter is the sleeper hit: it probes your services from the outside, exactly the way your users experience them. "The container is running" and "the service responds" are different facts, and blackbox checks the second one.

The alert that's actually useful

The first alert everyone writes is "CPU > 90%." It's also the first alert everyone mutes, because a CPU spike that recovers in a minute is noise. The alerts that earn their keep are the ones about availability and trends:

1
# prometheus.yml — alerting rules
2
groups:
3
  - name: homelab
4
    rules:
5
      - alert: ServiceDown
6
        expr: probe_success == 0
7
        for: 2m
8
        labels:
9
          severity: critical
10
        annotations:
11
          summary: "{{ $labels.instance }} is down"
12

13
      - alert: DiskFilling
14
        expr: disk_used_bytes / disk_total_bytes > 0.85
15
        for: 30m
16
        labels:
17
          severity: warning
18
        annotations:
19
          summary: "{{ $labels.mountpoint }} > 85%"

Note the for: clauses. for: 2m means "down for two consecutive minutes" — a blip during a restart doesn't page you. for: 30m on disk means the alert fires on a trend, not a moment, which is exactly when you still have time to act. If your alerts fire on moments, you'll mute them; if they fire on trends, you'll fix things.

The dashboard philosophy

Grafana dashboards are where monitoring goes to die if you're not careful — 40 panels of lines nobody reads. My rules after a year:

Three panels per service, max. Uptime, resource usage, and a trend. If a service needs ten panels, it needs its own dashboard.
Red/green, not rainbow. The dashboard should tell you at a glance what's wrong. Everything healthy = green; anything broken = red; the in-between is for clicking into.
Alert-backed, not dashboard-driven. The dashboard is for after the alert wakes you up. The alert is the system that's actually watching.

Logs with Loki: cheap and searchable

Loki is the log database that respects a 16 GB budget: it indexes labels, not full text, so it stays small and fast on modest hardware. Promtail ships container logs to it, and the payoff is the query that saved me twice already:

1
{job="docker"} |= "panic" |= "media-server"

"Show me every panic in the media server's logs, ever." On Elasticsearch that's a whole cluster; on Loki it's a query against a few hundred MB. For homelab scale, Loki is the correct answer, full stop.

The lessons

A year in, the stack has caught: a disk filling from a runaway backup (30-minute warning alert, fixed at leisure), a reverse-proxy cert that stopped renewing (blackbox probe went red, fixed in five minutes), and one actual crash (ServiceDown alert at 3 a.m., which I silenced and investigated at 8 a.m. — the alert did its job: I chose when to deal with it).

The honest lessons:

Start with blackbox probes. Before metrics, before logs — know when your services stop responding. Everything else is refinement.
Alert on trends, not moments. for: 2m and for: 30m are the difference between alerts you act on and alerts you mute.
One weekend to set up, forever to tune. The initial stack took an afternoon; the tuning — thresholds, alert fatigue, dashboard pruning — is the ongoing cost, and it's worth paying.
The stack must be boring. No beta software, no hand-rolled exporters, no experiment-as-infrastructure. The monitoring system that breaks is worse than no monitoring system.

The takeaway

Observability on a homelab budget is not a compromise — it's a discipline problem, and the free stack solves it completely. Metrics for health, logs for history, alerts for trends, all in one compose file on the box that was already running the services. The best part isn't the dashboards or the notifications. It's the peace of mind that comes from knowing your machines will tell you when they're sick — instead of finding out the way I used to, from the outside, when everything was already down.