Observability — Healthcheck, Metrics & Monitoring

Gubernator exposes all observability endpoints on port 4002, which is public (no authentication required) to allow easy scraping by Prometheus and other monitoring tools.

Port 4002 is intentionally public

This port is designed for internal infrastructure monitoring. In production, firewall it from the public internet and only expose it to your Prometheus / monitoring network.

---

Endpoints Summary

Endpoint	Method	Description
/health	GET	JSON health check — use for load balancers and readiness probes
/metrics	GET	Prometheus-format metrics (Gubernator + Go runtime)
/swagger/index.html	GET	Interactive Swagger UI for the REST API

---

Health Check

HTTP Endpoint

curl http://localhost:4002/health

Response when healthy:

{"status": "healthy"}

The endpoint always returns 200 OK as long as the Gubernator process is running. It can be used as:

Docker HEALTHCHECKDocker ComposeKubernetes ProbeCLI / Script

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD ["/app/gbnt", "health"]

This is already built into the official Dockerfile and is completely self-contained (requires no external shell, curl, or wget).

services:
  gubernator:
    image: gubernator:latest
    healthcheck:
      test: ["CMD", "/app/gbnt", "health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

livenessProbe:
  httpGet:
    path: /health
    port: 4002
  initialDelaySeconds: 10
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health
    port: 4002
  initialDelaySeconds: 5
  periodSeconds: 10

# Wait until Gubernator is healthy before running commands
until curl -sf http://localhost:4002/health; do
  echo "Waiting for Gubernator..."
  sleep 2
done
echo "Gubernator is healthy!"

---

Prometheus Metrics

Available Metrics

Gubernator Custom Metrics

Metric	Type	Description
gbnt_total_nodes	Gauge	Number of nodes currently registered in the cluster
gbnt_total_tasks	Gauge	Number of tasks (containers) scheduled in the cluster

These update every 15 seconds via the internal startWatchtowers() loop.

Go Runtime Metrics (automatic)

Metric	Description
go_goroutines	Number of running goroutines
go_memstats_alloc_bytes	Heap memory in use
go_memstats_sys_bytes	Total memory obtained from OS
go_gc_duration_seconds	GC pause durations (p50, p75, p99)
go_info	Go version information
process_cpu_seconds_total	CPU time consumed
process_open_fds	Number of open file descriptors

Scraping Manually

# Raw Prometheus text format
curl http://localhost:4002/metrics

# Filter only Gubernator metrics
curl -s http://localhost:4002/metrics | grep "^gbnt_"

Example output:

# HELP gbnt_total_nodes Current number of nodes registered in the cluster.
# TYPE gbnt_total_nodes gauge
gbnt_total_nodes 2
# HELP gbnt_total_tasks Current number of tasks scheduled in the cluster.
# TYPE gbnt_total_tasks gauge
gbnt_total_tasks 5

---

Prometheus Configuration

Minimal scrape config

Add this to your existing prometheus.yml:

scrape_configs:
  - job_name: 'gubernator'
    scrape_interval: 15s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['<gubernator-host>:4002']
        labels:
          service: 'gubernator'

Replace <gubernator-host> with: - localhost — if Prometheus runs on the same machine - gubernator — if running in the same Docker Compose network - host.docker.internal — if Prometheus is in Docker and Gubernator is on the host (Mac/Windows)

---

Monitoring Stack (Prometheus + Grafana)

The repository includes a ready-to-use monitoring/ directory with:

monitoring/
├── docker-compose.yml                          # Gubernator + Prometheus + Grafana
├── prometheus/
│   └── prometheus.yml                          # Pre-configured scrape job
└── grafana/
    ├── provisioning/
    │   ├── datasources/prometheus.yml          # Auto-connects Prometheus
    │   └── dashboards/dashboards.yml           # Auto-loads dashboards
    └── dashboards/
        ├── gubernator.json                     # Pre-built Gubernator dashboard
        └── network_dashboard.json              # Pre-built Network Monitor dashboard

Start the Full Stack

# Build Gubernator image first (if not using Docker Hub)
docker build -t gubernator:latest .

# Launch everything
cd monitoring/
docker compose up -d

Verify

# Check all containers are healthy
docker compose ps

# Check Gubernator is being scraped
curl 'http://localhost:9090/api/v1/query?query=up{job="gubernator"}' | python3 -m json.tool

Access

Service	URL	Role
Gubernator Web UI	http://localhost:4001	Dashboard (Monitoring, Loki Logs, Tracing, Topology, Security)
Loki Logs Explorer	http://localhost:4001 / Loki Logs	Real-time cluster & container log explorer
Prometheus	http://localhost:9090	Metrics collection & PromQL engine
Monitoring (Grafana)	http://localhost:3000 or /grafana/	Host & container telemetry dashboards
Jaeger UI	http://localhost:16686 or /jaeger/	Distributed tracing explorer
Jaeger OTLP gRPC	localhost:4317	OpenTelemetry gRPC trace ingestion
Jaeger OTLP HTTP	http://localhost:4318	OpenTelemetry HTTP trace ingestion
Gubernator Metrics	http://localhost:4002/metrics	Internal Prometheus metrics
Health Check	http://localhost:4002/health	Healthcheck probe endpoint

---

Grafana Telemetry Dashboards

💡 Logs Navigation Update: Logs are now explored directly within the dedicated Loki Logs Explorer tab in the Gubernator Dashboard, allowing Grafana to focus purely on high-frequency time-series metrics and hardware telemetry.

Two dashboards are provisioned automatically: Gubernator — Cluster Overview and Gubernator — Network Monitor.

1. Cluster Overview

The Gubernator — Cluster Overview dashboard shows:

┌────────────────────────────────────────────────────────────────┐
│  Active Nodes  │  Total Tasks  │  Status  │  Goroutines        │
│      ██ 1      │     ██ 3      │  ✅ UP   │      ██ 12         │
├────────────────────────────────────────────────────────────────┤
│  Nodes & Tasks Over Time        │  Memory Usage                │
│  ─────────────────────────────  │  ─────────────────────────── │
│  nodes ─────── 1                │  heap ────────────────        │
│  tasks ╱──────╱ 3               │  sys  ─────────────────       │
├────────────────────────────────────────────────────────────────┤
│  Goroutines Over Time  │  GC Rate  │  GC Pause Duration        │
└────────────────────────────────────────────────────────────────┘

Panels included:

Panel	Metric	Type
Active Nodes	gbnt_total_nodes	Stat
Total Tasks	gbnt_total_tasks	Stat
Gubernator Status	up{job="gubernator"}	Stat (UP/DOWN)
Goroutines	go_goroutines	Stat
Nodes & Tasks Over Time	gbnt_total_nodes, gbnt_total_tasks	Time series
Memory Usage	go_memstats_alloc_bytes, go_memstats_sys_bytes	Time series
Goroutines	go_goroutines	Time series
GC Rate	rate(go_gc_duration_seconds_count[5m])	Time series
GC Pause Duration	go_gc_duration_seconds p50/p99	Time series

2. Network Monitor

The Gubernator — Network Monitor dashboard combines metrics from node-exporter (host systems) and cAdvisor (containers) to provide a unified view of all network traffic. It shows:

• Global Network Traffic (In/Out) across the entire cluster
• Traffic sorted by Node/Host
• Network bandwidth per Container (Pod)

This allows visualizing both underlying node infrastructure and overlay container traffic simultaneously.

---

Jaeger Distributed Tracing

The SRE stack includes Jaeger (gbnt-monitor-jaeger) accepting OpenTelemetry traces over: - OTLP gRPC: localhost:4317 - OTLP HTTP: http://localhost:4318/v1/traces - Jaeger UI: Embedded at /jaeger/ on port 4001 or direct port :16686.

To generate synthetic OpenTelemetry trace traffic across various scenarios (Checkout, User Auth, Search, Error Handling), run the included traffic generator scripts:

# Python OpenTelemetry Traffic Generator:
python3 examples/example-jaeger/generate_traces.py --count 15 --scenario all --target otlp

# POSIX Shell Trace Generator (curl):
./examples/example-jaeger/send_traces.sh 10

📘 See Distributed Tracing Example (example-jaeger) for a full multi-service tutorial with domain jaeger.gbnt.local.

---

Tear Down

cd monitoring/
docker compose down       # stop containers, keep volumes
docker compose down -v    # stop containers AND delete volumes (reset all data)