Prometheus + Grafana explained (without the jargon)

Every Kubernetes tutorial says “add Prometheus + Grafana.” Almost none explain what they are.

Real answer in 30 seconds.

The 30-second explanation

Prometheus = a robot that walks around your cluster every 15 seconds and asks each app: “How are you doing right now?” Each app answers with numbers (CPU used, requests received, errors). Prometheus writes those numbers with timestamps.

Grafana = a picture-maker that reads Prometheus’s notebook and draws graphs.

Alertmanager = the guy who phones you at 2 AM when something’s broken.

That’s it. Everything else is detail.

Real-world analogy

Imagine you own 10 restaurants.

• Prometheus = the manager who visits every restaurant every 15 minutes and writes down: “Restaurant 3: 42 customers, 2 complaints, 1 chef sick.” Keeps notes for a year.
• Grafana = the office wall with charts: “Complaints per restaurant this week.”
• Alertmanager = calls YOUR phone at 2 AM based on rules like “call me if any restaurant has >5 complaints.”

Install both on minikube (5 min)

Use the kube-prometheus-stack Helm chart — bundles everything:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

Wait 2 minutes. Verify:

kubectl get pods -n monitoring

You’ll see 6 pods running. Each one does a specific job.

What each pod does (plain words)

See Grafana in your browser

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

Browser: http://localhost:3000

• Username: admin
• Password: prom-operator

Sidebar → Dashboards → Browse → click Kubernetes / Compute Resources / Cluster

You’ll see live CPU/memory graphs of your minikube. All this data was auto-collected. Zero config from you.

The 3 verbs to know

1. Scrape — Prometheus asks apps for numbers (HTTP GET on /metrics)
2. Query — you ask Prometheus “what did X look like at 3 PM?” using PromQL
3. Visualize — Grafana draws that query as a chart

PromQL — the query language (5 examples)

up                                    # 1 for healthy targets, 0 for down
kube_pod_info                         # info about every pod
sum(rate(http_requests_total[5m]))    # requests per second across all apps
node_memory_MemAvailable_bytes/1024/1024/1024   # available memory in GB
kube_pod_container_status_restarts_total > 0    # pods that restarted

Run these in the Prometheus UI:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Browser: http://localhost:9090

Adding YOUR app to Prometheus

Two steps:

1. Your app exposes /metrics endpoint (Prometheus client libraries handle this — Python, Go, Node, Java all have them)
2. Create a ServiceMonitor YAML telling Prometheus to scrape your service

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  namespace: monitoring
  labels:
    release: prometheus       # must match the Prometheus's serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: myapp              # match your Service's labels
  namespaceSelector:
    matchNames: [default]
  endpoints:
  - port: web
    path: /metrics
    interval: 30s

Apply. Wait 1 min. Prometheus starts scraping your app.

Alerts

PrometheusRule CRD defines alerts:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-alerts
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
  - name: myapp
    rules:
    - alert: MyAppHighErrorRate
      expr: sum(rate(myapp_requests_total{status="500"}[5m])) > 1
      for: 2m
      labels: { severity: warning }
      annotations:
        summary: "MyApp returning too many 5xx"

Fire → Alertmanager routes → Slack/email/PagerDuty.

The 3 things beginners miss

1. Labels are how everything joins. app: myapp on Deployment must match Service selector must match ServiceMonitor. One mismatch = no data.
2. Prometheus doesn’t retain forever. Default is 15 days. Add Thanos or Grafana Mimir for months/years.
3. Cardinality kills memory. Don’t use labels with unbounded values (user_id, request_id, timestamp). Prometheus RAM explodes.

Prometheus vs Datadog vs Splunk

For Kubernetes: Prometheus is the default. Companies switch to Datadog when they want zero ops overhead.

What you actually do day-to-day

• Rarely: install this whole stack (once per cluster)
• Sometimes: add /metrics to your app + ServiceMonitor
• Often: open Grafana → find why something’s slow
• Sometimes: write alert rules

80% of value comes from just looking at pre-built dashboards.

Start here

Install kube-prometheus-stack (5 min). Open Grafana. Click through 3 pre-built dashboards. That’s it — you now understand Kubernetes observability.

Deeper YAML + PromQL comes later, only when you need it.