Observability concepts

Observability concepts — three signals, the pull model, and checking `up` first

Mental model: when something breaks in production, observability is your ability to ask "why?"
from the outside. It rests on three signals, each answering a different question. Metrics are
numbers over time (request rate, memory, CPU) — cheap to store, perfect for dashboards and
alerts; they tell you THAT something is wrong. Logs are discrete events with detail; they tell
you WHAT happened on one request. Traces follow a single request across services; they tell you
WHERE the time went. Prometheus owns the metrics pillar: it stores numeric time series, each one
a metric name plus labels (dimensions like job, instance, method, status), so one name fans out
into many series you slice with PromQL.

1 · The three signals — reach for the right one
    Metrics  -> numbers over time (rate, memory, CPU). Aggregatable, cheap, alertable.
                Use for: dashboards, alerts, "is anything wrong right now?"
    Logs     -> discrete timestamped events with detail. Use for: "what exactly happened
                on THIS request / at this moment?" (the error message, the stack trace).
    Traces   -> one request stitched across services (spans). Use for: "where did the time
                go / which hop failed?" in a distributed call path.
    # Rule of thumb: metrics tell you SOMETHING is wrong and alert you; you then drop into
    #   logs for the detail, and traces to localize a slow/failing hop across services.

2 · Prometheus's PULL model — it scrapes, targets don't push
    # The key design choice: Prometheus PULLS. On a fixed interval it reaches out over HTTP and
    #   scrapes each target's /metrics endpoint — one HTTP GET per interval, no agent pushing in.
    #   Each target exposes its numbers as plain text on /metrics; that's the contract.
    curl -s http://node-exporter:9100/metrics        # what Prometheus GETs each interval
    # PULL vs PUSH:
    #   PULL (Prometheus)  -> Prometheus owns the schedule and the target list, so it always
    #                         knows exactly who it is SUPPOSED to be talking to. A target that
    #                         vanishes is visible as a failed scrape (see `up` below).
    #   PUSH (statsd-style)-> targets send whenever they like; a silent target just goes quiet
    #                         and you can't tell "healthy but idle" from "dead."
    # Push has a place (short-lived batch jobs -> the Pushgateway), but the default and the
    #   thing to reason about is the pull loop on an interval.

3 · Check the `up` metric FIRST — the health signal Prometheus generates for free
    # For every target it scrapes, Prometheus records a synthetic metric `up`:
    #   up == 1  -> the scrape succeeded (the target answered on /metrics)
    #   up == 0  -> the target was unreachable / the scrape failed
    # You don't instrument anything to get it — Prometheus writes it per target, per scrape.
    # In our setup, querying `up` returns one sample per target:
      up{job="node",       instance="node-exporter:9100"}  => 1     # scrape landing
      up{job="prometheus", instance="localhost:9090"}      => 1     # scrape landing
      up{job="down-demo",  instance="localhost:9999"}      => 0     # nothing answering there
    # up{job=node}=1 and up{job=prometheus}=1 -> those scrapes succeed.
    # up{job=down-demo}=0 -> nothing is listening on localhost:9999, so the scrape fails.
    # When something looks broken, `up == 0` is where you start.

4 · How to query it — PromQL, or the HTTP query API
    # a) PromQL (in the Prometheus expression browser or Grafana): just type the metric name.
        up                                 # every target's health, one series each
        up == 0                            # filter to only the DOWN targets
        sum by (job)(up)                   # health rolled up per job
        count(up)                          # how many targets exist  => 3
    # b) The instant-query HTTP API (scripts, curl, health checks):
        curl -s 'http://localhost:9090/api/v1/query?query=up'
        curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=up==0'
    # Each returned sample is a metric name + labels + value, e.g.
    #   {__name__="up", job="down-demo", instance="localhost:9999"} => 0
    # That name-plus-labels shape IS the multidimensional model PromQL slices on — the same
    #   `up` name fans into one series per (job, instance).

Notes:
  - `up` is SYNTHETIC: Prometheus generates it about the scrape itself, not something the
    target exports. You will never find `up` in a target's own /metrics text.
  - A metric is name + labels. Changing any label value (a new instance, a new job) makes a
    NEW time series. That is what lets you slice, but it is also how "cardinality" explodes.
  - Metrics answer THAT/how-much cheaply; they cannot tell you the exact request that failed —
    that is a job for logs (and traces for the cross-service path). Don't reach for logs to do
    a job metrics do better (rates/alerts), or metrics to do a job logs do (per-event detail).

Gotchas:
  - up == 1 means the scrape SUCCEEDED, not that the app is healthy — it only proves /metrics
    answered. An app can be broken while still serving its metrics endpoint (200 OK).
  - up == 0 is a scrape failure: wrong port, target down, firewall, or /metrics not exposed.
    Here it is job=down-demo on localhost:9999 — nothing is listening on that port.
  - PULL means Prometheus must be able to REACH the target's /metrics over the network. A
    target behind a NAT/firewall that Prometheus can't dial will read up=0 no matter how
    healthy the process is — this is the classic pull-model failure mode.
  - Don't confuse "no data" with up=0. If a target isn't in the scrape config at all, there is
    no `up` series for it — check the config, not just the value.

Verify:
    # 1) Ask Prometheus which scrapes are landing (expect node=1, prometheus=1, down-demo=0):
    curl -s 'http://localhost:9090/api/v1/query?query=up'
    # 2) Or in the expression browser, type:  up      then:  up == 0
    #    -> up == 0 should return exactly the down-demo target (localhost:9999).
    # 3) Confirm the pull contract by hitting a target's endpoint yourself:
    curl -s http://node-exporter:9100/metrics | head
    #    -> plain-text metric lines; that's what Prometheus scrapes each interval.