Shashikant shah

Tuesday, 28 July 2026

Metrics and Instrumentation.

What are metrics, and what are the many types of metrics?

Metrics ek prakaar ka numerical data hai jo kisi system, application, ya process ki state, performance, aur behavior ko track karta hai. Jaise:

"Jo data hume batata hai ki system kaisa perform kar raha hai, use Metrics kehte hain."

Metrics = Measurable numbers jo system ki health, performance aur resource usage ko monitor karne ke liye use hote hain.

  • CPU usage (%)
  • Memory consumption (MB)
  • HTTP requests count
  • Error rates
  • Network traffic, etc.

Time Series Data:-

Data ko time series form mein convert kar ke store karta hai Metrics.
Har time series data ka structure hota hai:

 For example:-

<metric_name>{labels} => value @ timestamp

http_requests_total{method="GET", handler="/home"}  10245  @  1627540200

  metric_name: http_requests_total

  labels: method="GET", handler="/home"

  value: 10245

  timestamp: 1627540200 (Unix time)


yeh format me metric storage hota hai .

metric_name{label="value"} value timestamp

http_requests_total{status="200"} 150 1712800000


Types of Metrics in Prometheus



















1.Counter

Sirf increment hota hai (reset ho sakta hai restart pe)

Events count karne ke liye use hota hai. jaise ki "Total HTTP requests"

PromQL Functions:

http_requests_total{method="POST"} 3456

increase(http_requests_total[1h])

2.Gauge(गेज)

Increase aur decrease dono ho sakta hai

Example Query : memory_usage_bytes, temperature_celsius

memory_usage_bytes 

2048000000

3.Histogram(हिस्टोग्राम)

Values ko buckets (like 0.5s, 1s, 2s...)  mein divide karta hai (distribution ke liye)

Response time, file size, temperature distribution jaise values ke liye useful hai

Example Query: Request duration

http_request_duration_seconds_bucket{le="1"} 542

http_request_duration_seconds_count 1000

http_request_duration_seconds_sum 780

1000 requests aaye

Total time = 780 seconds

542 requests ≤ 1s ke andar complete ho gaye

4.Summary

Percentile values deta hai, jaise: 50%, 95%, 99% response time

Histogram jaise hi, lekin quantile-based output deta hai

http_request_duration_seconds{quantile="0.95"} 1.3

95% requests 1.3 second ke andar complete hue

Metric Type

Definition

Use in PromQL

Example

Counter

Only increases (can reset to zero, e.g., after app restart)

- To calculate growth rate - To count errors, requests, or events

rate(http_requests_total[5m]) (request rate over 5 min)

Gauge

Can increase or decrease

- To check current values - To measure resource usage

node_memory_Active_bytes (current RAM usage)

Histogram

Splits observations into buckets (e.g., how long requests took to complete in milliseconds)

- To calculate latency distribution - To see how many requests fall into each bucket

rate(http_request_duration_seconds_bucket[5m])

Summary

Similar to Histogram, but directly provides quantiles (percentiles) (e.g., 95th percentile latency)

- To calculate request latency percentiles - Useful when approximate percentiles are needed

http_request_duration_seconds{quantile="0.95"}




What is a Label?

A label is a key–value pair attached to a metric to differentiate time series.

Example:

http_requests_total{method="GET", status="200"}

method="GET" and status="200" are labels.

The metric name is http_requests_total.

Filtering: Labels se specific data nikal sakte hai.

·       sirf GET requests dekhna

·       sirf server1 ka CPU usage

cpu_usage{host="server1"}


grouping: Data ko labels ke basis per aggregate (jodna/average krna) ho.

sum(http_requests_total) by (status)

Output:

  • status=200 → total
  • status=500 → total

 yeh detail nhi, balki summary/grouped data mil raha hai.

Cardinality issue:

Cardinality issue ka matlab hai bahut zyada unique time series generate ho jana, jiski wajah se monitoring system (jaise Prometheus) ki memory, CPU aur storage bahut consume hone lagti hai.

Metric + Labels combinations = Time Series

Problem kab aati hai?

jab labels bahut jadhad unique ho jata hai.

http_requests_total{user_id="12345"}
http_requests_total{user_id="67890"}

Aur 10 lakh users hain, to 10 lakh alag time series ban jayengi. Ye hi high cardinality hai.

Real-world Example

Suppose:

  • 100 Pods
  • 5 APIs
  • 4 HTTP methods
  • 5 Status codes

Total time series:

100 × 5 × 4 × 5 = 10,000

Ab agar user_id (1,00,000 users) add kar diya:

100 × 5 × 4 × 5 × 100,000 = 1,000,000,000 (100 crore time series)

Itni time series ko Prometheus efficiently handle nahi kar sakta.

Isse problem kya hoti hai?

  • RAM bahut consume hoti hai.
  • CPU usage badh jata hai.
  • Prometheus slow ho jata hai.
  • Queries execute hone me time lagta hai.
  • Disk storage jaldi bhar jati hai.
  • OOM (Out Of Memory) error bhi aa sakta hai.

High-cardinality labels avoid karo:

 Mat use karo:

  • user_id
  • session_id
  • request_id
  • transaction_id
  • email
  • mobile number
  • IP address (kabhi-kabhi)

    Bad query :
    http_requests_total{request_id="abc123"}

use karo:

  • status
  • method
  • host
  • region

Good Label Example

http_requests_total{method="GET", status="200", service="payment"}

Labels ki unique values limited hain.

Real Production Solution:

Production me organizations generally ye combination use karti hain:

  •  Dynamic labels avoid karna
  •  Metric relabeling se unwanted metrics/labels drop karna
  •  Exporters ko optimize karna
  •  Recording Rules use karna
  •  Retention policy set karna
  •  Long-term storage ke liye Thanos, Grafana Mimir, ya VictoriaMetrics use karna
  •  High-cardinality information ko metrics ke bajay logs me rakhna

Custom metrics:

Prometheus by default system metrics collect karta hai (CPU, RAM, Disk), but custom metrics aap define karte ho jaise:

  • number of login requests
  • API response time
  • payment failures
  • queue size
  • order processing count

Custom Metrics ka Flow

Application → /metrics endpoint → Prometheus → Grafana / Alerts

Saturday, 4 April 2026

Introduction of Prometheus ?













Prometheus is an open-source monitoring and alerting toolkit designed for recording real-time metrics in a time-series database (TSDB), built especially for cloud-native and container-based environments like Kubernetes.

It collects and stores metrics as time series data (i.e., values with timestamps), supports powerful querying via PromQL, and integrates with Grafana and Alertmanager.


Key Features of Prometheus:














Feature

Description

Time Series Storage

Stores metrics as time series with labels

Pull-based model

Prometheus scrapes metrics from targets, unlike push-based systems

PromQL

Built-in query language for filtering, calculations, and alert conditions

Service Discovery

Automatically finds services via Kubernetes, Consul, EC2, etc.

Visualization

Built-in graph UI; best used with Grafana dashboards

Alerting

Define alerts based on thresholds; send notifications via Alertmanager


 Why Use Prometheus?

  • Open source and widely used
  • Lightweight and easy to install
  • Perfect for microservices and containerized apps
  • Strong community and support ecosystem
  • Compatible with exporters for system, database, application monitoring

Prometheus Components?

Prometheus has several core components that work together to collect, store, query, and alert based on metrics.

Component

Description

Prometheus Server

Core component that collects (scrapes), stores, and queries metrics

Exporters

Services or agents that expose metrics in Prometheus format

Alerting Rules

PromQL-based rules to define alert conditions

Alertmanager

Manages alerts – grouping, silencing, routing to email, Slack, etc.

Pushgateway

Allows short-lived jobs to push metrics to Prometheus

Service Discovery

Auto-discovers scrape targets (e.g., Kubernetes pods, EC2 instances)

Prometheus UI

Built-in web interface to run queries, check targets, and view alerts

Grafana

External tool for beautiful dashboards and visualizations using Prometheus data

Prometheus vs. Other Monitoring Tools.

Feature

Prometheus

Graphite

Elastic Stack

Datadog

Architecture

Pull-based

Push & Pull

Pull

Agent-based

Primary Data Model

Time series

Time series

Logs

Custom metrics

Query Language

PromQL

Custom DSL

kibana query

Custom UI

Horizontal Scaling

Supported

Limited

Supported

Fully managed service

Open source

Yes

Yes

Yes

No


Basic Terminologies in Prometheus

basic terminologies in Prometheus explained simply, with practical context for better understanding:

Term

Meaning

Time Series

Prometheus stores metric values with timestamps, so it creates time series data.

Metric

Metric is a measured value collected from a system, like CPU, memory, or request count. (e.g., http_requests_total)

Label

Key-value pair to differentiate time series

Job

Logical group of scrape targets

Instance

A single scrape target (usually host:port)

Target

Actual endpoint Prometheus scrapes metrics from

Scraping

Prometheus pulling data from targets at regular intervals

PromQL

Prometheus Query Language to analyze and fetch data

Exporter

Service or agent that exposes metrics in Prometheus format

Recording Rule

Precomputed PromQL query result stored as a new time series

Alerting Rule

PromQL expression that triggers alerts when conditions are met

Alertmanager

Handles alerts: grouping, deduping, routing to email, Slack, etc.

Pushgateway

Allows short-lived jobs to push metrics to Prometheus

Retention

Duration Prometheus keeps time-series data (e.g., 15 days)

TSDB

Time Series Database used internally by Prometheus

Service Discovery

Automatically finds targets (Kubernetes, Consul, EC2, etc.)

Histogram

Metric type that counts observations in configurable buckets

Summary

Metric type similar to histogram but provides quantile estimation

Gauge

Metric that can go up or down (e.g., memory usage, temperature)

Counter

Monotonically increasing metric (e.g., number of requests)

Label Set

Complete set of labels attached to a metric time series

Expression Browser

Prometheus web UI for querying and visualizing time series data


Architecture of Prometheus.



1. Time Series

  • What: A series of metric values tracked over time.
  • Example: CPU usage of a server every 10 seconds.

2. Metric

  • What: The actual measurement name.
  • Example: http_requests_total (total HTTP requests)

3. Label

  • What: Key-value pair to give more information about a metric.
  • Example:
    http_requests_total{method="GET", status="200"}
    method and status are labels.

4. Job

  • What: A group of similar targets.
  • Example: All Node Exporters can be under job node.

5. Instance

  • What: A single target with address (host:port).
  • Example: 10.1.2.3:9100 for one Node Exporter.

6. Target

  • What: The actual endpoint Prometheus collects data from.
  • Includes: job, instance, labels.

7. Scraping

  • What: The process where Prometheus collects data from a target.

8. PromQL

  • What: Prometheus Query Language.
  • Use: To filter, calculate, and display data.
  • Example:
    rate(http_requests_total[5m])

9. Exporter

  • What: A tool that exposes metrics in a Prometheus-readable format.
  • Example:
    • node_exporter for Linux server metrics
    • mysqld_exporter for MySQL metrics

10. Recording Rule

  • What: Saves the result of a query as a new metric.
  • Why: Reduces query load and speeds up dashboards.

11. Alerting Rule

  • What: A rule that defines when an alert should fire.
  • Example:
    Alert when CPU usage > 90% for 5 minutes.

12. Alertmanager

  • What: Manages alerts – sends them via Email, Slack, etc.
  • Also handles:
    • Grouping
    • Silencing
    • Routing

13. Pushgateway

  • What: Allows short-lived jobs to send data to Prometheus.
  • Why needed: Those jobs finish before Prometheus can scrape them.

14. Retention

  • What: How long Prometheus stores data.
  • Default: 15 days (can be customized)

15. TSDB (Time Series DB)

  • What: Internal database where Prometheus stores its data.
  • Supports: Fast reads, writes, and compression.

16. Service Discovery

  • What: Automatically detects targets (like Kubernetes pods, EC2).
  • Benefit: No need to manually add each new server.

17. Histogram

  • What: Metric type that counts values in buckets.
  • Used for: Request duration, response sizes.

18. Summary

  • What: Like histogram but shows percentiles (e.g., 95th percentile).
  • Used for: Latency measurement.

19. Gauge

  • What: Metric that goes up and down.
  • Examples: Memory usage, temperature.

20. Counter

  • What: Only increases.
  • Examples: Total HTTP requests, total errors.

21. Label Set

  • What: All labels assigned to a time series.
  • Helps to: Identify and group metrics.

22. Expression Browser

  • What: Built-in UI in Prometheus to test and run queries.
  • Use: For debugging or checking real-time metrics.

Prometheus ki Limitations:


1. Long-Term Storage nahi hai.

Prometheus apna data local TSDB me store karta hai. Ye months ya years tak metrics store karne ke liye design nahi hua hai.
Solution: Thanos, Grafana Mimir, VictoriaMetrics

2. Scalability Limited hai.

Ek single Prometheus server ki capacity limited hoti hai.

Agar:

  • Bahut saare servers ho
  • Millions of metrics ho
  • Bahut zyada scrape targets ho

to CPU, RAM aur Disk usage bahut badh jata hai.

Solution: Multiple Prometheus + Thanos/Mimir

3. Built-in High Availability (HA) nahi hai

Agar Prometheus server down ho gaya to monitoring bhi ruk jayegi.

Iske liye manually multiple Prometheus servers configure karne padte hain.

4. High Cardinality Problem

Agar labels me unique values bahut zyada hain, jaise:

  • user_id
  • session_id
  • request_id

to Prometheus millions of time series bana deta hai.

Isse:

  • RAM bahut consume hoti hai.
  • CPU usage badhta hai.
  • Queries slow ho jati hain.
  • Kabhi-kabhi Prometheus crash bhi ho sakta hai.

Ye Prometheus ki sabse badi limitation mani jati hai.

5. Sirf Metrics Store karta hai

Prometheus sirf metrics collect karta hai.

Ye:

  • Logs
  • Traces
  • Events

store nahi karta.

Complete observability ke liye Loki, Tempo ya Jaeger jaise tools use kiye jate hain.

6. Pull Model par kaam karta hai

Prometheus target se metrics scrape karta hai.

Agar target firewall ya NAT ke piche ho aur Prometheus us tak na pahunch sake, to monitoring mushkil ho jati hai.

7. Complex Queries Slow ho sakti hain

Agar:

  • Bahut bada data ho
  • Long time range ho
  • High-cardinality metrics ho

to PromQL queries slow chal sakti hain aur zyada resources consume karti hain.


Tools used with Prometheus

1. Long-Term Storage & High Availability

Ye tools Prometheus ki storage aur scalability ki limitation ko solve karte hain.

  • Thanos (Sabse popular)
  • Grafana Mimir
  • VictoriaMetrics
  • Cortex (Ab kam use hota hai, Mimir ne kaafi had tak replace kar diya)

2. Visualization (Dashboard)

Prometheus khud basic UI deta hai, dashboards ke liye:

  • Grafana (Industry Standard)
  • Kibana (Agar Elasticsearch use ho)
  • Chronograf (InfluxDB ecosystem)

3. Alerting

Prometheus ke alerts manage karne ke liye:

  • Alertmanager (Official)
  • PagerDuty
  • Opsgenie
  • Slack
  • Microsoft Teams
  • Email
  • Webhook

4. Log Monitoring

Prometheus sirf metrics collect karta hai, logs ke liye:

  • Grafana Loki
  • Elasticsearch
  • OpenSearch
  • Splunk
  • Graylog

5. Distributed Tracing

Microservices tracing ke liye:

  • Grafana Tempo
  • Jaeger
  • Zipkin

6. OpenTelemetry

Aajkal OpenTelemetry bahut popular hai.

  • OpenTelemetry Collector
  • OpenTelemetry SDK

Ye metrics, logs aur traces ko collect karke Prometheus, Loki, Tempo ya doosre backends ko bhej sakta hai.


7. Kubernetes Monitoring

Kubernetes me commonly:

  • kube-state-metrics
  • Node Exporter
  • cAdvisor
  • Prometheus Operator
  • kube-prometheus-stack

8. Exporters

Prometheus directly applications se data nahi leta. Exporters use hote hain.

Examples:

  • Node Exporter
  • Blackbox Exporter
  • SNMP Exporter
  • JMX Exporter
  • PostgreSQL Exporter
  • MySQL Exporter
  • Redis Exporter
  • NGINX Exporter
  • HAProxy Exporter
  • Kafka Exporter
  • RabbitMQ Exporter

9. Service Discovery

Targets automatically discover karne ke liye:

  • Kubernetes Service Discovery
  • Consul
  • DNS Service Discovery
  • EC2 Service Discovery
  • Azure Service Discovery
  • GCE Service Discovery

10. Push-Based Metrics

Agar pull model possible na ho:

  • Pushgateway
  • OpenTelemetry Collector
  • Prometheus Agent

Enterprise Monitoring Stack (Most Common)

Applications / Servers

        |

   Exporters / OpenTelemetry

        |

    Prometheus

        |

   +-------------+

   | Alertmanager|

   +-------------+

        |

     Grafana

        |

+--------------------------+

| Thanos / Mimir /         |

| VictoriaMetrics          |

+--------------------------+

        |

  Long-Term Storage

        |

Logs --> Loki

Traces --> Tempo / Jaeger


Prometheus ke sath kaun-kaun se tools use hote hain.

  • Monitoring: Prometheus

  • Visualization: Grafana

  • Alerting: Alertmanager

  • Long-Term Storage & HA: Thanos, Grafana Mimir, VictoriaMetrics

  • Logs: Loki, Elasticsearch/OpenSearch

  • Tracing: Tempo, Jaeger

  • Telemetry Collection: OpenTelemetry Collector

  • Kubernetes: Prometheus Operator, kube-state-metrics, Node Exporter

  • Exporters: Node, PostgreSQL, MySQL, Kafka, Redis, NGINX, HAProxy, JMX, Blackbox Exporter