Metrics and Instrumentation.

What are metrics, and what are the many types of metrics?

Metrics ek prakaar ka numerical data hai jo kisi system, application, ya process ki state, performance, aur behavior ko track karta hai. Jaise:

"Jo data hume batata hai ki system kaisa perform kar raha hai, use Metrics kehte hain."

Metrics = Measurable numbers jo system ki health, performance aur resource usage ko monitor karne ke liye use hote hain.

• CPU usage (%)
• Memory consumption (MB)
• HTTP requests count
• Error rates
• Network traffic, etc.

Time Series Data:-

Data ko time series form mein convert kar ke store karta hai Metrics.
Har time series data ka structure hota hai:

 For example:-

<metric_name>{labels} => value @ timestamp

http_requests_total{method="GET", handler="/home"}  10245  @  1627540200

  metric_name: http_requests_total

  labels: method="GET", handler="/home"

  value: 10245

  timestamp: 1627540200 (Unix time)

yeh format me metric storage hota hai .

metric_name{label="value"} value timestamp

http_requests_total{status="200"} 150 1712800000

Types of Metrics in Prometheus

1.Counter

Sirf increment hota hai (reset ho sakta hai restart pe)

Events count karne ke liye use hota hai. jaise ki "Total HTTP requests"

PromQL Functions:

http_requests_total{method="POST"} 3456

increase(http_requests_total[1h])

2.Gauge(गेज)

Increase aur decrease dono ho sakta hai

Example Query : memory_usage_bytes, temperature_celsius

memory_usage_bytes 

2048000000

3.Histogram(हिस्टोग्राम)

Values ko buckets (like 0.5s, 1s, 2s...)  mein divide karta hai (distribution ke liye)

Response time, file size, temperature distribution jaise values ke liye useful hai

Example Query: Request duration

http_request_duration_seconds_bucket{le="1"} 542

http_request_duration_seconds_count 1000

http_request_duration_seconds_sum 780

1000 requests aaye

Total time = 780 seconds

542 requests ≤ 1s ke andar complete ho gaye

4.Summary

Percentile values deta hai, jaise: 50%, 95%, 99% response time

Histogram jaise hi, lekin quantile-based output deta hai

http_request_duration_seconds{quantile="0.95"} 1.3

95% requests 1.3 second ke andar complete hue

Metric Type	Definition	Use in PromQL	Example
Counter	Only increases (can reset to zero, e.g., after app restart)	- To calculate growth rate - To count errors, requests, or events	rate(http_requests_total[5m]) (request rate over 5 min)
Gauge	Can increase or decrease	- To check current values - To measure resource usage	node_memory_Active_bytes (current RAM usage)
Histogram	Splits observations into buckets (e.g., how long requests took to complete in milliseconds)	- To calculate latency distribution - To see how many requests fall into each bucket	rate(http_request_duration_seconds_bucket[5m])
Summary	Similar to Histogram, but directly provides quantiles (percentiles) (e.g., 95th percentile latency)	- To calculate request latency percentiles - Useful when approximate percentiles are needed	http_request_duration_seconds{quantile="0.95"}

What is a Label?

A label is a key–value pair attached to a metric to differentiate time series.

Example:

http_requests_total{method="GET", status="200"}

method="GET" and status="200" are labels.

The metric name is http_requests_total.

Filtering: Labels se specific data nikal sakte hai.

·       sirf GET requests dekhna

·       sirf server1 ka CPU usage

cpu_usage{host="server1"}

grouping: Data ko labels ke basis per aggregate (jodna/average krna) ho.

sum(http_requests_total) by (status)

Output:

• status=200 → total
• status=500 → total

 yeh detail nhi, balki summary/grouped data mil raha hai.

Cardinality issue:

Cardinality issue ka matlab hai bahut zyada unique time series generate ho jana, jiski wajah se monitoring system (jaise Prometheus) ki memory, CPU aur storage bahut consume hone lagti hai.

Metric + Labels combinations = Time Series

Problem kab aati hai?

jab labels bahut jadhad unique ho jata hai.

http_requests_total{user_id="12345"}
http_requests_total{user_id="67890"}

Aur 10 lakh users hain, to 10 lakh alag time series ban jayengi. Ye hi high cardinality hai.

Real-world Example

Suppose:

• 100 Pods
• 5 APIs
• 4 HTTP methods
• 5 Status codes

Total time series:

100 × 5 × 4 × 5 = 10,000

Ab agar user_id (1,00,000 users) add kar diya:

100 × 5 × 4 × 5 × 100,000 = 1,000,000,000 (100 crore time series)

Itni time series ko Prometheus efficiently handle nahi kar sakta.

Isse problem kya hoti hai?

• RAM bahut consume hoti hai.
• CPU usage badh jata hai.
• Prometheus slow ho jata hai.
• Queries execute hone me time lagta hai.
• Disk storage jaldi bhar jati hai.
• OOM (Out Of Memory) error bhi aa sakta hai.

High-cardinality labels avoid karo:

 Mat use karo:

• user_id
• session_id
• request_id
• transaction_id
• email
• mobile number
• IP address (kabhi-kabhi)
  
  Bad query :
  http_requests_total{request_id="abc123"}

use karo:

• status
• method
• host
• region

Good Label Example

http_requests_total{method="GET", status="200", service="payment"}


Labels ki unique values limited hain.

Real Production Solution:

Production me organizations generally ye combination use karti hain:

•  Dynamic labels avoid karna
•  Metric relabeling se unwanted metrics/labels drop karna
•  Exporters ko optimize karna
•  Recording Rules use karna
•  Retention policy set karna
•  Long-term storage ke liye Thanos, Grafana Mimir, ya VictoriaMetrics use karna
•  High-cardinality information ko metrics ke bajay logs me rakhna

Custom metrics:

Prometheus by default system metrics collect karta hai (CPU, RAM, Disk), but custom metrics aap define karte ho jaise:

• number of login requests
• API response time
• payment failures
• queue size
• order processing count

Custom Metrics ka Flow

Application → /metrics endpoint → Prometheus → Grafana / Alerts