Metrics Collection Methods in Prometheus

 Metrics Collection Methods in Prometheus

1.There are 3 main ways to collect data.

2.What are exporters?

3.Types of Exporters

4.Kubernetes (K8s) mein Exporter deploy karne ke multiple patterns hote hain.

5.Prometheus mein Exporter ke saath Target Configuration ke types

6.Prometheus Static Config for Node Exporter

7.Kubernetes Service Discovery

There are 3 main ways to collect data:

Source Type	Description
Exporter	Ready-made software jo system ya app ka metrics expose karta hai (Pull)
Client Libraries	Aapki application khud metrics expose karti hai using Prometheus client libraries (pull)
Pushgateway	Short-lived jobs apna data "push" karte hain Pushgateway pe, jise Prometheus scrape karta hai

i) .Exporters:-

Exporter ek software component hota hai jo kisi system, service, ya application ke internal metrics ko Prometheus-compatible format me expose karta hai, taaki Prometheus unhe scrape (pull) karke monitor kar sake.

global: 

 scrape_interval: 15s

Prometheus har 15 seconds mein configured target/exporter ko scrape karega.

Prometheus har 15 sec mein Exporter ko scrape karega aur us waqt Exporter jo current/latest metric value expose kar raha hai, Prometheus usko collect karega.

ii) Types of Exporters

• Official Exporters
• Community Exporters
• Blackbox Exporter
• Application-specific Exporters

 

1. Official Exporters

Ye exporters Prometheus project/community ke officially maintained ya recommended exporters hote hain.

Examples:

Node Exporter → Linux/Unix metrics

Blackbox Exporter → HTTP, HTTPS, TCP, ICMP, DNS probing

SNMP Exporter → Network devices

JMX Exporter → Java/JMX metrics

2. Community Exporters

Ye exporters community/developers ne develop aur maintain kiye hote hain.

Specific technologies/applications ko monitor karne ke liye.

Important: Community exporters use karte time maintenance status, compatibility, GitHub activity aur security jarur check karni chahiye.

Examples:

• PostgreSQL Exporter
• Redis Exporter
• Kafka Exporter
• Nginx Exporter
• HAProxy Exporter
• MongoDB Exporter

3. Blackbox Exporter

Ye thoda different hai.

Blackbox Exporter application ke internal metrics collect nahi karta. Ye externally service ko probe/check karta hai.

Bahar se service reachable hai ya nahi?

Prometheus phir Blackbox Exporter se probe results scrape karta hai.

For example:

Use:

• Website availability
• API availability
• Port check
• Ping
• DNS check
• SSL/TLS monitoring 

4. Application-specific Exporters

Ye kisi specific application/system ke metrics expose karne ke liye banaye jaate hain.

Examples:

Your Application

↓

Prometheus Client Library

↓

/metrics

↓

Prometheus

iii) Kubernetes (K8s) mein Exporter deploy karne ke multiple patterns hote hain.

1.DaemonSet

2.Deployment

3.Sidecar

4.StatefulSet

1.DaemonSet (Node-level exporters)

Jab exporter ko har node ka data chahiye, tab use DaemonSet me run karte hain.

·       Prometheus Node Exporter → CPU, Memory, Disk

·       CNI / network exporters

·       Log/agent type tools

2.Deployment (Service-level exporters)

Jab exporter kisi application/service ka data deta hai, tab Deployment use hota hai

·       Blackbox Exporter → HTTP/ICMP check

·       MySQL Exporter → DB metrics

·       Redis Exporter

3. Sidecar Pattern (Pod ke andar)

Kabhi exporter directly app ke pod ke saath run hota hai. App ke internal metrics access karne ke liye

·       Custom app exporter

·       Nginx + exporter sidecar

4. StatefulSet (Rare cases)

Kuch exporters ko state maintain karna hota hai

·       Kuch advanced monitoring tools

·       Cluster-level components

All Exporters Names:

iv) Prometheus mein Exporter ke saath Target Configuration ke types

(Prometheus Target Configuration & Service Discovery)

Configuration / Service Discovery	Meaning
Static Config	Server ka IP/hostname manually prometheus.yml mein define karte hain
File-based Service Discovery (File SD)	Targets ko separate YAML/JSON file mein define karte hain
DNS Service Discovery	DNS ke through targets discover hote hain
Kubernetes Service Discovery	Kubernetes API se Pods, Services, Nodes etc. automatically discover hote hain
EC2 Service Discovery	AWS EC2 instances automatically discover hote hain
Consul Service Discovery	Consul service registry se targets discover hote hain
HTTP Service Discovery	HTTP endpoint se target list discover hoti hai
OpenStack Service Discovery	OpenStack instances/services discover hote hain
Eureka Service Discovery	Eureka service registry se services discover hoti hain

 

1.Prometheus Static Config for Node Exporter

Static configs mean that we manually define the list of targets in the configuration file.

Step 1: node-exporter (DaemonSet)

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: node-exporter

  namespace: monitoring

spec:

  selector:

    matchLabels:

      app: node-exporter

  template:

    metadata:

      labels:

        app: node-exporter

    spec:

      hostNetwork: true

      hostPID: true

      containers:

        - name: node-exporter

          image: prom/node-exporter

          args:

            - "--path.rootfs=/host"

          ports:

            - containerPort: 9100

          volumeMounts:

            - name: root

              mountPath: /host

              readOnly: true

      volumes:

        - name: root

          hostPath:

            path: /

 

# kubectl apply -f node-exporter.yaml

# kubectl get pods -n monitoring -o wide

# kubectl describe pod node-exporter-m7bwh -n monitoring

Port: 9100/TCP

# ss -tulnp | grep 9100

 

Pod ke andar check

# kubectl exec -it <node-exporter-pod> -n monitoring -- netstat -tulnp | grep 9100

# kubectl get configmap -n monitoring

# kubectl edit configmap prometheus-config -n monitoring

- job_name: 'kubernetes-nodes'

  static_configs:

    - targets:

        - '192.168.56.149:9100'

        - '192.168.56.150:9100'

 

# kubectl rollout restart deployment prometheus -n monitoring

 

Prometheus UI → Targets

http://192.168.56.149:30090/targets

 

node_cpu_seconds_total

node_memory_MemAvailable_bytes

2.Kubernetes Service Discovery

global section :  

default settings hoti hain jo poore Prometheus par apply hoti hain.
scrape_interval : (default: 1 minute) Prometheus kitne interval pe data collect karega.

evaluation_interval:  alert rules kitni der me check honge.

scrape_timeout: agar target 10 sec me response nahi deta → fail

external_labels: multi-cluster me use hota hai. alerts me identify karne ke liye.

global:
scrape_interval: 15s

evaluation_interval: 15s

scrape_timeout: 10s

external_labels:
    monitor: 'prod-cluster'
    region: 'india'

scrape_configs :

targets define hote hain jaise servers/pods/services. Multiple jobs define kar sakte ho.
Kis-kis server/app se metrics collect karne hain.

job_name: ek label/naam dete hai jo ki target ka naam hoga.

Targets:  actual server / app ka address. yahin se metrics aayega

i). Static Discovery (basic): manually IP likhte ho.

For 4 types targets defined :

1.      Domain / DNS name (most common)

targets:
- node-exporter.monitoring.svc.cluster.local:9100

 

2.      IP Address

targets:
- 192.168.56.149:9100

 

3.      Multiple targets list

targets:
- 192.168.56.149:9100
- 192.168.56.150:9100

 

4.      Hostname (external / internal):DNS resolve hona chahiye

targets:
- my-server:9100

 

5.      Advance Labels ke sath

- targets: ['192.168.1.10:9100']
labels:
env: prod
app: node

 

 

ii) kubernetes_sd_configs: Kubernetes me auto discover karega

Kubernetes API se connect karke Pod, service, endpoints, node ko. Automatically discover karta hai.

Metadata fetch karta hai.  

Labels ke basis par filter karta hai.

Targets dynamically update karta hai.

deleted pods hata dega.

Agar prometheus ko kisi bahar ke kubernetes (k8s) se connect krna hai to niche diye gaye parameter follow krna hoga.

1)      Role : define karta hai kya discover karna hai.(Pod, service, endpoints, node )

2)      api_server: agar Prometheus cluster ke k8s ke bahar hai to k8s ko connect ke liye use kiya jayega.

3)      kubeconfig_file: k8s API kubeconfig se connect Karega.

4)      bearer_token_file: ye file me stored token se Prometheus Kubernetes API ko authenticate karta hai

5)      tls_config: Prometheus Kubernetes API ya target se secure (HTTPS) connection bnayega.

6)      ca_file: ye check karta hai ki jis server se connect ho rahe ho wo trusted hai ya nahi.

7)      insecure_skip_verify: SSL verification skip Karega

## for example  ##

kubernetes_sd_configs:
- role: <type>
api_server: https://<k8s-master-ip>:6443
kubeconfig_file: /root/.kube/config
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true

 

Agar Prometheus, kubernetes (k8s) ke andar hi hai, to niche diye gaye parameter follow krna hoga.

 Prometheus Auto Discovery = API access + Auth + Permission + Role + Filter

 

Internal K8s + Prometheus (Auto Discovery) – Required Points

1. role:  pod, service, endpoints, node define karta hai ki Prometheus kya discover kare

2.ServiceAccount: Prometheus pod ko ek ServiceAccount diya jata hai.(identity ke liye).

3.RBAC permission: Prometheus ko permission chahiye pods list , services , endpoints read ke liye.

4.Authentication (auto): Kubernetes automatically token inject karta hai:

5.TLS config (auto):  Secure API call hoti hai:

6.Relabeling (filtering):  Relabeling ka main kaam hota hai data ko filter aur modify karna during discovery/scraping.

i) source_labels: Kaunse labels se value uthani hai

source_labels: [__meta_kubernetes_pod_name]

ii) separator: Multiple labels ko join karne ke liye (default ;)

separator: ";"

iii) target_label: Result kis label me store hoga
target_label: instance

iv) regex: Matching condition define karta hai

regex: "true"

v) modulus: Hashing / sharding use-case me (advanced)

modulus: 2

vi) replacement: Replace karne ke liye value

replacement: "$1"

vii) action  (Most Important): Kya operation perform karna hai

·       keep → match hua to rakho

·       drop → match hua to hatao

·       replace → value change karo

·       labelmap → labels rename karo

·       labeldrop → label hatao

·       labelkeep → sirf selected labels rakho

 

7.Kubernetes side annotation: decide karta hai kaun monitor hoga

annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9100"

8.Verify targets

http://<prometheus>:9090/targets

Targets = UP

##

Note : Service and endpoints me different  kya hai ?

Client → Service → Endpoints → Pods

1.Endpoints = actual backend pods ke real IP + port

2.Service = ek stable “entry point” (load balancer) for pods.

 

Example:01

Sirf unhi pods ko monitor karo jinke annotation me prometheus.io/scrape = true likha ho"

scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod

relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true

Example:02

Only specific namespace scrape karo

Sirf prod namespace ke pods hi scrape honge

relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
action: keep
regex: prod

Example:03

Specific label wale pods hi scrape karo

Sirf app=nginx wale pods monitor honge

relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
  action: keep
  regex: nginx

Example:03

Path change karna (/metrics default hota hai)

Ab Prometheus /custom-metrics hit karega

annotations:
prometheus.io/path: "/custom-metrics"

 

relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)

Example:04

Label rename / add karna

env=default

 

relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
target_label: env

Common mistakes

  ❌ RBAC missing → kuch discover nahi hoga

  ❌ annotation nahi → scrape nahi hoga

  ❌ wrong role use

  ❌ Prometheus cluster ke bahar (without config)

 

iii) File-based Discovery : external file se targets load hote hain

file_sd_configs:
- files:
- /etc/prometheus/targets.json

# cat /etc/prometheus/targets.json

[
{
"targets": ["192.168.1.10:9100"]
}
]

iv) DNS Discovery : DNS se IP resolve hota hai. dynamic infra me useful.

dns_sd_configs:
- names:
- myapp.local
type: A
port: 9100

 

v) EC2 Discovery (AWS): AWS EC2 instances auto detect Karega.

ec2_sd_configs:
- region: ap-south-1

 

vi) Consul Discovery: Consul service registry use karta hai.

consul_sd_configs:
- server: 'localhost:8500'

 

vii) Azure / GCP Discovery: cloud-based auto discovery.

azure_sd_configs

gce_sd_configs

 

 

viii) HTTP Discovery (advanced): API se targets fetch Karega.

http_sd_configs:
- url: http://example.com/targets

 

Client Libraries:

The Prometheus client libraries are for the pull model, i.e. your application exposes an HTTP endpoint (usually /metrics) and Prometheus scrapes (pull) from it.

application running http://localhost:8000/metrics

Metric Types 

 - Counter 

 - Gauge 

 - Histogram 

 - Summary

Custom Metrics

Application ke business/application-specific metrics create kar sakte hain.

• http_requests_total 
• login_success_total
•  database_connections
•  payment_transactions_total

Pushgateway:

Pushgateway is used to collect metrics from short-lived jobs that cannot be reliably scraped directly by Prometheus.

Pushgateway Prometheus ecosystem ka ek component hai jo short-lived jobs/batch jobs se metrics receive karta hai.

Pushgateway kab use karein?

• Short-lived jobs
• Batch jobs
• Cron jobs
• One-time scripts
• Backup scripts
• Data processing jobs