What is Stateless vs. Statefulset in Kubernetes.

StatefulSets are API objects in Kubernetes that are used to manage stateful applications. There are two types of applications in Kubernetes, Stateful applications and stateless applications. There are two ways to deploy these applications:

1.     Deployment (for stateless applications).

2.     StatefulSets (for stateful applications).

What are StatefulSets application?

Kubernetes StatefulSets are a specialized workload API object designed for managing stateful applications. Unlike Deployments, which are suitable for stateless applications, StatefulSets are used for applications that require stable, unique network identifiers, persistent storage, and ordered, consistent deployment and scaling. StatefulSets are ideal for databases, distributed systems, and other services where each instance must be individually tracked and maintained.

• Stable Network Identity: StatefulSets provide each Pod with a unique, persistent identity (hostname) that is retained across rescheduling. This is crucial for applications where each instance must be uniquely addressable, such as databases.
• Ordered Deployment and Scaling: Pods in a StatefulSet are created, deleted, or scaled in a strict order, ensuring that the system remains consistent during these operations. This is essential for applications that require ordered startup, shutdown, or scaling processes.
• Persistent Storage: StatefulSets work in conjunction with PersistentVolumeClaims (PVCs) to provide persistent storage for each Pod. Each Pod gets its own persistent storage volume, which is not shared with other Pods and is retained even if the Pod is rescheduled or restarted.
• Graceful Rolling Updates: StatefulSets ensure that updates are applied to Pods in a controlled manner, maintaining the stability and availability of the application during updates.
• Pod IP Change on Deletion: When a central (master) Pod like Pod-1 is deleted and recreated, it receives a new IP address (e.g., 10.2.0.4). This change can break connections between the master Pod and other Pods (like Pod-2 and Pod-3) that rely on a stable IP for communication.
• High Availability Challenges: Even in a high-availability setup, where Pods are spread across different availability zones (AZs), connecting worker Pods (Pod-2 and Pod-3) to the master Pod (Pod-1) can fail if the master Pod’s IP changes. This issue can disrupt the overall functionality and reliability of the stateful application.

What are Stateless Applications?

A stateless application is an application that does not store client data or session state between requests. Each request from a client is treated independently, without relying on previous interactions.

Since there is no dependency on past requests, any instance of the application can handle any incoming request, making stateless applications highly scalable and resilient.

Key Characteristics of Stateless Applications.

1.No Persistent Storage → Data is not stored between sessions or requests.
2.Interchangeable Instances → Any instance can serve a request.
3.Scalability → Easy to scale horizontally by adding or removing instances.
4.Load Balancing Friendly → Requests can be routed to any available instance.
5.Failure Resilience → If one instance crashes, another can take over without data loss.
2.Interchangeable Instances → Any instance can serve a request.
3.Scalability → Easy to scale horizontally by adding or removing instances.
4.Load Balancing Friendly → Requests can be routed to any available instance.
5.Failure Resilience → If one instance crashes, another can take over without data loss. 

Here are some real-world examples of stateless applications:

1.Web Servers.
2.REST APIs & Microservices.
3.Proxy Servers & Load Balancers.
4.Serverless Functions.

Differences between StatefulSets and Deployment.

Deployment	StatefulSets
Deployment is used to deploy stateless applications.	StatefulSet is used to deploy stateful applications.
In Deployment, all pods are created parallelly.	In StatefulSets, the ports are created one by one.
When we scale down a deployment a random pod is picked up and deleted.	When we scale down StatefulSets, the last pod gets deleted.
In Deployment, a random name is assigned to the pods.	In StatefulSets, a sticky and predictable name is assigned.
In deployment all the ports use the same persistent volume.	In the statefull set each pod uses its own persistent volume.

Stateless vs. Statefulset in Kubernetes.

Feature	Stateless Applications (Deployment)	Stateful Applications (StatefulSet)
Storage	No persistent storage required.	Requires persistent storage.
Pod Identity	Pods are interchangeable.	Pods have unique, stable identities.
Scaling	Can scale up/down freely.	Scaling is ordered (e.g., mysql-0 before mysql-1).
Network Identity	Pods get random IPs and DNS names.	Pods get stable network identities (DNS).
Use Case	Web servers, APIs, stateless microservices.	Databases, distributed systems, etc.
Complexity	Easier to manage.	Harder to manage due to state and ordering.

 

MySQL StatefulSet with dynamic Persistent Volume (PV) provisioning using NFS (Network File System) .

1.NFS server Details:

# showmount -e 192.168.56.133

Export list for 192.168.56.133:

/exported/nfs-data *

2. Deploy the NFS Client Provisioner in Kubernetes

Kubernetes CSI (Container Storage Interface) driver for NFS.

Provisioner Name: nfs.csi.k8s.io

# helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts

# helm repo update

# helm repo list

# helm install nfs-csi csi-driver-nfs/csi-driver-nfs --namespace kube-system --create-namespace
# helm list -n kube-system

Note:- If remove repo from helm.  

# helm repo remove csi-driver-nfs

Note:- If remove nfs csi driver

# helm uninstall nfs-csi -n kube-system

Pods is created for CSI-NFS driver.

# kubectl get pods -n kube-system | grep nfs

 

CSI Driver is showing.

# kubectl get csidrivers

On which nodes is the CSI driver installed.

#kubectl get csinodes

 3. Create a Headless Service for MySQL.

A headless service provides a stable network identity for MySQL pods.

#  headless.yaml

apiVersion: v1

kind: Service

metadata:

  name: mysql

spec:

  clusterIP: None  # Headless service

  ports:

  - port: 3306

    name: mysql

  selector:

    app: mysql

# kubectl apply -f mysql-service.yaml

# kubectl get svc

 

4. Deploy MySQL StatefulSet with Dynamic NFS PVC

# vim mysql.yaml

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: mysql

spec:

  serviceName: "mysql"

  replicas: 3  # Adjust replicas as needed

  selector:

    matchLabels:

      app: mysql

  template:

    metadata:

      labels:

        app: mysql

    spec:

      containers:

      - name: mysql

        image: mysql:8

        ports:

        - containerPort: 3306

          name: mysql

        env:

        - name: MYSQL_ROOT_PASSWORD

          valueFrom:

            secretKeyRef:

              name: mysql-secret

              key: root-password

        volumeMounts:

        - name: mysql-storage

          mountPath: /var/lib/mysql

  volumeClaimTemplates:

  - metadata:

      name: mysql-storage

    spec:

      accessModes: ["ReadWriteMany"]

      storageClassName: "nfs-csi"

      resources:

        requests:

          storage: 10Gi

# kubectl apply -f mysql-statefulset.yaml

# kubectl get pods

 

 

 

# kubectl get pvc

# kubectl get pv

# kubectl get sts -o wide

 

5. Create a Secret for MySQL Credentials

# vim mysql-secret.yaml

apiVersion: v1

kind: Secret

metadata:

  name: mysql-secret

type: Opaque

data:

  root-password: cGFzc3dvcmQ=  # Base64 encoded password ('password')

# kubectl apply -f mysql-secret.yaml

# kubectl get secrets

6. Verify the Deployment

kubectl get pods

kubectl get pvc

kubectl get pv

kubectl get statefulset

kubectl get secrets

Check MySQL logs:

kubectl logs -f mysql-0

7. Access MySQL Inside Kubernetes (pass :- password)

# kubectl exec -it mysql-0 -- mysql -uroot -p

To connect from an external client:

# kubectl port-forward svc/mysql 3306:3306

mysql -h 127.0.0.1 -u root -p

Delete a pod and check once.

# kubectl delete pods mysql-0

 

Note :- If mysql-0 pods are deleted and then mysql-0 pods are created again, they will be mounted from the same volumes

Different Scenarios for MySQL StatefulSet in Kubernetes.

Scenario	Use Case	Pros	Cons
1️.Single MySQL Pod	Dev/Test	Simple	No HA
2️.Master-Slave	Read-heavy apps	Read scaling	Manual failover
3️.Master-Slave + Failover	Production	Auto-failover	Complex setup
4️.Multi-Master (Group Replication)	Write scaling	No single point of failure	High overhead
5️.Galera Cluster	Mission-critical apps	Synchronous replication	Complex setup

1️.Single MySQL Pod (Standalone)

• StatefulSet with 1 replica (mysql-0)
• No replication, no high availability (HA).
• Use Case: Simple applications, development, testing.

Setup Example

spec:

  replicas: 1  # Only one MySQL pod

Pros:

✅ Simple and easy to set up
✅ Uses Persistent Volumes (PV) for data persistence
✅ Suitable for small applications

Cons:

❌ No redundancy
❌ No auto-failover

 2️.MySQL Master-Slave Replication

• 1 Master (mysql-0), multiple Slaves (mysql-1, mysql-2, etc.)
• Slaves replicate data from the master.
• Use Case: Applications with high read queries.

Setup Example

spec:

  replicas: 3  # One master, two replicas

Pros:

✅ Read Scaling: Read-heavy applications can use replicas
✅ Persistence: Data remains available even if a slave fails

Cons:

❌ Single Point of Failure: If the master fails, manual intervention is needed
❌ Writes are limited to the master

3️.MySQL with Automatic Failover (Orchestrator)

• Replication + Automatic Failover
• Uses MySQL Orchestrator or ProxySQL.
• Use Case: Production databases needing high availability.

How It Works:

• Orchestrator monitors MySQL pods.
• If the master (mysql-0) fails, Orchestrator promotes a replica (mysql-1) to master.

Pros:

✅ Automatic Failover when master fails
✅ High Availability

Cons:

❌ More complex setup
❌ Requires additional components like Orchestrator

 

4️.MySQL Group Replication (Multi-Master)

• All nodes (mysql-0, mysql-1, mysql-2) act as both readers and writers
• MySQL automatically synchronizes data between them.
• Use Case: Applications needing write scaling.

Setup Example

Use MySQL InnoDB Cluster:

spec:

  replicas: 3  # All nodes are masters

Pros:

✅ No single point of failure
✅ Writes are distributed

Cons:

❌ Complex setup
❌ High network overhead

When to Choose MySQL Group Replication?

Use Case	Best Choice
General-purpose high-availability databases	✅ MySQL Group Replication
Multi-region cloud deployment	✅ MySQL Group Replication
Applications needing automatic failover	✅ MySQL Group Replication
E-commerce, SaaS, financial applications	✅ MySQL Group Replication
Multi-master write scalability	✅ MySQL Group Replication (Multi-Primary Mode)

5️.MySQL + Galera Cluster (Synchronous Replication)

• All nodes are active (multi-master)
• Uses Galera Cluster for synchronous replication.
• Use Case: Mission-critical applications needing zero data loss.

Pros:

✅ Fault Tolerant
✅ Write Scalability

Cons:

❌ Requires Galera Cluster setup
❌ Not compatible with all MySQL features 

When to Choose Which?

Use Case	Best Choice
Banking, finance, telecom, mission-critical apps	✅ MySQL Galera Cluster
E-commerce with high transaction volumes	✅ MySQL Galera Cluster
Multi-Region Cloud Deployment with failover support	✅ MySQL Group Replication
Multi-master writes with conflict resolution	✅ MySQL Group Replication
General purpose database replication	✅ MySQL Group Replication
Data Consistency > Performance	✅ MySQL Galera Cluster
Performance > Immediate Consistency	✅ MySQL Group Replication

MySQL Group Replication vs. Galera Cluster.

Static and Dynamic PV for PVC in Kubernetes.

A). How Static Persistent Volume (PV) and Persistent Volume Claim (PVC) work with HostPath in Kubernetes.

•  A hostPath volume mounts a file or directory from the host node’s filesystem into your pod.
• A hostPath PersistentVolume must be used only in a single-node cluster. Kubernetes does not support hostPath on a multi-node cluster currently.

i) First create a PV for hostpath.

# hostpath-pv.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

  name: pv-hostpath

  labels:

    type: local

spec:

  storageClassName: manual

  capacity:

    storage: 1Gi

  accessModes:

    - ReadWriteOnce

  hostPath:

    path: "/shashi_data"

# kubectl apply -f hostpath-pv.yaml

# kubectl get pv

ii) Create a PVC for hostpath.

# cat hostpath-pvc.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: pvc-hostpath

spec:

  storageClassName: manual

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 100Mi

# kubectl apply -f hostpath-pvc.yaml

# kubectl  get  pvc

iii) create a pod with hostpath. 

# cat busy_box.yaml

apiVersion: v1

kind: Pod

metadata:

  name: busybox

spec:

  volumes:

  - name: host-volume

    persistentVolumeClaim:

      claimName: pvc-hostpath

  containers:

  - image: busybox

    name: busybox

    command: ["/bin/sh"]

    args: ["-c", "sleep 600"]

    volumeMounts:

    - name: host-volume

      mountPath: /mydata

# kubectl apply -f busy_box.yaml

# kubectl get pods -o wide

# kubectl exec busybox -- ls /mydata

# kubectl exec busybox -- touch /mydata/test123

go to woker2 node.

# ls /shashi_data/

B). How to schedule HostPath in select nodes?

# kubectl apply -f hostpath-pv.yaml

# kubectl apply -f hostpath-pvc.yaml

Check node labels.

# kubectl get node worker1 --show-labels

Labels are added at the woker1 node.

# kubectl label node worker1 node_worker=true

# kubectl get node worker1 --show-labels

# vim busy_box.yaml

# kubectl apply -f busy_box.yaml

# kubectl get pods -o wide

# kubectl exec busybox -- touch /mydata/test71

# kubectl exec busybox -- ls /mydata

test71

# ls /shashi_data/

C). How to reuse a PersistentVolume (PV) in kubernetes.

By default, PVs have a reclaim policy of Retain, which means the PV is not automatically deleted or made available for reuse when the PVC is deleted.

# kubectl apply -f hostpath-pvc.yaml

How to fix “manual not found” Error

Edit the PV to remove the claimRef field, which ties the PV to the previous PVC:

# kubectl edit  pv pv-hostpath

# kubectl get pv

# kubectl apply -f hostpath-pvc.yaml

# kubectl get pv

# kubectl get pvc

# kubectl describe pvc pvc-hostpath

D). Create Static Persistent Volume PV and PVC for EBS AWS. 

1.We will manually create EBS volumes and create volumes in the same zone (Availability Zone : us-east-2b) as the cluster node.

2. Mount the volume vol-0cf28e25fd0db918e in the node and format the volume with ext4 filesystem. Will then Detach Volume  from node.

Disk → PV →  PVC ← Pods

i) Create a PV for  ebs . 

# vim ebs-pv.yaml

# kubectl  apply -f ebs-pv.yaml

# kubectl get pv

ii) Create a PVC for  ebs . 

#vim  ebs-pvc.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  labels:

    app: mysql

  name: pvc

spec:

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: 10Gi

  storageClassName: gp2

  volumeMode: Filesystem

  volumeName: pv-ebs-static

# kubectl  apply -f ebs-pvc.yaml

# kubectl get pvc

iii) Create a Pods for  ebs . 

# vim nginx.yaml 

apiVersion: v1

kind: Pod

metadata:

  name: nginx

spec:

  containers:

    - name: nginx

      image: nginx:latest

      ports:

        - name: http

          containerPort: 80

      volumeMounts:

        - name: data

          mountPath: /usr/share/nginx/html

  volumes:

    - name: data

      persistentVolumeClaim:

        claimName: pvc

# kubectl  apply -f nginx.yaml

# kubectl get pv,pvc,pods

# kubectl  exec -it nginx /bin/bash
# touch /usr/share/nginx/html/test.txt

E). Create Dynamic PVC and StorageClass for EBS AWS. 

reclaimPolicy: Unsupported value: "Recycle": supported values: "Delete", "Retain"

Disk ← storageClass ← PVC ← Pods

i) Create a storageClass.

# vim storageclass.yaml

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

  name: mystorageclass

provisioner: kubernetes.io/aws-ebs

parameters:

  type: gp2

  iopsPerGB: "10"

  fsType: ext4

reclaimPolicy: Delete

mountOptions:

  - debug

# kubectl apply -f  storageclass.yaml

# kubectl get sc

 

ii) Create a pvc for mysql.

# vim ebs-pvc_test.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  labels:

    app: mysql

  name: pvc

spec:

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: 10Gi

  storageClassName: mystorageclass

 

# kubectl apply -f ebs-pvc_test.yaml

# kubectl get pvc

 

iii) Create a pods.

# vim nginx.yaml

apiVersion: v1

kind: Pod

metadata:

  name: nginx

spec:

  containers:

    - name: nginx

      image: nginx:latest

      ports:

        - name: http

          containerPort: 80

      volumeMounts:

        - name: data

          mountPath: /usr/share/nginx/html

  volumes:

    - name: data

      persistentVolumeClaim:

        claimName: pvc

# kubectl apply -f nginx.yaml

# kubectl get pods,pv,pvc

# kubectl describe  pods nginx 

F). How to Allow Multiple Pods to Read/Write in AWS EFS Storage in Kubernetes.

1.In the security group for EFS will be allowed worker nodes Ips.

2. Click Attach Option its showing the access point URL.

3.if security group change in efs:-

              Network → manage → change security group.

4.install “sudo apt install nfs-common” in all node workers.

https://github.com/antonputra/tutorials/tree/main/lessons/041

# mkdir efs

# cd efs

1-namespace.yaml

2-service-account.yaml

3-clusterrole.yaml

4-clusterrolebinding.yaml

5-role.yaml

6-rolebinding.yaml

7-deployment.yaml

 

8-storage-class.yaml

# mkdir test

# cd test

1-namespace.yaml

2-persistent-volume-claim.yaml

3-foo-deployment.yaml

4-bar-deployment.yaml

# testing 

Run below command on woker1 and worker2 and data is available in below path on woker nodes.

# mount | grep “nfs” 

###### Dynamic PV and PVC with NFS on-premise  ###  

1️.Set up an NFS server (if not already available)
2️.Deploy the NFS CSI Driver
3️.Create an NFS StorageClass (for dynamic provisioning)
4️.Create a PersistentVolumeClaim (PVC)
5️.Deploy a Pod using the dynamically provisioned volume

 

1.Set up an NFS server (if not already available)

showmount -e <NFS_SERVER_IP>

2.Install the NFS CSI Driver in Kubernetes.

Kubernetes CSI (Container Storage Interface) driver for NFS.

Provisioner Name: nfs.csi.k8s.io

# helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts

# helm repo update

# helm repo list

# helm install nfs-csi csi-driver-nfs/csi-driver-nfs --namespace kube-system --create-namespace

# helm list -n kube-system

Note:- If remove repo from helm.  

# helm repo remove csi-driver-nfs

Note:- If remove nfs csi driver

# helm uninstall nfs-csi -n kube-system

Pods is created for CSI-NFS driver.

# kubectl get pods -n kube-system | grep nfs

 

CSI Driver is showing.

# kubectl get csidrivers

 

On which nodes is the CSI driver installed.

#kubectl get csinodes

 

3️.Create an NFS StorageClass.

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: nfs-csi

provisioner: nfs.csi.k8s.io

parameters:

  server: <NFS_SERVER_IP>   # Replace with actual NFS server IP

  share: /exported/nfs-data       # Path to NFS shared directory

reclaimPolicy: Retain

volumeBindingMode: Immediate

allowVolumeExpansion: true

# kubectl apply -f storageclass-nfs.yaml

# kubectl get storageclass

Or

# kubectl get sc

 

5.Create a PersistentVolumeClaim (PVC).

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: nfs-pvc

spec:

  accessModes:

    - ReadWriteMany

  resources:

    requests:

      storage: 5Gi

  storageClassName: nfs-csi

 

# kubectl apply -f pvc.yaml

# kubectl get pvc

# kubectl get pv

 

6.Deploy a Pod Using the Dynamically Provisioned Volume.

apiVersion: v1

kind: Pod

metadata:

  name: nfs-test-pod

spec:

  containers:

  - name: busybox

    image: busybox

    command: [ "sleep", "3600" ]

    volumeMounts:

    - mountPath: "/mnt/nfs"

      name: nfs-volume

  volumes:

  - name: nfs-volume

    persistentVolumeClaim:

      claimName: nfs-pvc

 

# kubectl apply -f pod.yaml

# kubectl get pods

# kubectl get pv,pvc

 

Verification :-

1️.Check Mounted Storage Inside the Pod

# kubectl exec -it nfs-test-pod -- df -h /mnt/nfs

2️.Create a Test File to Verify Persistence

kubectl exec -it nfs-test-pod -- touch /mnt/nfs/testfile.txt

Check  the nfs  Node:

 

Multiple pvc volume in nfs path.

# ls /exported/nfs-data/<volume>

# ls /exported/nfs-data/pvc-819ab3b0-18a0-4f33-beb5-747530c01957/

 

Note :-

1.If the pod is deleted and recreated it will use the same PVC volumes.

2.If PVC volumes are deleted and PVC volumes are created, a new PVC ID is created.

3. ReclaimPolicy: Delete parameter then if PVC is deleted then PV will also be deleted. (all data).

# kubectl get pv,pvc