Kubernetes Troubleshooting Guide

Comprehensive guide to diagnosing and resolving common Kubernetes issues.

Troubleshooting Workflow

1. Identify the Problem
   ↓
2. Check Pod Status
   ↓
3. View Logs
   ↓
4. Describe Resources
   ↓
5. Check Events
   ↓
6. Verify Configuration
   ↓
7. Test Connectivity
   ↓
8. Fix and Verify

Pod Issues

Pod Stuck in Pending

Symptoms:

kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
myapp     0/1     Pending   0          5m

Diagnosis:

# Describe pod
kubectl describe pod myapp

# Check events
kubectl get events --sort-by='.lastTimestamp'

# Check node resources
kubectl top nodes
kubectl describe nodes

Common Causes: 1. Insufficient Resources

Events:
  Warning  FailedScheduling  pod didn't fit on any node

Solution: Add nodes or reduce resource requests

1. Node Selector Mismatch

   nodeSelector:
     disktype: ssd  # No nodes with this label
   

   Solution: Fix node selector or add label to nodes

2. PVC Not Bound

   kubectl get pvc
   

   Solution: Check PVC status and storage class

Pod Stuck in ImagePullBackOff

Symptoms:

NAME      READY   STATUS             RESTARTS   AGE
myapp     0/1     ImagePullBackOff   0          2m

Diagnosis:

kubectl describe pod myapp

Common Causes: 1. Image Doesn't Exist

Failed to pull image "myapp:1.0": not found

Solution: Verify image name and tag

1. Authentication Required

   Failed to pull image: unauthorized
   

   Solution: Add imagePullSecrets

   imagePullSecrets:
   - name: regcred
   

2. Network Issues

   # Test from node
   docker pull myapp:1.0
   

Pod Stuck in CrashLoopBackOff

Symptoms:

NAME      READY   STATUS             RESTARTS   AGE
myapp     0/1     CrashLoopBackOff   5          5m

Diagnosis:

# View logs
kubectl logs myapp
kubectl logs myapp --previous

# Describe pod
kubectl describe pod myapp

# Check exit code
kubectl get pod myapp -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Common Causes: 1. Application Error

# Check logs for errors
kubectl logs myapp --previous

1. Missing Dependencies

   Error: Cannot connect to database
   

   Solution: Verify ConfigMap/Secret values

2. Liveness Probe Failing

   livenessProbe:
     httpGet:
       path: /healthz
       port: 8080
     initialDelaySeconds: 30  # Increase if app is slow to start
   

Pod in Error State

Diagnosis:

kubectl describe pod myapp
kubectl logs myapp

Common Causes: 1. Init Container Failed

kubectl logs myapp -c init-container

1. Volume Mount Issues

   Error: failed to mount volume
   

2. Security Context Issues

   Error: container has runAsNonRoot and image has non-numeric user
   

Service Issues

Service Not Accessible

Diagnosis:

# Check service
kubectl get svc myapp
kubectl describe svc myapp

# Check endpoints
kubectl get endpoints myapp

# Test from within cluster
kubectl run test --rm -it --image=busybox --restart=Never -- wget -O- http://myapp

Common Causes: 1. No Endpoints

kubectl get endpoints myapp
# NAME    ENDPOINTS   AGE
# myapp   <none>      5m

Solution: Check pod selector matches

kubectl get pods --show-labels
kubectl get svc myapp -o yaml | grep selector

1. Wrong Port

   ports:
   - port: 80
     targetPort: 8080  # Must match container port
   

2. Readiness Probe Failing

   kubectl describe pod myapp
   # Check readiness probe status
   

LoadBalancer Pending

Symptoms:

kubectl get svc
NAME    TYPE           EXTERNAL-IP   PORT(S)
myapp   LoadBalancer   <pending>     80:30080/TCP

Diagnosis:

kubectl describe svc myapp

Common Causes: 1. Cloud Provider Not Configured - Verify cloud controller manager is running - Check cloud provider credentials

1. Quota Exceeded
2. Check cloud provider quotas

3. Verify billing is active

4. Network Configuration

5. Check VPC/subnet configuration
6. Verify security groups

Networking Issues

Pod-to-Pod Communication

Test Connectivity:

# Get pod IPs
kubectl get pods -o wide

# Test from one pod to another
kubectl exec -it pod1 -- ping <pod2-ip>
kubectl exec -it pod1 -- curl http://<pod2-ip>:8080

Common Issues: 1. Network Policy Blocking

kubectl get networkpolicies
kubectl describe networkpolicy <policy-name>

1. CNI Plugin Issues

   kubectl get pods -n kube-system | grep -E 'calico|flannel|weave'
   kubectl logs -n kube-system <cni-pod>
   

DNS Issues

Test DNS:

# Test DNS resolution
kubectl run test --rm -it --image=busybox --restart=Never -- nslookup kubernetes.default

# Test service DNS
kubectl run test --rm -it --image=busybox --restart=Never -- nslookup myapp.default.svc.cluster.local

# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Common Issues: 1. CoreDNS Not Running

kubectl get pods -n kube-system -l k8s-app=kube-dns

1. DNS Configuration

   kubectl get configmap -n kube-system coredns -o yaml
   

2. Pod DNS Policy

   dnsPolicy: ClusterFirst  # Should be default
   

Storage Issues

PVC Stuck in Pending

Diagnosis:

kubectl get pvc
kubectl describe pvc myapp-pvc

Common Causes: 1. No Storage Class

kubectl get storageclass

Solution: Create or specify storage class

1. No Available PV

   kubectl get pv
   

   Solution: Create PV or use dynamic provisioning

2. Access Mode Mismatch

   accessModes:
   - ReadWriteOnce  # Must match PV
   

Volume Mount Failures

Diagnosis:

kubectl describe pod myapp
# Check events for mount errors

Common Issues: 1. Permission Denied

securityContext:
  fsGroup: 2000  # Add fsGroup

1. Volume Not Found

   kubectl get pvc
   kubectl get pv
   

Node Issues

Node NotReady

Diagnosis:

kubectl get nodes
kubectl describe node <node-name>

# Check kubelet
ssh <node>
systemctl status kubelet
journalctl -u kubelet -f

Common Causes: 1. Kubelet Not Running

systemctl start kubelet
systemctl enable kubelet

1. Disk Pressure

   df -h
   # Clean up disk space
   

2. Memory Pressure

   free -h
   # Add more memory or reduce workload
   

Node Disk Pressure

Diagnosis:

kubectl describe node <node-name>
# Check conditions

ssh <node>
df -h

Solution:

# Clean up images
docker system prune -a

# Clean up logs
journalctl --vacuum-time=3d

# Clean up pods
kubectl delete pods --field-selector=status.phase==Succeeded

Resource Issues

Out of Memory (OOM)

Symptoms:

kubectl get pods
NAME      READY   STATUS      RESTARTS   AGE
myapp     0/1     OOMKilled   1          2m

Diagnosis:

kubectl describe pod myapp
# Check last state: OOMKilled

kubectl top pod myapp

Solution:

resources:
  limits:
    memory: "512Mi"  # Increase memory limit
  requests:
    memory: "256Mi"

CPU Throttling

Diagnosis:

kubectl top pods
kubectl top nodes

# Check metrics
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

Solution:

resources:
  limits:
    cpu: "1000m"  # Increase CPU limit
  requests:
    cpu: "500m"

Configuration Issues

ConfigMap/Secret Not Found

Diagnosis:

kubectl get configmap
kubectl get secret

kubectl describe pod myapp
# Check events

Solution:

# Create missing ConfigMap
kubectl create configmap myapp-config --from-literal=key=value

# Verify reference
kubectl get pod myapp -o yaml | grep -A 5 configMap

Environment Variable Issues

Diagnosis:

# Check environment variables
kubectl exec myapp -- env

# Check pod spec
kubectl get pod myapp -o yaml | grep -A 10 env

RBAC Issues

Permission Denied

Diagnosis:

# Check permissions
kubectl auth can-i create pods
kubectl auth can-i create pods --as=system:serviceaccount:default:myapp-sa

# Check role bindings
kubectl get rolebindings
kubectl describe rolebinding <binding-name>

Solution:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-binding
subjects:
- kind: ServiceAccount
  name: myapp-sa
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

Debugging Tools

Debug Container

# Add ephemeral debug container
kubectl debug myapp -it --image=busybox --target=myapp

# Debug node
kubectl debug node/<node-name> -it --image=ubuntu

Network Debug Pod

kubectl run netshoot --rm -it --image=nicolaka/netshoot --restart=Never -- bash

# Inside pod
ping <ip>
nslookup <service>
curl http://<service>
traceroute <ip>

Port Forward for Debugging

# Forward pod port
kubectl port-forward myapp 8080:80

# Forward service port
kubectl port-forward svc/myapp 8080:80

# Test locally
curl http://localhost:8080

Useful Commands

Get Events

# All events
kubectl get events --sort-by='.lastTimestamp'

# Events for specific resource
kubectl get events --field-selector involvedObject.name=myapp

# Watch events
kubectl get events --watch

Check Resource Usage

# Node resources
kubectl top nodes

# Pod resources
kubectl top pods

# Pod resources in namespace
kubectl top pods -n production

View Logs

# Current logs
kubectl logs myapp

# Previous container logs
kubectl logs myapp --previous

# Follow logs
kubectl logs -f myapp

# Logs from specific container
kubectl logs myapp -c container-name

# Logs with timestamps
kubectl logs myapp --timestamps

# Tail logs
kubectl logs myapp --tail=100

Describe Resources

kubectl describe pod myapp
kubectl describe node <node-name>
kubectl describe svc myapp
kubectl describe pvc myapp-pvc

Common Error Messages

"ImagePullBackOff"

• Image doesn't exist
• Wrong image name/tag
• Authentication required
• Network issues

"CrashLoopBackOff"

• Application error
• Missing dependencies
• Liveness probe failing
• Wrong command/args

"Pending"

• Insufficient resources
• Node selector mismatch
• PVC not bound
• Taints/tolerations

"Error"

• Init container failed
• Volume mount issues
• Security context issues
• Command failed

"OOMKilled"

• Memory limit too low
• Memory leak in application
• Increase memory limits

Best Practices

1. Enable Logging
2. Centralized logging
3. Structured logs

4. Appropriate log levels

5. Monitor Resources

6. Set up metrics collection
7. Create alerts

8. Regular reviews

9. Use Health Checks

10. Liveness probes
11. Readiness probes

12. Startup probes

13. Document Issues

14. Keep runbooks
15. Document solutions

16. Share knowledge

17. Test Changes

18. Test in non-production
19. Use dry-run
20. Gradual rollouts

References

• Troubleshooting Applications
• Troubleshooting Clusters
• Debug Running Pods