## Introduction I keep this reference open in a tab whenever I'm working with Kubernetes. These are the commands I actually use when diagnosing cluster issues, organized by problem type. Context matters in debugging—I've tried to organize this in a way that matches how you'd actually troubleshoot. ## Pod Issues ### Pod Won't Start or Stays in Pending ```bash # Get detailed pod info kubectl describe pod -n # Check pod events specifically kubectl get events -n --field-selector involvedObject.name= # Check resource requests vs node availability kubectl describe nodes kubectl top nodes ``` **Common causes:** Resource limits exceeded, node affinity mismatches, PVC not bound. ### Pod Crashes or Restarts Constantly ```bash # Check restart count and reason kubectl get pods -n -o wide kubectl describe pod -n # View the last crash logs kubectl logs -n --previous # Stream current logs (useful for immediate crashes) kubectl logs -n -f # Check if it's an OOMKill kubectl describe pod -n | grep -A 5 "Last State" ``` **Common causes:** Out of memory, uncaught exceptions, health check failures, dependency unavailable. ### Pod Stuck in CrashLoopBackOff ```bash # Get the full error message kubectl logs -n --previous --tail=100 # Check if it's a startup probe issue kubectl get pod -n -o yaml | grep -A 10 "startupProbe" # Increase startup probe initialDelaySeconds if the app is slow to start kubectl patch pod -n -p \ '{"spec":{"containers":[{"name":"","startupProbe":{"initialDelaySeconds":60}}]}}' ``` ### Check Pod Resource Usage ```bash # Real-time resource usage (requires metrics-server) kubectl top pods -n # Per-container breakdown kubectl top pods -n --containers # Most resource-hungry pods cluster-wide kubectl top pods -A --sort-by=memory ``` ## Service Connectivity Issues ### Service Won't Route Traffic to Pods ```bash # Check service endpoints kubectl get endpoints -n kubectl describe service -n # Verify label selectors match your pods kubectl get pods -n -L app= kubectl get service -n -o yaml | grep -A 5 "selector:" # Test DNS resolution from inside a pod kubectl run -it --rm debug --image=busybox --restart=Never -- \ nslookup ..svc.cluster.local # Test connectivity to the service kubectl run -it --rm debug --image=busybox --restart=Never -- \ wget -O- :8080 ``` ### DNS Not Resolving ```bash # Check CoreDNS pod status kubectl get pods -n kube-system -l k8s-app=kube-dns kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 # Restart CoreDNS if needed kubectl rollout restart deployment/coredns -n kube-system # Test DNS from a pod (should resolve immediately) kubectl exec -n -- nslookup kubernetes.default ``` ### Ingress Not Routing Traffic ```bash # Check ingress resource kubectl describe ingress -n # Verify the ingress controller is running kubectl get pods -n ingress-nginx (or your ingress controller namespace) # Check ingress controller logs kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50 # Test the ingress backend kubectl describe ingress -n # Then manually test one of the service endpoints ``` ## Log Analysis ### View Logs from Multiple Pods ```bash # All pods in a deployment kubectl logs -n -l app= --tail=100 # Follow logs from multiple pods kubectl logs -n -l app= -f # Logs from previous container (after crash) kubectl logs -n -l app= --previous ``` ### Search Logs for Errors ```bash # Error logs from the last 10 minutes kubectl logs -n --timestamps=true \ | grep -i error \ | tail -20 # Logs with timestamps for correlation kubectl logs -n --timestamps=true | tail -50 ``` ### Streaming Logs with Context ```bash # Follow logs and show 50 lines of context kubectl logs -n -f --tail=50 # Follow logs from multiple pods in a deployment for pod in $(kubectl get pods -n -l app=myapp -o name); do kubectl logs "$pod" -n -f & done ``` ## Performance & Resource Debugging ### Node Performance ```bash # Node resource usage kubectl top nodes # Detailed node info kubectl describe node # Check node conditions kubectl get nodes -o wide kubectl describe node | grep -A 10 "Conditions" # Find pods using most resources on a node kubectl get pods -A -o wide | grep kubectl top pods -A | grep ``` ### Disk and Storage Issues ```bash # Check PVC status kubectl get pvc -n kubectl describe pvc -n # Check PV status kubectl get pv kubectl describe pv # Find pods using a particular PVC kubectl get pods -n -o json | \ jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="") | .metadata.name' ``` ## Quick Reference Table | Problem | Command | Purpose | |---------|---------|---------| | Pod won't start | `kubectl describe pod ` | Get detailed status and events | | Pod keeps restarting | `kubectl logs --previous` | See crash logs | | Check pod health | `kubectl get pods -n -o wide` | Quick status overview | | Service not routing | `kubectl get endpoints ` | Verify pods are registered | | DNS issues | `kubectl get pods -n kube-system` | Check CoreDNS health | | Resource usage | `kubectl top pods -A` | Identify resource hogs | | Recent errors | `kubectl logs --tail=100` | Check latest logs | | Follow live logs | `kubectl logs -f` | Stream in real-time | ## Debugging Strategies 1. **Start with status**: Always run `kubectl describe` first—it shows events and conditions. 2. **Check logs early**: Most issues show up in logs. Look for exceptions, errors, or warnings. 3. **Verify selectors**: Label mismatches are a common cause of connectivity issues. 4. **Test DNS**: If services can't find each other, DNS is usually the culprit. 5. **Monitor resources**: Out of memory and CPU throttling cause subtle bugs. 6. **Use temporary debug pods**: The busybox image is perfect for testing network connectivity from inside the cluster. ## Useful Aliases ```bash # Add to your ~/.bashrc or ~/.zshrc alias k='kubectl' alias kdesc='kubectl describe' alias klogs='kubectl logs' alias kget='kubectl get' alias kctx='kubectl config current-context' ``` These commands have saved me countless hours of debugging. When something breaks in Kubernetes, these are the first things I reach for.