a fully-equipped docker image with diagnostic tools for kubernetes cluster troubleshooting

   docker   |   rocky linux 9   |   kubectl

Overview

konnect is a comprehensive diagnostic container built on Rocky Linux 9, designed specifically for troubleshooting and debugging issues within Kubernetes clusters. Rather than installing tools ad-hoc when problems arise, konnect provides a pre-built, production-ready environment with a curated selection of networking, debugging, and observability tools.

This container serves as a swiss-army knife for platform engineers, SREs, and DevOps practitioners who need to quickly diagnose cluster issues, inspect network traffic, debug services, or interact with various infrastructure components.

Development

Prerequisites

• Docker and Docker Compose for building and running the container
• hadolint for Dockerfile linting (optional, for development)
• shellcheck for shell script validation (optional, for development)
• commitlint for commit message validation (optional, for development)

Building

The container is built using multi-stage builds for optimal layer caching and image size management.

Using Docker Compose (Recommended)

docker compose up -d --build && docker compose logs -f -t

Using Make for Interactive Shell

make start-and-enter

This will build the container with full output, start it in detached mode, and drop you into a zsh shell inside the running container.

Manual Docker Build

docker build -t konnect:latest .
docker run -it konnect:latest zsh

Development Setup

To set up the development environment with linting and git hooks:

make setup
make hooks

This installs:

• commitlint for enforcing conventional commit messages
• hadolint for Dockerfile best practices
• shellcheck for shell script validation
• Git hooks for automated checks

Installation Stages

The Dockerfile uses a three-stage build process:

Stage 1: Configuration (stage1)

• Updates Rocky Linux 9 base system
• Installs EPEL (Extra Packages for Enterprise Linux) repository
• Configures package manager for optimal package installation

Stage 2: Package Installation (stage2)

• Installs all tools and utilities from pkglist.txt
• Downloads and installs architecture-specific binaries (supports both x86_64 and ARM64)
• Sets up language runtimes and platform-specific tools

Stage 3: Shell Customization (stage3)

• Configures zsh as the default shell
• Installs Oh My Zsh for enhanced shell experience
• Applies user customizations for improved terminal usability

Included Tools

Core Languages & Runtimes

Tool	Version	Purpose
go	Latest from dnf	Go programming language for running Go-based tools and scripts
python3	3.x	Python runtime for automation scripts and utilities
java-21-openjdk-headless	Java 21	Java runtime for running JVM-based tools (e.g., Pulsar client)

Shell & Utilities

Tool	Purpose
zsh	Enhanced shell with better scripting capabilities and user experience
oh-my-zsh	Framework for managing zsh configuration with plugins and themes
git	Version control for cloning repositories and managing configurations
sudo	Privilege escalation for administrative tasks
jq	JSON processor for parsing and manipulating JSON data from APIs and logs
yq	YAML processor for parsing Kubernetes manifests and config files
fzf	Fuzzy finder for interactive command-line searching
ripgrep	Ultra-fast text search tool for finding patterns in logs and files

Networking & Diagnostics

Tool	Purpose
dnsutils	DNS troubleshooting tools including dig, nslookup, and host for debugging DNS resolution issues
iputils	Essential networking utilities including ping, arping, and tracepath for basic connectivity testing
net-tools	Classic networking tools including ifconfig, netstat, route, and arp for network configuration inspection
traceroute	Packet route tracing to identify network path and latency issues between hosts
mtr	Combined traceroute and ping tool for continuous network path analysis
iftop	Real-time network bandwidth monitoring by interface and connection
tcpdump	Packet capture and analysis for deep network debugging and protocol inspection

Databases & Data Stores

Tool	Purpose
mysql	MySQL client for connecting to and querying MySQL/MariaDB databases
redis	Redis CLI for interacting with Redis caches and debugging connection issues

API & Messaging

Tool	Version	Purpose
evans	v0.10.11	Interactive gRPC client for testing and debugging gRPC services in the cluster
apache-pulsar	4.0.5	Apache Pulsar client tools for interacting with Pulsar messaging systems

System Utilities

Tool	Purpose
screen	Terminal multiplexer for managing multiple shell sessions within a single connection
which	Locates executables in the system PATH

Use Cases

Network Diagnostics

DNS Troubleshooting

# Check DNS resolution with detailed output
dig service-name.namespace.svc.cluster.local

# Query specific DNS record types
dig service-name.namespace.svc.cluster.local A +short

# Use nslookup for quick lookups
nslookup service-name.namespace.svc.cluster.local

# Check DNS with host command
host service-name.namespace.svc.cluster.local

Connectivity Testing

# Test basic connectivity
ping -c 4 pod-ip-address

# Test connectivity with ARP ping
arping -c 4 -I eth0 gateway-ip

# Trace network path to a destination
traceroute service-name.namespace.svc.cluster.local

# Use tracepath for path MTU discovery
tracepath service-name.namespace.svc.cluster.local

# Combined traceroute and ping for continuous monitoring
mtr pod-ip-address

Network Monitoring & Analysis

# Monitor real-time network bandwidth by interface
iftop -i eth0

# Show network connections and routing tables
netstat -tulpn

# Display network interfaces and their configurations
ifconfig

# Show ARP cache
arp -a

# View routing table
route -n

# Capture packets for analysis
tcpdump -i any -w capture.pcap port 8080

# Capture HTTP traffic
tcpdump -i any -A 'tcp port 80 or tcp port 443'

# Monitor specific pod traffic
tcpdump -i any host pod-ip-address

Service & Application Debugging

gRPC Services

# Test gRPC services interactively
evans --host grpc-service --port 50051

# List available gRPC services
evans --host grpc-service --port 50051 --reflection list

# Call specific gRPC method
evans --host grpc-service --port 50051 --reflection call ServiceName.MethodName

Database Connectivity

# Connect to MySQL database
mysql -h mysql-service -u user -p database_name

# Execute MySQL query directly
mysql -h mysql-service -u user -p -e "SELECT * FROM table_name;"

# Test Redis connection
redis-cli -h redis-service ping

# Get Redis key value
redis-cli -h redis-service GET key_name

# Monitor Redis commands in real-time
redis-cli -h redis-service MONITOR

# Check Redis info
redis-cli -h redis-service INFO

Messaging Systems

# List Pulsar topics
pulsar-client --url pulsar://pulsar-service:6650 list-topics

# Consume messages from a Pulsar topic
pulsar-client --url pulsar://pulsar-service:6650 consume persistent://tenant/namespace/topic

# Produce test messages to Pulsar
pulsar-client --url pulsar://pulsar-service:6650 produce persistent://tenant/namespace/topic --messages "test message"

Data Processing & Configuration

JSON & YAML Processing

# Parse JSON logs and filter by level
kubectl logs pod-name | jq 'select(.level == "error")'

# Extract specific JSON fields
kubectl logs pod-name | jq '.timestamp, .message'

# Pretty print JSON
echo '{"key":"value"}' | jq '.'

# Parse and query Kubernetes manifests
kubectl get pod pod-name -o yaml | yq '.spec.containers[].image'

# Filter YAML by field
kubectl get deployment deployment-name -o yaml | yq '.spec.replicas'

# Convert YAML to JSON
cat config.yaml | yq -o json

Log Analysis & Search

# Search logs for errors with ripgrep
kubectl logs pod-name | rg "ERROR|FATAL"

# Case-insensitive search
kubectl logs pod-name | rg -i "warning"

# Search with context lines
kubectl logs pod-name | rg -A 3 -B 3 "exception"

# Interactive search through logs
kubectl logs pod-name | fzf

# Search through multiple files
rg "pattern" /var/log/

Development & Scripting

Python Utilities

# Run Python scripts for automation
python3 script.py

# Use Python for quick calculations or data processing
python3 -c "import json; print(json.dumps({'key': 'value'}, indent=2))"

# Install additional Python packages temporarily
pip3 install --user package-name

Go Tools

# Run Go-based diagnostic tools
go run tool.go

# Build Go utilities on-the-fly
go build -o binary tool.go

Session Management

Terminal Multiplexing

# Start a new screen session
screen -S debug-session

# List screen sessions
screen -ls

# Reattach to a screen session
screen -r debug-session

# Detach from screen session (Ctrl-A, D)

# Run long-running commands in screen
screen -dmS monitoring-session iftop -i eth0

System Investigation

File & Process Discovery

# Locate executables
which kubectl

# Find files by pattern
find /etc -name "*.conf"

# Search for patterns across files
rg "pattern" /etc/

# Use fuzzy finder to search command history
history | fzf

Version Control

# Clone configuration repositories
git clone https://github.com/user/repo.git

# Check repository status
cd repo && git status

# Pull latest configurations
git pull origin main

General Troubleshooting

Deploying konnect

# Enter the container in a running cluster
kubectl run konnect --image=your-registry/konnect:latest -it --rm -- zsh

# Attach to an existing konnect pod
kubectl exec -it konnect-pod-name -- zsh

# Use as a debug container (Kubernetes 1.23+)
kubectl debug -it problematic-pod --image=your-registry/konnect:latest --target=container-name

# Deploy as a long-running pod for persistent debugging
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: konnect
spec:
  containers:
  - name: konnect
    image: your-registry/konnect:latest
    command: ["sleep", "infinity"]
EOF

Common Workflows

# Debug DNS issues: test resolution and trace routing
dig problematic-service.namespace.svc.cluster.local
mtr $(dig +short problematic-service.namespace.svc.cluster.local | head -1)

# Investigate network performance: capture and analyze traffic
tcpdump -i any -w debug.pcap host service-ip &
# ... reproduce issue ...
kill %1
# Download debug.pcap for analysis in Wireshark

# Test service connectivity end-to-end
ping service-ip
traceroute service-ip
redis-cli -h service-ip PING
mysql -h service-ip -u user -p -e "SELECT 1;"

# Analyze application logs for patterns
kubectl logs deployment/app-name --all-containers=true | rg "ERROR|WARN" | jq -r '.message' | sort | uniq -c | sort -rn

Architecture Support

The container build process automatically detects and supports both x86_64 (AMD64) and ARM64 (aarch64) architectures, ensuring compatibility across various Kubernetes node types including:

• Traditional x86_64 servers
• ARM-based instances (AWS Graviton, Azure Ampere, etc.)
• Apple Silicon (M1/M2/M3) for local development

Contributing

This project uses conventional commits and automated git hooks for quality assurance.

Commit Message Format

<type>(<scope>): <subject>

<body>

Types: feat, fix, docs, style, refactor, test, chore

Adding New Tools

To add a new tool to the container:

1. For dnf packages: Add the package name to scripts/pkglist.txt
2. For binary downloads: Add installation logic to scripts/install_packages.sh
3. Update documentation: Add the tool to the "Included Tools" section above with its purpose
4. Test the build: make start-and-enter

License

See LICENSE for details.