seaweedfs/test/kafka/kafka-client-loadtest

Kafka Client Load Test for SeaweedFS

This comprehensive load testing suite validates the SeaweedFS MQ stack using real Kafka client libraries. Unlike the existing SMQ tests, this uses actual Kafka clients (sarama and confluent-kafka-go) to test the complete integration through:

• Kafka Clients → SeaweedFS Kafka Gateway → SeaweedFS MQ Broker → SeaweedFS Storage

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────────┐
│   Kafka Client  │    │  Kafka Gateway   │    │   SeaweedFS MQ      │
│   Load Test     │───▶│  (Port 9093)     │───▶│   Broker            │
│   - Producers   │    │                  │    │                     │
│   - Consumers   │    │  Protocol        │    │   Topic Management  │
│                 │    │  Translation     │    │   Message Storage   │
└─────────────────┘    └──────────────────┘    └─────────────────────┘
                                                             │
                                                             ▼
                                                ┌─────────────────────┐
                                                │  SeaweedFS Storage  │
                                                │  - Master           │
                                                │  - Volume Server    │
                                                │  - Filer            │
                                                └─────────────────────┘

Features

🚀 Multiple Test Modes

• Producer-only: Pure message production testing
• Consumer-only: Consumption from existing topics
• Comprehensive: Full producer + consumer load testing

📊 Rich Metrics & Monitoring

• Prometheus metrics collection
• Grafana dashboards
• Real-time throughput and latency tracking
• Consumer lag monitoring
• Error rate analysis

🔧 Configurable Test Scenarios

• Quick Test: 1-minute smoke test
• Standard Test: 5-minute medium load
• Stress Test: 10-minute high load
• Endurance Test: 30-minute sustained load
• Custom: Fully configurable parameters

📈 Message Types

• JSON: Structured test messages
• Avro: Schema Registry integration
• Binary: Raw binary payloads

🛠 Kafka Client Support

• Sarama: Native Go Kafka client
• Confluent: Official Confluent Go client
• Schema Registry integration
• Consumer group management

Quick Start

Prerequisites

• Docker & Docker Compose
• Make (optional, but recommended)

1. Run Default Test

make test

This runs a 5-minute comprehensive test with 10 producers and 5 consumers.

2. Quick Smoke Test

make quick-test

1-minute test with minimal load for validation.

3. Stress Test

make stress-test  

10-minute high-throughput test with 20 producers and 10 consumers.

4. Test with Monitoring

make test-with-monitoring

Includes Prometheus + Grafana dashboards for real-time monitoring.

Detailed Usage

Manual Control

# Start infrastructure only
make start

# Run load test against running infrastructure
make test TEST_MODE=comprehensive TEST_DURATION=10m

# Stop everything
make stop

# Clean up all resources
make clean

Using Scripts Directly

# Full control with the main script
./scripts/run-loadtest.sh start -m comprehensive -d 10m --monitoring

# Check service health
./scripts/wait-for-services.sh check

# Setup monitoring configurations
./scripts/setup-monitoring.sh

Environment Variables

export TEST_MODE=comprehensive        # producer, consumer, comprehensive  
export TEST_DURATION=300s            # Test duration
export PRODUCER_COUNT=10              # Number of producer instances
export CONSUMER_COUNT=5               # Number of consumer instances  
export MESSAGE_RATE=1000              # Messages/second per producer
export MESSAGE_SIZE=1024              # Message size in bytes
export TOPIC_COUNT=5                  # Number of topics to create
export PARTITIONS_PER_TOPIC=3         # Partitions per topic

make test

Configuration

Main Configuration File

Edit config/loadtest.yaml to customize:

• Kafka Settings: Bootstrap servers, security, timeouts
• Producer Config: Batching, compression, acknowledgments
• Consumer Config: Group settings, fetch parameters
• Message Settings: Size, format (JSON/Avro/Binary)
• Schema Registry: Avro/Protobuf schema validation
• Metrics: Prometheus collection intervals
• Test Scenarios: Predefined load patterns

Example Custom Configuration

test_mode: "comprehensive"
duration: "600s"  # 10 minutes

producers:
  count: 15
  message_rate: 2000
  message_size: 2048
  compression_type: "snappy"
  acks: "all"

consumers:
  count: 8
  group_prefix: "high-load-group"
  max_poll_records: 1000

topics:
  count: 10
  partitions: 6
  replication_factor: 1

Test Scenarios

1. Producer Performance Test

make producer-test TEST_DURATION=10m PRODUCER_COUNT=20 MESSAGE_RATE=3000

Tests maximum message production throughput.

2. Consumer Performance Test

# First produce messages
make producer-test TEST_DURATION=5m

# Then test consumption
make consumer-test TEST_DURATION=10m CONSUMER_COUNT=15

3. Schema Registry Integration

# Enable schemas in config/loadtest.yaml
schemas:
  enabled: true
  
make test

Tests Avro message serialization through Schema Registry.

4. High Availability Test

# Test with container restarts during load
make test TEST_DURATION=20m &
sleep 300
docker restart kafka-gateway

Monitoring & Metrics

Real-Time Dashboards

When monitoring is enabled:

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Key Metrics Tracked

• Throughput: Messages/second, MB/second
• Latency: End-to-end message latency percentiles
• Errors: Producer/consumer error rates
• Consumer Lag: Per-partition lag monitoring
• Resource Usage: CPU, memory, disk I/O

Grafana Dashboards

• Kafka Load Test: Comprehensive test metrics
• SeaweedFS Cluster: Storage system health
• Custom Dashboards: Extensible monitoring

Advanced Features

Schema Registry Testing

# Test Avro message serialization
export KAFKA_VALUE_TYPE=avro
make test

The load test includes:

• Schema registration
• Avro message encoding/decoding
• Schema evolution testing
• Compatibility validation

Multi-Client Testing

The test supports both Sarama and Confluent clients:

// Configure in producer/consumer code
useConfluent := true  // Switch client implementation

Consumer Group Rebalancing

• Automatic consumer group management
• Partition rebalancing simulation
• Consumer failure recovery testing

Chaos Testing

chaos:
  enabled: true
  producer_failure_rate: 0.01
  consumer_failure_rate: 0.01
  network_partition_probability: 0.001

Troubleshooting

Common Issues

Services Not Starting

# Check service health
make health-check

# View detailed logs
make logs

# Debug mode
make debug

Low Throughput

• Increase MESSAGE_RATE and PRODUCER_COUNT
• Adjust batch_size and linger_ms in config
• Check consumer max_poll_records setting

High Latency

• Reduce linger_ms for lower latency
• Adjust acks setting (0, 1, or "all")
• Monitor consumer lag

Memory Issues

# Reduce concurrent clients
make test PRODUCER_COUNT=5 CONSUMER_COUNT=3

# Adjust message size  
make test MESSAGE_SIZE=512

Debug Commands

# Execute shell in containers
make exec-master
make exec-filer  
make exec-gateway

# Attach to load test
make attach-loadtest

# View real-time stats
curl http://localhost:8080/stats

Development

Building from Source

# Set up development environment
make dev-env

# Build load test binary
make build

# Run tests locally (requires Go 1.21+)
cd cmd/loadtest && go run main.go -config ../../config/loadtest.yaml

Extending the Tests

1. Add new message formats in internal/producer/
2. Add custom metrics in internal/metrics/
3. Create new test scenarios in config/loadtest.yaml
4. Add monitoring panels in monitoring/grafana/dashboards/

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass: make test
5. Submit a pull request

Performance Benchmarks

Expected Performance (on typical hardware)

Scenario	Producers	Consumers	Rate (msg/s)	Latency (p95)
Quick	2	2	200	<10ms
Standard	5	3	2,500	<20ms
Stress	20	10	40,000	<50ms
Endurance	10	5	10,000	<30ms

Results vary based on hardware, network, and SeaweedFS configuration

Tuning for Maximum Performance

producers:
  batch_size: 1000
  linger_ms: 10
  compression_type: "lz4"
  acks: "1"  # Balance between speed and durability

consumers:  
  max_poll_records: 5000
  fetch_min_bytes: 1048576  # 1MB
  fetch_max_wait_ms: 100

Comparison with Existing Tests

Feature	SMQ Tests	Kafka Client Load Test
Protocol	SMQ (SeaweedFS native)	Kafka (industry standard)
Clients	SMQ clients	Real Kafka clients (Sarama, Confluent)
Schema Registry	❌	✅ Full Avro/Protobuf support
Consumer Groups	Basic	✅ Full Kafka consumer group features
Monitoring	Basic	✅ Prometheus + Grafana dashboards
Test Scenarios	Limited	✅ Multiple predefined scenarios
Real-world	Synthetic	✅ Production-like workloads

This load test provides comprehensive validation of the SeaweedFS Kafka Gateway using real-world Kafka clients and protocols.

---

Quick Reference

# Essential Commands
make help                    # Show all available commands
make test                    # Run default comprehensive test  
make quick-test              # 1-minute smoke test
make stress-test             # High-load stress test
make test-with-monitoring    # Include Grafana dashboards
make clean                   # Clean up all resources

# Monitoring
make monitor                 # Start Prometheus + Grafana
# → http://localhost:9090 (Prometheus)
# → http://localhost:3000 (Grafana, admin/admin)

# Advanced
make benchmark               # Run full benchmark suite
make health-check            # Validate service health
make validate-setup          # Check configuration