|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## Project Overview |
| 6 | + |
| 7 | +This is the **Redis Kafka Connect** project - a set of Kafka Connect connectors for Redis that provides both source and sink functionality. The project enables bidirectional data flow between Redis and Apache Kafka. |
| 8 | + |
| 9 | +### Key Components |
| 10 | + |
| 11 | +- **RedisSinkConnector**: Writes data from Kafka topics to Redis (supports regular Redis keys, JSON documents, and streams) |
| 12 | +- **RedisStreamSourceConnector**: Reads data from Redis streams and publishes to Kafka topics |
| 13 | +- **RedisKeysSourceConnector**: Monitors Redis key changes via keyspace notifications and publishes to Kafka topics |
| 14 | + |
| 15 | +## Build System |
| 16 | + |
| 17 | +This project uses **Gradle** as the build system with a multi-project structure: |
| 18 | + |
| 19 | +### Core Commands |
| 20 | + |
| 21 | +```bash |
| 22 | +# Build the project |
| 23 | +./gradlew build |
| 24 | + |
| 25 | +# Run tests |
| 26 | +./gradlew test |
| 27 | + |
| 28 | +# Run all tests (including integration tests) |
| 29 | +./gradlew redis-kafka-connect:allTests |
| 30 | + |
| 31 | +# Create distribution JAR |
| 32 | +./gradlew redis-kafka-connect:shadowJar |
| 33 | + |
| 34 | +# Run test coverage |
| 35 | +./gradlew redis-kafka-connect:jacocoTestReport |
| 36 | + |
| 37 | +# Check license headers |
| 38 | +./gradlew license |
| 39 | + |
| 40 | +# Format license headers |
| 41 | +./gradlew licenseFormat |
| 42 | + |
| 43 | +# Create Confluent Hub archive |
| 44 | +./gradlew redis-kafka-connect:createConfluentArchive |
| 45 | +``` |
| 46 | + |
| 47 | +### Testing |
| 48 | + |
| 49 | +The project includes comprehensive test suites: |
| 50 | +- Unit tests: `./gradlew test` |
| 51 | +- Integration tests with Redis Enterprise and Redis Stack: Use `redis-kafka-connect:allTests` |
| 52 | +- Tests use TestContainers for Redis instances |
| 53 | + |
| 54 | +## Architecture |
| 55 | + |
| 56 | +### Module Structure |
| 57 | + |
| 58 | +``` |
| 59 | +core/redis-kafka-connect/ |
| 60 | +├── src/main/java/com/redis/kafka/connect/ |
| 61 | +│ ├── RedisKeysSourceConnector.java # Keys source connector |
| 62 | +│ ├── RedisSinkConnector.java # Sink connector |
| 63 | +│ ├── RedisStreamSourceConnector.java # Stream source connector |
| 64 | +│ ├── common/ # Shared configuration and utilities |
| 65 | +│ │ ├── RedisConfig.java # Redis connection configuration |
| 66 | +│ │ └── RedisConfigDef.java # Configuration definitions |
| 67 | +│ ├── sink/ # Sink connector implementation |
| 68 | +│ │ ├── RedisSinkTask.java # Main sink task |
| 69 | +│ │ └── RedisSinkConfig.java # Sink configuration |
| 70 | +│ └── source/ # Source connector implementations |
| 71 | +│ ├── AbstractRedisSourceConnector.java |
| 72 | +│ ├── RedisKeysSourceTask.java # Keys monitoring task |
| 73 | +│ └── RedisStreamSourceTask.java # Stream reading task |
| 74 | +``` |
| 75 | + |
| 76 | +### Key Design Patterns |
| 77 | + |
| 78 | +- **AbstractRedisSourceConnector**: Base class for all source connectors, handles common functionality like task configuration and versioning |
| 79 | +- **RedisConfig**: Centralized Redis connection configuration supporting standalone, cluster, SSL/TLS, and authentication |
| 80 | +- **Task-based architecture**: Each connector delegates actual work to corresponding Task classes |
| 81 | +- **Modular configuration**: Each connector has its own ConfigDef for type-safe configuration |
| 82 | + |
| 83 | +## Development Environment |
| 84 | + |
| 85 | +### Docker Development Setup |
| 86 | + |
| 87 | +Use the provided `run.sh` script for a complete development environment: |
| 88 | + |
| 89 | +```bash |
| 90 | +./run.sh |
| 91 | +``` |
| 92 | + |
| 93 | +This script: |
| 94 | +- Starts Redis, Kafka, and Kafka Connect via Docker Compose |
| 95 | +- Configures sample connectors |
| 96 | +- Provides UI access at http://localhost:9021 (Confluent Control Center) or http://localhost:8000 |
| 97 | + |
| 98 | +### Manual Docker Setup |
| 99 | + |
| 100 | +```bash |
| 101 | +# Start services |
| 102 | +docker compose up -d |
| 103 | + |
| 104 | +# Stop services |
| 105 | +docker compose down |
| 106 | +``` |
| 107 | + |
| 108 | +## Testing Strategy |
| 109 | + |
| 110 | +### Test Categories |
| 111 | + |
| 112 | +1. **Unit Tests**: Test individual components in isolation |
| 113 | +2. **Integration Tests**: Test against real Redis instances using TestContainers |
| 114 | +3. **Enterprise Tests**: Test Redis Enterprise features |
| 115 | +4. **Stack Tests**: Test Redis Stack features (JSON, Search, etc.) |
| 116 | + |
| 117 | +### Running Specific Tests |
| 118 | + |
| 119 | +```bash |
| 120 | +# Run single test class |
| 121 | +./gradlew test --tests "StreamSourceConnectorTest" |
| 122 | + |
| 123 | +# Run integration tests only |
| 124 | +./gradlew redis-kafka-connect:allTests -Dtest.include.tags="integration" |
| 125 | +``` |
| 126 | + |
| 127 | +## Configuration |
| 128 | + |
| 129 | +### Redis Connection |
| 130 | + |
| 131 | +The connectors support multiple Redis deployment types: |
| 132 | +- **Standalone**: Single Redis instance |
| 133 | +- **Cluster**: Redis Cluster setup |
| 134 | +- **SSL/TLS**: Secure connections with certificate validation |
| 135 | +- **Authentication**: Username/password and certificate-based auth |
| 136 | + |
| 137 | +### Common Configuration Properties |
| 138 | + |
| 139 | +- `redis.uri`: Redis connection URI (e.g., `redis://localhost:6379`) |
| 140 | +- `redis.cluster`: Enable cluster mode |
| 141 | +- `redis.tls`: Enable SSL/TLS |
| 142 | +- `redis.username`/`redis.password`: Authentication credentials |
| 143 | + |
| 144 | +## Dependencies |
| 145 | + |
| 146 | +### Core Dependencies |
| 147 | + |
| 148 | +- **Lettuce**: Redis client library (configured via `lettucemodVersion` in gradle.properties) |
| 149 | +- **Kafka Connect API**: For connector framework integration |
| 150 | +- **Spring Batch Redis**: For batch processing utilities |
| 151 | +- **TestContainers**: For integration testing |
| 152 | + |
| 153 | +### Version Management |
| 154 | + |
| 155 | +Key versions are managed in `gradle.properties`: |
| 156 | +- Java compatibility: JDK 17 |
| 157 | +- Lettuce version: 6.5.2.RELEASE |
| 158 | +- TestContainers Redis: 2.2.3 |
| 159 | + |
| 160 | +## Release Process |
| 161 | + |
| 162 | +The project uses JReleaser for release management: |
| 163 | +- Confluent Hub distribution via `createConfluentArchive` task |
| 164 | +- GitHub releases with automated changelog generation |
| 165 | +- Version management through Gradle properties |
0 commit comments