Parallel Consumer

Shields.io variant (for comparison):

astubbs/parallel-consumer

What it is

A parallel Apache Kafka consumer that processes multiple messages concurrently from a single consumer instance. Higher throughput and lower latency than traditional partition-based parallelism.

How it works

Traditional Kafka consumers process messages one partition at a time – if you want more parallelism, you add more partitions. Parallel Consumer flips this: one consumer instance processes many messages in parallel, while preserving ordering where it matters (by key, by partition, or not at all depending on your needs).

var options = ParallelConsumerOptions.<String, String>builder()
    .ordering(KEY)
    .maxConcurrency(1000)
    .consumer(kafkaConsumer)
    .build();

var pc = ParallelStreamProcessor.createEosStreamProcessor(options);
pc.poll(record -> processRecord(record));

The talk

Watch the Kafka Summit Europe 2021 presentation where I introduced this as Confluent Labs Parallel Consumer Client.

Timeline

• April 2026 – Picking the project back up on my fork. Re-establishing it as a properly-maintained community open source project, with renewed commitment to reviewing PRs, cutting releases, and carrying the work forward.
• 2024 – An incident on the Confluent repository reset the accumulated star count to 123.
• August 2023 – Peak star count of 548 stars on the Confluent repository (Wayback Machine snapshot).
• February 2023 – Left Confluent. The Confluent repository entered a quieter period.
• 2020-2023 – Developed and maintained the project at Confluent as Confluent Labs Parallel Consumer Client. Gathered a community, gave the Kafka Summit Europe 2021 talk, reached 479 stars by early 2023.
• May 2020 – Project started. First commits go up as a draft “generic async message processor” for Kafka.

Repositories

The project’s active home – carrying it forward as a fully open source effort with renewed commitment to community maintenance:

Original location at Confluent: