githubEdit

Stream Connector Version Matrix

A version and packaging matrix for all stream ingestion connectors shipped with Apache Pinot.

This page provides a quick-reference matrix mapping each stream connector to its Maven module, artifact ID, underlying client library version, and consumer factory class.

Connector matrix

Stream
Connector Module
Maven Artifact
Client Library Version
Consumer Factory Class
Status

Apache Kafka 3.x

pinot-kafka-3.0

org.apache.pinot:pinot-kafka-3.0

kafka-clients 3.9.2

org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory

Default, included in binary distribution

Apache Kafka 4.x

pinot-kafka-4.0

org.apache.pinot:pinot-kafka-4.0

kafka-clients 4.1.1

org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory

Included in binary distribution

Amazon Kinesis

pinot-kinesis

org.apache.pinot:pinot-kinesis

AWS SDK 2.42.16 (software.amazon.awssdk:kinesis)

org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory

Included in binary distribution

Apache Pulsar

pinot-pulsar

org.apache.pinot:pinot-pulsar

pulsar-client 4.0.9

org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory

Optional, enable with -Dplugins.include=pinot-pulsar

circle-info

All version numbers above are from the Pinot master branch (1.5.0-SNAPSHOT). Released Pinot versions may ship slightly different client library versions. Check the pom.xml of the corresponding module in your Pinot release for the exact version.

Compatibility notes

Kafka 3.x connector (pinot-kafka-3.0)

  • Compatible with Kafka brokers version 2.x and above.

  • Uses the Scala-based Kafka library alongside kafka-clients.

  • This is the recommended connector for most deployments.

Kafka 4.x connector (pinot-kafka-4.0)

  • Requires Kafka brokers version 4.0 or above.

  • Uses the pure-Java kafka-clients library only (no Scala dependency).

  • Designed for KRaft-mode Kafka clusters that have removed ZooKeeper.

  • Uses Testcontainers for integration testing instead of the embedded Kafka server.

Amazon Kinesis connector (pinot-kinesis)

  • Uses AWS SDK v2 (software.amazon.awssdk).

  • Supports both key-based and IAM role-based authentication.

  • Included in the default Pinot distribution.

Apache Pulsar connector (pinot-pulsar)

  • Not included in the default binary distribution. Enable with -Dplugins.include=pinot-pulsar at startup, or add the plugin JAR to the plugins directory.

  • Uses the Apache Pulsar client library.

  • Supports token-based, OAuth2, and TLS authentication.

Removed connectors

Former Module
Removed In
Migration Path

pinot-kafka-0.9

Pre-1.0

Migrate to pinot-kafka-3.0

pinot-kafka-2.0

Pre-1.0

Migrate to pinot-kafka-3.0 (or pinot-kafka-4.0 for Kafka 4.x brokers)

To migrate, update stream.kafka.consumer.factory.class.name in your table config from the old class to the new one. No other stream config changes are required.

Last updated

Was this helpful?