arrow-left

All pages
gitbookPowered by GitBook
1 of 14

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Stream ingestion

This page contains guides related to importing data from Apache Kafka using stream ingestion.

Data import

This section is an overview of the various options for importing data into Pinot.

There are multiple options for importing data into Pinot. These guides are ready-made examples that show you step-by-step instructions for importing records into Pinot, supported by our plugin architecture. These guides are meant to get you up and running with imported data as quick as possible. Pinot supports multiple file input formats without needing to change anything other than the file name. Each example imports a ready-made dataset so you can see how things work without needing to bring your own dataset.

hashtag
Pinot File Systems

These guides will show you how to import data from a supported file system.

Import from ADLS (Azure)chevron-rightImport from GCPchevron-rightImport from HDFSchevron-right

hashtag
Pinot Input Formats

These guides will show you how to import data from a Pinot supported input format.

hashtag
Pinot Stream Ingestion

This guide will show you how to import data using stream ingestion from Apache Kafka topics.

Import from CSVchevron-right
Import from JSONchevron-right
Import from Avrochevron-right
Import from Parquetchevron-right
Import from Thriftchevron-right
Import from Kafkachevron-right

Input formats

This section contains a collection of guides that will show you how to import data from a Pinot supported input format.

Import from GCP

This guide shows you how to import data from GCP (Google Cloud Platform).

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

Import from Kafka

This guide shows you how to ingest a stream of records from an Apache Kafka topic into a Pinot table.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

Import from Parquet

This guide shows you how to import records from a Parquet file into Pinot.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture
VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture
VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture

Import from HDFS

This guide shows you how to import data from HDFS.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

Import from Thrift

This guide shows you how to import records into Pinot using a Thrift file.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture
We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.
circle-info

While you're waiting, please take a look at our section on plugin architecture.

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh

Import from ORC

This guide shows you how to import records into Pinot using ORC file format.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

Import from Avro

This guide shows you how to import records into Pinot using Avro file format.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

Import from CSV

This guide shows you how to import a CSV file of records into Pinot.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

File systems

This section contains a collection of short guides to show you how to import from a Pinot supported file system.

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture
VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture
We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.
circle-info

While you're waiting, please take a look at our section on plugin architecture.

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh

Import from ADLS (Azure)

This guide shows you how to import data from files stored in Azure Data Lake Storage (ADLS)

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.

circle-info

While you're waiting, please take a look at our section on .

Import from JSON

This guide shows you how to import a JSON file of records into Pinot.

hashtag
Coming soon

circle-exclamation

This guide is a work in progress.

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh
plugin architecture
We're actively working on improving our documentation. This doc will be available very soon. Please check back in a day or two for more details.
circle-info

While you're waiting, please take a look at our section on plugin architecture.

VERSION=0.3.0
wget https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz
tar vxf apache-pinot-incubating-*-bin.tar.gz
cd apache-pinot-incubating-*-bin
bin/quick-start-batch.sh