LogoLogo
release-0.4.0
release-0.4.0
  • Introduction
  • Basics
    • Concepts
    • Architecture
    • Components
      • Cluster
      • Controller
      • Broker
      • Server
      • Minion
      • Tenant
      • Table
      • Schema
      • Segment
    • Getting started
      • Frequent questions
      • Running Pinot locally
      • Running Pinot in Docker
      • Running Pinot in Kubernetes
      • Public cloud examples
        • Running on Azure
        • Running on GCP
        • Running on AWS
      • Manual cluster setup
      • Batch import example
      • Stream ingestion example
    • Data import
      • Stream ingestion
        • Import from Kafka
      • File systems
        • Import from ADLS (Azure)
        • Import from HDFS
        • Import from GCP
      • Input formats
        • Import from CSV
        • Import from JSON
        • Import from Avro
        • Import from Parquet
        • Import from Thrift
        • Import from ORC
    • Feature guides
      • Pinot data explorer
      • Text search support
      • Indexing
    • Releases
      • 0.3.0
      • 0.2.0
      • 0.1.0
    • Recipes
      • GitHub Events Stream
  • For Users
    • Query
      • Pinot Query Language (PQL)
        • Unique Counting
    • API
      • Querying Pinot
        • Response Format
      • Pinot Rest Admin Interface
    • Clients
      • Java
      • Golang
  • For Developers
    • Basics
      • Extending Pinot
        • Writing Custom Aggregation Function
        • Pluggable Streams
        • Pluggable Storage
        • Record Reader
        • Segment Fetchers
      • Contribution Guidelines
      • Code Setup
      • Code Modules and Organization
      • Update Documentation
    • Advanced
      • Data Ingestion Overview
      • Advanced Pinot Setup
    • Tutorials
      • Pinot Architecture
      • Store Data
        • Batch Tables
        • Streaming Tables
      • Ingest Data
        • Batch
          • Creating Pinot Segments
          • Write your batch
          • HDFS
          • AWS S3
          • Azure Storage
          • Google Cloud Storage
        • Streaming
          • Creating Pinot Segments
          • Write your stream
          • Kafka
          • Azure EventHub
          • Amazon Kinesis
          • Google Pub/Sub
    • Design Documents
  • For Operators
    • Basics
      • Setup cluster
      • Setup table
      • Setup ingestion
      • Access Control
      • Monitoring
      • Tuning
        • Realtime
        • Routing
    • Tutorials
      • Build Docker Images
      • Running Pinot in Production
      • Kubernetes Deployment
      • Amazon EKS (Kafka)
      • Amazon MSK (Kafka)
      • Batch Data Ingestion In Practice
  • RESOURCES
    • Community
    • Blogs
    • Presentations
    • Videos
  • Integrations
    • ThirdEye
    • Superset
    • Presto
  • PLUGINS
    • Plugin Architecture
    • Pinot Input Format
    • Pinot File System
    • Pinot Batch Ingestion
    • Pinot Stream Ingestion
Powered by GitBook
On this page
  • Overview
  • Pinot
  • How to build a docker image
  • How to publish a docker image
  • Kubernetes Examples
  • Pinot Presto
  • How to build
  • How to push
  • Configuration
  • Volumes
  • Kubernetes Examples
  • Pinot Superset
  • How to build
  • How to push
  • Configuration
  • Volumes
  • Kubernetes Examples

Was this helpful?

Edit on Git
Export as PDF
  1. For Operators
  2. Tutorials

Build Docker Images

PreviousTutorialsNextRunning Pinot in Production

Last updated 4 years ago

Was this helpful?

Overview

The scripts to build Pinot related docker images is located at .

You can access those scripts by running below command to checkout Pinot repo:

git clone git@github.com:apache/incubator-pinot.git pinot
cd pinot/docker/images

You can find current supported 3 images in this directory:

  • Pinot: Pinot all-in-one distribution image

  • Pinot-Presto: Presto image with Presto-Pinot Connector built-in.

  • Pinot-Superset: Superset image with Pinot connector built-in.

Pinot

This is a docker image of .

How to build a docker image

There is a docker build script which will build a given Git repo/branch and tag the image.

Usage:

./docker-build.sh [Docker Tag] [Git Branch] [Pinot Git URL]

This script will check out Pinot Repo [Pinot Git URL] on branch [Git Branch] and build the docker image for that.

The docker image is tagged as [Docker Tag].

Docker Tag: Name and tag your docker image. Default is pinot:latest.

Git Branch: The Pinot branch to build. Default is master.

Pinot Git URL: The Pinot Git Repo to build, users can set it to their own fork. Please note that, the URL is https:// based, not git://. Default is the Apache Repo: https://github.com/apache/incubator-pinot.git.

  • Example of building and tagging a snapshot on your own fork:

./docker-build.sh pinot_fork:snapshot-5.2 snapshot-5.2 https://github.com/your_own_fork/pinot.git
  • Example of building a release version:

./docker-build.sh pinot:release-0.1.0 release-0.1.0 https://github.com/apache/incubator-pinot.git
  • Example of building current master branch as a snapshot:

./docker-build.sh apachepinot/pinot:0.3.0-SNAPSHOT master

How to publish a docker image

Script docker-push.sh publishes a given docker image to your docker registry.

In order to push to your own repo, the image needs to be explicitly tagged with the repo name.

./docker-push.sh apachepinot/pinot:latest
  • You can also tag a built image, then push it.

docker tag pinot:release-0.1.0 apachepinot/pinot:release-0.1.0
docker push apachepinot/pinot:release-0.1.0

Script docker-build-and-push.sh builds and publishes this docker image to your docker registry after build.

./docker-build-and-push.sh apachepinot/pinot:latest master https://github.com/apache/incubator-pinot.git

Kubernetes Examples

Pinot Presto

This docker build project is specialized for Pinot.

How to build

Usage:

./docker-build.sh [Docker Tag] [Git Branch] [Presto Git URL]

This script will check out Presto Repo [Presto Git URL] on branch [Git Branch] and build the docker image for that.

The docker image is tagged as [Docker Tag].

Docker Tag: Name and tag your docker image. Default is pinot-presto:latest.

Git Branch: The Presto branch to build. Default is master.

Presto Git URL: The Presto Git Repo to build, users can set it to their own fork. Please note that, the URL is https:// based, not git://. Default is the Apache Repo: https://github.com/prestodb/presto.git.

How to push

docker push apachepinot/pinot-presto:latest

Configuration

Volumes

The image defines two data volumes: one for mounting configuration into the container, and one for data.

The configuration volume is located alternatively at /home/presto/etc, which contains all the configuration and plugins.

The data volume is located at /home/presto/data.

Kubernetes Examples

Pinot Superset

How to build

Please modify file Makefile to change image and superset_version accordingly.

Below command will build docker image and tag it as superset_version and latest.

make latest

You can also build directly with docker build command by setting arguments:

docker build \
    --build-arg NODE_VERSION=latest \
    --build-arg PYTHON_VERSION=3.6 \
    --build-arg SUPERSET_VERSION=0.34.1 \
    --tag apachepinot/pinot-superset:0.34.1 \
    --target build .

How to push

make push

Configuration

Place this file in a local directory and mount this directory to /etc/superset inside the container. This location is included in the image's PYTHONPATH. Mounting this file to a different location is possible, but it will need to be in the PYTHONPATH.

Volumes

The image defines two data volumes: one for mounting configuration into the container, and one for data (logs, SQLite DBs, &c).

The configuration volume is located alternatively at /etc/superset or /home/superset; either is acceptable. Both of these directories are included in the PYTHONPATH of the image. Mount any configuration (specifically the superset_config.py file) here to have it read by the app on startup.

The data volume is located at /var/lib/superset and it is where you would mount your SQLite file (if you are using that as your backend), or a volume to collect any logs that are routed there. This location is used as the value of the SUPERSET_HOME environmental variable.

Kubernetes Examples

Example of publishing an image to dockerHub repo.

Example of building and publishing an image to dockerHub repo.

Please refer to for deployment examples.

Docker image for with Pinot integration.

Follow the provided by Presto for writing your own configuration files under etc directory.

Please refer to as k8s deployment example.

Docker image for with Pinot integration.

This docker build project is based on Project and specialized for Pinot.

Follow the provided by Apache Superset for writing your own superset_config.py.

Please refer to as k8s deployment example.

here
Apache Pinot
apachepinot/pinot
apachepinot/pinot
Kubernetes Quickstart
Presto
instructions
presto-coordinator.yaml
Superset
docker-superset
instructions
superset.yaml