Running on AWS

Search…

Running on AWS

This guide provides a quick start for running Pinot on Amazon Web Services (AWS).

This document provides the basic instruction to set up a Kubernetes Cluster on Amazon Elastic Kubernetes Service (Amazon EKS)​
1. Tooling Installation
1.1 Install Kubectl
Please follow this link (https://kubernetes.io/docs/tasks/tools/install-kubectl) to install kubectl.
For Mac User
1
brew install kubernetes-cli
Copied!
Please check kubectl version after installation.
1
kubectl version
Copied!
QuickStart scripts are tested under kubectl client version v1.16.3 and server version v1.13.12
1.2 Install Helm
Please follow this link (https://helm.sh/docs/using_helm/#installing-helm) to install helm.
For Mac User
1
brew install kubernetes-helm
Copied!
Please check helm version after installation.
1
helm version
Copied!
This QuickStart provides helm supports for helm v3.0.0 and v2.12.1. Please pick the script based on your helm version.
1.3 Install AWS CLI
​
Please follow this link (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html#install-tool-bundled) to install AWS CLI.
For Mac User
1
curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-macos.zip" -o "awscliv2.zip"
2
unzip awscliv2.zip
3
sudo ./aws/install
Copied!
1.4 Install Eksctl
Please follow this link (https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html#installing-eksctl) to install AWS CLI.
For Mac User
1
brew tap weaveworks/tap
2
brew install weaveworks/tap/eksctl
Copied!
2. (Optional) Login to your AWS account.
For first time AWS user, please register your account at https://aws.amazon.com/.
Once created the account, you can go to AWS Identity and Access Management (IAM) to create a user and create access keys under Security Credential tab. 
1
aws configure
Copied!
Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will override  AWS configuration stored in file ~/.aws/credentials
3. (Optional) Create a Kubernetes cluster(EKS) in AWS 
The script below will create a 1 node cluster named pinot-quickstart in us-west-2 with a t3.xlarge machine for demo purposes:
1
EKS_CLUSTER_NAME=pinot-quickstart
2
eksctl create cluster \
3
--name ${EKS_CLUSTER_NAME} \
4
--version 1.16 \
5
--region us-west-2 \
6
--nodegroup-name standard-workers \
7
--node-type t3.xlarge \
8
--nodes 1 \
9
--nodes-min 1 \
10
--nodes-max 1 \
11
--node-ami auto
Copied!
You can monitor the cluster status via this command:
1
EKS_CLUSTER_NAME=pinot-quickstart
2
aws eks describe-cluster --name ${EKS_CLUSTER_NAME}
Copied!
Once the cluster is in ACTIVE status, it's ready to be used.
4. Connect to an existing cluster
Simply run below command to get the credential for the cluster pinot-quickstart that you just created or your existing cluster.
1
EKS_CLUSTER_NAME=pinot-quickstart
2
aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME}
Copied!
To verify the connection, you can run:
1
kubectl get nodes
Copied!
5. Pinot Quickstart
Please follow this Kubernetes QuickStart to deploy your Pinot Demo.
6. Delete a Kubernetes Cluster
1
EKS_CLUSTER_NAME=pinot-quickstart
2
aws eks delete-cluster --name ${EKS_CLUSTER_NAME}
Copied!

Running on GCP

Hdfs as Deep Storage

Last modified 10mo ago

Copy link

Contents

1. Tooling Installation

2. (Optional) Login to your AWS account.

3. (Optional) Create a Kubernetes cluster(EKS) in AWS

4. Connect to an existing cluster

5. Pinot Quickstart

6. Delete a Kubernetes Cluster