Azure Data Lake Storage
This guide shows you how to import data from files stored in Azure Data Lake Storage (ADLS)
You can enable the Azure Data Lake Storage using the plugin pinot-adls
. In the controller or server, add the config -
By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify -Dplugins.include
, you need to put all the plugins you want to use, e.g. pinot-json
, pinot-avro
, pinot-kafka-2.0...
Azure Blob Storage provides the following options -
accountName
: Name of the azure account under which the storage is createdaccessKey
: access key required for the authenticationfileSystemName
- name of the filesystem to use i.e. container name (container name is similar to bucket name in S3)enableChecksum
- enable MD5 checksum for verification. Default isfalse
.
Each of these properties should be prefixed by pinot.[node].storage.factory.class.abfss.
where node
is either controller
or server
depending on the config
e.g.
Examples
Job spec
Controller config
Server config
Minion config
Last updated