.parquetfiles can now be found in
/path/to/batch_inputdirectory. You can now upload this directory to S3 either using their UI or running the command
pinot-admin.shCLI for these purpose.
sparkand configured the appropriate runners for each of our steps. We also need a temporary
stagingDirfor our spark job. This directory is cleaned up after our job has executed.