Broker GRPC API

Apart from HTTP based query endpoint, Pinot also supports gRPC endpoint in broker.

Enable GRPC query entrypoint in broker

Add gRPC port config in pinot broker to enable the BrokerGrpcServer:

pinot.broker.grpc.port=8010

if you want to enable TLS, then use below configs:

pinot.broker.grpc.tls.enabled=true
pinot.broker.grpc.tls.port=8020

// Server side TLS is used. The client does not present a certificate.
// The server only verifies the client’s connection via its own certificate and doesn’t validate the client’s identity via TLS.
// Common in public APIs, where only the server needs to be trusted.
pinot.broker.grpctls.client.auth.enabled=false

// Config TLS keystore and truststore
pinot.broker.grpctls.keystore.path=/home/pinot/tls-store/keystore-internal.jks
pinot.broker.grpctls.keystore.password=changeit
pinot.broker.grpctls.keystore.type=JKS
pinot.broker.grpctls.truststore.path=/home/pinot/tls-store/truststore.jks
pinot.broker.grpctls.truststore.password=changeit
pinot.broker.grpctls.truststore.type=JKS

Broker GRPC Clients

Below are the examples of usage for pinot-java-client and pinot-jdbc-client .

Java Grpc Client

package org.apache.pinot.client.examples;

import java.io.IOException;
import java.util.Properties;
import org.apache.pinot.client.ConnectionFactory;
import org.apache.pinot.client.ResultSetGroup;
import org.apache.pinot.client.grpc.GrpcConnection;


public class PinotBrokerGrpcClientExample {

  private PinotBrokerGrpcClientExample() {
  }
  
  public static void main(String[] args)
      throws IOException {
    GrpcConnection grpcConnection = ConnectionFactory.fromControllerGrpc(new Properties(), "localhost:9000");
    ResultSetGroup resultSetGroup = grpcConnection.execute("SELECT * FROM airlineStats limit 1000");
    for (int i = 0; i < resultSetGroup.getResultSetCount(); i++) {
      org.apache.pinot.client.ResultSet resultSet = resultSetGroup.getResultSet(i);
      for (int rowId = 0; rowId < resultSet.getRowCount(); rowId++) {
        System.out.print("Row Id: " + rowId + "\t");
        for (int colId = 0; colId < resultSet.getColumnCount(); colId++) {
          System.out.print(resultSet.getString(rowId, colId) + "\t");
        }
        System.out.println();
      }
    }
    grpcConnection.close();
  }
}

JDBC Grpc Client

The main usage difference here is scheme is changed to pinotgrpc .

And there are two parameters to set per connection basis:

1. blockRowSize: the number of rows per block that grpc response will return, default is 10000.

2. compression: the compression algorithm over the wire, default is ZSTD, other options:

   1. LZ4_FAST: fast than LZ4_HIGH but not as high compression rate

   2. LZ4_HIGH

   3. ZSTD (Default)

   4. DEFLATE

   5. GZIP

   6. SNAPPY

   7. PASS_THROUGH/NONE: no compression, fast but could be large data transfer over the wire.

We did a very rough benchmark to compress 97k(9.2MB) rows with block size 10K.

CompressionType	Compression Ratio	Response Latency(ms)	Latency Ratio
LZ4_FAST	42.19%	477.6	103.02%
LZ4_HIGH	30.04%	977.0	210.74%
ZSTD	26.10%	485.4	104.70%
DEFLATE	25.64%	810.0	174.72%
GZIP	25.64%	811.8	175.11%
SNAPPY	42.56%	461.2	99.48%
NONE	100%	463.6	100%

We also tried for ZSTD the same data set with different block size:

BlockRowSize	Compression Ratio	Avg Execution Time(ms)
100	30.48%	539.6
1000	27.56%	478.3
10000	26.10%	485.4
100000	25.73%	561.2

Below is a sample code to use JDBC driver.

package org.apache.pinot.client.examples;

import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;


public class PinotBrokerGrpcJdbcClientExample {

  private PinotBrokerGrpcJdbcClientExample() {
  }

  public static void main(String[] args)
      throws IOException {
    try (Connection connection = DriverManager.getConnection("jdbc:pinotgrpc://localhost:9000?blockRowSize=100");
        Statement statement = connection.createStatement();
        ResultSet resultSet = statement.executeQuery("SELECT * FROM airlineStats limit 1000")) {
      // Print results
      ResultSetMetaData metaData = resultSet.getMetaData();
      int columnCount = metaData.getColumnCount();
      while (resultSet.next()) {
        System.out.print("Row Id: " + resultSet.getRow() + "\t");
        for (int i = 1; i <= columnCount; i++) {
          System.out.print(metaData.getColumnName(i) + ": " + resultSet.getString(i) + "\t");
        }
        System.out.println();
      }
    } catch (SQLException e) {
      e.printStackTrace();
    }
  }
}