Broker GRPC API

Apart from HTTP based query endpoint, Pinot also supports gRPC endpoint in broker.

Enable GRPC query entrypoint in broker

Add gRPC port config in pinot broker to enable the BrokerGrpcServer:

pinot.broker.grpc.port=8010

if you want to enable TLS, then use below configs:

pinot.broker.grpc.tls.enabled=true
pinot.broker.grpc.tls.port=8020

// Server side TLS is used. The client does not present a certificate.
// The server only verifies the client’s connection via its own certificate and doesn’t validate the client’s identity via TLS.
// Common in public APIs, where only the server needs to be trusted.
pinot.broker.grpctls.client.auth.enabled=false

// Config TLS keystore and truststore
pinot.broker.grpctls.keystore.path=/home/pinot/tls-store/keystore-internal.jks
pinot.broker.grpctls.keystore.password=changeit
pinot.broker.grpctls.keystore.type=JKS
pinot.broker.grpctls.truststore.path=/home/pinot/tls-store/truststore.jks
pinot.broker.grpctls.truststore.password=changeit
pinot.broker.grpctls.truststore.type=JKS

Broker GRPC Clients

Below are the examples of usage for pinot-java-client and pinot-jdbc-client .

Java Grpc Client

The main difference of the usage here is that ConnectionFactory will return a GrpcConnection instead of Connection.

Below is a sample code to use the java client: GrpcConnection:

package org.apache.pinot.client.examples;

import java.io.IOException;
import java.util.Properties;
import org.apache.pinot.client.ConnectionFactory;
import org.apache.pinot.client.ResultSetGroup;
import org.apache.pinot.client.grpc.GrpcConnection;


public class PinotBrokerGrpcClientExample {

  private PinotBrokerGrpcClientExample() {
  }
  
  public static void main(String[] args)
      throws IOException {
    Properties properties = new Properties();
    properties.put("encoding", "JSON");
    properties.put("compression", "ZSTD");
    properties.put("blockRowSize", "10000");
    GrpcConnection grpcConnection = ConnectionFactory.fromControllerGrpc(properties, "localhost:9000");
    // GrpcConnection grpcConnection = ConnectionFactory.fromZookeeperGrpc(properties, "localhost:2123/QuickStartCluster");
    // GrpcConnection grpcConnection = ConnectionFactory.fromHostListGrpc(properties, List.of("localhost:8010"));
    ResultSetGroup resultSetGroup = grpcConnection.execute("SELECT * FROM airlineStats limit 1000");
    for (int i = 0; i < resultSetGroup.getResultSetCount(); i++) {
      org.apache.pinot.client.ResultSet resultSet = resultSetGroup.getResultSet(i);
      for (int rowId = 0; rowId < resultSet.getRowCount(); rowId++) {
        System.out.print("Row Id: " + rowId + "\t");
        for (int colId = 0; colId < resultSet.getColumnCount(); colId++) {
          System.out.print(resultSet.getString(rowId, colId) + "\t");
        }
        System.out.println();
      }
    }
    grpcConnection.close();
  }
}

JDBC Grpc Client

The main usage difference here is scheme is changed to pinotgrpc.

Below is a sample code to use JDBC driver.

package org.apache.pinot.client.examples;

import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;


public class PinotBrokerGrpcJdbcClientExample {

  private PinotBrokerGrpcJdbcClientExample() {
  }

  public static void main(String[] args)
      throws IOException {
    try (Connection connection = DriverManager.getConnection("jdbc:pinotgrpc://localhost:9000?blockRowSize=10000&encoding=JSON&compression=ZSTD");
        Statement statement = connection.createStatement();
        ResultSet resultSet = statement.executeQuery("SELECT * FROM airlineStats limit 1000")) {
      // Print results
      ResultSetMetaData metaData = resultSet.getMetaData();
      int columnCount = metaData.getColumnCount();
      while (resultSet.next()) {
        System.out.print("Row Id: " + resultSet.getRow() + "\t");
        for (int i = 1; i <= columnCount; i++) {
          System.out.print(metaData.getColumnName(i) + ": " + resultSet.getString(i) + "\t");
        }
        System.out.println();
      }
    } catch (SQLException e) {
      e.printStackTrace();
    }
  }
}

Client Configurations

Below are the parameters to set per connection basis:

  1. blockRowSize: the number of rows per block that grpc response will return, default is 10000.

  2. compression: the compression algorithm over the wire, default is ZSTD, other options:

    1. LZ4_FAST: fast than LZ4_HIGH but not as high compression rate

    2. LZ4_HIGH

    3. ZSTD (Default)

    4. DEFLATE

    5. GZIP

    6. SNAPPY

    7. PASS_THROUGH/NONE: no compression, fast but could be large data transfer over the wire.

  3. encoding: how to do serialization and deserialization for ResultTable transport between BrokerGrpcServer and Grpc Client, default is JSON, other options:

    1. JSON(Default)

    2. ARROW

Benchmark

We did a simple benchmark of query:

SELECT * FROM airlineStats limit 1000

Will compress 97k (9.2MB) rows with block size 10K.

Compression

CompressionType
Compression Ratio
Response Latency(ms)
Latency Ratio

LZ4_FAST

42.19%

477.6

103.02%

LZ4_HIGH

30.04%

977.0

210.74%

ZSTD

26.10%

485.4

104.70%

DEFLATE

25.64%

810.0

174.72%

GZIP

25.64%

811.8

175.11%

SNAPPY

42.56%

461.2

99.48%

NONE

100%

463.6

100%

Block Size

We also tried for ZSTD the same data set with different block size:

BlockRowSize
Compression Ratio
Avg Latency(ms)

100

30.48%

539.6

1000

27.56%

478.3

10000

26.10%

485.4

100000

25.73%

561.2

Encoding

Tested with ZSTD compression for the same query but different encoding and block size:

BlockRowSize
Encoding
Compression
Avg Latency(ms)
Bytes
Compression Ratio

1000

JSON

ZSTD

473

2559246

27.56%

10000

JSON

ZSTD

473

2424082

26.10%

100000

JSON

ZSTD

594

2389717

25.73%

1000

ARROW

ZSTD

372

3054993

32.90%

10000

ARROW

ZSTD

367

2705245

29.13%

100000

ARROW

ZSTD

447

2809744

30.26%

Last updated

Was this helpful?