Continuous Java Flight Recorder (JFR)

Continuous JFR runbook for Pinot with dynamic cluster-level on/off control

This page is the runbook for running continuous Java Flight Recorder (JFR) in Pinot.

Pinot supports cluster-level runtime control through ContinuousJfrStarter, so operators can turn recording on/off or adjust settings without restarting Pinot processes.

What is Java Flight Recorder (JFR)?

Java Flight Recorder (JFR) is a profiling and diagnostics framework built into the JDK. It records events from the JVM and application (for example CPU, memory, allocation, GC, thread, and lock events) into .jfr files with low production overhead.

In Java 8, JFR was a commercial feature and older documentation may mention -XX:+UnlockCommercialFeatures -XX:+FlightRecorder. Since Java 11, JFR is part of OpenJDK and does not require commercial flags.

Official deployment model

Run one long-lived recording in each Pinot JVM process (Controller, Broker, Server, Minion) and control it with pinot.jfr.* cluster config.

Configure with cluster config

Key	Default	Description
pinot.jfr.enabled	false	Enables/disables continuous recording.
pinot.jfr.configuration	default	JFR settings profile (default, profile, or custom JFR config).
pinot.jfr.name	pinot-continuous	Recording name.
pinot.jfr.dumpOnExit	true	Dumps recording on JVM exit.
pinot.jfr.toDisk	true	Stores recording repository on disk.
pinot.jfr.maxSize	2GB	Max recording size when toDisk=true; supports human-readable values (for example 512MB, 2GB) or raw bytes.
pinot.jfr.maxAge	P7D	Max event age (ISO-8601 duration) when toDisk=true.
pinot.jfr.directory	system temp directory	JFR repository path, applied through JFR.configure repositorypath=....
pinot.jfr.dumpPath	unset	Default JFR dump path, applied through JFR.configure dumppath=....

Example:

pinot.jfr.enabled=true
pinot.jfr.configuration=default
pinot.jfr.name=pinot-continuous
pinot.jfr.dumpOnExit=true
pinot.jfr.toDisk=true
pinot.jfr.maxSize=2GB
pinot.jfr.maxAge=PT24H
pinot.jfr.directory=/var/log/pinot/jfr-repository
pinot.jfr.dumpPath=/var/log/pinot/jfr-dumps

Notes:

• configuration=default is low-overhead and production-safe.

• Use configuration=profile only during active investigations.

• maxAge and maxSize cap footprint and history.

• Configuration changes are applied dynamically; Pinot restarts the active recording in-process.

Behavior of ContinuousJfrStarter

• Starts/stops one recording per Pinot JVM based on pinot.jfr.enabled.

• Reacts to pinot.jfr.* config updates at runtime.

• Applies runtime JFR options via the DiagnosticCommand MBean (JFR.configure) for repository and dump paths.

• Starts/stops recordings via the DiagnosticCommand MBean (JFR.start and JFR.stop).

• If the DiagnosticCommand MBean is unavailable, Pinot logs a warning and skips JFR operations instead of failing startup.

• Does not rotate dump files, does not run a cleaner thread, and does not schedule periodic dumps.

• Uses standard JFR lifecycle controls (same model as JVM-native JFR), with Pinot cluster config as the control plane.

Operational checks

List JVMs:

jcmd -l

Inspect recordings:

jcmd <pid> JFR.check

Incident workflow

Capture a point-in-time dump without restarting the process:

jcmd <pid> JFR.dump name=pinot-continuous filename=/var/log/pinot/jfr/pinot-incident-$(date +%Y%m%d-%H%M%S).jfr

Take additional dumps as needed during the incident timeline.

Alternative startup via JVM options

If you prefer static startup-only configuration, you can configure JFR in JAVA_OPTS:

-XX:StartFlightRecording=name=pinot-continuous,settings=default,disk=true,maxage=24h,maxsize=2g,dumponexit=true

Use this only when dynamic cluster-level toggling is not required.

Handling large recordings

When a recording is too large to transfer or inspect as one file, split it:

jfr disassemble --output /tmp/jfr-chunks <file.jfr>

Share only relevant chunks for triage.

Retention and tuning

• Start with configuration=default.

• Increase maxAge for longer timeline retention.

• Increase maxSize for high event-volume workloads.

• Keep host-level cleanup policies for operator-created dump files.

• Use explicit timestamped names for ad hoc dump files.

Common pitfalls

• Assuming Pinot automatically rotates JFR dump files.

• Running without disk budget guardrails (maxAge and maxSize).

• Leaving configuration=profile enabled permanently.

Minimal operator checklist

• pinot.jfr.enabled=true applied in cluster config.

• pinot.jfr.* values validated for footprint (toDisk, maxSize, maxAge).

• jcmd <pid> JFR.check validated post-deploy.

• Incident dump command tested in non-prod.

• Retention and cleanup policy applied to operator-created dump files.

Related

• Monitoring — Metrics, Prometheus, and Grafana.

• Configuration Reference / Monitoring Metrics — Pinot metrics reference.

Opening and analyzing JFR files

• jfr summary <file.jfr> for high-level stats.

• jfr view <file.jfr> for aggregated views.

• jfr print <file.jfr> for detailed events (--json and --xml supported).

• Java Mission Control (JMC) for interactive analysis.

• jdk.jfr.consumer.RecordingFile for programmatic analysis.

Quick sanity check:

jfr summary /var/log/pinot/jfr/pinot-incident-20260310-120000.jfr

External references

• Package jdk.jfr (Java SE 21) — Overview of the JFR API: defining events, controlling Flight Recorder, and the jdk.jfr package.

• The jfr command — Command-line tool to view, print, and summarize .jfr files (JDK 21+).

• Using JDK Flight Recorder with Java Mission Control — Recording and inspecting flights with the JMC GUI; includes an overview of JFR and how to analyze recordings.

• JDK Mission Control (JMC) — Download — Standalone JMC build for opening and analyzing .jfr recordings.

• RecordingFile (jdk.jfr.consumer) — API for reading and parsing .jfr files programmatically.