Ingestion Aggregations

Many data analytics use-cases only need aggregated data. For example, data used in charts can be aggregated down to one row per time bucket per dimension combination.
Doing this results in much less storage and better query performance. Configuring this for a table is done via the Aggregation Config in the table config.
Aggregation Config
The aggregation config controls the aggregations that happen during realtime data ingestion. Offline aggregations must be handled separately.
Below is a description of the config, which is defined in the ingestion config of the table config.
1
{
2
  "tableConfig": {
3
    "tableName": "...",
4
    "ingestionConfig": {
5
      "aggregationConfigs": [{
6
        "columeName": "aggregatedFieldName",
7
        "aggregationFunction": "<aggregationFunction>(<originalFieldName>)"
8
      }]
9
    }
10
  }
11
}
Copied!
Example Scenario
Here is an example of sales data, where only the daily sales aggregates per product are needed.
Example Input Data
1
{"customerID":205,"product_name": "car","price":"1500.00","timestamp":1571900400000}
2
{"customerID":206,"product_name": "truck","price":"2200.00","timestamp":1571900400000}
3
{"customerID":207,"product_name": "car","price":"1300.00","timestamp":1571900400000}
4
{"customerID":208,"product_name": "truck","price":"700.00","timestamp":1572418800000}
5
{"customerID":209,"product_name": "car","price":"1100.00","timestamp":1572505200000}
6
{"customerID":210,"product_name": "car","price":"2100.00","timestamp":1572505200000}
7
{"customerID":211,"product_name": "truck","price":"800.00","timestamp":1572678000000}
8
{"customerID":212,"product_name": "car","price":"800.00","timestamp":1572678000000}
9
{"customerID":213,"product_name": "car","price":"1900.00","timestamp":1572678000000}
10
{"customerID":214,"product_name": "car","price":"1000.00","timestamp":1572678000000}
Copied!
Schema
Note that the schema only reflects the final table structure.
1
{
2
  "schemaName": "daily_sales_schema",
3
  "dimensionFieldSpecs": [
4
    {
5
      "name": "product_name",
6
      "dataType": "STRING"
7
    }
8
  ],
9
  "metricSpecs": [
10
    {
11
      "name": "sales_count",
12
      "dataType": "LONG"
13
    },
14
    {
15
      "name": "total_sales",
16
      "dataType": "DOUBLE"
17
    }
18
  ],
19
  "dateTimeFieldSpecs": [
20
    {
21
      "name": "daysSinceEpoch",
22
      "dataType": "LONG",
23
      "format": "1:MILLISECONDS:EPOCH",
24
      "granularity": "1:MILLISECONDS"
25
    }
26
  ]
27
}
Copied!
Table Config
1
{
2
  "tableName": "daily_sales",
3
  "ingestionConfig": {
4
    "transformConfigs": [
5
      {
6
        "columnName": "daysSinceEpoch",
7
        "transformFunction": "toEpochDays(timestamp)"
8
      }
9
    ],
10
    "aggregationConfigs": [
11
      {
12
        "columnName": "total_sales",
13
        "aggregationFunction": "SUM(price)"
14
      },
15
      {
16
        "columnName": "sales_count", 
17
        "aggregationFunction": "COUNT(*)"
18
      }
19
    ]
20
  }
21
}
Copied!
Example Final Table
product_name
sales_count
total_sales
daysSinceEpoch
car
2
2800.00
18193
truck
1
2200.00
18193
truck
1
700.00
18199
car
2
3300.00
18200
truck
1
800.00
18202
car
3
3700.00
18202
Requirements
The following are required for ingestion aggregation to work:
​Stream ingestion type must be lowLevel.
All metrics must have aggregation configs.
All metrics must be noDictionaryColumns.
Allowed Aggregation Functions
function name
notes
MAX
​
MIN
​
SUM
​
COUNT
Specify as COUNT(*)
DISTINCTCOUNTHLL
Not available yet, but coming soon
Frequently Asked Questions
Why not use a Startree?
Startrees can only be added to realtime segments after the segments has sealed, and creating startrees is CPU-intensive. Ingestion Aggregation works for consuming segments and uses no additional CPU.
Startrees take additional memory to store, while ingestion aggregation stores less data than the original dataset.
When to not use ingestion aggregation?
If the original rows in non-aggregated form are needed, then ingestion-aggregation cannot be used.
I already use the aggregateMetrics setting?
The aggregateMetrics works the same as Ingestion Aggregation, but only allows for the SUM function.
The current changes are backward compatible, so no need to change your table config unless you need a different aggregation function.
Does this config work for offline data?
Ingestion Aggregation only works for realtime ingestion. For offline data, the offline process needs to generate the aggregates separately.
Why do all metrics need to be aggregated?
If a metric isn't aggregated then it will result in more than one row per unique set of dimensions.

Data Ingestion Overview

Ingestion Transformations

Last modified 15d ago

Copy link

Contents

Allowed Aggregation Functions

Frequently Asked Questions

Why not use a Startree?

When to not use ingestion aggregation?

I already use the aggregateMetrics setting?

Does this config work for offline data?

Why do all metrics need to be aggregated?