groupfrom the Meetup events Quickstart example. Note this object has two child fields, and the child
groupis a nested array with the element of object type.
groupwith complex type to JSON, you can add the following config to table config.
transformConfigstransforms the object
groupto a JSON string
group_json, which then creates the JSON indexing with config
jsonIndexColumns. To read the full spec, please check out this file. Also note that
groupis a reserved keyword in SQL, and that's why it's quoted in the
maxLengthof the field
group_jsonon the schema, because by default, a string column has a limited length. For example,
group. For the deatils about the supported JSON function, please check out this guide).
JSON_EXTRACT_SCALARis needed to extract the values in the GROUP BY and ORDER BY clauses, which invokes the function evaluation.
ingestionConfig. For example:
complexTypeConfig, all the map objects will be flattened to direct fields automatically. And with
unnestFields, a record with the nested collection will unnest into multiple records. For instance, the example in the beginning will transform into two rows with this configuration example.
groupis flattened to field
group.group_id. The default value of the delimiter is
., you can choose other delimiter by changing the configuration
complexTypeConfig. This flattening rule also apllies on the maps in the collections to be unnested.
groupis unnested into the top-level, and convert the output to a collection of two rows. Note the handling of the nested field within
group_topics, and the eventual top-level field of
group.group_topics.urlkey. All the collections to unnest shall be included in configuration
fieldsToUnnest, the ingestion by default will serialize them into JSON string, except for the array of primitive values, which will be ingested as multi-value column by default. The behavior is defined in config
collectionNotUnnestedToJsonwith default value to
NON_PRIMITIVE. Other behaviors include (1)
ALL, which aslo convert the array of primitive values to JSON string; (2)
NONE, this does not do conversion, but leave it to the users to use transform functions for handling.
.is a reserved character in SQL, so you need to quote the flattened column.
fieldsToUnnestsimilar to the ones in
complexTypeConfig. And this will simulate the complex-type handling rules on the Avro schema and output the Pinot schema in the file specified in