Delta Tables Schema - Open Telemetry

tip

See Delte Egress Sink page to learn more about sending data to Delta Tables in general.

note

The Open Telemetry schema for Delta Egress is currently in preview and can be configured only via API (not Portal or CLI).

When using this schema, multiple tables is created at the Directory Path specified in the Egress Route configuration. Each table contains data from a respective OpenTelemetry signal type.

Logs: /logs
Metrics: /metrics
Traces: /traces

The incoming message's payload is expected to be in Protobuf format satisfying Open Telemetry protocol for each signal type:

Schema for Logs

Name	Type	Example	Description
`stream_group_name`	string	group-a	Name of the stream group the message was sent into.
`stream_name`	string	telemetry	Name of the stream the message was sent into.
`device_id`	string	robot-125	Id of the device that sent the message.
`batch_id`	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
`message_id`	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
`workspace_id`	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
`ingress_enqueued_date_time`	timestamp	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.
`otel_date`	date	2023-12-05	The UTC date extracted from from otel_time_unix_nano.
`otel_attributes`	map<string, string>	{"service": "example"}	Merged attributes from resource, scope and data point level.
`otel_time_unix_nano`	long	1724063701707354000	Time when the event occurred.
`otel_observed_time_unit_nano`	long	1724063701707316000	Time when the event was observed by the collection system.
`otel_severity_text`	string	WARN	The severity text (also known as log level). The original string representation as it is known at the source.
`otel_severity_number`	long	15	Numerical value of the severity, normalized to values described in Open Telemetry's Log Data Model.
`otel_body`	string	Event happened.	A value containing the body of the log record. Primitive values (string, int, double, bool) are serialized directly to their string representation. Bytes are encoded to base64. Array or KeyValueList values are serialized to JSON.

Spark SQL - interpretable schema

stream_group_name STRING,
stream_name STRING,
site_id STRING,
device_id STRING,
batch_id STRING,
message_id STRING,
workspace_id STRING,
ingress_enqueued_date_time TIMESTAMP,
otel_date DATE,
otel_attributes MAP<STRING, STRING>,
otel_time_unix_nano LONG,
otel_observed_time_unit_nano LONG,
otel_severity_text STRING,
otel_severity_number LONG,
otel_body STRING

Partition key columns

Table is partitioned by otel_data column.

Schema for Metrics

Name	Type	Example	Description
`stream_group_name`	string	group-a	Name of the stream group the message was sent into.
`stream_name`	string	telemetry	Name of the stream the message was sent into.
`device_id`	string	robot-125	Id of the device that sent the message.
`batch_id`	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
`message_id`	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
`workspace_id`	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
`ingress_enqueued_date_time`	timestamp	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.
`otel_date`	date	2023-12-05	The UTC date extracted from from otel_time_unix_nano.
`otel_attributes`	map<string, string>	{"service": "example"}	Merged attributes from resource, scope and record level.
`otel_time_unix_nano`	long	1724063701707354000	Moment corresponding to when the data point's aggregate value was captured.
`otel_start_time_unix_nano`	long	1724063701708323000	Indicates to consumers the start time for points with cumulative and delta aggregation temporality.
`otel_name`	string	system.filesystem.usage	Name of the metric.
`otel_unit`	string	s	Unit in which the metric value is reported. Follows the format described by https://unitsofmeasure.org/ucum.html.
`otel_instrument`	string	gauge	Type of the metric instrument: `gauge`, `sum`, `histogram`. Indicates what following columns are available.
`otel_gauge_value`	double	42.1	Gauge value.
`otel_gauge_flags`	long
`otel_sum_value`	double	42.1	Sum value.
`otel_sum_flags`	long
`otel_sum_is_monotonic`	bool	true	`true` if sum is monotonic, `false` otherwise.
`otel_sum_aggregation_temporality`	string	Cumulative	Describes if the aggregator reports delta changes since last report time, or cumulative changes since a fixed start time.
`otel_histogram_count`	long	42	Number of values in the population.
`otel_histogram_sum`	double	1252	Sum of the values in the population.
`otel_histogram_min`	double	2	Minimal value in the population.
`otel_histogram_max`	double	91	Maximal value in the population.
`otel_histogram_bucket_counts`	long[]	[20, 21, 38, 89]	Contains the count values of histogram for each bucket.
`otel_histogram_explicit_bounds`	double[]	[0.1, 0.2, 0.3]	Specifies buckets with explicitly defined bounds for values.
`otel_histogram_flags`	long
`otel_histogram_aggregation_temporality`	string	Cumulative	Describes if the aggregator reports delta changes since last report time, or cumulative changes since a fixed start time.

Spark SQL - interpretable schema

stream_group_name STRING,
stream_name STRING,
site_id STRING,
device_id STRING,
batch_id STRING,
message_id STRING,
workspace_id STRING,
ingress_enqueued_date_time TIMESTAMP,
otel_date DATE,
otel_attributes MAP<STRING, STRING>,
otel_time_unix_nano LONG,
otel_start_time_unix_nano LONG,
otel_name STRING,
otel_unit STRING,
otel_instrument STRING,
otel_gauge_value DOUBLE,
otel_gauge_flags LONG,
otel_sum_value DOUBLE,
otel_sum_flags LONG,
otel_sum_is_monotonic BOOLEAN,
otel_sum_aggregation_temporality STRING,
otel_histogram_count LONG,
otel_histogram_sum DOUBLE,
otel_histogram_min DOUBLE,
otel_histogram_max DOUBLE,
otel_histogram_bucket_counts ARRAY<LONG>,
otel_histogram_explicit_bounds ARRAY<DOUBLE>,
otel_histogram_flags LONG,
otel_histogram_aggregation_temporality STRING

Partition key columns

Table is partitioned by otel_data column.

Schema for Traces

Name	Type	Example	Description
`stream_group_name`	string	group-a	Name of the stream group the message was sent into.
`stream_name`	string	telemetry	Name of the stream the message was sent into.
`device_id`	string	robot-125	Id of the device that sent the message.
`batch_id`	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
`message_id`	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
`workspace_id`	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
`ingress_enqueued_date_time`	timestamp	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.
`otel_date`	date	2023-12-05	The UTC date extracted from from otel_start_time_unix_nano.
`otel_attributes`	map<string, string>	{"service": "example"}	Merged attributes from resource, scope and span level.
`otel_start_time_unix_nano`	long	1724063701707354000	Start time of the span. On the client side, this is the time kept by the local machine where the span execution starts. On the server side, this is the time when the server's application handler starts running
`otel_end_time_unix_nano`	long	1724063701708323000	End time of the span. On the client side, this is the time kept by the local machine where the span execution ends. On the server side, this is the time when the server application handler stops running
`otel_span_name`	string	get_account/{accountId}	A description of the span's operation.
`otel_span_kind`	string	Client	Distinguishes between spans generated in a particular context.
`otel_span_id`	string	7370616e2d69640b	A unique identifier for a span within a trace. The ID is an 8 bytes encoded to base-16.
`otel_parent_span_id`	string	7370616e2d69640a	Identifier of the parent span. The ID is 16 bytes encded to base-16.
`otel_trace_id`	string	74726163652d69642d746573742d3132	A unique identifier for a trace. The ID is 16 bytes encded to base-16.
`otel_trace_state`	string	example-state	Conveys information about request position in multiple distributed tracing graphs.
`otel_status_code`	string	Ok	Status code.
`otel_status_message`	string	example-message	A developer-facing human readable error message.
`otel_flags`	long
`otel_events`	string[]		An event can be thought of as a structured log message (or annotation) on a span, typically used to denote a meaningful, singular point in time during the span's duration. Please see details about its format below.
`otel_links`	string[]		Links is a collection of Links, which are references from this span to a span in the same or different trace. Please see details about its format below.

The items in otel_events and otel_links arrays are serialized to JSON strings with the format described by the following examples:

JSON body of events:

{
    "name": "operation_failed",
    "timeUnixNano": "1724063701708323000",
    "attributes": {"service": "example", "exception.message": "example", "exception.stacktrace": "example"},
}

JSON body of links:

{
    "traceId": "74726163652d69642d746573742d3132",
    "spanId": "7370616e2d69640b",
    "traceState": "example-state",
    "attributes": {"service": "example"},
    "flags": 0,
}

Spark SQL - interpretable schema

stream_group_name STRING,
stream_name STRING,
site_id STRING,
device_id STRING,
batch_id STRING,
message_id STRING,
workspace_id STRING,
ingress_enqueued_date_time TIMESTAMP,
otel_date DATE,
otel_attributes MAP<STRING, STRING>,
otel_start_time_unix_nano LONG,
otel_end_time_unix_nano LONG,
otel_span_name STRING,
otel_span_kind STRING,
otel_span_id STRING,
otel_parent_span_id STRING,
otel_trace_id STRING,
otel_trace_state STRING,
otel_status_code STRING,
otel_status_message STRING,
otel_flags LONG,
otel_events ARRAY<STRING>,
otel_links ARRAY<STRING>

Partition key columns

Table is partitioned by otel_data column.

Schema for Logs​

Schema for Metrics​

Schema for Traces​

Schema for Logs

Schema for Metrics

Schema for Traces