Delta Tables Schema - Open Telemetry

tip

See Delte Egress Sink page to learn more about sending data to Delta Tables in general.

When using this schema, multiple tables is created at the Directory Path specified in the Egress Route configuration. Each table contains data from a respective OpenTelemetry signal type.

• Logs: /logs
• Metrics: /metrics
• Traces: /traces

The incoming message's payload is expected to be in Protobuf format satisfying Open Telemetry protocol for each signal type:

• Logs: ExportLogsServiceRequest
• Metrics: ExportMetricsServiceRequest
• Traces: ExportTraceServiceRequest

Schema for Logs

Name	Type	Example	Description
stream_group_name	string	group-a	Name of the stream group the message was sent into.
stream_name	string	telemetry	Name of the stream the message was sent into.
device_id	string	robot-125	Id of the device that sent the message.
batch_id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
message_id	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
workspace_id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
ingress_enqueued_date_time	timestamp	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.
otel_date	date	2023-12-05	The UTC date extracted from from otel_time_unix_nano.
otel_attributes	map<string, string>	{"service": "example"}	Merged attributes from resource, scope and data point level.
otel_time_unix_nano	long	1724063701707354000	Time when the event occurred.
otel_observed_time_unit_nano	long	1724063701707316000	Time when the event was observed by the collection system.
otel_severity_text	string	WARN	The severity text (also known as log level). The original string representation as it is known at the source.
otel_severity_number	long	15	Numerical value of the severity, normalized to values described in Open Telemetry's Log Data Model.
otel_body	string	Event happened.	A value containing the body of the log record. Primitive values (string, int, double, bool) are serialized directly to their string representation. Bytes are encoded to base64. Array or KeyValueList values are serialized to JSON.

Spark SQL - interpretable schema

stream_group_name STRING,
stream_name STRING,
site_id STRING,
device_id STRING,
batch_id STRING,
message_id STRING,
workspace_id STRING,
ingress_enqueued_date_time TIMESTAMP,
otel_date DATE,
otel_attributes MAP<STRING, STRING>,
otel_time_unix_nano LONG,
otel_observed_time_unit_nano LONG,
otel_severity_text STRING,
otel_severity_number LONG,
otel_body STRING

Partition key columns

Table is partitioned by otel_date column. The value for this column is extracted from the LogRecord.time_unix_nano field of the OTEL protobuf message.

Schema for Metrics

Name	Type	Example	Description
stream_group_name	string	group-a	Name of the stream group the message was sent into.
stream_name	string	telemetry	Name of the stream the message was sent into.
device_id	string	robot-125	Id of the device that sent the message.
batch_id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
message_id	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
workspace_id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
ingress_enqueued_date_time	timestamp	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.
otel_date	date	2023-12-05	The UTC date extracted from from otel_time_unix_nano.
otel_attributes	map<string, string>	{"service": "example"}	Merged attributes from resource, scope and record level.
otel_time_unix_nano	long	1724063701707354000	Moment corresponding to when the data point's aggregate value was captured.
otel_start_time_unix_nano	long	1724063701708323000	Indicates to consumers the start time for points with cumulative and delta aggregation temporality.
otel_name	string	system.filesystem.usage	Name of the metric.
otel_unit	string	s	Unit in which the metric value is reported. Follows the format described by https://unitsofmeasure.org/ucum.html.
otel_instrument	string	gauge	Type of the metric instrument: gauge, sum, histogram. Indicates what following columns are available.
otel_gauge_value	double	42.1	Gauge value.
otel_gauge_flags	long
otel_sum_value	double	42.1	Sum value.
otel_sum_flags	long
otel_sum_is_monotonic	bool	true	true if sum is monotonic, false otherwise.
otel_sum_aggregation_temporality	string	Cumulative	Describes if the aggregator reports delta changes since last report time, or cumulative changes since a fixed start time.
otel_histogram_count	long	42	Number of values in the population.
otel_histogram_sum	double	1252	Sum of the values in the population.
otel_histogram_min	double	2	Minimal value in the population.
otel_histogram_max	double	91	Maximal value in the population.
otel_histogram_bucket_counts	long[]	[20, 21, 38, 89]	Contains the count values of histogram for each bucket.
otel_histogram_explicit_bounds	double[]	[0.1, 0.2, 0.3]	Specifies buckets with explicitly defined bounds for values.
otel_histogram_flags	long
otel_histogram_aggregation_temporality	string	Cumulative	Describes if the aggregator reports delta changes since last report time, or cumulative changes since a fixed start time.

Spark SQL - interpretable schema

stream_group_name STRING,
stream_name STRING,
site_id STRING,
device_id STRING,
batch_id STRING,
message_id STRING,
workspace_id STRING,
ingress_enqueued_date_time TIMESTAMP,
otel_date DATE,
otel_attributes MAP<STRING, STRING>,
otel_time_unix_nano LONG,
otel_start_time_unix_nano LONG,
otel_name STRING,
otel_unit STRING,
otel_instrument STRING,
otel_gauge_value DOUBLE,
otel_gauge_flags LONG,
otel_sum_value DOUBLE,
otel_sum_flags LONG,
otel_sum_is_monotonic BOOLEAN,
otel_sum_aggregation_temporality STRING,
otel_histogram_count LONG,
otel_histogram_sum DOUBLE,
otel_histogram_min DOUBLE,
otel_histogram_max DOUBLE,
otel_histogram_bucket_counts ARRAY<LONG>,
otel_histogram_explicit_bounds ARRAY<DOUBLE>,
otel_histogram_flags LONG,
otel_histogram_aggregation_temporality STRING

Partition key columns

Table is partitioned by otel_date column. The value for this column is extracted from the NumberDataPoint.time_unix_nano field or HistogramDataPoint.time_unix_nano field of the OTEL protobuf message.

Schema for Traces

Name	Type	Example	Description
stream_group_name	string	group-a	Name of the stream group the message was sent into.
stream_name	string	telemetry	Name of the stream the message was sent into.
device_id	string	robot-125	Id of the device that sent the message.
batch_id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
message_id	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
workspace_id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
ingress_enqueued_date_time	timestamp	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.
otel_date	date	2023-12-05	The UTC date extracted from from otel_start_time_unix_nano.
otel_attributes	map<string, string>	{"service": "example"}	Merged attributes from resource, scope and span level.
otel_start_time_unix_nano	long	1724063701707354000	Start time of the span. On the client side, this is the time kept by the local machine where the span execution starts. On the server side, this is the time when the server's application handler starts running
otel_end_time_unix_nano	long	1724063701708323000	End time of the span. On the client side, this is the time kept by the local machine where the span execution ends. On the server side, this is the time when the server application handler stops running
otel_span_name	string	get_account/{accountId}	A description of the span's operation.
otel_span_kind	string	Client	Distinguishes between spans generated in a particular context.
otel_span_id	string	7370616e2d69640b	A unique identifier for a span within a trace. The ID is an 8 bytes encoded to base-16.
otel_parent_span_id	string	7370616e2d69640a	Identifier of the parent span. The ID is 16 bytes encded to base-16.
otel_trace_id	string	74726163652d69642d746573742d3132	A unique identifier for a trace. The ID is 16 bytes encded to base-16.
otel_trace_state	string	example-state	Conveys information about request position in multiple distributed tracing graphs.
otel_status_code	string	Ok	Status code.
otel_status_message	string	example-message	A developer-facing human readable error message.
otel_flags	long
otel_events	string[]		An event can be thought of as a structured log message (or annotation) on a span, typically used to denote a meaningful, singular point in time during the span's duration. Please see details about its format below.
otel_links	string[]		Links is a collection of Links, which are references from this span to a span in the same or different trace. Please see details about its format below.

The items in otel_events and otel_links arrays are serialized to JSON strings with the format described by the following examples:

JSON body of events:

{
    "name": "operation_failed",
    "timeUnixNano": "1724063701708323000",
    "attributes": {"service": "example", "exception.message": "example", "exception.stacktrace": "example"},
}

JSON body of links:

{
    "traceId": "74726163652d69642d746573742d3132",
    "spanId": "7370616e2d69640b",
    "traceState": "example-state",
    "attributes": {"service": "example"},
    "flags": 0,
}

Spark SQL - interpretable schema

stream_group_name STRING,
stream_name STRING,
site_id STRING,
device_id STRING,
batch_id STRING,
message_id STRING,
workspace_id STRING,
ingress_enqueued_date_time TIMESTAMP,
otel_date DATE,
otel_attributes MAP<STRING, STRING>,
otel_start_time_unix_nano LONG,
otel_end_time_unix_nano LONG,
otel_span_name STRING,
otel_span_kind STRING,
otel_span_id STRING,
otel_parent_span_id STRING,
otel_trace_id STRING,
otel_trace_state STRING,
otel_status_code STRING,
otel_status_message STRING,
otel_flags LONG,
otel_events ARRAY<STRING>,
otel_links ARRAY<STRING>

Partition key columns

Table is partitioned by otel_date column. The value for this column is extracted from the Span.start_time_unix_nano field of the OTEL protobuf message.