Event Streaming for Observability (OpenTelemetry)

Stream audit logs and execution traces to your own infrastructure for compliance, analytics, and observability.

Overview

Relevance AI delivers events in OpenTelemetry (OTEL) format - an open, vendor-neutral standard for telemetry data. This provides several advantages:

Interoperability: OTEL is supported by most major observability platforms, making it easy to route data to your existing tools
Future-proof: As an industry standard backed by CNCF, OTEL ensures your data pipelines won’t be locked into proprietary formats
Rich semantics: Built-in support for traces, logs, and metrics with standardized attribute naming conventions
Correlation: Trace IDs link related events across agent invocations, LLM calls, and workforce executions

You don’t need to run an OTEL collector to use this feature. Events are delivered directly to your S3 bucket where you can:

Query them directly using Athena, BigQuery, or similar tools
Ingest into your data lake (Snowflake, Databricks, etc.)
Forward to any OTEL-compatible backend for visualization and alerting

Enterprise customers can enable PII redaction to automatically protect sensitive information in logs. Contact your Account Manager to learn more.

S3 is currently the only supported destination. Support for direct OTEL collector endpoints is on our roadmap.

Setup

Prerequisites

AWS account with permissions to create S3 buckets and bucket policies
Relevance AI Enterprise plan

1. Create an S3 Bucket

Create a bucket in the same AWS region as your Relevance data:

Relevance Region	AWS Region
Australia	`ap-southeast-2` (Sydney)
Europe	`eu-west-2` (London)
US	`us-east-1` (N. Virginia)

2. Configure Bucket Policy

Add this policy to allow Relevance to write events to your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowRelevanceEventConsumer",
      "Effect": "Allow",
      "Principal": {
        "AWS": "RELEVANCE_EVENT_CONSUMER_ROLE_ARN"
      },
      "Action": ["s3:PutObject", "s3:GetBucketLocation"],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    }
  ]
}

Replace:

YOUR_BUCKET_NAME: Your S3 bucket name
RELEVANCE_EVENT_CONSUMER_ROLE_ARN: Contact your Relevance team for the region-specific IAM role ARN

3. Provide Configuration to Relevance

Send your Account Manager or support team:

Bucket Name: Your S3 bucket name
Region: AWS region of the bucket
Prefix: S3 prefix for events (e.g., relevance-events/)

PII Redaction (Enterprise Feature)

PII (Personally Identifiable Information) redaction is an org-level feature that automatically scrubs sensitive information like email addresses, phone numbers, credit card numbers, and names before your event data leaves the platform and is delivered to your S3 destination. PII redaction applies at the point of data export, specifically when telemetry and audit logs are being written to your S3 bucket. It does not apply to live agent conversations in real-time or data stored internally on Relevance AI’s side. Think of it as a “scrub before delivery” mechanism for your downstream data pipeline.

PII redaction is a contractual Enterprise feature. Contact your Account Manager to enable this capability for your organization.

What gets redacted

There are two layers of redaction: 1. Structured fields (always scrubbed automatically) The following fields are redacted from exported data regardless of whether PII scrubbing is explicitly enabled:

User email addresses
IP addresses
Device info

2. Unstructured text (scrubbed when PII redaction is enabled for your org) When PII redaction is enabled, the following fields are scanned and scrubbed using a Presidio-powered ML engine:

LLM input messages (what you send to an agent/tool)
LLM output messages (what the agent responds)
System instructions

Supported PII entities

The following entity types can be detected. By default, all supported entities are targeted. You can optionally limit detection to specific entity types via configuration.

EMAIL_ADDRESS

PHONE_NUMBER

PERSON

CREDIT_CARD

US_SSN

IP_ADDRESS

IBAN_CODE

URL

DATE_TIME

LOCATION

Redaction actions

When PII is detected, the following actions can be applied:

Action	Description
`replace`	Substitutes the detected value with a placeholder like `<EMAIL_ADDRESS>`. This is the default.
`mask`	Replaces the detected value with `****`
`redact`	Removes the detected value entirely
`hash`	Hashes the detected value (one-way)

Prerequisites

Before configuring PII redaction, ensure:

OTEL log shipping is enabled for your organization
The jsonl_to_otel_json transformation is applied
Your Enterprise plan includes PII redaction (contact your Account Manager)

Configuration

Field	Type	Required	Description
`enabled`	boolean	Yes	Enable or disable PII redaction
`action`	string	Yes	Action to take on detected PII. Options: `"replace"`, `"redact"`, `"mask"`, or `"hash"`
`entities`	string[]	No	Specific PII entity types to target. If omitted, all supported entities are used
`target_fields`	string[]	No	Specific data fields to scan for PII
`score_threshold`	float	No	Minimum confidence level for PII detection (0.0–1.0). Higher values require more confidence

There is currently no self-serve UI to toggle PII redaction. It is configured at the organization level by the Relevance AI team. Contact your Account Manager to have it enabled.

Delivery Format

Format: Gzipped JSON following the OpenTelemetry JSON specification. File path pattern:

{prefix}/customer-otel-{type}-formatted/org_id={org_id}/dt={YYYY-MM-DD}/year={YYYY}/month={MM}/day={DD}/hour={HH}/minute={MM}/{type}_{org_id}_{timestamp}_{uuid}.json.gz

Event types:

logs - Audit logs for administrative and lifecycle events
traces - Execution traces for agents, workforces, and LLM completions

Logs

Audit logs provide a complete record of activity across your organization for security monitoring, compliance reporting, and operational visibility. Events capture who did what, when, and from where. Enterprise customers can enable PII redaction to automatically protect sensitive information in logs.

We’re actively expanding our event coverage. If there are specific operations or attributes you’d like to see, let your Account Manager know.

Structure

{
  "resourceLogs": [{
    "scopeLogs": [{
      "logRecords": [...]
    }]
  }]
}

Base Attributes

Included on all log records:

Attribute	Type	Required	Description
`relevance_ai.organization_id`	string	Yes	Organization ID
`relevance_ai.project_id`	string	Yes	Project ID
`relevance_ai.user_id`	string	No	User ID that performed the action
`relevance_ai.user_email`	string	No	Email of the user
`relevance_ai.user_type`	string	No	Type of user (`user`, `api_key`, etc.)
`relevance_ai.ip_address`	string	No	IP address of the request
`relevance_ai.device_info`	string	No	User agent / device information

Log Record Properties

Property	Type	Description
`timeUnixNano`	int64	Timestamp in nanoseconds since Unix epoch
`severityNumber`	int	OTEL severity level (9 = Info)
`severityText`	string	`"Info"`
`body.stringValue`	string	The event name
`attributes`	array	Key-value pairs with base + event attributes

Supported Events

Agent Events

agent_created - Emitted when a new agent is created (from scratch, cloned, or duplicated).

Attribute	Type	Required	Description
`relevance_ai.event.agent_id`	string	Yes	ID of the newly created agent
`relevance_ai.event.cloned_from_agent_id`	string	No	Source agent ID if cloned
`relevance_ai.event.cloned_from_region`	string	No	Source region if cloned
`relevance_ai.event.cloned_from_project_id`	string	No	Source project if cloned

agent_updated - Emitted when an agent’s configuration is updated directly (outside draft/publish workflow).

Attribute	Type	Required	Description
`relevance_ai.event.agent_id`	string	Yes	ID of the updated agent

agent_deleted - Emitted when an agent is permanently deleted.

Attribute	Type	Required	Description
`relevance_ai.event.agent_id`	string	Yes	ID of the deleted agent

agent_draft_saved - Emitted when work-in-progress changes are saved to an agent.

Attribute	Type	Required	Description
`relevance_ai.event.agent_id`	string	Yes	ID of the agent being edited
`relevance_ai.event.version_id`	string	Yes	ID of the draft version saved

agent_published - Emitted when agent changes are published to make them live.

Attribute	Type	Required	Description
`relevance_ai.event.agent_id`	string	Yes	ID of the agent being published
`relevance_ai.event.version_id`	string	Yes	ID of the version now live

Tool Events

tool_created - Emitted when a new tool is created (from scratch or cloned).

Attribute	Type	Required	Description
`relevance_ai.event.tool_id`	string	Yes	ID of the newly created tool
`relevance_ai.event.cloned_from_tool_id`	string	No	Source tool ID if cloned
`relevance_ai.event.cloned_from_region`	string	No	Source region if cloned
`relevance_ai.event.cloned_from_project_id`	string	No	Source project if cloned

tool_deleted - Emitted when a tool is permanently deleted.

Attribute	Type	Required	Description
`relevance_ai.event.tool_id`	string	Yes	ID of the deleted tool

tool_draft_saved - Emitted when work-in-progress changes are saved to a tool.

Attribute	Type	Required	Description
`relevance_ai.event.tool_id`	string	Yes	ID of the tool being edited
`relevance_ai.event.version_id`	string	No	ID of the draft version saved

tool_published - Emitted when tool changes are published to make them live.

Attribute	Type	Required	Description
`relevance_ai.event.tool_id`	string	Yes	ID of the tool being published
`relevance_ai.event.version_id`	string	Yes	ID of the version now live

Workforce Events

workforce_created - Emitted when a new workforce is created (from scratch, cloned, or duplicated).

Attribute	Type	Required	Description
`relevance_ai.event.workforce_id`	string	Yes	ID of the newly created workforce
`relevance_ai.event.cloned_from_workforce_id`	string	No	Source workforce ID if cloned
`relevance_ai.event.cloned_from_region`	string	No	Source region if cloned
`relevance_ai.event.cloned_from_project_id`	string	No	Source project if cloned

workforce_deleted - Emitted when a workforce is permanently deleted.

Attribute	Type	Required	Description
`relevance_ai.event.workforce_id`	string	Yes	ID of the deleted workforce

workforce_draft_saved - Emitted when work-in-progress changes are saved to a workforce.

Attribute	Type	Required	Description
`relevance_ai.event.workforce_id`	string	Yes	ID of the workforce being edited
`relevance_ai.event.version_id`	string	No	ID of the draft version saved

workforce_published - Emitted when workforce changes are published to make them live.

Attribute	Type	Required	Description
`relevance_ai.event.workforce_id`	string	Yes	ID of the workforce being published
`relevance_ai.event.version_id`	string	No	ID of the version now live

Permission Events

project_user_role_updated - Emitted when a user’s role within a project is updated. This occurs when an admin changes another user’s access level in the project settings.

Attribute	Type	Required	Description
`relevance_ai.event.target_user_id`	string	Yes	ID of the user whose role was changed
`relevance_ai.event.project_role`	string	Yes	The new project role assigned to the user

organization_user_role_updated - Emitted when a user’s role within an organization is updated. This occurs when an admin changes another user’s organization-level access.

Attribute	Type	Required	Description
`relevance_ai.event.target_user_id`	string	Yes	ID of the user whose role was changed
`relevance_ai.event.organization_role`	string	Yes	The new organization role assigned to the user

Example Log Record

{
  "resourceLogs": [{
    "scopeLogs": [{
      "logRecords": [{
        "timeUnixNano": 1768742472616000000,
        "severityNumber": 9,
        "severityText": "Info",
        "body": { "stringValue": "agent_deleted" },
        "attributes": [
          { "key": "relevance_ai.organization_id", "value": { "stringValue": "f6fb76a4-..." }},
          { "key": "relevance_ai.project_id", "value": { "stringValue": "3acde218-..." }},
          { "key": "relevance_ai.user_id", "value": { "stringValue": "941afb4d-..." }},
          { "key": "relevance_ai.user_email", "value": { "stringValue": "user@example.com" }},
          { "key": "relevance_ai.user_type", "value": { "stringValue": "user" }},
          { "key": "relevance_ai.ip_address", "value": { "stringValue": "119.18.1.156" }},
          { "key": "relevance_ai.device_info", "value": { "stringValue": "Mozilla/5.0..." }},
          { "key": "relevance_ai.event.agent_id", "value": { "stringValue": "3d47a9f5-..." }}
        ],
        "droppedAttributesCount": 0,
        "traceId": "",
        "spanId": ""
      }]
    }]
  }]
}

Traces

Traces track execution flows for agents, workforces, and LLM completions. Use traces to understand performance, debug issues, and analyze agent behavior.

We’re actively expanding our trace coverage. If there are specific spans or attributes you’d like to see, let your Account Manager know.

Structure

{
  "resourceSpans": [{
    "scopeSpans": [{
      "spans": [...]
    }]
  }]
}

Resource Attributes

Resource attributes identify the service producing telemetry data. These attributes are attached to all spans and logs exported from Relevance AI.

Attribute	Type	Value	Description
`service.name`	string	`"Relevance AI"`	Logical name of the service producing telemetry

This attribute is applied to all spans - chat, invoke_agent, multi_agent_system_trigger, and condition_trigger. In your observability tool, you can filter all Relevance activity with a single query like service.name = "Relevance AI", cleanly separating Relevance spans from your own services. Example structure:

{
  "resourceSpans": [{
    "resource": {
      "attributes": [
        { "key": "service.name", "value": { "stringValue": "Relevance AI" } }
      ]
    },
    "scopeSpans": [{ "spans": [...] }]
  }]
}

Base Attributes

Included on all spans:

Attribute	Type	Required	Description
`relevance_ai.organization_id`	string	Yes	Organization ID
`relevance_ai.project_id`	string	Yes	Project ID
`relevance_ai.user_id`	string	No	User ID that triggered the run
`relevance_ai.user_email`	string	No	Email of the user

Span Properties

Property	Type	Description
`traceId`	string	32-character hex trace identifier
`spanId`	string	16-character hex span identifier
`parentSpanId`	string	Parent span ID (if child span)
`name`	string	Span name (e.g., `invoke_agent My Agent`)
`kind`	int	Span kind (3 = CLIENT)
`startTimeUnixNano`	int64	Start time in nanoseconds
`endTimeUnixNano`	int64	End time in nanoseconds
`attributes`	array	Key-value pairs with span details
`status.code`	int	1 = OK, 2 = ERROR

Trace Hierarchy

Spans share traceId and link via parentSpanId:

multi_agent_system_trigger
├── condition_trigger
└── invoke_agent
    ├── chat
    └── invoke_agent (sub-agent)
        └── chat

Supported Spans

`invoke_agent`

Records a complete agent conversation/invocation. Name: invoke_agent or invoke_agent {agent_name}

Attribute	Type	Required	Description
`gen_ai.operation.name`	string	Yes	`"invoke_agent"`
`gen_ai.agent.id`	string	Yes	Agent UUID
`gen_ai.agent.name`	string	No	Agent name
`gen_ai.agent.description`	string	No	Agent description
`gen_ai.conversation.id`	string	Yes	Conversation/task UUID
`relevance_ai.agent.relevance_model`	string	Yes	Relevance model ID
`relevance_ai.agent.runtime`	string	Yes	`"default"` or `"phone_call"`
`relevance_ai.agent.metadata`	object	Yes	Custom metadata key-value pairs
`relevance_ai.agent.knowledge_used`	array	Yes	Knowledge set IDs used
`relevance_ai.agent.escalation.reason`	string	No	Escalation reason
`relevance_ai.agent.escalation.context`	string	No	Escalation context
`relevance_ai.agent.version_id`	string	No	The active version ID of the agent at invocation time. Use this to compare performance across agent versions

`chat`

Records a single LLM inference call. Name: chat {model_name}

This span was previously named llm_completion. If you have existing dashboards or queries filtering on gen_ai.operation.name = "llm_completion" or span names containing llm_completion, update them to use "chat". The rename aligns with the OpenTelemetry GenAI semantic conventions.

Attribute	Type	Required	Description
`gen_ai.operation.name`	string	Yes	`"chat"`
`gen_ai.request.model`	string	Yes	Requested model ID
`gen_ai.request.temperature`	int	No	Temperature setting
`gen_ai.request.max_tokens`	int	No	Max tokens setting
`gen_ai.system_instructions`	string	No	System prompt
`gen_ai.input.messages`	string	Yes	JSON-stringified input messages
`gen_ai.tool.definitions`	string	Yes	JSON-stringified tool definitions (use `JSON.parse()` to access as structured data)
`gen_ai.response.id`	string	Yes	Provider’s response ID
`gen_ai.response.model`	string	Yes	Actual model used
`gen_ai.response.finish_reasons`	array	Yes	Finish reasons (e.g., `["stop"]`)
`gen_ai.output.messages`	string	Yes	JSON-stringified output messages
`gen_ai.usage.input_tokens`	int	Yes	Input tokens used
`gen_ai.usage.output_tokens`	int	Yes	Output tokens generated

Attribute: gen_ai.tool.definitions

gen_ai.tool.definitions was previously an array type (arrayValue). It is now a JSON string (stringValue). If you consume this field in your pipeline or queries, you need to call JSON.parse() on the value to work with the tool definitions as structured data. The content is identical - the same tool definitions are present, just serialized as a string for better compatibility across OTEL exporters and backends.

`multi_agent_system_trigger`

Records a complete workforce execution. Name: multi_agent_system_trigger

Attribute	Type	Required	Description
`gen_ai.operation.name`	string	Yes	`"multi_agent_system_trigger"`
`relevance_ai.workforce.workforce_id`	string	Yes	Workforce UUID
`relevance_ai.workforce.workforce_task_id`	string	Yes	Task UUID (for correlation)
`relevance_ai.workforce.type`	string	Yes	`"chat"` or `"default"`
`relevance_ai.workforce.metadata`	object	Yes	Custom metadata
`relevance_ai.workforce.status`	string	Yes	Final status
`relevance_ai.workforce.version_id`	string	No	The active version ID of the workforce at trigger time. Use this to compare performance across workforce versions

`condition_trigger`

Records a condition node evaluation in a workforce. Name: condition_trigger

Attribute	Type	Required	Description
`gen_ai.operation.name`	string	Yes	`"condition_trigger"`
`relevance_ai.condition.workforce_node_id`	string	Yes	Node ID in workforce
`relevance_ai.condition.workforce_node_label`	string	No	Node label/name
`relevance_ai.condition.input`	string	Yes	JSON-stringified condition input
`relevance_ai.condition.decisions`	array	Yes	Decision objects (see below)
`relevance_ai.condition.reasoning`	string	No	LLM reasoning

Decision object:

{ "node": { "id": "node-id", "name": "Node Name" }, "will_run": true }

Attribute Value Types

Type	JSON Format
String	`{ "stringValue": "text" }`
Integer	`{ "intValue": 123 }`
Double	`{ "doubleValue": 1.23 }`
Boolean	`{ "boolValue": true }`
Array	`{ "arrayValue": { "values": [...] }}`
Object	`{ "kvlistValue": { "values": [{ "key": "k", "value": {...} }] }}`

Resources

Understanding OpenTelemetry

What is OpenTelemetry? - Official introduction to OTEL concepts

Compatible Observability Platforms

OTEL is supported by most major observability platforms. See the OpenTelemetry Registry for a full list of compatible vendors and integrations. For a detailed example of ingesting OTEL data, see Honeycomb’s OpenTelemetry guide.

Querying OTEL Data Directly

OTEL JSON files in S3 can be queried directly using SQL tools like AWS Athena.

​Overview

​Setup

​Prerequisites

​1. Create an S3 Bucket

​2. Configure Bucket Policy

​3. Provide Configuration to Relevance

​PII Redaction (Enterprise Feature)

​What gets redacted

​Supported PII entities

EMAIL_ADDRESS

PHONE_NUMBER

PERSON

CREDIT_CARD

US_SSN

IP_ADDRESS

IBAN_CODE

URL

DATE_TIME

LOCATION

​Redaction actions

​Prerequisites

​Configuration

​Delivery Format

​Logs

​Structure

​Base Attributes

​Log Record Properties

​Supported Events

​Agent Events

​Tool Events

​Workforce Events

​Permission Events

​Example Log Record

​Traces

​Structure

​Resource Attributes

​Base Attributes

​Span Properties

​Trace Hierarchy

​Supported Spans

​invoke_agent

​chat

​multi_agent_system_trigger

​condition_trigger

​Attribute Value Types

​Resources

​Understanding OpenTelemetry

​Compatible Observability Platforms

​Querying OTEL Data Directly

Overview

Setup

Prerequisites

1. Create an S3 Bucket

2. Configure Bucket Policy

3. Provide Configuration to Relevance

PII Redaction (Enterprise Feature)

What gets redacted

Supported PII entities

Redaction actions

Prerequisites

Configuration

Delivery Format

Logs

Structure

Base Attributes

Log Record Properties

Supported Events

Agent Events

Tool Events

Workforce Events

Permission Events

Example Log Record

Traces

Structure

Resource Attributes

Base Attributes

Span Properties

Trace Hierarchy

Supported Spans

`invoke_agent`

`chat`

`multi_agent_system_trigger`

`condition_trigger`

Attribute Value Types

Resources

Understanding OpenTelemetry

Compatible Observability Platforms

Querying OTEL Data Directly