Announcing OpenSearch Data Prepper 2.12: Additional source and sinks for your data ingestion needs - OpenSearch

Announcing OpenSearch Data Prepper 2.12: Additional source and sinks for your data ingestion needs

By [David Venable](/content/author/david-venable/ "Posts by David Venable"/index.html), [Krishna Kondaka](/content/author/krishna-kondaka/ "Posts by Krishna Kondaka"/index.html), [Huy Pham](/content/author/huy-pham/ "Posts by Huy Pham"/index.html), [Shenoy Pratik Gurudatt](/content/author/shenoy-pratik-gurudatt/ "Posts by Shenoy Pratik Gurudatt"/index.html) July 1, 2025 August 25th, 2025

OpenSearch Data Prepper 2.12 is now available for download! This release includes a new way to ingest OpenTelemetry (OTel) data as well as two new sinks.

Unified OTLP source

Data Prepper now includes a unified OpenTelemetry protocol (OTLP) source that streamlines telemetry data ingestion through a single, consolidated configuration. This source supports multiple protocols, seamlessly handling both gRPC and HTTP (with proto encoding) endpoints. It enables ingestion of OTel logs, traces, and metrics through exposed OTLP endpoints, simplifying configuration management and improving the efficiency of the data processing pipeline.

Additionally, to help you process the different signal types that OTel provides, Data Prepper now includes the getEventType() function. This feature enables dynamic classification and conditional routing of events within pipelines for more flexible and intelligent processing. Specifically for the otlp source, you can use this to route different types to different pipelines.

This sample pipeline shows a basic otlp source that routes logs, metrics, and traces to three different pipelines for processing:

source:
  otlp:
  route:
    - logs: 'getEventType() == "LOG"'
    - traces: 'getEventType() == "TRACE"'
    - metrics: 'getEventType() == "METRIC"'

Amazon SQS sink

Data Prepper now supports Amazon Simple Queue Service (Amazon SQS) as an output sink. Amazon SQS is a widely adopted message queuing service designed for decoupling producers and consumers in distributed systems. It is especially suited for lightweight, structured messages that require timely delivery and reliable processing.

Sending Data Prepper output to SQS enables seamless communication between producers and consumers of data. Sending data directly to SQS is significantly faster and more efficient than traditional approaches such as sending output to an Amazon Simple Storage Service (Amazon S3) bucket and configuring an SQS notification on that bucket. With the new SQS sink, Data Prepper bypasses the overhead of writing to S3 and triggering SQS indirectly, reducing latency and improving responsiveness. This eliminates the need to configure S3 event notifications, write intermediate files, or manage bucket lifecycle rules. You can now go straight from processing to queuing with a clean, minimal configuration.

Here’s how to get started with the SQS sink:

sink:
  - sqs:
        queue_url: <queue-url>
        codec:
          json:
        aws:
          region: <region>
          sts_role_arn: <role>

OTLP sink for AWS X-Ray

You can now enhance your observability pipeline’s interoperability by seamlessly exporting processed trace data to AWS X-Ray through Data Prepper’s new OTLP sink plugin. This integration enables organizations to leverage Data Prepper’s powerful transformation and enrichment capabilities while maintaining compliance with OTel standards and sending data directly to AWS X-Ray using the OTLP format. The OTLP sink currently supports exporting spans to AWS X-Ray endpoints, with future versions planned to support sending spans, metrics, and logs to any OTLP protobuf-compatible endpoint. The plugin is designed for high performance, sustaining up to 3,500 transactions per second with sub-150ms p99 latency while using minimal system resources. Built with production reliability in mind, it features configurable retry logic with exponential backoff, gzip compression for efficient data transfer, and comprehensive metrics for monitoring pipeline health. Here’s how to get started with the OTLP sink for AWS X-Ray:

source:
  otel_trace_source:
sink:
  - otlp:
      endpoint: "https://xray.{region}.amazonaws.com/v1/traces"
      aws: { }

Maven releases

Many community members have expressed interest in using various Data Prepper features as libraries. To help support the broader community, the Data Prepper team is now publishing all Data Prepper libraries to Maven Central.

The following Maven groups are available to the community:

Other features and improvements

Getting started

Thanks to our contributors!

Thanks to the following community members who contributed to this release!