Data Ingestion Using AWS OpenSearch

Data Ingestion Using AWS OpenSearch

Learn how to move OpenSearch data ingestion from an operational bottleneck to a scalable, cost-efficient foundation for search and analytics.
10 mins read
Nov 13, 2026
Share

Data ingestion is a core part of designing an Amazon OpenSearch Service workload. The ingestion design affects the performance, cost, and reliability of the search and analytics workload. A poorly configured ingestion pipeline can cause indexing backlogs, latency spikes, and, in some failure scenarios, data loss. Choosing between direct API ingestion and managed ingestion services is an architectural decision that affects operations and scalability. Understanding the trade-offs among control, operational overhead, and scalability helps teams design a resilient ingestion pipeline. This newsletter examines the primary data ingestion patterns for Amazon OpenSearch Service.

Understanding OpenSearch data ingestion#

Effective data ingestion is the critical entry point for leveraging Amazon OpenSearch Service. Without a well-architected pipeline, the speed and relevance of search queries suffer, and operational costs can escalate unexpectedly.

Available ingestion methods range from direct API calls and self-managed Logstash to managed services such as Amazon OpenSearch Ingestion (OSI) and Amazon Kinesis Data Firehose. The primary challenge lies in selecting a method that aligns with your data's volume, velocity, and transformation requirements.

Common pitfalls often derail production deployments. These include under-provisioned ingestion capacity leading to high latency, inadequate error handling causing data loss, and inefficient batching strategies that overload the destination cluster. A robust ingestion strategy anticipates these issues, ensuring data is indexed reliably and cost-effectively. Understanding this landscape is the first step toward building a high-performance search application.

To help visualize these pathways, here’s a high-level architectural diagram showing various data sources and ingestion methods:

The architectural landscape of OpenSearch ingestion
The architectural landscape of OpenSearch ingestion

This diagram shows how different data sources flow through ingestion pipelines into an Amazon OpenSearch Service domain. Next, we examine direct API ingestion and the operational considerations it introduces.

Direct API and legacy ingestion methods#

Direct ingestion via the OpenSearch Bulk REST API or self-managed Logstash instances offers maximum control but comes with significant operational overhead. These methods are often used for low-volume development environments, proof-of-concept projects, or integrations where a managed ingestion service is not a good fit. The primary responsibility for scaling, error handling, and monitoring falls entirely on your team. You must manually provision compute resources, like EC2 instances for Logstash, manage software updates, and implement custom logic for retries and dead-letter queues.

In a production environment, this manual approach becomes a liability. For example, a sudden spike in data volume can overwhelm a fixed-size Logstash cluster, leading to data backlogs or outright loss. While direct API calls provide granular control over every request, they require careful client-side implementation of batching, compression, and retry logic, adding complexity to your application code. For most production use cases, the operational burden of these methods outweighs their flexibility.

Let's look at a code example to see how a direct bulk request is structured:

C++
import json
def create_bulk_payload(documents, index_name):
"""
Formats a list of documents into a JSON payload suitable for the OpenSearch Bulk API.
Each document requires an '_id' field for the 'index' action.
"""
bulk_data = []
for doc in documents:
# Action/metadata line for 'index' operation
metadata = {"index": {"_index": index_name, "_id": doc["_id"]}}
bulk_data.append(json.dumps(metadata))
# Document source line
bulk_data.append(json.dumps(doc))
return "\n".join(bulk_data) + "\n" # Bulk API requires a trailing newline
# Example usage:
documents_to_index = [
{"_id": "1", "title": "My first document", "text": "This is some example text."},
{"_id": "2", "title": "Another document", "text": "More text for indexing."},
{"_id": "3", "title": "Third entry", "text": "Final piece of content."},
]
target_index = "my-test-index"
# generated_payload = create_bulk_payload(documents_to_index, target_index)
# print(generated_payload)
# This payload can be sent as the body of an HTTP POST request to _bulk endpoint
}

Direct ingestion is manageable at low volumes but requires more operational work as throughput grows. Managed ingestion services reduce much of this operational overhead.

Fully managed OpenSearch ingestion#

Amazon OpenSearch Ingestion (OSI) is a fully managed, serverless data collector and pipeline that delivers data to OpenSearch clusters and S3 buckets. It abstracts away the underlying infrastructure, freeing you from managing and scaling data collector fleets like Logstash or Fluentd. OSI automatically provisions and scales resources based on workload, using a capacity model built on Ingestion OpenSearch Compute Units (OCUs). An OCU is a measure of data processing capacity within an OSI pipeline where one OCU provides approximately 15 GiB of memory and 2 vCPUs. This elasticity ensures the pipeline can handle fluctuating data volumes without manual intervention, preventing bottlenecks that often plague self-managed solutions.

OSI supports data transformation and enrichment within managed ingestion pipelines. Declarative YAML configurations define multi-stage pipelines that can parse, filter, mask, and enrich data during ingestion. This is particularly valuable for complex use cases, such as preparing data for vector search by managing vector embedding size and model inference cost through integrations with models hosted on Amazon SageMaker or Bedrock. For observability, OSI pipelines can now send metrics to Amazon Managed Service for Prometheus, providing deep visibility into pipeline performance.

To better understand the trade-offs, here’s a comparison between self-managed Logstash and OSI:

Feature

Amazon OpenSearch Ingestion (OSI)

Self-Managed Logstash

Operational Overhead

Fully managed, serverless. AWS handles infrastructure, patching, scaling, upgrades, and monitoring.

Requires manual provisioning, infrastructure management, software installs/upgrades, patching, HA, monitoring, and backups.

Scaling and Capacity

Automatic scaling based on min/max OCUs. Max OCUs: 96 (stateless), 48 (stateful). Higher with persistent buffering (384 stateless, 192 stateful). Distributed across AZs.

Horizontal scaling by adding more nodes. Throughput is influenced by filter complexity. Requires manual setup for buffering and persistence.

Cost Model

Billed per allocated OCU (compute resources), even if idle. Not based on data volume.

Incurs all infrastructure costs (servers, compute, storage, network), potential licensing, and significant engineering time.

Transformation Features

Built on Data Prepper 2.7. Supports filtering, enrichment, transformation, and aggregation (stateless/stateful processors). Multiple sinks (OpenSearch, S3). Various codecs and persistent buffering.

Rich plugin ecosystem (200+ inputs, filters, outputs). Includes filters like grok, mutate, geoip. Supports conditional routing, codecs, and configurable batching.

Service Level/Availability

99.9% availability SLA (for production configuration: minimum 2 OCUs distributed across AZs).

User-implemented SLA; it depends entirely on manual redundancy, failover mechanisms, and monitoring setup. No built-in vendor SLA.

Key Metrics

Max Stateless OCUs: 96 (no buffer), 384 (buffer). Max Stateful OCUs: 48 (no buffer), 192 (buffer). Minimum OCU: 1 (2 for SLA/push-based sources). Engine: Data Prepper 2.7.

Maximum capacity is dependent on the number and size of deployed nodes. Minimum is one node, but more are required for resilience. Engine: User-deployed Logstash version (typically 7.x/8.x).

OSI provides a compelling alternative to self-managed pipelines, but for high-volume, real-time streaming data, another service often comes into play.

Streaming data with Kinesis Data Firehose#

Amazon Kinesis Data Firehose is a fully managed service designed to reliably load high-volume streaming data into destinations like Amazon S3, Redshift, and OpenSearch Service. Its primary advantage is its simplicity and automatic scaling, making it a robust choice for use cases such as application log streaming or IoT telemetry. Firehose handles all the underlying stream management, including scaling throughput to match the rate of incoming data from sources like Kinesis Data Streams or direct PUT operations. A key feature is its configurable buffering strategy, which batches incoming records based on either time, for example, every 60 seconds, or size, for example, every 5 MB. This batching improves OpenSearch indexing efficiency by combining many small writes into fewer, larger bulk requests.

When integrating with OpenSearch, proper configuration is essential to prevent pipeline stalls. You must align the Data Firehose delivery stream's capacity with the OpenSearch cluster's indexing capabilities, paying close attention to the relationship between the cluster's shard count and the managed service's max OCU limits. If Firehose attempts to push data faster than the cluster can index it, it can lead to backpressure, a condition where a downstream system cannot process data as fast as an upstream system sends it, causing data to queue up and potentially leading to processing delays or data loss. We have seen cases where misaligned capacities led to significant backlogs, underscoring the importance of Firehose's retry mechanisms and S3 backup options for data resilience. To mitigate this, Firehose includes built-in retry mechanisms and supports backing up all or only failed records to an S3 bucket, creating a resilient data lake architecture.

A visual representation can help clarify this data flow:

A typical Kinesis Data Firehose pipeline for OpenSearch
A typical Kinesis Data Firehose pipeline for OpenSearch

While Firehose excels at streaming, other managed services address different ingestion patterns, such as event-driven processing from object storage.

To further strengthen your streaming pipeline skills, practice Building a Real-Time Data Pipeline with Kinesis and S3 Tables while configuring Kinesis Data Firehose delivery streams and querying the ingested data in Apache Iceberg format with Athena.

Other managed ingestion paths#

Beyond OSI and Firehose, AWS provides other managed options tailored to specific ingestion patterns. For event-driven workflows, an S3-triggered AWS Lambda function is a common and effective solution. When a new object is created in an S3 bucket, an event notification can invoke a Lambda function that reads the object, transforms its content, and indexes it into OpenSearch. This pattern is excellent for batch-oriented data, such as periodic log file drops or data exports, where processing is tied to file creation events rather than a continuous stream. However, it can become less cost-effective for high volumes of very small files due to the per-invocation cost of Lambda.

Another powerful pattern is the use of zero-ETL integrations, such as those available for Amazon DynamoDB and Amazon DocumentDB. Zero-ETL is a type of data integration that eliminates the need for manual Extract, Transform, and Load (ETL) processes by enabling near real-time data synchronization between a source and a destination. These integrations automatically stream data changes from the source database to an OpenSearch index with minimal configuration. This is ideal for use cases where you need to add full-text search capabilities to an existing NoSQL database without building and maintaining a complex data synchronization pipeline. While incredibly simple for their intended purpose, these integrations offer limited options for in-flight data transformation compared to a full-fledged service like OSI.

Note: Zero-ETL integrations are optimized for simplicity and low-latency synchronization, not for complex data enrichment or aggregation tasks.

Here’s a brief code example illustrating the S3-Lambda pattern:

C++
// Import necessary AWS SDK clients and HTTP client for Node.js
const { S3Client, GetObjectCommand } = require('@aws-sdk/client-s3');
const fetch = require('node-fetch'); // Requires 'node-fetch' package installed (Lambda layer/dependency)
// Initialize S3 client for object retrieval
const s3Client = new S3Client({ region: process.env.AWS_REGION || 'us-east-1' });
// Configuration for OpenSearch endpoint and index (use environment variables in production)
const OPENSEARCH_ENDPOINT = process.env.OPENSEARCH_ENDPOINT || 'https://YOUR_OPENSEARCH_ENDPOINT.amazonaws.com'; // e.g., vpc-my-domain-123.us-east-1.es.amazonaws.com
const OPENSEARCH_INDEX = process.env.OPENSEARCH_INDEX || 'my-s3-data';
exports.handler = async (event) => {
// Extract S3 bucket and object key from the triggering S3 PutObject event
const record = event.Records[0];
const bucketName = record.s3.bucket.name;
const objectKey = decodeURIComponent(record.s3.object.key.replace(/\+/g, ' '));
try {
// Retrieve the object content from S3
const s3GetObjectResponse = await s3Client.send(new GetObjectCommand({
Bucket: bucketName,
Key: objectKey,
}));
const objectContent = await s3GetObjectResponse.Body.transformToString();
// Parse the retrieved content as JSON
const documents = JSON.parse(objectContent);
// Construct OpenSearch bulk request body in NDJSON format
let bulkRequestBody = '';
// Ensure documents is an array, even if the S3 file contained a single JSON object
const documentsArray = Array.isArray(documents) ? documents : [documents];
documentsArray.forEach(doc => {
// Generate a unique ID if not present in the document
const id = doc.id || doc._id || Math.random().toString(36).substring(2);
// Append action line and document line for bulk request
bulkRequestBody += JSON.stringify({ index: { _index: OPENSEARCH_INDEX, _id: id } }) + '\n';
bulkRequestBody += JSON.stringify(doc) + '\n';
});
// Send the constructed bulk request to OpenSearch
const osResponse = await fetch(`${OPENSEARCH_ENDPOINT}/${OPENSEARCH_INDEX}/_bulk`, {
method: 'POST',
headers: { 'Content-Type': 'application/x-ndjson' },
body: bulkRequestBody
});
// Handle potential errors from the OpenSearch response
if (!osResponse.ok) {
const errorDetails = await osResponse.text();
throw new Error(`OpenSearch bulk request failed: ${osResponse.status} ${osResponse.statusText}. Details: ${errorDetails}`);
}
const osResponseBody = await osResponse.json();
console.log('OpenSearch bulk indexing successful:', osResponseBody);
return {
statusCode: 200,
body: JSON.stringify(`Successfully processed s3://${bucketName}/${objectKey} and indexed to OpenSearch.`)
};
} catch (error) {
console.error(`Error processing s3://${bucketName}/${objectKey}:`, error);
throw error; // Re-throw the error to indicate Lambda execution failure
}
};
}

With several viable options available, the next step is to develop a framework for choosing the best one for your workload.

Choosing the right ingestion strategy#

Selecting the optimal ingestion method requires a careful evaluation of your specific requirements across several dimensions: data volume and velocity, transformation complexity, operational tolerance, and cost sensitivity. There is no single best ingestion method. The choice depends on trade-offs among requirements. For instance, while a direct API approach offers maximum control, it carries the highest operational burden. Conversely, a zero-ETL integration is operationally simple but provides minimal transformation capabilities. A critical mistake is creating an under-provisioned ingestion capacity, where the pipeline cannot keep up with the data source, leading to indexing lag and stale data.

Effective buffer strategies are essential for any high-throughput pipeline. Services like Kinesis Data Firehose and OSI manage this automatically, but for Lambda-based or direct ingestion, you must implement buffering yourself. For machine learning applications, early architectural decisions around vector embedding size have a drastic impact on future costs and latency. A vector embedding is a numerical representation of text, images, or other data in a high-dimensional space, used in similarity search and AI applications. Larger embeddings require more storage and compute for indexing and querying. Key design questions include how fresh the data must be, how complex the transformations are, and who will manage the infrastructure. These requirements help determine an ingestion pipeline that balances performance and cost.

The following table provides a decision matrix to guide your selection process:

Criterion

Kinesis Data Firehose

AWS S3 + Lambda (Event-Triggered Processing)

Data Volume

Handles high streaming data volumes; auto-scales.

Small to moderate file counts; many invocations for small files.

Transformation Needs

Optional Lambda, format conversion (JSON/Parquet/ORC), compression, encryption.

Very flexible; arbitrary transformations via custom Lambda code.

Latency/Real-Time Req.

Near real-time; buffering introduces delay (default 60s for transform).

Near real-time (file write triggers Lambda); depends on execution.

Buffering/Batching

Buffers records (0.2-3MB, 0-900s interval; default 1MB/60s).

Per object/event; no built-in buffering (requires custom logic).

Operational Overhead

Low-to-medium; managed service, but configuration/Lambda for features.

Medium to high; custom code, manage concurrency, retries, scaling.

Limits/Constraints

Record size ≤ 6MB (with transform); 500 active partitions for dynamic partitioning.

Lambda payload/execution time limits; invocation rate/concurrency limits.

Cost Considerations

Per GB ingested; additional for format conversion, dynamic partitioning, VPC.

Lambda invocations/compute + S3 storage/requests; scales with processing.

To make these concepts more concrete, let's look at a common and practical implementation pattern.

S3 to OpenSearch pipeline example#

A common and robust pattern for ingesting data is moving files from Amazon S3 into OpenSearch Service using an OpenSearch Ingestion (OSI) pipeline. This architecture is well-suited for data that arrives in batches, such as application logs, transaction records, or data exports. The process begins when a new object is created in a designated S3 bucket. This event can be configured to send a notification to an Amazon Simple Queue Service (SQS) queue. The OSI pipeline is configured with S3 as its source, polling this SQS queue for new object notifications.

Once OSI receives a notification, it retrieves the corresponding object from S3. Inside the OSI pipeline configuration, you define one or more processors to parse and transform the data. For example, you might use the grok processor to parse unstructured log lines, the rename processor to standardize field names, or the flatten processor to simplify nested JSON structures. After processing, the data is sent to the OpenSearch sink, which is configured with the endpoint of your OpenSearch cluster and the target index name. This serverless pipeline provides a scalable, resilient, and low-overhead solution for S3-based data.

Practical tip: Use a separate SQS queue as a buffer between S3 and OSI. This decouples the components and improves resilience against downstream processing delays.

Here’s a diagram illustrating this specific pipeline architecture:

S3 to OpenSearch ingestion via an OSI pipeline
S3 to OpenSearch ingestion via an OSI pipeline

Building the pipeline is the first step. Ensuring it runs reliably in production requires ongoing attention to performance and cost.

Optimizing for production readiness#

Operating a data ingestion pipeline requires ongoing attention after deployment. It requires ongoing monitoring and optimization to maintain reliability and cost efficiency. The most critical practice is to monitor key performance indicators proactively. For ingestion pipelines, tracking metrics like records-in error rates helps catch data format issues promptly. Similarly, monitoring bulkRequestLatency can identify indexing delays before they escalate. Setting up alerts on these metrics allows you to detect and respond to problems before they impact users. Proper capacity planning is equally crucial. You must understand the relationship between pipeline capacity, for example, OCUs in OSI, and cluster capacity, for example, shard count and node size, to prevent bottlenecks.

To manage costs and performance over time, implement index life cycle management (ILM) policies in OpenSearch. ILM is a feature that automates index life cycle tasks over time. Policies can automate actions such as index rollover, migration from hot storage to UltraWarm and cold storage, and deletion of older indexes. ILM can automatically transition older, less frequently accessed data to more cost-effective storage tiers or delete it entirely, preventing uncontrolled index growth.

Finally, use an iterative approach. As the application evolves, data volumes may change, and new ML models may alter embedding dimensions or vector structure. Regularly test and adjust your pipeline configuration to ensure it continues to meet your performance and cost objectives. A crucial point often overlooked is configuring a dead-letter queue (DLQ) in your pipeline to capture failed records, preventing data loss and enabling later analysis.


Try the Cloud Lab "Using Amazon OpenSearch Service for Data Ingestion" to get hands-on practice building an S3-to-OpenSearch ingestion pipeline, configuring IAM roles, triggering SQS events, and setting up an OSI pipeline for real-time data ingestion.


Effective data ingestion helps support reliable, efficient, and cost-conscious OpenSearch Service workloads. By carefully evaluating your unique requirements and leveraging the right AWS services, you can design a pipeline that scales with your data and ensures its integrity for the long term.


Written By:
Naeem
Free Edition
AWS CodeCommit is back—and hopefully for good
AWS CodeCommit’s 2024 de-emphasis reshaped source control and CI/CD across AWS. With CodeCommit now back in GA and a clear roadmap, teams can reassess AWS-native workflows and pipeline design.
7 mins read
Feb 6, 2026