Cloud PlatformsConfirmedAWS Lambda now supports queues for Apache Kafka in self-managed event sources, enabling greater scalability and resilience for event-driven workloads by decoupling parallelism from partition count.
AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings (ESM). This capability extends to Kafka clusters that customers operate themselves and to managed offerings such as Confluent Cloud. The feature allows customers to scale processing for Kafka workloads beyond the partition count of a topic, increasing throughput during traffic spikes and improving processing resilience.
Queue support for self-managed Kafka ESMs is built on Kafka share groups, which are generally available in Apache Kafka 4.2 and later. The launch enables multiple event pollers to consume concurrently from the same partition, with each event poller supporting up to 10 concurrent invocations. This decouples parallelism from partition count with no changes required to producers, topic configuration, or function code.
Queue mode does not preserve partition ordering and is intended for workloads that do not require strict ordering. The capability requires a Kafka cluster with share groups enabled, such as Apache Kafka 4.2 or later, and is available in all AWS commercial Regions where self-managed Kafka ESM is available. Customers can create a self-managed Kafka ESM with ConsumptionMode set to Queue through the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK.
Standard Lambda pricing applies. Customers who configure provisioned mode for their self-managed Kafka ESM also pay for Event Poller Unit (EPU) usage. Previously, Kafka ESMs used the Kafka consumer group protocol, which assigns each partition to exactly one consumer within a consumer group. Customers needing higher parallelism had to over-partition topics or move traffic to a separate queuing system, adding infrastructure to operate and producer code to maintain.
The introduction of queue-based consumption for self-managed Kafka event sources represents a fundamental shift in how Lambda scales Kafka workloads. Under the traditional consumer group protocol, the partition count of a topic establishes a hard ceiling on parallelism: each partition can be assigned to only one consumer within a group. This constraint forced architects into a dilemma — either over-partition topics at creation time (incurring operational overhead and potential partition imbalance) or accept throughput bottlenecks during traffic spikes. The new queue mode, built on Kafka share groups (KIP-932), removes this ceiling by allowing multiple event pollers to read from the same partition concurrently.
From an engineering perspective, the decoupling of parallelism from partition count has several direct consequences. First, it eliminates the need for predictive partition planning. Teams no longer need to estimate peak throughput and pre-provision partitions accordingly; instead, Lambda's event poller scaling can respond to actual load. Second, it reduces the blast radius of slow or failing invocations. In consumer group mode, a single slow record blocks progress on its entire partition. With queue mode, other pollers can continue processing subsequent records from the same partition, improving tail latency and overall resilience. Third, it simplifies producer logic: producers no longer need to implement custom partitioning strategies or route to alternative queuing systems to achieve higher parallelism.
The constraint that queue mode does not preserve partition ordering is a deliberate trade-off documented in the announcement. Workloads that require per-key ordering — such as financial transaction processing or stateful aggregations keyed by entity ID — must continue using consumer group mode or implement ordering guarantees at the application layer. The announcement explicitly identifies suitable use cases: image and video encoding, fraud detection, notification delivery, and telemetry ingestion. These share a common characteristic: each event is an independent unit of work where processing speed matters more than strict ordering. Engineers should evaluate their ordering requirements before migrating existing ESMs to queue mode.
The pricing model introduces a new cost dimension for provisioned mode users: Event Poller Units (EPUs). While standard Lambda pricing applies to on-demand invocations, provisioned mode customers pay for EPU usage in addition to standard charges. The announcement does not disclose the EPU pricing rate or the mapping between poller count and EPU consumption. This information gap means teams running provisioned mode cannot fully model cost implications without consulting the Lambda pricing page or documentation. For on-demand workloads, the cost model remains unchanged — pay per request and duration — but the increased concurrency potential may increase total invocation count during burst periods.
Adopting queue mode requires a Kafka cluster running Apache Kafka 4.2 or later with share groups enabled. This version requirement may constrain adoption for teams on older self-managed clusters or managed services that have not yet upgraded. The announcement does not specify whether AWS-managed Kafka offerings (such as Amazon MSK) support this feature, nor does it detail the share group configuration steps required on the cluster side. Engineers operating self-managed clusters should verify share group availability and any associated broker configuration changes before enabling queue mode.
The concurrency model — up to 10 concurrent invocations per event poller, with multiple pollers per partition — introduces new scaling dynamics. Lambda's poller scaling behavior for queue mode is not described in the announcement. In consumer group mode, Lambda scales pollers based on partition count and lag metrics. With queue mode, the scaling signals and limits may differ. Teams should monitor invocation concurrency, throttle rates, and poller metrics after migration to understand the new scaling profile. The announcement does not disclose maximum poller counts per ESM or per partition.
Error handling and retry semantics may differ between consumption modes. The announcement states that a slow or failed event no longer holds up other events in the same partition, implying per-record failure isolation. However, it does not specify whether failed records are retried within the same poller, re-queued for another poller, or sent to a dead-letter queue. The Lambda Developer Guide section on polling Kafka queues (referenced in the announcement) would need to be consulted for these details. Teams with strict exactly-once or at-least-once processing requirements should verify the delivery semantics before switching modes.
The announcement emphasizes that queue mode requires no changes to producers, topic configuration, or function code. This is a significant operational advantage: existing Lambda functions processing Kafka events can switch consumption modes by updating the ESM's ConsumptionMode parameter. However, the ordering guarantee change means that functions relying on partition ordering for correctness will exhibit different behavior. Teams must audit their function logic for ordering dependencies — such as stateful aggregations, deduplication based on partition sequence, or downstream systems expecting ordered delivery — before migrating.
The announcement does not address whether an existing ESM can be switched from consumer group mode to queue mode in place, or whether a new ESM must be created. It also does not describe any migration tooling, rollback procedures, or coexistence patterns (e.g., running both modes on the same topic with different consumer groups). The reference to the Lambda Developer Guide suggests these operational details are documented there. Engineers planning migration should consult that guide for step-by-step procedures and any limitations on mode switching.
For teams using infrastructure-as-code (CloudFormation, SAM, CDK), the ConsumptionMode parameter addition means template updates are straightforward. The announcement lists all major IaC tools as supported. However, it does not specify the exact parameter name or value format for each tool, nor whether existing ESM resources support in-place updates or require replacement. CDK and CloudFormation users should verify the resource specification for AWS::Lambda::EventSourceMapping to understand update behavior.