Confluent and Amazon EventBridge for Broad Event Distribution

Introduction

Confluent has established itself as a leader in event streaming, providing not only a robust platform but also a rich portfolio of pre-built connectors. These connectors act as bridges, effortlessly channeling data between a multitude of systems, from databases and applications to cloud services. This extensive portfolio empowers users to weave together their data landscapes with remarkable ease and flexibility.

Building on this foundation, the Amazon EventBridge Sink Connector significantly amplifies Confluent's reach within the Amazon Web Services (AWS) ecosystem. Imagine EventBridge as a highly efficient postal service within AWS that routes those messages to a vast network of AWS services—from serverless functions and data lakes to machine learning platforms and notification systems. This powerful combination allows you to seamlessly extend your Confluent event streams to a far broader array of AWS capabilities, unlocking new avenues for building responsive, interconnected, and data-driven applications with unparalleled agility and scope.

Use Cases

To help visualize how Amazon EventBridge and Confluent work together in practice, consider these four real-world use cases.

• Financial services company: A global bank processes millions of transactions per day and needs to detect suspicious activity in real time. By streaming transactional data through Confluent Platform and forwarding relevant events into EventBridge, the bank can trigger fraud detection–related AWS Lambda functions that stop/intercept fraudulent transactions, invoke automated alerts via Amazon Simple Notification Service (SNS), buffer messages in Amazon Simple Queue Service (SQS), and update downstream analytics in Amazon Redshift—all within seconds of the triggering transaction.

  ‎ 

• Manufacturing company: A factory floor is outfitted with thousands of Internet of Things (IoT) sensors that continuously emit telemetry—temperature, vibration, throughput, and more—into an on-premises Confluent Platform cluster. Using the Amazon EventBridge Sink Connector, these sensor events are seamlessly shipped into EventBridge, where they initiate cloud-based quality-control workflows such as AWS Step Functions for anomaly remediation and feed historical data into Amazon S3 (via Amazon Kinesis Firehose) for long-term trend analysis.

  ‎ 

• Healthcare company: A network of clinics and hospitals captures patient vitals and device alarms in near-real time. Confluent Platform streams these clinical events into Amazon EventBridge, triggering automated patient-monitoring alerts with Amazon SNS. Alternatively, Amazon Pinpoint (with the help of AWS Lambda) can send push or text notifications to on-call personnel when critical thresholds are crossed. All this data, both raw events and the resulting triggered events, is simultaneously stored securely in AWS HealthLake for compliance and audit purposes.

How to Set Up the Integration

These instructions will walk you through setting up the Amazon EventBridge Sink Connector on Confluent Platform that is installed locally on your computer. We’ll start by setting up the AWS resources, followed by Confluent Platform.

AWS Setup

1. Install and configure the AWS CLI. This tutorial assumes that Confluent Platform and the connector will be deployed on your local machine, and instructions for this follow. First, prepare your machine’s AWS CLI. You can install it here. Once installed, set up your AWS Identity and Access Management (IAM) credentials or optionally create a profile used by the connector to assume an IAM role and send events to EventBridge. In the default configuration, the connector requires only EventBridge PutEvents permissions as described here.

   aws configure

2. Create an Amazon EventBridge event bus. Navigate to Amazon EventBridge in the AWS console. For now, leave all values as default and leave the “Resource-based policy” section blank.

   ‎ 

3. Create a rule. Within your new event bus, create a rule. This rule will be the logic that filters and routes events on the event bus. For the sake of this example, you’ll allow all events and supply an event pattern that allows events based on the AWS account ID. For the target, create a new CloudWatch log group. This will make it quick and easy to verify the flow of events later. All other fields can be left as default or blank.

   ‎ 

   The value for “account” is the account number of your AWS account.

   ‎ 

   ‎ 

Confluent Platform Setup

1. Install Confluent Platform and populate the Apache Kafka® topics. For the purposes of this blog post, follow Steps 1-3 of this Confluent Platform Quick Start (non-Docker local install). Then open http://localhost:9021/ to access the user interface of the locally hosted Confluent Platform. Note: These commands must be executed from the same terminal/shell where you configured your AWS credentials (see Step 1) so that the connector can retrieve the IAM credentials upon start.

   ‎ 

2. Install Confluent Hub client. Use the documentation to install the Confluent Hub client.

   ‎ 

3. Install the Amazon EventBridge Sink Connector. Run the following one directory level above where you saved the Confluent Platform directory. This will make the connector available for deployment in Confluent Platform.

   ‎ 

   confluent-hub install aws/kafka-eventbridge-sink:v1.5.0

   ‎ 

   After installing the connector via the Confluent Hub CLI tool, you’ll see a new tile labeled “EventBridgeSinkConnector.”

   ‎ 

4. Activate the sink connector. With the connector now available within Confluent Platform, go ahead and deploy the connector. Use below JSON as a template for the “Upload connector config file” option. Note: You may need to change the value.converter based on the schema you’re using, such as using the Confluent Schema Registry with Avro or JSON_SR for your DataGen connector. This configuration also assumes that IAM credentials are correctly configured (Step 1) in the shell where the connector will be executed. Details, including how to use AWS CLI profiles, can be found here.

   ‎ 

   {
     "name": "EventBridgeSinkConnectorConnector_0",
     "config": {
       "connector.class": "software.amazon.event.kafkaconnector.EventBridgeSinkConnector",
       "tasks.max": "1",
       "value.converter:": "io.confluent.connect.avro.AvroConverter",
       "value.converter.schema.registry.url":"http://localhost:8081",
       "topics": "pageviews",
       "aws.eventbridge.connector.id": "confluent-connector-bus",
       "aws.eventbridge.region": "us-east-1",
       "aws.eventbridge.eventbus.arn": "<YOUR_EVENTBUS_ARN>"
     }
   }

Setup Confirmation

1. Navigate back to the AWS console and into the CloudWatch service. Find the log group that was set as the target for the event bus. Messages from Confluent Platform should now be present. Note: Unless you updated the auto.offset.reset configuration for the Amazon EventBridge Sink Connector, the connector by default will read from the latest records (i.e., any new records that come into the topic after the connector has started listening on the Kafka topic). If you’d like the connector to read from the beginning of the Kafka topic and the new records that came afterward, you can set the auto.offset.reset configuration attribute to earliest.

   ‎ 

Best Practices for Production

1. Align EventBridge service quotas with your peak Kafka throughput to avoid throttling. Optionally, tune Kafka Connect consumer settings (poll intervals, batch sizes) to match processing latency and avoid connector stalls/throttling. Details can be found here.

   ‎ 

2. The EventBridge connector uses automatic retries through the underlying AWS software development kit configurable) to guard against temporary network issues. However, setting up dead-letter (topic) handling for the connector is recommended to avoid failure, such as from head-of-line blocking. Details can be found here.

   ‎ 

3. Enforce least-privilege IAM by scoping PutEvents permissions to only the specific EventBridge bus. Optionally, you can provide your own authorization mechanism for advanced control and integration scenarios using the aws.eventbridge.auth.credentials_provider.class configuration property.

   ‎ 

4. EventBridge currently limits event sizes to 256 KB. For larger records, use the built-in Amazon S3 offloading capabilities to send oversized payloads to S3, as described here.

   ‎ 

5. Leverage and optionally tweak logging configuration for insights into warnings, failures, and latency, as described here.

More information on all configuration settings can be found here.

Conclusion

The Amazon EventBridge Sink Connector provides a highly scalable and reliable mechanism for distributing records consumed from Kafka topics to a wide array of AWS services. This capability provides streaming architects and developers with optionality for integrating with the AWS ecosystem. By seamlessly connecting Kafka's real-time data streams with services like AWS Lambda, SQS, SNS, and Step Functions, as well as many others that are accessible through Amazon EventBridge, organizations can construct sophisticated, responsive, and loosely coupled applications, ultimately leading to more agile and interconnected data-driven solutions.

Apache®, Apache Kafka®, and Kafka® are registered trademarks of the Apache Software Foundation. No endorsement by the Apache Software Foundation is implied by the use of these marks.