First, you may not know the Kinesis Data Firehose service. Here's the AWS definition: Amazon Kinesis Data Firehose is an Extract, Transform, and Load (ETL) service that captures, transforms, and reliably delivers streaming data to data lakes, data stores, and analytics services. (https://aws.amazon.com/kinesis/data-firehose/). - Source: dev.to / over 1 year ago

As you can see in the diagram, we are feeding all events from Event Bus via a catch-all rule into Kinesis Data Firehose. Firehose is a fully managed service that streams into specific destinations like Data Warehouses or Data Lakes. Unlike it's bigger brother of using Kinesis Data Streams directly, there are no setting up of shards and it's mostly configuration free. We are only defining a buffer interval which is... - Source: dev.to / over 1 year ago

When using EventBridge I always log all events to an S3 bucket for auditing, analytics and debugging purposes. A super easy method to do this is to create a Kinesis Data Firehose stream and create a rule that captures all events that points to the Firehose stream. The Firehose stream can then flush the events on S3 in an interval/size of choice based on configuration. - Source: dev.to / almost 2 years ago

Have you looked at Kinesis Firehose? It was pretty much build for this use case although you will still need to see if you can define a partitioning scheme probably in combination with an S3 Select query to meet your query requirements. https://aws.amazon.com/kinesis/data-firehose/?nc=sn&loc=0. - Source: Hacker News / almost 2 years ago

Is continuous backup important ? e.g. If the stuff fails for one day and you lose that day's upload is that ok? Do you want it to push updates more frequently than once a day? If you want to continuously push updates then Kinesis Firehose might be worth looking into. Source: over 2 years ago

Hi u/CoolFounder! I think hotglue (https://hotglue.xyz) could fit your use case super well - I would be happy to chat about it. Source: over 2 years ago

If you would like to check out the easiest way to offer your users native integrations inside of your product, you can check out hotglue’s site here. - Source: dev.to / over 2 years ago

Hey jumping in here as someone who works at a company that helps people building products offer integrations to their users (hotglue). Wanted to ask why you thought of building the integration yourself instead of using a tool to do it? Source: over 2 years ago

I’m a little biased here as one of the co-founders, but hotglue is meant to help companies like yours get data in from your customers! Feel free to give us a look at https://hotglue.xyz - happy to answer any questions you have about integrations in general as well. - Source: Hacker News / over 2 years ago

I haven’t experienced using AirByte yet but based on the use case I believe you’re mentioning (user-facing integrations in your product), I think hotglue (https://hotglue.xyz) may be a good fit. Disclaimer: I am one of the founders but thought I’d jump in here in case you hadn’t heard about us yet! - Source: Hacker News / over 2 years ago