Meltano Reviews

Amazon Kinesis Firehose Reviews

Meltano mentions (26)

• AI product development is being held back by data engineering

  Hey HN, Arch CEO here! Our team has been working at the intersection of data engineering and software engineering for a few years now with Meltano (https://meltano.com), and this year, the rise in Generative AI has made it clear that the bottleneck in unlocking the potential value of data has shifted from data integration on data teams to data engineering on software teams, so we’ve decided to do something about... - Source: Hacker News / 8 months ago

• How useful is Airbytes in production pipelines?

  We use Meltano for (EL) and Prefect for scheduling. Is not click-ops, but works very well for us! Behind the scenes Meltano wraps up Singer spec similarly like Airbyte does with its connectors. Before that we tried Airbyte (~5 months ago?) and it was so bad.. We could not choose the columns to replicate and the connectors were unstable i.e. Skipping data, all sort of odd errors and so on.. Source: about 1 year ago

• Ask HN: Who is hiring? (May 2023)

  Meltano's all-remote team and community of thousands are on a mission to enable everyone to realize the full potential of their data. To this end, we are bringing software engineering best practices to data teams in the form of an open-source DataOps platform that we envision becoming the foundation of every team's ideal data stack. Our public company handbook (https://handbook.meltano.com/) has all the details on... - Source: Hacker News / about 1 year ago

• Ask HN: Who is hiring? (April 2023)

  Meltano | Full-Time | Remote | https://meltano.com Meltano's all-remote team and community of thousands are on a mission to enable everyone to realize the full potential of their data. To this end, we are bringing software engineering best practices to data teams in the form of an open-source DataOps platform that we envision becoming the foundation of every team's ideal data stack. Our public company handbook... - Source: Hacker News / about 1 year ago

• If dbt is the "T" part of an "ELT", what do you use for "EL"?

  We switched from AWS Glue to Meltano for the EL part of ELT and it's been a joy to use. We're moving so much faster now. Source: over 1 year ago

Amazon Kinesis Firehose mentions (6)

• What is an event-driven architecture and why storing events is important ?

  First, you may not know the Kinesis Data Firehose service. Here's the AWS definition: Amazon Kinesis Data Firehose is an Extract, Transform, and Load (ETL) service that captures, transforms, and reliably delivers streaming data to data lakes, data stores, and analytics services. (https://aws.amazon.com/kinesis/data-firehose/). - Source: dev.to / over 1 year ago

• Serverless Event Driven AI as a Service

  As you can see in the diagram, we are feeding all events from Event Bus via a catch-all rule into Kinesis Data Firehose. Firehose is a fully managed service that streams into specific destinations like Data Warehouses or Data Lakes. Unlike it's bigger brother of using Kinesis Data Streams directly, there are no setting up of shards and it's mostly configuration free. We are only defining a buffer interval which is... - Source: dev.to / over 1 year ago

• Logging EventBridge events to S3 with Firehose

  When using EventBridge I always log all events to an S3 bucket for auditing, analytics and debugging purposes. A super easy method to do this is to create a Kinesis Data Firehose stream and create a rule that captures all events that points to the Firehose stream. The Firehose stream can then flush the events on S3 in an interval/size of choice based on configuration. - Source: dev.to / almost 2 years ago

• S3 Isn't Getting Cheaper

  Have you looked at Kinesis Firehose? It was pretty much build for this use case although you will still need to see if you can define a partitioning scheme probably in combination with an S3 Select query to meet your query requirements. https://aws.amazon.com/kinesis/data-firehose/?nc=sn&loc=0. - Source: Hacker News / almost 2 years ago

• Advice on S3 Application

  Is continuous backup important ? e.g. If the stuff fails for one day and you lose that day's upload is that ok? Do you want it to push updates more frequently than once a day? If you want to continuously push updates then Kinesis Firehose might be worth looking into. Source: over 2 years ago