AWS Batch Reviews

Dagster Reviews

AWS Batch mentions (14)

• Looking for a decent (self hostable) program to orchestrate scripts, notify on failures, etc

  After moving off Jenkins, I moved everything to AWS Batch with Fargate. This works quite well, but it is proving to be a little expensive, as I have to pay for:. Source: 12 months ago

• Hosting strategy suggestions

  If you're looking for more control over your infrastructure and want to run a full computing environment, EC2 might be the right choice for you. With EC2, you have complete control over the operating system, network, and storage, which can be useful if you need to install custom software or use specific hardware configurations. Additionally, EC2 + Batch processing provide a wider range of instance types, including... Source: about 1 year ago

• Questions for bioinformatics researchers that use AWS

  AWS Batch is the equivalent of a university cluster you submit to with slurm/sge/lsf/etc. But does not use those schedulers as AWS has their own. Source: over 1 year ago

• Scheduling "Fetch & Run" Batch Jobs with AWS Batch and CloudWatch Rules

  Developers frequently use batch computing to access significant amounts of processing power. You may perform batch computing workloads in the AWS Cloud with the aid of AWS Batch, a fully managed service provided by AWS. It is a powerful solution that can plan, schedule, and execute containerized batch or machine learning workloads across the entire spectrum of AWS compute capabilities, including Amazon ECS, Amazon... - Source: dev.to / over 1 year ago

• can you run OS applications in lambda layers?

  As others mentioned, you *can*. It might be easier with AWS Batch (https://aws.amazon.com/batch/) depending on what you're trying to do. Source: over 1 year ago

Dagster mentions (3)

• How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

  Instead of the custom orchestrator I used, a proper orchestration tool should replace it like Apache Airflow, Dagster, ..., etc. - Source: dev.to / 10 days ago

• AI Strategy Guide: How to Scale AI Across Your Business

  Level 1 of MLOps is when you've put each lifecycle stage and their intefaces in an automated pipeline. The pipeline could be a python or bash script, or it could be a directed acyclic graph run by some orchestration framework like Airflow, dagster or one of the cloud-provider offerings. AI- or data-specific platforms like MLflow, ClearML and dvc also feature pipeline capabilities. - Source: dev.to / about 2 months ago

• What are some open-source ML pipeline managers that are easy to use?

  I would recommend the following: - https://www.mage.ai/ - https://dagster.io/ - https://www.prefect.io/ - https://metaflow.org/ - https://zenml.io/home. Source: about 1 year ago