Apache Airflow

Apache Airflow Review

Pros:

Easy to use, understandable interface and very fast to run

Cons:

very high costs for its use, A long waiting time on your generator

Deploy/schedule/manage DAG's easily with Airflow

Comments: A stable open source product solution to manage and execute batch jobs.

Pros:

Airflow being served as a product in leading multi cloud environment is one of the big plus . I also comes at a cheaper rate with lots of features to manage the daily data load pipelines.

Cons:

The UI could have been improved by showing statistics of the task in more cleaner way. The version management of the batch that gets deployed should definitely be improved.

Take control of your data processing workflows with Apache Airflow.

Comments: Apache Airflow is a highly useful platform for orchestrating and coordinating complex data processing workflows. While there may be a steep learning curve and some configuration issues, the benefits in terms of flexibility, scalability, and monitoring make it a valuable tool for many data and analytics teams.

Pros:

Airflow is highly flexible and customizable, making it suitable for a wide range of use cases. Users can define their own tasks and workflows and tailor them according to their specific needs. It is designed to handle large workflows and big data sets. It can scale horizontally across multiple servers to handle large volumes of data processing. In addition, it provides a web user interface for monitoring and managing workflows in real-time. Users can check the status of their tasks and receive alerts about any issues or failures. It has a large community of developers and users, which means there is a wealth of documentation, tutorials, and resources available online.

Cons:

Airflow can be difficult to learn and properly configure, especially for those who do not have prior experience in programming or server administration. Some users have reported issues when setting up Airflow in different environments, which can result in stability or performance problems. Even though Airflow can work with a diverse range of tools and technologies, it may not be compatible with certain tools used by certain teams.

Review of Airflow

Comments: Airflow is a well known and well implemented scheduler that can be use by developpers but also other non IT collabs.

Pros:

Airflow is one of the top scheduler on the market. It's easy to implement your python script for the deployment in production.

Cons:

The only lack on Airflow so far is that it doesn't give you an overview on all you repository.

The best workflow management tool that is loaded with features.

Comments: It deals with the challenge of managing processes and acts as a bridge to allow data to move from the source to the extremely dynamic destination without a lot of human intervention, allowing for straightforward problem-solving when it occurs.

Pros:

I have noticed that Apache has been releasing constant updates to Airflow and has been making it more feature rich and useful, so kudos to the updates and support. Enables the management of a variety of tasks, workflows, and dependencies, including the monitoring of them. It is possible to create original logic and routines that can be rapidly combined with other tools with just a few lines of code. Sensitive data can be managed when the right roles and privileges are given, which is really helpful.

Cons:

When the team gets bigger, there can be a tonne of DAGs from different teams filling up the DAG folder; organising these DAGs can be challenging and messy.

A Powerful open-source ETL Solution that doesn't stretch your pockets.

Pros:

Overall, Apache Airflow is a powerful platform for creating and managing data workflows. But what I like the most is that it is open-source and also allows us to define workflows as code, which makes it easy to maintain, version control, and share with others. This also makes it easy to track changes and roll back changes when necessary.

Cons:

The only thing I don't like about airflow is that if someone is not so good at programming for them the learning curve is steep and because it is open-source we need to do custom setup on our own servers which is good but also complex.

easy platform

Pros:

the SW helps and easy to use, no special training requires for using it.

Cons:

the main problem in this SW is the communication that fails often

Review of Apache Airflow

Comments: Airflow solved several problems for our business, namely notifications and retry behavior. In addition, it offered integrations with several services that lowered the amount of development time or knowledge we needed. We have since transitioned most of our standalone scripts to this system simply because it does everything we have needed it to.

Pros:

Apache Airflow made our automations a lot more straightforward to maintain. There were issues we were seeing where our cronjobs were occasionally failing and there were no methods in place to automatically notify on these issues. Similarly, we also were looking for a solution to retry scripts if they failed, especially for tasks that were running overnight. The UI was also a pleasant upgrade from our CLI-based task scheduling system.

Cons:

The scheduling aspect could occasionally be slow, especially depending on how many tasks you may have. If you require tasks to be scheduled exactly at a certain time, there could be several second delays before the scheduler will pick up tasks so that should be kept in mind. The learning curve on some parts of Airflow could also be confusing if trying to learn or teach new developers.

Powerful workflow orchestration tool

Comments: Overall, Airflow has allowed us to develop and maintain a large number of automated workflows, both for data engineering and data science. It gave us a force multiplier for each data engineering hire which makes our team a lot more efficient compared to before.

Pros:

I like the whole concept of DAGs which was made prominent, I think, by Airflow in the data engineering workspace. Airflow is really good at this, both in executing and also in monitoring. It makes dev and ops a lot easier for both data engineering and data science workflows.

Cons:

Task scheduling and execution used to be a bit slow, but this has been fixed since version 2.0. So far, I don't have anything else to not like about Airflow.

Versatile to set up any workflow

Comments: Helped our data engineering teams to seamlessly build and monitor pipelines

Pros:

great UI; good integration with others; just python knowledge is required

Cons:

No versioning; architecture is a little difficult for first time users

Arvind Pant’s review

Comments: Overall I like airflow. Very easy to implement and monitor each task.

Pros:

Things I like the most1.Task level monitoring is amazing. 2. Dag implementation is very easy

Cons:

There is no task level notification anything fails. Also need integration with tools like slack for failure notifications.