Pachyderm

Rethinking Data in AI and ML

Comments: Like any tool, Pachyderm is no silver bullet for the entire AI/ML stack. However, from a data processing and management perspective, it has fulfilled every application requirement I've needed it for and continues to be a flexible tool in meeting additional requirements. For example, after having computed some results from a pipeline, I needed to serve these results to an existing application. Pachyderm made this simple by exposing the data through a built-in S3 REST API. Since the application was already compatible with S3, Pachyderm served as a drop-in replacement for an S3 bucket. For anyone that strives to design clean and straightforward AI/ML architectures, I can definitely recommend Pachyderm as a must for the foundational data component.

Pros:

AI/ML production systems typically consist of multiple data processing steps organized as a DAG. Many automation frameworks manage these DAGs as tightly coupled steps ordered by _code execution_. What I like so much about Pachyderm is that it approaches DAG management as loosely coupled steps ordered by _data dependencies_. This alternative way of thinking has enabled me to design AI/ML architectures with data at the center, which has revolutionized the development and production workflows I've participated in. I can confidently store, process, and otherwise manage the data because Pachyderm provides a solid foundation for data provenance, data versioning, data storage patterns, and efficient incremental processing. Since AI/ML models are effectively a form of data, model versioning and management can be built as an extension of Pachyderm's data foundation. Furthermore, I really like that Pachyderm is powered by Kubernetes, because it passes on important architectural properties to Pachyderm, such as high scalability, robustness, efficiency, and portability (i.e. cloud agnosticism). I can containerize my pipelines, quickly test them locally through Docker Desktop or minikube, then scale them up to massive amounts of data in an on-prem or cloud cluster. If autoscaling is supported in a cloud cluster, I can especially reap the benefits of cost efficiency because I only pay for the compute resources I use.

Cons:

- In 1.X versions of Pachyderm, there are a few performance pain points, especially around handling very small files when uploading/downloading to/from a repo. These pain points have been significantly improved in Pachyderm 2.X. - Also in 1.X, debugging pipeline failures can sometimes be challenging without extra tools or integrating external logging services. Pachyderm 2.X improves upon this as well. - When Pachyderm processes data files in a pipeline, it groups the files into logical structures called datums for provenance and data efficiency reasons, and then it invokes the pipeline on each datum. This is necessary for scalability, but the downside is that each invocation of the pipeline incurs an overhead cost of just starting the processing code. The bright side is that there are several straightforward ways to engineer around the problem. It's also important to recognize that the impact of the problem is minimized by the benefits of incremental processing(i.e. only processing data that has changed on future pipeline runs). - This isn't necessarily a problem, but prospective buyers should be aware that although compute costs may go down due to incremental processing, storage costs may go up due to storing multiple versions of data.

Game changer for handling dynamic data

Comments: Pachyderm meets many previously unmet needs for our organization, including complete data provenance, automatic handling of data change, and modular/portable processing architecture, which facilitates the joint development of processing pipelines between software developers and scientists. Pachyderm engineers have been extremely responsive to our issues and development requests, and we plan to work well into the future with this software.

Pros:

Perhaps the most important aspect we benefit from operationally is the awareness and automatic handling of data change. Generation of our data products involves multiple processing steps and several sources of data and metadata that enter the processing sequence at various points and may change at any time. Pachyderm automatically knows what has changed and triggers downstream (re)processing, removing the need for error-prone human management.

Cons:

In Pachyderm 1.X there was a relatively high amount of overhead associated with processing each datum. Our data typically consists of small but numerous datums, and we needed to artificially combine datums for performance. However, Pachyderm has been working with us on this issue and we expect to see big improvements in 2.0 and beyond.

Great in theory

Comments: We achieved some of our goals with Pachyderm. However, we were really hoping to spend more time on solving the problems directly related with our goal. Instead, we spent a significant amount on time solving problems with Pachyderm and tailoring our problem to it.

Pros:

Great concept, really fits what we would like to do. Re-computing only the pieces where the data has changed is super valuable.

Cons:

Working with it in practice is very hard. We would like to use Pachyderm also for research, developing research pipelines that can be executed easily on big amounts of data on the cluster. However, during research/development, pipelines naturally crash often. Translating something that works locally to something that works in pachyderm has several scenarios in which it can fail. Inspecting those types of errors is incredibly difficult, unless you invest a significant amount of time into setting up logging/monitoring manually.

Pachyderm is a great data processing platform on cloud.

Comments: We have used Pachyderm for more than a year. Overall experience is Good. We love the core technology and features provided by Pachyderm. We experienced frustrated issues, like the download speed, deployment, system stability. We get excellent support from the Pachyderm team all the time.

Pros:

Data Driven Automation. It supports incremental data processing. Reproducibility. Perfectly match our tech stacks: K8s, S3. Community facing.

Cons:

We expect fully automated data replication/export to external storage system. The logging & debugging support could be improved.

Scalable machine learning without the mlops

Pros:

The ability to scale model builds in native python is something that has been missing in this space until now. Utilizing spark and/or dask comes with a large amount of overhead that can be avoided leveraging pachyderm.

Cons:

The learning curve is quite steep since there are some core concepts that are foundational to understand before using pachyderm.

Pachyderm for data pipelines

Pros:

Pachyderm pipelines are an intuitive way to split and process data concurrently using autoscaling compute clusters. Writing a program to interact with data in a pipeline is straightforward due to working similar to a native filesystem, requiring no additional libraries or integrations.

Cons:

We ran into issues with Pachyderm that required deleting and recreating pipelines. As an upside, support was very responsive to resolving our problems and providing upgrades to Pachyderm.

The missing ingredient for reproducible research

Comments: I'm a big fan of the pachyderm approach; it's young software and needs to be understood a little to get the best out of it; but when stuff works, it works so damn well.

Pros:

The systematic recording of provenance for training and benchmarking results.

Cons:

When things go wrong, it's hard to diagnose.