Daft

Why we included this project

Teams that build training or inference pipelines from raw media usually end up wiring together separate tools for images, audio, video, and the tabular metadata that describes them. Daft folds that into one dataframe API: you load a folder of images from S3, run an embedding model or an LLM prompt over the rows, and write the result to Parquet or Iceberg without leaving the framework. The Python interface sits on a Rust core, so you get vectorized execution and out-of-core processing and skip the JVM overhead of Spark-style stacks. It scales from a laptop to distributed clusters on Ray or Kubernetes, which matters for small teams that want a single code path from prototype to production. If multimodal ETL, dataset curation, or feature generation at scale is your daily work, this is worth a serious look.