datachain

Why we included this project

Large piles of images, PDFs, or other files in object storage are awkward to query, version, or hand to an AI agent in any reproducible way. DataChain keeps the bytes in S3, GCS, or Azure and builds typed, versioned datasets over them, letting you filter, join, and similarity-search millions of records at sub-second speed without moving anything. Its parallel Python engine runs your own functions across the files with checkpoint recovery and incremental updates, so a rerun only processes what changed. For agent workflows it can generate an LLM-readable knowledge base and serve it over MCP, which is how tools like Claude Code or Codex get grounded context on your real data instead of a stale snapshot. ML engineers and agent builders dealing with large unstructured collections will find this worth trying.