TopOpenSourceAI
zh
← Back to scenarios

AI Data Infrastructure & Storage

Open-source storage engines, data fabrics, and infrastructure layers engineered for high-throughput AI pipelines, covering checkpointing, dataloading, and KV-cache systems.

42 projects

See methodology for ranking rules; order uses public GitHub metrics within this scenario.

21–40 of 42

Rank Project Stars Forks
21 datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

2.8K 156
22 supabase-py

Python Client for Supabase. Query Postgres from Flask, Django, FastAPI. Python user authentication, security policies, edge functions, file storage, and realtime data streaming. Good first issue.

2.6K 540
23 spider

Get web data for AI agents and LLMs - fast, efficient, and reliable with Rust

2.7K 284
24 openlake

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

2.3K 417
25 OpenMLDB

OpenMLDB is an open-source machine learning database that provides a feature platform computing consistent features for training and inference.

1.7K 332
26 alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud

7.2K 2.9K
27 infinispan

Infinispan is an open source data grid platform and highly scalable NoSQL cloud data store.

1.3K 652
28 rocketmq-rust

🚀Apache RocketMQ build in Rust🦀. Faster, safer, and with lower memory usage. ⭐ Star to support our work❤️!

1.5K 245
29 pixeltable

Unified multimodal backend for AI data apps

1.6K 219
30 datalevin

A simple, fast and versatile Datalog database

1.5K 85
31 database-build

In-browser Postgres sandbox with AI assistance (formerly postgres.new)

3.0K 274
32 nocturne_memory

A lightweight, rollbackable, and visual Long-Term Memory Server for MCP Agents. Say goodbye to Vector RAG and amnesia. Empower your AI with persistent, graph-like structured memory across any model, session, or tool. Drop-in replacement for OpenClaw.

1.3K 166
33 duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

1.3K 93
34 ai-data-extraction

extract all your personal data history from cursor, codex, claude-code, windsurf, and trae

1.3K 106
35 dataclaw

Agent harness to publish your agent chat history as Huggingface datasets.

2.1K 233
36 docarray

Represent, send, store and search multimodal data

3.1K 243
37 fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

2.2K 103
38 webdataset

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

3.2K 234
39 postgresml

Postgres with GPUs for ML/AI apps.

6.8K 364
40 img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

4.4K 375
< Previous
/ 3
Next >