sycamore

Why we included this project

Sycamore sits in the gap between raw document files and the vector stores or search engines you actually query. It is an open source processing engine that partitions PDFs, presentations, transcripts, and manuals, then enriches and chunks them so the result is ready to load into OpenSearch, Elasticsearch, Pinecone, Qdrant, Weaviate, or DuckDB. The DocSet abstraction is the interesting part: you describe partitioning, cleaning, and loading as a declarative dataflow, which makes it easy to try different chunking strategies without writing glue code. You can also pick which LLMs power the enrichment transforms. Aryn's DocParse service and an open DETR vision model handle complex pages with embedded tables and figures, so the semantic structure of the document survives into the chunks. Teams building RAG or hybrid search over messy enterprise documents will appreciate the notebook-driven workflow and the Ray backend for moving from prototype to production.