How your documents become searchable

The librarian who files by meaning has one quirk: it files index cards, not books. Hand it a 200-page PDF and it cannot do anything useful until someone transcribes the book and cuts the text into cards of a readable size. Ingestion is that someone. It opens whatever you hand it, a PDF, a Word document, plain text, or the address of a web page, extracts the words, and cuts them into passages called chunks. Each chunk gets its own meaning fingerprint and its own place in your node's knowledge base.

Every card also gets a note on the back recording exactly where it came from: which document, which page for PDFs, and the precise character range within the text. Search results carry that origin note with them, so a match is never just a floating passage; your node always knows which document and page it is quoting.

Handing your node the same document twice costs nothing. It recognizes content it has already filed and skips it instead of doing the work again.

Where the scissors fall matters. Cut too coarsely and a card holds three unrelated topics, which blurs its fingerprint; cut too finely and no card holds enough context to answer anything. You choose the strategy per upload:

Fixed-size cuts every chunk to the same length, with a configurable overlap so a fact sitting on a boundary appears intact on both neighboring cards. It is predictable and fast, and works well for logs and uniform reference material.

Sentence-boundary, the default, packs whole sentences up to the size target and never cuts mid-sentence, so every card reads naturally. It also carries trailing sentences over as overlap, keeping context connected across cards.

Semantic asks the embedding model itself where the scissors go. Adjacent sentences are compared by meaning, and a new chunk starts where the topic shifts. It costs more compute at upload time, and rewards documents that move between subjects, like meeting notes and policy manuals.

Parsing, chunking, and embedding all run in-process on the platform, using the same verified, in-process model described in Semantic Search. Your documents are not sent to a parsing service or an embedding API, and there is no per-page or per-token fee.

Web page ingestion is deliberately suspicious. Before your node fetches a URL, and again at every redirect, it checks that the address resolves somewhere public. A page that tries to redirect your node toward private infrastructure, such as internal services or cloud metadata endpoints, is refused. Every ingestion is also written to the platform's audit trail: who ingested what, into which node, and how it was chunked.

Ingestion reads text, not pictures of text: a scanned PDF with no embedded text layer yields nothing, because there is no OCR step. Complex layouts flatten; a table survives as words but loses its grid. Sentence detection is a heuristic and will occasionally split at an abbreviation. Web ingestion reads the page as served, so content behind logins or rendered entirely by JavaScript is out of reach, and uploads are capped at 10 MB per document.