Understand the Core Ingestion Pipeline Components
The Ingestion Pipeline is composed of the following core components:
Create an Ingestion Pipeline
An ingestion pipeline parses documents, splits content into chunks, optionally enriches or compiles it, and sends the results to an index. Create and manage pipelines under Agent > Ingestion pipeline.
Configure the Parser Component
Parser extracts document content into structured items for a downstream Chunker. The available controls depend on the file category and the selected parser.
Configure the Chunker Component
After Parser, choose Token Chunker or Title Chunker through the outgoing connection point. After the Chunker, add Transformer or Compiler to enrich the chunks, or Indexer to index them.
Configure the Transformer Component
Transformer uses an LLM to enrich document chunks with keywords, questions, tags, summaries, and metadata. Configure it on the Agent > Ingestion pipeline canvas. One Transformer can enable several extraction functions at the same time.
Configure the Indexer Component
Indexer prepares chunk data for full-text and embedding search. Connect it after the Chunker, Transformer, or Compiler and use the pipeline with a knowledge base to index documents.
Validate the Pipeline
Publish a pipeline version, then parse a sample document in a connected knowledge base to check the results. The pipeline canvas has no Run button; document processing is started from the knowledge base.
Connect the Pipeline to a Knowledge Base
Publish a pipeline version before applying a custom pipeline to a Dataset. Saving a draft alone does not make it selectable for document processing.