Understand the Core Ingestion Pipeline Components
The Ingestion Pipeline is composed of the following core components:
Create an Ingestion Pipeline
1. Go to the Agent page.
Configure the Parser Component
The Parser component converts your files into structured text while preserving layout, tables, headers and other formatting.
Configure the Chunker Component
The Chunker component intelligently splits text. Its goal is to prevent overflow of the AI context window and improve semantic accuracy in hybrid search.
Configure the Transformer Component
The Transformer component is designed to bridge the "semantic gap". In general, it uses AI models to add semantic metadata, making your content easier to discover during retrieval.
Configure the Indexer Component
The Indexer component indexes data for optimal retrieval. It is the final step, writing processed data into search engines such as Infinity, Elasticsearch and OpenSearch.
Test Run
On the pipeline canvas, click Run to upload a sample file and view step-by-step results.
Connect the Pipeline to a Knowledge Base
1. When creating or editing a knowledge base, find the parsing method section.