Knowledge bases Innovation Release
A knowledge base is a vector store created by a pipeline whose last step is KnowledgeBase. It holds embeddings generated from your source data and supports semantic similarity search for retrieval-augmented generation (RAG) and other AI workloads.
Knowledge base list
Navigate to Sovereign AI > Knowledge Bases to see all knowledge bases in your project. Each entry shows:
Name: The knowledge base identifier, derived from the pipeline name.
Origin: The cluster and database where the knowledge base resides.
Model: The embedding model used to generate vectors.
Source: The source table feeding the pipeline.
Processing mode: The pipeline's current processing mode (On Demand, Live, or Background).
Status: The pipeline's processing status (Up To Date, Processing, Stale, Has Errors, Failed, and others).
Knowledge base detail
Select a knowledge base to view its detail page.
Record counts
| Metric | Description |
|---|---|
| Source records | Total rows in the source table |
| Embeddings | Number of vector embeddings stored in the knowledge base |
| Unprocessed records | Source rows that haven't yet been processed |
When the pipeline is operating as expected, the unprocessed count is zero (for Background and Live modes) or equal to the source count minus embeddings (for On Demand mode).
Associated pipeline
The detail page shows the pipeline that feeds this knowledge base, including its step configuration and processing parameters. In the current release, each knowledge base is associated with exactly one pipeline (1:1 relationship).
Multi-pipeline knowledge bases
AI Database (AIDB) 7 supports multiple pipelines feeding a single knowledge base (N:1 relationship). This capability isn't yet available through Pipeline Designer and is planned for a future release. See Limitations for details. When available, the knowledge base detail page will list all associated pipelines.
Testing retrieval
The knowledge base detail page includes a built-in query tool for testing semantic search without writing SQL.
Running a test query
On the knowledge base detail page, select Run Test Query.
In the Perform Test Query Retrieval dialog, enter your search text in the Search Query field.
Optionally adjust the Number of Results parameter.
Click Run Query.
The query tool encodes your text using the knowledge base's embedding model, performs a vector similarity search, and returns the top matching records with their distance scores. Lower distance scores indicate closer semantic matches. The results appear in the same dialog beneath the query fields, with each matching record shown as a row containing the source key and its distance score.
Interpreting results
Each result includes:
Source key: The key column value from the original source row, allowing you to trace results back to source data.
Distance: The similarity distance between the query vector and the result vector, using the knowledge base's configured distance operator (L2 by default).
Test queries are useful for validating that your pipeline is producing meaningful embeddings and that the chunking strategy produces segments of appropriate granularity. If results seem poor, consider adjusting the ChunkText step's desired length or trying a different embedding model.
Knowledge base lifecycle
Knowledge bases are created automatically when you deploy a pipeline with a KnowledgeBase terminal step. They can't be created independently through Pipeline Designer.
When you delete a pipeline that has a KnowledgeBase step, the associated knowledge base and its vector table are also deleted. All stored embeddings are permanently removed.
Warning
Deleting a knowledge base is irreversible. All vector data is permanently removed. If the knowledge base is used by Langflow flows or other consumers, those integrations will break.
Using knowledge bases with Langflow
Knowledge bases created through Pipeline Designer are accessible from the Langflow-based Gen AI builder. The Langflow KB component connects to AIDB knowledge bases using the same aidb.retrieve_key() and aidb.retrieve_text() functions that power the Pipeline Designer query tool.
To use a Pipeline Designer knowledge base in Langflow:
Create and populate a knowledge base through Pipeline Designer as described above.
In Langflow, add a KB component to your flow.
Configure the component with the database connection details and the knowledge base name.
The component queries the knowledge base for semantically similar content based on the flow's input.
Langflow database identity
Langflow components that connect to Postgres use the database credentials configured within the flow, not a system-managed role. The Postgres role used by a Langflow flow determines which knowledge bases and tables it can access through standard Postgres privilege rules. If the configured role does not have SELECT access to the knowledge base's underlying vector table, queries will fail with a permission error. Granting the Langflow role inheritance from visual_pipeline_user (for example, GRANT visual_pipeline_user TO langflow_role) provides access to all VPU-owned objects.
Troubleshooting
No embeddings after pipeline runs
If the source record count is positive but the embedding count remains zero:
Check the pipeline status for errors. "Has Errors" or "Failed" status indicates processing problems.
Verify that the embedding model is reachable from the pipeline's cluster. Hybrid Manager (HM)-hosted and HM-proxied models are only reachable from primary-location clusters. If the pipeline runs on a secondary location or self-managed cluster, those models will fail. See Executing pipelines: How models reach pipeline steps.
If using an HM-hosted model on the primary location, confirm the KServe InferenceService is healthy.
Check that the source data column contains valid content for the configured pipeline steps.
Review the AIDB error log for detailed error messages. Connect to the database and run
SELECT * FROM aidb.get_error_logs('your_pipeline_name');.
Unprocessed count not decreasing
If the unprocessed count remains static:
Verify the processing mode isn't On Demand. If On Demand, trigger a manual run or switch to Background or Live.
For Background mode, confirm the sync interval has elapsed since the last data change.
Check for pipeline-blocking errors that halt processing entirely.
Poor retrieval quality
If test queries return irrelevant results:
Review your chunking strategy. Chunks that are too large or too small produce suboptimal embeddings.
Consider a different embedding model. Some models perform better on specific content types.
Verify that the source data is clean and relevant. HTML tags, boilerplate text, or corrupt content degrade embedding quality. Consider adding a ParseHTML step before chunking.