Pipelines retrievers reference

Suggest edits

This reference documentation for Pipelines retrievers includes information on the functions and views available in the aidb extension related to retrievers.

Views

`aidb.retrievers`

The aidb.retrievers view shows information about the retrievers that were created in the database.

Column	Type	Description
id	integer
name	text	Name of the retriever.
vector_table_name	text	Name of the table where the embeddings are stored. Gets newly created if it doesn’t exist. Managed by aidb.
vector_table_key_column	text	Column to use to store the key that references the key in source data when computing embeddings. We recommend using the default and letting aidb manage this table.
vector_table_vector_column	text	Column to store embeddings in. We recommend using the default and letting aidb manage this table.
model_name	text	Name of the model to use for embedding computation and retrievals.
topk	integer	Default number of results to return during a retrieve. Similar to LIMIT in SQL.
distance_operator	aidb.DistanceOperator	During retrieval, the vector operation to use to compare the vectors.
options	jsonb	Currently unused.
processing_mode	aidb.pipelineprocessingmode	Automated processing mode
source_type	text	Type of source data the retriever is working with. Can be 'Table' or 'Volume'.
source_table_name	regclass	Name of the table that has the source data Pipelines computes embeddings for and retrieves from. Only applicable to retrievers configured with aidb.create_retriever_for_table().
source_table_data_column	text	Column name in the source table that Pipelines computes embeddings for. This is also the column that's returned in retrieve operations.
source_table_data_column_type	aidb.RetrieverSourceDataFormat	Type of data the retriever is working with. Uses type `aidb.RetrieverSourceDataFormat`. Only relevant for table-based retrievers. In the case of a volume-based retriever, the format/type information is discovered from the volume.
source_table_key_column	text	Column to use as key for storing the embedding in the vector table. This provides a reference from the embedding to the source data.
source_volume_name	text	Name of the volume to use as a data source. Only applicable to retrievers configured with aidb.create_retriever_for_volume().

Types

`aidb.DistanceOperator`

The aidb.DistanceOperator type is an enum that represents the distance operators that can be used during retrieval.

Value	Description
L2	Euclidean distance
Inner	Inner product
Cosine	Cosine similarity
L1	L1 distance
Hamming	Hamming distance
Jaccard	Jaccard distance

SQL definition:

CREATE TYPE DistanceOperator AS ENUM (
  'L2',
  'InnerProduct',
  'Cosine',
  'L1',
  'Hamming',
  'Jaccard'
);

`aidb.RetrieverSourceDataFormat`

The aidb.RetrieverSourceDataFormat type is an enum that represents the data formats that can be used as source data.

Value	Description
Text	Text data
Image	Image data
Pdf	PDF data

SQL definition:

CREATE TYPE RetrieverSourceDataFormat AS ENUM (
  'Text',
  'Image',
  'Pdf'
);

`aidb.PipelineProcessingMode`

The aidb.PipelineProcessingMode type is an enum used to define which processing mode is configured for a Pipeline (e.g. a Retriever or a Preparer).

Value	Description
Live	New data is processed immediately while being added (using Postgres Triggers)
Background	Currently unused
Disabled	No automated processing

CREATE TYPE PipelineProcessingMode AS ENUM (
	'Live',
	'Background',
	'Disabled'
);

Functions

`aidb.create_retriever_for_table`

Creates a retriever for a given table.

Parameters

Parameter	Type	Default	Description
name	TEXT	Required	Name of the retriever
model_name	TEXT	Required	Name of the model to use
source_table	regclass	Required	Name of the table to use as source
source_data_column	TEXT	Required	Column name in source table to use
source_data_type	aidb.RetrieverSourceDataFormat	Required	Type of data in that column ("Text"."Image","PDF")
source_key_column	TEXT	'id'	Column to use as key to reference the rows
vector_table	TEXT	NULL
vector_data_column	TEXT	'embeddings'
vector_key_column	TEXT	'id'
topk	INTEGER	1
distance_operator	aidb.distanceoperator	'L2'
options	JSONB	'{}'::JSONB	Options
index_type	TEXT	'vector'	Type of index to use for the vector table.

Index_types

If index_type is set to vector, the system will automatically create a hnsw index on the vector table based on the distance operator used in the retriever. This is the default index type. The vector index type is only able to support 2000 dimensions or less. If more dimensions are needed, the index type should be set to disabled.
distance_operator index_type
L2 vector_l2_ops
InnerProduct vector_ip_ops
Cosine vector_cosine_ops
L1 vector_l1_ops
If index_type is set to ivfflat, the system will create a IVFFlat index on the vector table.
If index_type is set to disabled, no index will be created.

distance_operator	index_type
L2	`vector_l2_ops`
InnerProduct	`vector_ip_ops`
Cosine	`vector_cosine_ops`
L1	`vector_l1_ops`

Example

SELECT aidb.create_retriever_for_table(
               name => 'test_retriever',
               model_name => 'simple_model',
               source_table => 'test_source_table',
               source_data_column => 'content',
               source_data_type => 'Text',
       );

`aidb.create_retriever_for_volume`

Creates a retriever for a given PGFS volume.

Parameters

Parameter	Type	Default	Description
name	TEXT	Required	Name of the retriever
model_name	TEXT	Required	Name of the model
source_volume_name	TEXT	Required	Name of the volume
vector_table	TEXT	NULL	Name of the vector table
vector_data_column	TEXT	'embeddings'	Name of the vector column
vector_key_column	TEXT	'id'	Name of the key column
topk	INTEGER	1	Number of results to return
distance_operator	aidb.distanceoperator	'L2'	Distance operator
options	JSONB	'{}'::JSONB	Options
index_type	TEXT	'vector'	Type of index to use for the vector table.

Index_types

If index_type is set to vector, the system will automatically create a hnsw index on the vector table based on the distance operator used in the retriever. This is the default index type. The vector index type is only able to support 2000 dimensions or less. If more dimensions are needed, the index type should be set to disabled.
distance_operator index_type
L2 vector_l2_ops
InnerProduct vector_ip_ops
Cosine vector_cosine_ops
L1 vector_l1_ops
If index_type is set to ivfflat, the system will create a IVFFlat index on the vector table.
If index_type is set to disabled, no index will be created.

distance_operator	index_type
L2	`vector_l2_ops`
InnerProduct	`vector_ip_ops`
Cosine	`vector_cosine_ops`
L1	`vector_l1_ops`

Example

SELECT aidb.create_retriever_for_volume(
               name => 'demo_vol_retriever',
               model_name => 'simple_model',
               source_volume_name => 'demo_bucket_vol'
       );

`aidb.set_retriever_auto_processing`

Sets the processing mode for this Retriever. This function is used to enable and disable auto-embedding: Live mode enables auto-embedding and Disabled disables it.

Parameters

Parameter	Type	Default	Description
retriever_name	TEXT		Name of retriever for which to enable auto-embedding
mode	aidb.PipelineProcessingMode		Desired processing mode

Example

SELECT aidb.set_retriever_auto_processing('test_retriever', 'Live');
SELECT aidb.set_retriever_auto_processing('test_retriever', 'Disabled');

`aidb.bulk_embedding`

Generates embeddings for all data in a given table if there's existing data in the table.

Parameters

Parameter	Type	Default	Description
retriever_name	TEXT		Name of retriever for which to generate embeddings

Example

edb=# select aidb.bulk_embedding('test_retriever');

Output

INFO:  bulk_embedding_text found 3 rows in retriever test_retriever
 bulk_embedding
----------------

(1 row)

`aidb.retrieve_key`

Retrieves a key from matching embeddings without looking up the source data.

Parameters

Parameter	Type	Default	Description
retriever_name	TEXT		Name of retriever to use for retrieval
query_string	TEXT		Query string to use for retrieval
number_of_results	INTEGER	0	Number of results to return

Example

SELECT * FROM aidb.retrieve_key('test_retriever', 'shoes', 2);

Output

key  |      distance
-------+--------------------
 43941 | 0.2938963414490189
 19337 | 0.3023805122617119
(2 rows)

`aidb.retrieve_text`

Retrieves the source text data from matching embeddings by joining the embeddings with the source table.

Parameters

Parameter	Type	Default	Description
retriever_name	TEXT		Name of retriever to use for retrieval
query_string	TEXT		Query string to use for retrieval
number_of_results	INTEGER	0	Number of results to return

Returns

Column	Type	Description
`key`	text	Key of the retrieved data
`value`	text	Value of the retrieved data
`distance`	double precision	Distance of the retrieved data from the query

Example

SELECT * FROM aidb.retrieve_text('test_retriever', 'jacket', 2);

Output

key  |                       value                        |      distance
-------+----------------------------------------------------+--------------------
 19337 | United Colors of Benetton Men Stripes Black Jacket | 0.2994317672742334
 55018 | Lakme 3 in 1 Orchid  Aqua Shine Lip Color          | 0.3804609668507203
(2 rows)

`aidb.delete_retriever`

Deletes only the retriever's configuration from the database.

Parameters

Parameter	Type	Default	Description
retriever_name	TEXT		Name of retriever to delete

Example

select aidb.delete_retriever('test_retriever');

Output

 delete_retriever
------------------

(1 row)

`aidb.create_volume`

Creates a volume from a PGFS storage location for use as a data source in retrievers.

Parameters

Parameter	Type	Description
name	TEXT	Name of the volume to create
server_name	TEXT	Name of the storage location to use for the volume
path	TEXT	Path to the volume in the storage location
mime_type	TEXT	Type of data in the volume (Text or Image)

Example

select aidb.create_volume('demo_bucket_vol', 'demo_bucket', 'demo_bucket/demo_folder', 'Text');

Output

 create_volume
---------------
(1 row)

`aidb.list_volumes`

Lists all the volumes that were created in the database.

Example

select * from aidb.list_volumes();

`aidb.delete_volume`

Deletes a volume from the database.

Parameters

Parameter	Type	Default	Description
volume_name	TEXT		Name of the volume to delete

Example

select aidb.delete_volume('demo_bucket_vol');

Output

 delete_volume
-------------
(1 row)

← Prev

Pipelines reference - Preparers

↑ Up

Pipelines reference

Pipelines PGFS reference

Could this page be better? Report a problem or suggest an addition!

Pipelines retrievers reference

Views

aidb.retrievers

Types

aidb.DistanceOperator

aidb.RetrieverSourceDataFormat

aidb.PipelineProcessingMode

Functions

aidb.create_retriever_for_table

Parameters

Index_types

Example

aidb.create_retriever_for_volume

Parameters

Index_types

Example

aidb.set_retriever_auto_processing

Parameters

Example

aidb.bulk_embedding

Parameters

Example

aidb.retrieve_key

Parameters

Example

aidb.retrieve_text

Parameters

Returns

Example

aidb.delete_retriever

Parameters

Example

aidb.create_volume

Parameters

Example

aidb.list_volumes

Example

aidb.delete_volume

Parameters

Example

← Prev

↑ Up

Next →

`aidb.retrievers`

`aidb.DistanceOperator`

`aidb.RetrieverSourceDataFormat`

`aidb.PipelineProcessingMode`

`aidb.create_retriever_for_table`

`aidb.create_retriever_for_volume`

`aidb.set_retriever_auto_processing`

`aidb.bulk_embedding`

`aidb.retrieve_key`

`aidb.retrieve_text`

`aidb.delete_retriever`

`aidb.create_volume`

`aidb.list_volumes`

`aidb.delete_volume`