indexers.main

indexers.main

Provides functionality for creating a VectorStore from a CSV file.

Defines the VectorStore class, which is used to model and create vector databases from CSV (text) files using a Vectoriser object.

This class requires a Vectoriser object from the vectorisers submodule to convert the CSV’s text data into vector embeddings which are then stored in the VectorStore objects.

Key Features: - Batch processing of input files to handle large datasets. - Support for CSV file format - Integration with a custom embedder for generating vector embeddings. - Support for user-defined hooks for preprocessing and postprocessing. - Logging for tracking progress and handling errors during processing.

VectorStore Class:

  • The VectorStore class is initialised with a Vectoriser object and a CSV knowledgebase.
  • Additional columns in the CSV may be specified as metadata to be included in the vector database.
  • Upon creation, the VectorStore is saved in parquet format for efficient, and quick reloading via the VectorStore’s .from_filespace() method.
  • A new piece of text data (or label) can be queried against the VectorStore in the following ways:
    • .search(): to find the most semantically similar pieces of text in the vector database.
    • .reverse_search(): to find all examples in the knowledgebase that have a given label.
    • .embed(): to generate a vector embedding for a given piece of text data using the vectoriser.
  • ‘Hook’ methods may be specified to perform pre-processing on input data before embedding, and post-processing on the output of the search methods.

Classes

Name Description
VectorStore Models and creates vector databases from CSV text files.

VectorStore

indexers.main.VectorStore(
    file_name,
    data_type,
    vectoriser,
    batch_size=_BATCH_SIZE,
    meta_data=None,
    output_dir=None,
    overwrite=False,
    skip_save=False,
    hooks=None,
    quiet_mode=False,
)

Models and creates vector databases from CSV text files.

Converts a knowledgebase (CSV file) to a DataFrame, embeds the text column in batches, storing the resulting vectors for querying. Once built, the store supports semantic search via .search(), label-based lookup via .reverse_search(), and direct embedding via .embed(). The index can be persisted to disk and reloaded later using .from_filespace().

Attributes

Name Type Description
file_name str Path to the input file used to build the vectors dataframe.
data_type str Format of the input file. Currently only “csv” is supported.
vectoriser VectoriserBase Vectoriser instance used to convert text into vector embeddings.
batch_size int The batch size to pass to the vectoriser when embedding.
meta_data dict | None Mapping of extra CSV column names to extract to their Python types (e.g. {“source”: str}). Values are Python types.
output_dir str | None Directory where vectors.parquet and metadata.json are written. Defaults to the input file stem when None is passed. Ignored when skip_save=True.
skip_save bool If False, saves the VectorStore to disk after creation. If True, keeps it in memory only (for testing or ephemeral use cases). Defaults to False.
vectors pl.DataFrame | None Polars DataFrame containing the full knowledgebase table with columns: label, text, uuid, embeddings, and any columns specified in meta_data. None until the index is built.
vector_shape int Number of dimensions in the vector embeddings.
num_vectors int Total number of rows stored in the VectorStore.
vectoriser_class str The type of Vectoriser used to create embeddings.
hooks dict A dictionary of user-defined hooks for preprocessing and postprocessing.
quiet_mode bool Whether to minimise verbose output, such as progress bars.

Methods

Name Description
embed Generates vector embeddings from a VectorStoreEmbedInput object.
from_filespace Creates a VectorStore instance from a saved filespace folder.
reverse_search Looks up documents in vectors by label.
search Queries the vectors attribute for the most similar documents.
embed
indexers.main.VectorStore.embed(query)

Generates vector embeddings from a VectorStoreEmbedInput object.

Accepts a VectorStoreEmbedInput object and generates vector embeddings for its text content using the vectoriser attribute. Any preprocessing hooks set on the instance are applied to the input before embedding, and any postprocessing hooks are applied to the output before it is returned.

Parameters
Name Type Description Default
query VectorStoreEmbedInput Input object containing the text to be embedded and their corresponding ids. required
Returns
Name Type Description
VectorStoreEmbedOutput VectorStoreEmbedOutput Output object containing the generated embeddings together with their corresponding ids and original texts.
Raises
Name Type Description
DataValidationError If invalid arguments are passed.
HookError If a preprocessing or postprocessing hook raises an exception.
ClassifaiError If the embedding operation fails.
from_filespace
indexers.main.VectorStore.from_filespace(
    folder_path,
    vectoriser,
    batch_size=None,
    hooks=None,
    quiet_mode=False,
)

Creates a VectorStore instance from a saved filespace folder.

Reads metadata.json and vectors.parquet from folder_path using fsspec, so both local and remote paths (e.g. gs://) are supported. The vectoriser class name stored in metadata.json must match the class name of the supplied vectoriser object. The instance is constructed via object.__new__, so init is never called and no embeddings are generated.

Note: the returned instance does not have output_dir or skip_save attributes set. vector_shape and num_vectors are read directly from metadata.json without being cross-checked against the actual contents of the parquet file.

WarningKnown issue (v1.1.0, patched in v1.1.1)

Loading a VectorStore whose metadata.json was produced by v1.0.0 (which did not persist batch_size) will raise a DataValidationError because batch_size is listed as a required metadata key. Passing batch_size as an argument does not bypass this check.

Fix: upgrade to v1.1.1+, which resolves this issue.

Workaround: If you are unable to update from v1.1.0 to a later version, this issue may be circumvented by manually editing the metadata.json file to include a batch_size field (e.g., "batch_size": 128). We advise against this approach. Editing metadata.json directly is not recommended as a general practice and can lead to issues.

Parameters
Name Type Description Default
folder_path str Path to the folder containing metadata.json and vectors.parquet. Supports any fsspec-compatible path (local, gs://, etc.). required
batch_size int | None Overrides the batch_size stored in metadata. Defaults to None, which uses the value from metadata.json. None
vectoriser An object with a callable .transform(texts) method. Its class name must match the vectoriser_class value stored in metadata.json. required
hooks dict | None A dictionary of user-defined hooks for preprocessing and postprocessing. Defaults to None. None
quiet_mode bool Whether to minimise verbose output, such as progress bars. Defaults to False. False
Returns
Name Type Description
VectorStore A VectorStore instance with vectors populated from the parquet file. file_name, data_type, and batch_size are all set to None.
Raises
Name Type Description
DataValidationError If folder_path is not a non-empty string, does not point to an existing directory, if metadata.json is missing or malformed, or if vectors.parquet is missing, empty, or does not contain the required columns.
OptionalDependencyError If the user attempts to use a gs:// path without having gcsfs installed.
ConfigurationError If vectoriser does not have a callable .transform() method, if the fsspec path cannot be resolved, or if the vectoriser class name does not match the one stored in metadata.json.
IndexBuildError If metadata.json or vectors.parquet cannot be read or parsed, or if the instance cannot be constructed.
search
indexers.main.VectorStore.search(query, n_results=10, batch_size=None)

Queries the vectors attribute for the most similar documents.

Queries are processed in batches of batch_size, with each batch embedded using vectoriser.transform() and scored against all stored document embeddings via dot-product similarity (equivalent to cosine similarity when embeddings are L2-normalised). The top n_results documents are returned for each query, ordered by descending score.

Any preprocessing hooks set on the instance are applied to the input before searching, and any postprocessing hooks are applied to the output before it is returned.

Parameters
Name Type Description Default
query VectorStoreSearchInput The input object containing the text query or list of queries to search for, with ids. required
n_results int Number of top results to return for each query. Defaults to 10. 10
batch_size int The batch size for processing queries. Defaults to the batch_size set during initialisation. None
Returns
Name Type Description
VectorStoreSearchOutput VectorStoreSearchOutput The output object containing search results with columns for query_id, query_text, doc_label, doc_text, rank, score, and any associated metadata columns.
Raises
Name Type Description
DataValidationError Raised if invalid arguments are passed.
ConfigurationError Raised if the VectorStore is not initialised.
HookError Raised if user-defined hooks fail.
ClassifaiError Raised if there is a package-specific error during the search operation.
VectorisationError Raised if query embedding fails.