evaluation.main

evaluation.main

This module evaluates one or more classifai.indexers.VectorStore instances on a ground-truth labelled dataset.

A typical evaluation run will perform the following sequence of actions;

  1. Create an Evaluation instance with a ground-truth dataset and a list of metrics.
  2. Run a batched top-1 VectorStore.search over all queries in the provided dataset.
  3. Merge the ground-truth labels into the retrieved results.
  4. Compute the specified multiclass, single-label classification metrics.
  5. If specified, save results and provide access to individual metric results.

Evaluation is (currently with future updates pending) framed as retrieval-as-classification: for each query, the label of the top retrieved document (doc_label) is treated as the model prediction, and the provided dataset label is treated as the ground truth (ground_truth_label).

DataFrames

Ground-truth input (ground_truths) must include:

  • qid (str): Unique query identifier.
  • text (str): Query text.
  • label (str): Ground-truth label.

Search evaluation output (results_df) is expected to include:

  • query_id (str): Query identifier (automatically generated for, and extracted from VectorStoreSearchInput dataclass).
  • query_text (str): Query text.
  • doc_label (str): Predicted label (label of retrieved doc).
  • doc_text (str): Retrieved document text.
  • rank (int): Rank of the retrieved document (>= 0).
  • score (float): Similarity score from the vector store.
  • ground_truth_label (str): Ground-truth label merged in from ground_truths.

Metrics

Metric functions are defined in classifai.evaluation.metrics and can be added to an Evaluation instance by their names. Supported metric names are:

  • “accuracy”
  • “macro_recall”
  • “macro_precision”
  • “macro_f1”

Raises

Name Type Description
InvalidMetricError Raised when requested metric names cannot be parsed.
EvaluationError Raised when validation, vectorstore execution, result validation, or metric computation fails.

Classes

Name Description
Evaluation Evaluation class for assessing the performance of vectorstores against ground truth data.
MetricType Available classification metrics.

Evaluation

evaluation.main.Evaluation(
    ground_truths,
    metrics,
    batch_size=8,
    save_output=False,
)

Evaluation class for assessing the performance of vectorstores against ground truth data.

This class provides methods to evaluate vectorstores using specified metrics, validate inputs, and save results. It supports batch processing and allows for detailed inspection of individual metric results.

Attributes

Name Type Description
ground_truths pd.DataFrame DataFrame containing ‘qid’, ‘text’, and ‘label’ columns.
batch_size int Batch size for vectorstore search operations.
save_output bool Whether to save evaluation results to a file.
parsed_metrics dict Dictionary of parsed metrics to compute.
metric_results dict Dictionary of individual metric results for detailed inspection.

Methods

Name Description
evaluate Evaluate multiple VectorStore instances on ground truth data and compute metrics.
evaluate
evaluation.main.Evaluation.evaluate(
    vectorstores,
    vectorstore_names,
    output_file=None,
    overwrite=False,
)

Evaluate multiple VectorStore instances on ground truth data and compute metrics.

This method validates the input, evaluates each VectorStore instance or callable, computes metrics, and optionally saves the results to a CSV file.

Parameters
Name Type Description Default
vectorstores list[VectorStore | Callable[[], VectorStore]] A list of VectorStore instances or callables that return VectorStore instances. required
vectorstore_names list[str] A list of unique names corresponding to the VectorStore instances. The length must match the vectorstores list. required
output_file str | None The file path to save the evaluation results as a CSV file. Must end with “.csv”. If None, results are not saved unless self.save_output is True. None
overwrite bool Whether to overwrite the output file if it already exists. Default is False. False
Returns
Name Type Description
pd.DataFrame pd.DataFrame: A DataFrame containing the evaluation results for all VectorStore instances.
Raises
Name Type Description
ValueError If input validation fails (e.g., mismatched lengths, invalid types, or duplicate names).
EvaluationError If any step of the evaluation process fails, such as instantiating a VectorStore, running a search, validating search results, or computing metrics.
ClassifaiError If saving the results to a file fails.
Notes
  • Each VectorStore instance or callable is processed sequentially.
  • Metrics are computed using self.parsed_metrics, and results are stored in self.metric_results.

MetricType

evaluation.main.MetricType()

Available classification metrics.

Functions

Name Description
parse_metrics Parse a list of metric names and return a dictionary mapping metric names to their corresponding functions.

parse_metrics

evaluation.main.parse_metrics(metrics)

Parse a list of metric names and return a dictionary mapping metric names to their corresponding functions.

Parameters

Name Type Description Default
metrics list[str] A list of metric names as strings. required

Returns

Name Type Description
dict[str, Metric] dict[str, Metric]: A dictionary where the keys are the metric names (as strings) and the values are
dict[str, Metric] the corresponding Metric instances.

Raises

Name Type Description
ValueError If a metric name in the input list is invalid, an error is raised with a list of valid metrics.