evaluation.main.Evaluation

evaluation.main.Evaluation(
    ground_truths,
    metrics,
    batch_size=8,
    save_output=False,
)

Evaluation class for assessing the performance of vectorstores against ground truth data.

This class provides methods to evaluate vectorstores using specified metrics, validate inputs, and save results. It supports batch processing and allows for detailed inspection of individual metric results.

Attributes

Name	Type	Description
ground_truths	pd.DataFrame	DataFrame containing ‘qid’, ‘text’, and ‘label’ columns.
batch_size	int	Batch size for vectorstore search operations.
save_output	bool	Whether to save evaluation results to a file.
parsed_metrics	dict	Dictionary of parsed metrics to compute.
metric_results	dict	Dictionary of individual metric results for detailed inspection.

Methods

Name	Description
evaluate	Evaluate multiple VectorStore instances on ground truth data and compute metrics.

evaluate

evaluation.main.Evaluation.evaluate(
    vectorstores,
    vectorstore_names,
    output_file=None,
    overwrite=False,
)

Evaluate multiple VectorStore instances on ground truth data and compute metrics.

This method validates the input, evaluates each VectorStore instance or callable, computes metrics, and optionally saves the results to a CSV file.

Parameters

Name	Type	Description	Default
vectorstores	list[VectorStore | Callable[[], VectorStore]]	A list of VectorStore instances or callables that return VectorStore instances.	required
vectorstore_names	list[str]	A list of unique names corresponding to the VectorStore instances. The length must match the vectorstores list.	required
output_file	str | None	The file path to save the evaluation results as a CSV file. Must end with “.csv”. If None, results are not saved unless self.save_output is True.	None
overwrite	bool	Whether to overwrite the output file if it already exists. Default is False.	False

Returns

Name	Type	Description
	pd.DataFrame	pd.DataFrame: A DataFrame containing the evaluation results for all VectorStore instances.

Raises

Name	Type	Description
	ValueError	If input validation fails (e.g., mismatched lengths, invalid types, or duplicate names).
	EvaluationError	If any step of the evaluation process fails, such as instantiating a VectorStore, running a search, validating search results, or computing metrics.
	ClassifaiError	If saving the results to a file fails.

Notes

• Each VectorStore instance or callable is processed sequentially.
• Metrics are computed using self.parsed_metrics, and results are stored in self.metric_results.