fixml.modules.llm_eval.consistency_eval

Classes

ConsistencyEvaluator

Module Contents

class fixml.modules.llm_eval.consistency_eval.ConsistencyEvaluator

evaluate(models, num_test_runs=2, verbose=False)

Input the initialized TestEvaluator models, test run num_test_runs times to obtain the result models = [{‘name’: ‘model_no1’, ‘model’: {{model object}}}, …]

get_completeness_score_dist()

Obtain the distribution of the Test Completeness scores

get_consistency_dist()

Obtain the distribution of the consistency per checklist item