FELM is a meta-benchmark designed to evaluate the factuality evaluation benchmarks for Large Language Models (LLMs).
FELM addresses the critical need for evaluating the evaluators that assess the factuality of text generated by Large Language Models (LLMs). This research area is crucial for identifying potential errors in LLM outputs and guiding the development of more reliable models. However, the tools and methods used to evaluate factuality themselves require proper assessment to ensure progress and foster advancements in the field. To mitigate the current impediments in evaluating factuality evaluators, FELM introduces a comprehensive benchmark. It involves collecting responses generated from various LLMs and annotating their factuality with fine-grained labels. The benchmark covers five distinct domains: World Knowledge, Science/Technology, Writing/Recommendation, Reasoning, and Math. Prompts for these domains are sourced from standard datasets like TruthfulQA, online platforms, ChatGPT generation, or drafted by the authors, with responses obtained from ChatGPT.
Platforms: Web
View full FELM: Benchmarking Factuality Evaluation of Large Language Models profile on Tools-Radar | Browse Research tools | Alternatives to FELM: Benchmarking Factuality Evaluation of Large Language Models
Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com