AlpacaEval is an LLM-based automatic evaluation system that ranks instruction-following language models based on their win rates.
AlpacaEval is an automatic evaluation system designed to assess the instruction-following capabilities of large language models (LLMs). It utilizes an LLM-based auto-annotator (GPT-4 Preview for AlpacaEval 2.0) to compare model responses against reference responses from models like Davinci003 or GPT-4 Preview. The system calculates win rates, which are displayed on a leaderboard, and has shown high agreement with human annotations. The evaluation is based on the AlpacaFarm evaluation set, which focuses on general user instructions. The leaderboard aims to provide a fast, cheap, and reliable method for comparing LLMs. The platform welcomes community contributions for new models, evaluators, and evaluation sets, particularly encouraging harder instruction distributions and safety testing of LLMs. However, AlpacaEval has limitations, including a correlation between auto-annotator win rates and response length, and the AlpacaFarm eval set primarily consists of simple instructions. It does not comprehensively evaluate all model abilities or assess model safety.
Platforms: Web
View full AlpacaEval Leaderboard profile on Tools-Radar | Browse Research tools | Alternatives to AlpacaEval Leaderboard
Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com