InfiBench

Categories: Coding & Developer Tools, Research, Data Analysis | Pricing: Free | Official Website ↗

InfiBench is an open-source benchmark for evaluating the question-answering capabilities of code Large Language Models (LLMs).

InfiBench is a comprehensive, open-source benchmark designed to evaluate the ability of code Large Language Models (LLMs) to answer freeform, real-world questions within the code domain. Unlike existing benchmarks that primarily focus on code generation, InfiBench addresses the broader spectrum of capabilities expected from code LLMs, including understanding and responding to diverse coding-related inquiries. The benchmark comprises 234 high-quality questions sourced from Stack Overflow, covering 15 programming languages. It employs four model-free automatic metrics—keywords matching, blank filling, unit testing, and dialogue similarity—with evaluation criteria carefully defined by domain experts. InfiBench has been used to systematically evaluate over 100 code LLMs, providing insights into their performance and potential areas for improvement. It is continuously expanding to promote more scientific and systematic practices in code LLM evaluation.

Key Features

234 high-quality Stack Overflow questions
Covers 15 programming languages
Four model-free automatic evaluation metrics (keywords matching, blank filling, unit testing, dialogue similarity)
Systematic evaluation of over 100 code LLMs
Open-source code repository
Leaderboard for evaluated models

Pros

Focuses on freeform question-answering, addressing a gap in existing code LLM benchmarks.
Uses real-world questions from Stack Overflow for practical relevance.
Employs diverse, model-free evaluation metrics for robust assessment.
Supports a wide range of 15 programming languages.
Fully open-source, encouraging community contribution and transparency.

Cons

Currently only supports Linux environments.
Evaluation protocol uses fixed sampling temperature and top p, limiting exploration of other hyperparameters.
Requires converting model weights to Hugging Face Transformers format.
Evaluation of models with >30B parameters is done only once due to resource limits.

Use Cases

Benchmarking new code LLM architectures
Comparing the question-answering abilities of various code LLMs
Identifying strengths and weaknesses of code LLMs in real-world scenarios
Contributing to the scientific evaluation of AI models

Best For

AI researchers
Code LLM developers
Academics evaluating code LLMs
Organizations comparing code LLM performance

Platforms: Linux

Watch demo on YouTube ↗

View full InfiBench profile on Tools-Radar | Browse Coding & Developer Tools tools | Alternatives to InfiBench

Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com