A benchmark of 1,616 diverse NLP tasks and a transformer model (Tk-Instruct) for cross-task generalization via declarative instructions.
Super-NaturalInstructions is a research project introducing a benchmark dataset comprising 1,616 diverse Natural Language Processing (NLP) tasks, each accompanied by expert-written instructions. This collection covers 76 distinct task types, including classification, extraction, infilling, sequence tagging, text rewriting, and text composition. The primary goal of this benchmark is to enable rigorous evaluation of how well NLP models can generalize to a variety of unseen tasks when provided with explicit task instructions. Alongside the benchmark, the project also introduces Tk-Instruct, a transformer model specifically trained to follow various in-context instructions, which can be either plain language task definitions or k-shot examples. Experiments conducted with Tk-Instruct demonstrate its ability to outperform existing instruction-following models, such as InstructGPT, by over 9% on the Super-NaturalInstructions benchmark, despite being significantly smaller in scale. The research further analyzes generalization capabilities in relation to scaling parameters like the number of observed tasks, instances per task, and model sizes. The dataset and model are intended to foster advancements in developing more general-purpose NLP models.
Platforms: Web
View full Super-NaturalInstructions profile on Tools-Radar | Browse Research tools | Alternatives to Super-NaturalInstructions
Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com