← Back to Tools-Radar

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Categories: Text & Writing, Research  |  Pricing: Free  |  Official Website ↗

PEGASUS is a Transformer encoder-decoder model designed for abstractive text summarization, achieving state-of-the-art results.

PEGASUS is a Transformer encoder-decoder model developed by Google Research for abstractive text summarization. It utilizes a novel self-supervised pre-training objective called 'gap-sentence generation,' where the model is trained to recover masked sentences from documents. This approach encourages the model to learn language understanding, information compression, and language generation skills relevant to summarization. The model has achieved state-of-the-art performance on 12 diverse summarization datasets, including news articles, scientific papers, and legal documents. A key finding is its 'sample efficiency,' meaning it can achieve near state-of-the-art performance with a relatively small number of fine-tuning examples (e.g., 1000 examples), significantly reducing the cost and scale of supervised data collection. Human evaluations suggest that PEGASUS can produce summaries comparable to human-written ones on certain datasets. Google has released the training code and model checkpoints on GitHub to support further research and ensure reproducibility. This includes fine-tuning code, allowing researchers to adapt PEGASUS to other summarization datasets and explore its capabilities in various applications.

Key Features

• Abstractive text summarization
• Gap-sentence generation pre-training objective
• Transformer encoder-decoder architecture
• State-of-the-art performance on diverse datasets
• Sample-efficient fine-tuning
• Human-like summary generation
• Rudimentary symbolic reasoning (e.g., counting)

Pros

• Achieves state-of-the-art results on multiple summarization benchmarks.
• Requires significantly fewer fine-tuning examples for strong performance.
• Generates abstractive summaries, not just extractive snippets.
• Produces summaries comparable to human quality on some datasets.
• Open-sourced code and model checkpoints for research and reproducibility.

Cons

• Counting ability is rudimentary and does not generalize perfectly.
• Requires technical expertise to implement and fine-tune.
• Not a ready-to-use end-user application, but a research model.
• Performance metrics (ROUGE) are proxies and don't capture all aspects of quality.
• Developed in 2020, newer models might exist.

Use Cases

• Developing new abstractive summarization systems
• Benchmarking summarization model performance
• Researching self-supervised learning for NLP tasks
• Creating summaries for various document types (news, scientific papers, legal docs)

Best For

• NLP researchers
• Machine learning engineers
• Academics studying text summarization
• Developers building summarization features

Platforms: Web

Watch demo on YouTube ↗

View full PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization profile on Tools-Radar | Browse Text & Writing tools | Alternatives to PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com