Real-Time, Automatic Sign Language Detection for Video Conferencing

Categories: Productivity, Research | Pricing: Free | Official Website ↗

A research project demonstrating a real-time sign language detection model for video conferencing to identify signers as active speakers.

This Google Research project introduces a real-time sign language detection model designed to improve accessibility in video conferencing for sign language users. The model addresses the challenge of signers being overlooked by typical video conferencing applications that prioritize spoken audio. It uses a lightweight architecture to minimize computational load, making it suitable for client-side integration. The system works by first employing PoseNet for human pose estimation, reducing video input to a set of body landmarks. It then calculates frame-to-frame optical flow from these landmarks to quantify user motion, which is fed into an LSTM-based classification model. A proof-of-concept web demo leverages tf.js to run the models in the browser and transmits an inaudible ultrasonic audio tone to video conferencing applications when signing is detected, effectively 'fooling' the application into recognizing the signer as an active speaker. The training code and web demo source code are available on GitHub.

Key Features

Real-time sign language detection
Human pose estimation for input reduction
LSTM-based classification model
Ultrasonic audio tone generation to trigger active speaker
Lightweight model for client-side processing
Web-based demo available

Pros

Enables sign language users to be recognized as active speakers in video conferences
Lightweight model minimizes impact on call quality
Open-source code available for further development
Achieves high accuracy (up to 91.5%) in detection
Works in real-time at 30 frames per second

Cons

Requires a virtual audio cable setup for full functionality
Relies on 'tricking' video conferencing apps with ultrasonic audio
Currently a research demo, not a fully integrated product
Limited research on sign language detection in general
Accuracy can vary with different contexts and users

Use Cases

Improving accessibility in online meetings for deaf and hard-of-hearing individuals
Facilitating smoother communication flow in video calls involving sign language
Research and development of real-time gesture recognition systems
Integration into future video conferencing platforms

Best For

Sign language users
Video conferencing platforms seeking accessibility features
Researchers in human-computer interaction and machine perception
Developers interested in real-time pose estimation applications

Platforms: Web

Watch demo on YouTube ↗

View full Real-Time, Automatic Sign Language Detection for Video Conferencing profile on Tools-Radar | Browse Productivity tools | Alternatives to Real-Time, Automatic Sign Language Detection for Video Conferencing

Tools-Radar is a free directory of 10,000+ AI tools — discover, compare, and choose the right AI software for your needs. Visit tools-radar.com