- Punjabi University develops system to convert spoken Punjabi into Indian Sign Language (ISL) in real time.
- Powered by speech recognition, NLP, and 3D animation, the tool ensures accurate and natural ISL gestures.
- Available as a Progressive Web App, it supports multilingual input and aims to expand to other languages.
In a groundbreaking step toward inclusive communication, researchers at Punjabi University, Patiala, have developed an innovative system that automatically converts spoken Punjabi into Indian Sign Language (ISL). The project, funded by the Department of Science and Technology (DST), Government of India, is designed to bridge communication gaps between hearing-impaired and non-hearing-impaired individuals, opening new avenues for accessibility in education, healthcare, and public services.
The initiative is spearheaded by Williamjeet Singh, Project Supervisor and Assistant Professor at the Department of Computer Science and Engineering, with support from Amandeep Singh, Research Project Assistant. The system integrates speech recognition, natural language processing, and synthetic animation to convert spoken Punjabi into ISL gestures in real time.
Speaking about the innovation, Williamjeet Singh emphasized that communication is central to human interaction, yet hearing-impaired individuals face persistent barriers due to the limited knowledge of ISL among the general population. “This system empowers the hearing-impaired community by enabling seamless communication in key areas like education and healthcare. It addresses the challenge of exclusion by ensuring that spoken Punjabi can be instantly transformed into meaningful ISL gestures”, he said.
The technology employs the Hamburg Notation System (HamNoSys) for representing sign gestures and Signing Gesture Markup Language (SiGML) to generate animated 3D sign sequences. It also incorporates an ISL dictionary that captures both manual gestures (hand movements) and non-manual features (facial expressions and body movements), ensuring a natural and accurate representation of signs.
Amandeep Singh explained that the system combines synthetic animation and video-based sign presentation to deliver high accuracy in sign conversion. A novel optimization technique further enhances the precision of translating spoken words into structured ISL scripts. “Rigorous testing has demonstrated its efficiency, accuracy, and usability, making it a reliable platform for real-time communication. As a Progressive Web Application, it supports multilingual speech input and is accessible via both web and mobile devices. Beyond communication, it also acts as an educational tool for learning new signs”, he said.
The research team envisions scaling the system to support multiple Indian languages, thereby extending its reach to a larger audience. Plans are underway to develop a dedicated mobile application for wider accessibility.
Also Read: How Micro-Content Is Rewiring Education in Asia
Congratulating the team, Vice Chancellor Jagdeep Singh said, “This innovation promises to transform the lives of millions of hearing-impaired individuals in India and beyond, fostering greater inclusion and opportunities for interaction”.
With its potential to bridge critical communication gaps, the Punjabi-to-ISL system stands as a pioneering model of how advanced technology can create a more inclusive society by empowering those who rely on sign language.
