Approximately 10,000 of deaf people in Switzerland primarily use sign language as a form of communication, facing isolation and obstacles in daily interaction with public/semi-public entities, where the assistance of an interpreter is expensive and not always available.
Ticino, on the other side, stands out as one of the few cantons to have officially recognized sign language, leading a positive change toward the inclusion of all citizens.
In this context lies the project DEEP: Augmented Reality metaverse for assisting deafness people for institutional interaction by exploiting digital video forensics, funded by , the Swiss Innovation Agency. Led by Alessandro Trivilini, PhD, it is based on a fundamental principle of sensitivity to the people involved and the importance of product accessibility following the motto "Nothing for us, without us."
"Inclusion is not an empty word. With this project, I personally experienced the complexity of human interactions that deaf people encounter on a daily basis while communicating with other individuals. My technical and relational skills have been fundamental to understanding the peculiarities of Italian sign language, allowing the creation of a unique body of data, destined to be a valuable contribution to the entire scientific community," says Elisa Colletti, scientific collaborator at SIF.
During the project, a comprehensive collection of 36,694 videos depicting hand movements, body positions, and facial expressions essential for a complete representation of Italian Sign Language (LIS) was compiled. This corpus was developed with the participation of 14 deaf individuals and 3 hearing individuals, and their privacy was safeguarded through adherence to a rigorous forensic protocol.
"In such cases, software development cannot be reduced to simple coding in an artificial intelligence model. LIS is a very rich language in which the use of hands is as important as facial expressiveness and body movements. Having designed and developed the computer architecture needed to facilitate the accessibility and usability of the entire system has challenged my engineering skills, stimulating an innovative approach to solving the problem related to human interaction with the digital world," continues Nicolas Tagliabue, scientific collaborator of SIF and software architect of the entire project.
Roberto Tedesco, SIF Researcher, also echoes him: "In recent years, machine translation systems have made great strides. However, the peculiarities of LIS make the design of a translation system into Italian extremely challenging. At the same time, these very specificities have allowed me to tackle novel problems and deepen my knowledge of the latest neural models applied to video, text and voice."
This project paves the way for new developments toward other sign languages, for example German and French, as well as new contexts of use to increase communicative independence of deaf citizens.
"I am very proud of the results we have achieved from this innovative project. My company has always tried to improve the daily lives of people with disabilities, taking care of the smallest details with the help of new technologies to offer concrete benefits. The fact that the Grand Council has recognized LIS as an official language is confirmation that my vision has found institutional backing toward a real problem," concludes Daniele Raffa, CEO of .