Hangzhou, China — Cloud computing company Alibaba Cloud is stepping up its inclusivity efforts, especially during the Para Asian Games scheduled for next month, with the introduction of Xiaomo, a digital sign language avatar specifically designed for the hearing-impaired.

Xiaomo is seamlessly integrated into Alibaba Group’s payment gateway, Alipay, as a mini-app. It is also incorporated into the Asian Games news channel to enhance the accessibility of announcements during the event.

This virtual avatar facilitates two-way translation between sign language and the Chinese spoken language. Xiaomo can capture sign language and translate it into spoken words, as well as convert spoken language into sign language.

Alibaba beefs up DingTalk messaging app for Hangzhou Asian Games
Alibaba Cloud hosts core digital systems for Hangzhou Asian Games

“For people with hearing difficulties, communication barriers are a significant challenge,” said Matt Zhang, algorithm engineer at Alibaba Cloud. “However, Xiaomo now offers a novel solution to bridge this gap, making the Games more inclusive.”

Deep neural network

Zhang acknowledged that one of the major challenges in developing Xiaomo was creating a comprehensive dataset of 25,000 sign language translations. To overcome this hurdle, the company employed a deep neural network and a lip movement module to enhance Xiaomo’s translation accuracy.

Sign language has its own complexity and nuances, which makes digital sign language translation even more challenging. In addition to leveraging visual recognition and motion tracking capabilities, Alibaba Cloud collaborated with individuals who are hearing-impaired and sign language experts to compile the sign language translation dataset.

Throughout the two-year development process, the company also tackled the challenge of machine translation. Zhang explained that they refined the system to simultaneously select and translate words to reduce ambiguity and optimize accuracy.

Xiaomo has been developed to empower more individuals to participate in the Games. This includes enabling audiences to independently navigate the Games, book their tickets, and purchase from shops. Participants of the Asian Para Games will now be able to interact more seamlessly with one another and seek assistance as needed.

During the Asian Para Games, Xiaomo will serve as a valuable tool for participants with hearing impairments, facilitating interactions with volunteers at the venues for various forms of assistance. This includes requesting directions, seeking medical assistance, and obtaining support for game viewing.