Dezeen Magazine

Fingerspelling.xyz app by Hello Monday

Fingerspelling.xyz app helps people to learn the sign language alphabet

Creative studio Hello Monday has created an app that teaches and corrects hand positions in real-time to make it easier for people to learn the sign language alphabet.

Hello Monday worked with the American Society for Deaf Children to design Fingerspelling.xyz, an app that runs in web browsers.

Fingerspelling.xyz home screen showing a 3D model of a hand with its fingers crossed
Fingerspelling.xyz is an app that runs in web browsers

Not unlike the popular language-learning app Duolingo, Fingerspelling.xyz is able to assess its user's attempts to learn words in real-time.

The app makes use of a person's webcam to track their hand movements as they attempt the hand positions of the American Sign Language (ASL) alphabet, while an algorithm trained on images of correct hand positions analyses their accuracy.

Fingerspelling.xyz set-up screen prompting users to choose whether they are right- or left-handed
The app gives instruction in the sign language alphabet tailored to the user

"The fingerspelling game is a great way to introduce the basics of ASL in a fun and playful way," said Hello Monday founding partner Anders Jessen.

According to Jessen, the technology used by the app offers a more engaging learning experience than the traditional methods of learning the sign language alphabet – through reading or watching videos.

Fingerspelling.xyz level-one home screen giving instructions to look at the hand on screen and copy its hand shape
The app targets parents of deaf children, who are unlikely to know sign language

"The game leverages advanced hand recognition technology, matched with machine learning, to give you real-time feedback via the webcam for each sign and word you spell correctly," Jessen said.

"It's fun to think about how this highly trained machine-learning model, which researchers have spent countless hours on training, now does the opposite – it now trains us back, and makes us better at fingerspelling."

A level one screen tells the user they will be learning to spell the word 'able'
To start, users are shown a word they will be learning to spell letter by letter

In ASL, fingerspelling is used for proper nouns, or when a person doesn't know the sign for a word.

On a purple and butter-yellow interface, Fingerspelling.xyz starts by giving the user a series of words, along with a 3D model of a hand showing the position required for each letter.

The app then analyses the user's attempt, gives feedback until they achieve the correct alignment, and then lets them progress through the levels.

Hello Monday promises that webcam data isn't stored or sent anywhere.

Screen tells the user to place their hand where it is visible in their webcam
Fingerspelling.xyz makes use of the computer's webcam to track the user's gestures

The studio is especially targeting the app at parents of deaf children, so that they can encourage their child's development by introducing them to sign language at an early stage.

It points to statistics that show that two to three of every 1,000 children are born deaf or hard of hearing in the US, and 90 per cent of them are born to hearing parents.

FingerspA splitscreen shows the letter A and a 3D hand in a fist salute shape on the left side and webcam view of a user's flat palm on the right
A 3D model of a hand shows the correct shape, which users try to emulate

However, 72 per cent of families do not sign with their deaf children, and Hello Monday and the American Society for Deaf Children hope to change that.

"We are hoping that this can become useful for anyone wanting to learn Fingerspelling – and the next steps after this would be to take a course on Fingerspelling/ASL," said Jessen.

A splitscreen shows the letter A and a 3D hand in a fist salute shape on the left side and webcam view of a matching fist shape on the right
An algorithm analyses the user's hand shape and guides them to correct it

Hello Monday is a creative studio focused on digital products and experiences. Its recent work has included designing a virtual Summer Show exhibition for the Bartlett School of Architecture in 2020.

In 2011, a deaf couple had their house in Tokyo designed to enable them to sign to each other and their children through the walls. It is covered with nearly a hundred small windows.