BUas students develop AI tool for learning Dutch Sign Language

Learning by doing   

The projects were developed during the End-of-the-Block Challenges: intensive assignments lasting one to two weeks that take place between the regular teaching blocks. Students work independently, in small groups, and deliver tangible results for real clients, from both the business sector and non-profit organisations. Many of these projects develop into full-scale projects in the third year, or lead to work placements and graduation projects. 

Making sign language accessible   

The brief was submitted by Nienke Fluitman of Gebaren.nl, a certified NGT teacher and entrepreneur specialising in inclusivity and accessibility. She identified a significant lack of interactive online learning materials for NGT and proposed a platform where users can practise finger spelling via a camera, spelling out names and words that do not have their own sign. In the Netherlands, approximately 30,000 people use NGT as their primary language. 

The results exceeded expectations. Students developed applications incorporating gamification elements, state-of-the-art AI techniques such as action recognition, dynamic time warping (DTW) and landmark detection, and real-time feedback on individual hand gestures. The apps are designed for non-technical users. Students built their own training data by taking photos of each other whilst making hand gestures. 

"I had a deaf friend at secondary school and always wanted to learn sign language. This challenge was my chance, and I actually learned the alphabet. I kept practising and created loads of training sessions because I enjoyed it so much." — Khaled, ADS&AI student 

"I had no experience with computer vision and didn’t know how it worked. It was exciting, because I wanted to know how a machine ‘sees’. Recognising hand gestures seems simple for a human, but turned out to be quite complicated to implement." — Timur, ADS&AI student 

"I set these students a challenge: can you build a model that helps people learn the finger spelling alphabet, with real-time feedback? The result was astonishing. They built a model that provides real-time feedback on individual fingers as a letter is formed. They’ve even started work on a game, and how cool would it be if that works out too. But what perhaps touched me most is that all these students have learned the finger spelling alphabet and used their technical skills for a social cause. In my view, that’s an absolute win" — Nienke Fluitman, NGT teacher and accessibility advisor, Gebaren.nl 

Following the success of this challenge, BUas is organising a follow-up week to further refine the tools developed. We also expect the project to be developed into a full-scale third-year project. 

Interested in collaborating with ADS&AI? 

BUas ADS&AI is looking for new clients for challenges, semester projects, work placements, and graduation projects. Organisations wishing to explore how AI and data science can benefit them are welcome to get in touch.