Letter by letter
Technical · January 2023
BoilerMake X
My first big hackathon was at Purdue, January 2023. Reeve, Arleen and I wanted to build video calls that a deaf person could join without bringing an interpreter. The full version of that is machine translation of sign language, which was way past a weekend. So we scoped down to the part we could actually do: fingerspelling. You sign a letter, the other side sees the letter.
It worked, letter by letter. Slow, but readable, and it ran in a normal browser call with nothing installed. We were honest in the submission that this was the foundation layer and not the product.
SignLink
The interesting problem was turning a hand on camera into a letter without training a model, because we didn't have the data or the time.
Technical
MediaPipe gives 21 landmarks per hand. We took 20 of them and computed joint angles with plain vector algebra, then matched the angle vector against a hand-coded table of ASL letter poses. No training, just trig and a lookup. Calls ran over WebRTC with PeerJS handling the peer connections, so the recognition loop stayed client-side next to the video.
The matcher was brittle in ways you'd expect. Letters that differ by a thumb tuck collided at bad camera angles, and motion letters like J and Z don't fit a static pose table at all. That's the honest boundary of the approach.
Personal
Accessibility hacks at hackathons are usually built for the judges. We tried to keep asking the other question, whether a deaf person would actually sit through a call on this. The answer for the weekend version was probably not yet, and saying that out loud in the submission felt more useful than pretending otherwise.