01 · Accessibility · Computer Vision
31.5ms
median on-device detection
Touch-only navigation for people who are DeafBlind. A chest camera sees, a laptop in a backpack thinks, and two Joy-Cons on the wrists steer: left buzz means left.
The Exhibit
01·Problem
Most accessibility tech assumes you can either see or hear: tools for blind people talk, tools for Deaf people show. For someone who is DeafBlind, both channels are closed. Getting to a place, a waiting car, or a water bottle across the room needs directions that arrive through touch alone, with nothing to translate.
02·Approach
- 01
A 20 Hz guidance controller fuses compass heading, GPS, an OpenStreetMap walking route (OSRM foot router), camera detections and obstacle estimates, then decides which wrist to buzz: left, right, both for forward, a double buzz for arrival, nothing for stop.
- 02
Chest iPhone runs YOLOv10s in Safari over WebGPU (WebAssembly fallback) and streams live H.264 video to the laptop over WebRTC through a Cloudflare tunnel.
- 03
Mac server forks three model workers: YOLOv8n finds the requested object, Depth Anything V2 plus a floor fit flags obstacles YOLO has no name for (walls, poles, boxes), and CLIP ViT-B/32 with a trained head asks "is this a Waymo?"
- 04
Three modes: walk to a saved place along real sidewalks, walk to a waiting car (GPS toward a beacon phone, then camera for the last meters), or find an everyday object and turn the wearer until the camera spots it.
- 05
Nintendo Joy-Cons driven straight from Chrome over WebHID. Gemini parses voice commands and vague requests ("something to drink" → water bottle) but never steers; ElevenLabs narrates for onlookers.
03·Architecture
Swipe to explore
04·Results
31.5ms
median YOLOv8n round trip (~32 fps)
96%
Waymo classifier precision at 0.7% FPR
0.992
classifier ROC-AUC (50× 5-fold CV)
05·Stack
JavaScript · Node.js · WebRTC · WebHID · WebGPU · YOLOv10s · YOLOv8n · Depth Anything V2 · CLIP · Transformers.js · OpenStreetMap · Gemini · ElevenLabs