02 · Computer Vision · Robotics
68ms
end-to-end perception latency
Turn pediatric rehab into a clinical boss fight — an autonomous AI therapy companion that tracks movement, coaches kids by name, and paces sessions with a modified Roomba.
The Exhibit
01·Problem
Physical therapy is repetitive and discouraging for kids, which kills engagement and slows progress. Most CV pipelines are too slow for live coaching, and coordinating vision, voice feedback, and robot pacing as separate systems is hard to get right in a single session.
02·Approach
- 01
Real-time perception loop in FastAPI: YOLOv8n person detection → MediaPipe pose → EMA-smoothed joint angles → MJPEG dashboard stream at ~15 FPS on-device (M3 Pro CPU).
- 02
Deterministic clinical logic with hysteresis rep counting (squats, arm raises, side reaches) and 0.8s debounce — 100% accuracy on 13 scripted angle test cases.
- 03
Intel DPT depth estimation runs async on a worker thread so follow logic never blocks the main perception loop.
- 04
Six-agent orchestration (patient, clinical, assistant, director, pacer, audio): Gemini 2.5 Flash coaching via Google ADK Runner on rep events; ElevenLabs TTS streamed over WebSocket.
- 05
QR / bracelet scan loads therapy profiles; dashboard shows live agent trace, rep counter, and session scorecard.
- 06
ESP32 WebSocket bridge translates pacer spatial data into Roomba motor commands — simulation mode when hardware is offline.
03·Architecture
Swipe to explore
04·Results
68ms
avg end-to-end latency (p50)
~15 FPS
on-device perception (CPU)
100%
rep logic accuracy (13 test cases)
05·Stack
Python · FastAPI · YOLOv8n · MediaPipe · Intel DPT · Google ADK · Gemini 2.5 Flash · ElevenLabs · WebSocket · ESP32 · OpenCV