05 · Voice AI
100%
allergy conflicts caught
Your health, handled. An autonomous medical proxy that cross-references patient records against live webcam scans and places real Twilio pharmacy calls on the patient's behalf.
The Exhibit
01·Problem
Managing prescriptions, refill timing, and allergy safety is overwhelming — especially for patients juggling multiple medications or relying on caregivers. A single missed interaction or an unnoticed allergen can have serious consequences.
02·Approach
- 01
Onboarding pipeline uses Gemini 2.0 Flash to parse uploaded medical PDFs into a structured JSON patient dossier (allergies, meds, caregiver contacts).
- 02
Visual Pill Scanner uses GPT-4o-mini Vision to identify medication from a webcam frame and cross-check it against known allergies, firing a CRITICAL alert before consumption.
- 03
Real-time telephony loop places outbound Twilio calls to the pharmacy for refills. Low-latency ElevenLabs TTS (eleven_turbo_v2_5) handles the voice, with a fallback persuasion flow if the pharmacist declines.
- 04
FastAPI backend hosts the glassmorphism patient dashboard and runs an APScheduler cron job to flag upcoming refills based on fill dates and days supply.
- 05
All LLM requests are routed through a single OpenRouter API layer covering both Gemini (documents) and GPT-4o-mini (chat + vision).
03·Architecture
Swipe to explore
04·Results
100%
allergy conflicts blocked by Vision AI
3
autonomous loops (safety, call, scheduler)
Zero
human intervention required for refill calls
05·Stack
Python · FastAPI · Twilio Voice · ElevenLabs TTS · GPT-4o-mini Vision · Gemini 2.0 Flash · OpenRouter · APScheduler · Tailwind CSS · GSAP