UScribe

My role: technical product owner and builder: I decided what UScribe should do and built it.

The problem: physicians spend a large part of their day writing notes instead of looking at patients. UScribe listens to the visit, writes the structured note for them, and sends it to the EHR.

How a visit flows:

1. Visit audio is captured and stored in Cloud Storage
2. Whisper large-v3-turbo on autoscaling Vertex AI GPUs transcribes 12 minutes of audio in about 30 seconds
3. Claude Haiku 5.5 writes a 29-field SOAP note in under 60 seconds
4. Cloud Functions and Pub/Sub pipelines extract clinical entities and validate the note, with dead-letter retries
5. The note routes to eClinicalWorks over HL7/FHIR, through an OAuth 2.0 connection I built
6. Clinician edits are tracked by edit distance

The hardest part: knowing whether a note is good

The hardest part of UScribe wasn't building it, it was knowing whether a note was good. A clinical note never has one right answer, and an LLM never writes the same note twice, so exact matching was useless for the note as a whole. I split the problem in two:

• Note quality, judged by closeness: 1,000 transcript and gold-standard note pairs, drawn from synthetic and publicly available de-identified data, so no real patient records were used. Each generated note was scored on how close it came to the gold note.
• Medical codes, judged by exact match: ICD-10 diagnosis codes, RxNorm codes in medication suggestions and CVX codes for immunizations had to match exactly, because a near-miss code is a wrong code.

How closeness was scored: I built an evaluation matrix around how clinicians actually write and review their notes, and a generated note had to pass it.

• LLM judge for routine runs: small changes and repeated tests were scored by an LLM judge against the matrix.
• Clinicians for major changes: any major change also went to clinician review.
• Keeping the judge honest: clinicians reviewed a random sample of judged notes, their scores were compared with the LLM judge's, and the gaps were used to improve the matrix.

In production, clinician edit distance showed how much each section still needed fixing, and changes guided by it cut edits per note by 14%.

What I'd do differently

Most of our delays weren't technical. They came from coordinating with the EHR vendor's developers, whose approval and testing cycles ran on their own timeline. Next time I'd build those response times into the plan from day one. I'd also talk to clinicians more often, well before the product was ready: early conversations surface real needs and shape the scope far better than a finished demo does.