USOpenloop Health · 2026
AI Assist for a telehealth funnel builder
An AI editing assistant on Vertex AI that safely edits live funnel pages, with the production model picked through a 315-call evaluation harness.
Team Lead · owner from spike to production
- eval calls to choose the model
- 315eval calls to choose the model
- shipped with 100+ new tests
- 6 PRsshipped with 100+ new tests
- Gemini tool definitions
- 4Gemini tool definitions
Context
The situation
Openloop runs a white-label telehealth platform serving 600+ insurance plans and hundreds of client brands. Marketers build patient funnels in an in-house page builder.
The problem. Marketers wanted to change copy and layout by describing the change, but the builder’s page structure is strict: one bad edit can break a live funnel that patients are using right now. The feature needed privacy review before any patient-facing content touched a model.
Approach
Key decisions
- 01
Choose the model on evidence, not taste
Built an evaluation harness of 21 real utterances across 5 model and prompt variants — 315 calls — and picked the production setup from the results.
- 02
Tools instead of free text
The model answers through 4 Gemini tool definitions behind a tRPC endpoint, so every suggestion is a structured operation rather than HTML to paste.
- 03
A patch engine as the safety boundary
Suggested operations go through a patch engine that validates and applies them to live page content, so the page structure cannot be corrupted.
- 04
Scope before code
Ran the technical research and privacy review first, then split the work into 10 tickets so it could ship incrementally.
How it works
- 1
Marketer prompt
Plain-language change request in the builder
- 2
tRPC endpoint
Page context + privacy-reviewed payload
- 3
Gemini on Vertex AI
4 tool definitions → structured operations
- 4
Patch engine
Validates and applies operations safely
- 5
Live page
Accept, reject or undo in the panel
Outcome
Results
- Shipped in 6 PRs with 100+ new tests.
- Panel UI with 4 response states, progressive loading, rejection handling and undo — marketers accept or reject edits without touching structure.
- Same period: release gatekeeper across sprints, 40+ code reviews, 2 production hotfixes.
Next case study
AI question reviewer for exam content
Quill Learn