A hospital choosing an AI model now has four options, from paying a frontier lab per token to running a model on its own servers. The right choice depends on where patient data can go and how much work the model will do.
Quick answer: at typical hospital volumes, paying per token is usually far cheaper than buying a GPU server. Run your own model when data control requires it or when the server will stay busy, not because it looks cheaper on paper.
Four ways to run a model
Where the patient data goes
Frontier model, standard terms
Your application
Frontier lab API
Short-term retention for abuse monitoring
US or global processing
Frontier model, zero data retention
Your application
Frontier lab API
Nothing kept after the response, where eligible
Usually US or global processing
Open model, rented inference
Your application
Hosting provider
Together, Fireworks, Hugging Face and others
Provider's region
Choose one that signs your agreement
Open model, your own server
Your application
Model server
Mac Studio up to NVIDIA H200 or B200
Stays in your network
How far behind are open models?
Behind, but less than the price gap suggests. On physician-graded health conversations, GLM-5.3 trails the best frontier model by about 13 points.
HealthBench Professional score
General intelligence: Artificial Analysis Intelligence Index v5
Coding: SWE-bench Verified (%)
Epoch AI puts open models about four months behind. The Stanford AI Index shows the closed-model lead widening to 3.3% in March 2026, up from 0.5% in August 2024.
How big is a GLM-5 model?
You may have seen GLM-5 described as an 850 GB download. No GLM-5 release is that size. The closest model at that size is DeepSeek V4 Pro.
Model weights on disk (GB)
Check the licence too. GLM-5, 5.1, 5.2 and 5.3-Flash use MIT, while GLM-5.3 has custom terms that add a security review for very large hosting providers.
What fits on what
| Hardware | Memory | Runs | Purchase price |
|---|---|---|---|
| Mac Studio M5 Ultra | 512 GB unified | GLM-5.2 4-bit, at the edge. GLM-5.3-Flash 4-bit | US$22,299, 1 TB |
| 8× NVIDIA H100 | 640 GB | GLM-5.3-Flash. Not GLM-5.3 FP8 | Reliable new prices not published |
| 8× NVIDIA H200 | 1,128 GB | GLM-5.3 FP8 | US$350,000 to 420,000 |
| 8× NVIDIA B200 | 1,440 GB | GLM-5.3 with full 1M context | US$390,704 configured |
What tokens cost
Output price per million tokens
A worked example: scribe notes at one hospital
Hospitals do not publish their token use, so this is our illustrative model and not measured data. Change any input and the answer changes.
Assumptions for a mid-size hospital
- 60,000
- AI scribe notes a month
- 8K / 1.5K
- input and output tokens per note
- US$385K
- 8× H200 server
- US$7.2K
- server running cost a month
2,000 a day. Our assumption
Single pass. Our assumption
Middle of the street price range
10.2 kW colocation plus a quarter of an engineer
Total cost over three years at 60,000 notes a month
- Opus 5.5 tokens
- GLM-5.3, rented
- 4 Mac Studios
- 8× H200 server
View data as a table
| Period | Opus 5.5 tokens | GLM-5.3, rented | 4 Mac Studios | 8× H200 server |
|---|---|---|---|---|
| 0 | US$0 | US$0 | US$89.2K | US$385K |
| 3 | US$11.2K | US$3.2K | US$93.1K | US$406.6K |
| 6 | US$22.3K | US$6.4K | US$97K | US$428.2K |
| 9 | US$33.5K | US$9.6K | US$100.9K | US$449.7K |
| 12 | US$44.6K | US$12.8K | US$104.8K | US$471.3K |
| 15 | US$55.8K | US$16K | US$108.7K | US$492.9K |
| 18 | US$67K | US$19.2K | US$112.6K | US$514.5K |
| 21 | US$78.1K | US$22.4K | US$116.5K | US$536K |
| 24 | US$89.3K | US$25.6K | US$120.4K | US$557.6K |
| 27 | US$100.4K | US$28.8K | US$124.3K | US$579.2K |
| 30 | US$111.6K | US$32K | US$128.2K | US$600.8K |
| 33 | US$122.8K | US$35.2K | US$132.1K | US$622.3K |
| 36 | US$133.9K | US$38.4K | US$136K | US$643.9K |
At this volume the server never catches up. Tokens cost US$3,720 a month on Opus 5.5 and US$1,860 on Sonnet 5.5, while the server costs US$7,192 a month to run before you pay for it. Haiku 5.5 would cost about US$93 a month.
Heavier workloads narrow the gap. A multi-pass pipeline can use 37,000 to 44,000 input tokens and 10,000 to 13,000 output tokens per note, per Hyperscribe. That is at the high end.
The same comparison with a heavy multi-pass pipeline
- Opus 5.5 tokens
- Sonnet 5.5 tokens
- GLM-5.3, rented
- 8× H200 server
View data as a table
| Period | Opus 5.5 tokens | Sonnet 5.5 tokens | GLM-5.3, rented | 8× H200 server |
|---|---|---|---|---|
| 0 | US$0 | US$0 | US$0 | US$385K |
| 3 | US$48K | US$24K | US$13.1K | US$406.6K |
| 6 | US$96K | US$48K | US$26.1K | US$428.2K |
| 9 | US$144K | US$72K | US$39.2K | US$449.7K |
| 12 | US$192K | US$96K | US$52.2K | US$471.3K |
| 15 | US$240K | US$120K | US$65.3K | US$492.9K |
| 18 | US$288K | US$144K | US$78.3K | US$514.5K |
| 21 | US$336K | US$168K | US$91.4K | US$536K |
| 24 | US$384K | US$192K | US$104.4K | US$557.6K |
| 27 | US$432K | US$216K | US$117.5K | US$579.2K |
| 30 | US$480K | US$240K | US$130.6K | US$600.8K |
| 33 | US$528K | US$264K | US$143.6K | US$622.3K |
| 36 | US$576K | US$288K | US$156.7K | US$643.9K |
Monthly cost by volume
- Opus 5.5 tokens
- GLM-5.3, rented
- 8× H200, over 3 years
View data as a table
| Period | Opus 5.5 tokens | GLM-5.3, rented | 8× H200, over 3 years |
|---|---|---|---|
| 20K | US$1.2K | US$356 | US$17.9K |
| 60K | US$3.7K | US$1.1K | US$17.9K |
| 100K | US$6.2K | US$1.8K | US$17.9K |
| 140K | US$8.7K | US$2.5K | US$17.9K |
| 180K | US$11.2K | US$3.2K | US$17.9K |
| 220K | US$13.6K | US$3.9K | US$17.9K |
| 260K | US$16.1K | US$4.6K | US$17.9K |
| 300K | US$18.6K | US$5.3K | US$17.9K |
| 340K | US$21.1K | US$6.1K | US$17.9K |
Months to pay off a server, compared with Opus 5.5 tokens
View data as a table
| Period | Months to pay back |
|---|---|
| 50% | 92 |
| 60% | 60 |
| 70% | 44 |
| 80% | 35 |
| 90% | 29 |
| 100% | 25 |
Against rented GLM-5.3 the server never pays back on cost. Even fully loaded, the same work would cost about US$6,536 a month rented, less than the US$7,192 it takes to run the server.
Renting a whole server
Dedicated GPUs sit between tokens and ownership. You get an isolated machine with no purchase.
Renting versus buying 8× H200
- Rent, running all month
- Rent, 10 hours a day
- Buy and colocate
View data as a table
| Period | Rent, running all month | Rent, 10 hours a day | Buy and colocate |
|---|---|---|---|
| 0 | US$0 | US$0 | US$385K |
| 3 | US$87.6K | US$36K | US$406.6K |
| 6 | US$175.2K | US$72K | US$428.2K |
| 9 | US$262.8K | US$108K | US$449.7K |
| 12 | US$350.4K | US$144K | US$471.3K |
| 15 | US$438K | US$180K | US$492.9K |
| 18 | US$525.6K | US$216K | US$514.5K |
| 21 | US$613.2K | US$252K | US$536K |
| 24 | US$700.8K | US$288K | US$557.6K |
| 27 | US$788.4K | US$324K | US$579.2K |
| 30 | US$876K | US$360K | US$600.8K |
| 33 | US$963.6K | US$396K | US$622.3K |
| 36 | US$1.05M | US$432K | US$643.9K |
Dedicated GPU rental, per GPU per hour
Privacy and where processing happens
Frontier labs do not train on API data by default and they will sign a BAA. The harder question for Canadian organizations is where the model actually runs.
Newest models and Canadian processing
| Provider | Retention | Processing in Canada |
|---|---|---|
| Anthropic API | Zero retention available, not for Fable or Mythos | No. Global or US only |
| OpenAI API | 30-day abuse logs. Zero retention on eligible endpoints | Storage only, not processing |
| Google Gemini API | Caching up to 24 hours unless configured | Only older models: 3.5 Flash, 2.5 Flash, 2.5 Pro |
| Azure OpenAI | Set by your enterprise agreement | GPT-6.x is Global only. Regional Canada East has older models |
| AWS Bedrock | Set by your enterprise agreement | Claude Haiku 5.5 only. Sonnet and Opus route to US or global |
| Open model on Canadian GPUs | You decide | Yes |
What regulators expect, whichever route you choose
Sources
Ontario IPC
AI scribe guidance, January 2026
Privacy Commissioner of Canada
Cross-border and generative AI principles
Quebec
Assessment before data leaves the province
HIPAA
If you handle US patient data
Your organization stays accountable
Moving processing to a vendor does not move responsibility
You need
A privacy impact assessment
A contract covering access, retention and subcontractors
Breach notice and monitoring terms
Notice to patients where required
Sources: IPC guidance on AI scribes, OPC cross-border guidelines, OPC generative AI principles. This is not legal advice, so have counsel confirm how PHIPA and your provincial rules apply.
Which one should you choose?
What limits your decision most?
Best possible quality
Frontier model with zero retention
Sonnet 5.5 or GPT-6 Astra lead the health benchmarks
Low cost with no hardware
Open model, rented
GLM-5.3 at US$1.40 in and US$4.40 out
A small pilot, private
Mac Studio
Fits GLM-5.3-Flash 4-bit. Limited concurrency
Data must stay in Canada
Open model on Canadian GPUs
Dedicated rental first. Buy once usage is proven
Busy, steady, sensitive workloads
Your own 8× H200 or B200
Worth it when the server stays busy
A sensible order to do this in
Measure real tokens
Run a pilot and count what a note actually uses
Test on your cases
Compare models on your documents, not only leaderboards
Set your data boundary
Counsel decides what can leave and where
Pick the cheapest route inside it
Then revisit once volumes are known
We build and deploy all four routes for Canadian hospitals and clinics, and we would rather talk you out of a server you do not need. Learn more about our healthcare software work and how we secure open-weight deployments.
