I've done a good bit of research lately on how to put an AI model in front of two kinds of data at once: clinical notes and the medical images that go with them. X-rays, CT scans, tissue samples under a microscope. The data is protected health information, so the first thing you hear is "we can't use hosted AI for this."
That sentence turned out to be half right, and the wrong half took the most reading to untangle. This is the write-up I wish I had found at the start: the research I did as an engineer, for other engineers, architects, and the technical people at healthcare organizations, health-tech startups, the contract research firms that run clinical trials, and consultancies who get handed this exact problem.
The short version: finding a model that can look at an x-ray and read the chart is the easy part. Deciding where that model runs, and under what contract, is the whole job.
The model question is mostly solved
"Multimodal" means the model takes more than one kind of input in a single request, usually text plus images. The class you want is vision-language models, and every current frontier model from Anthropic, OpenAI, Google, and the rest is one. Sending a chest x-ray alongside the written report and asking "does the report match the image?" is a normal API call, the same shape as sending a screenshot of a bug.
So the answer to "is there a model that can do this" is yes, and has been for a while. The questions that remain take work:
- Accuracy on your task. A general vision benchmark tells you nothing about how a model handles your specific kind of image. You need your own evaluation set.
- Edge cases. Poor image quality, a patient who does not look like the training data, a smudge from the scanner. What the model does when it should say "I can't tell" matters more than the happy path.
- DICOM and 3D scans. Medical images ship in DICOM, the file format hospitals use, and a CT or MRI is a stack of image slices with patient metadata in the file headers. Most APIs want a PNG or JPEG. Somebody has to pick which slices to send, convert them, and strip the metadata. I underestimated this part when I started reading.
- Where the model is allowed to run. This is the rest of the post.
One framing point. None of this makes the model a diagnostic device. A model that sorts which cases a doctor looks at first, drafts a report for a doctor to sign, checks whether a patient qualifies for a clinical trial, or pulls structured data out of scanned records still needs a human reviewing the output, validation on your own data, and often a separate regulatory review. FDA's rules for software that acts as a medical device, the FDA record-keeping rules for trials (21 CFR Part 11), the clinical trial quality standard (GCP), and the ethics board that oversees a study (the IRB) all have opinions here. They are out of scope for this post, and the hosting decision does not settle them.
The HIPAA vocabulary you need
Most of the confusion I ran into came from people (me included, early on) using these words loosely. Plain-English versions:
PHI, protected health information. Any health information that can be tied to a specific person, held by a healthcare organization or one of its vendors. An x-ray with a patient ID on it, a doctor's note, a trial participant's record. If you can tie it to a person, it is PHI.
Covered entity. A healthcare provider, a health insurer, or a clearinghouse (the middlemen that process claims and billing). The HIPAA Privacy and Security Rules apply to these organizations directly.
Business associate. Anyone outside the covered entity's own workforce who handles PHI on its behalf. That covers a cloud provider processing PHI, a model API vendor receiving it, and a consultancy building the system on real data. A business associate's own subcontractors who touch PHI are business associates too, all the way down.
BAA, business associate agreement. The written contract HIPAA requires between a covered entity and each business associate, and between associates and their subcontractors. In plain terms, it says what the vendor is allowed to do with the data, requires them to protect it, requires them to tell you about breaches, lets the government inspect them, makes their own vendors sign the same terms, and lets you walk away if they break it.
HIPAA-eligible versus HIPAA-compliant. This is the one that tripped me up early on. Cloud providers label services "HIPAA-eligible," meaning "we will cover this under our BAA." It does not mean your workload is compliant. The cloud provider secures the platform. Securing what you build on it is your job: configuration, access control, encryption, logging, how long you keep data, and the documentation proving all of it. Eligibility is where compliance starts.
Enforcement. Nobody certifies you as HIPAA-compliant up front. The federal health privacy office investigates after a breach or a complaint, and both the healthcare organization and its vendors can be fined, and in bad cases prosecuted.
De-identification. HIPAA gives you two approved ways to scrub data so it no longer points at a person: strip out a fixed list of 18 identifiers (names, dates, record numbers, and so on), or have a qualified expert certify that the chance of identifying anyone is very small. The legal names for those are Safe Harbor and Expert Determination. Data that passes either test is no longer PHI, and you can send it to any model you like with no BAA. The catch with medical images is that identifiers hide in two places you might not think to look: text stamped onto the picture itself (a patient name in the corner of an x-ray) and the metadata stored inside the image file. Renaming the file fixes neither. And plenty of clinical trial work needs to keep patients identifiable, which is why the rest of this post exists.
Three places the model can run
Once the data is PHI and stays PHI, there are three realistic answers to "where does the model run." Bring the model to the data on hardware you control, or send the data to a hosted model under a BAA with one of the two clouds that have paved this road. Each path trades capability, ops load, and paperwork differently, and for image inputs the paperwork is not equal across clouds.
Path A: bring the model to your data
Run open-weight vision-language models on GPUs you control: on-prem servers, a private cloud tenancy, or dedicated GPU instances inside your own private cloud network. Serve them with vLLM or a packaged inference container (NVIDIA's NIM-style microservices are the usual enterprise wrapper).
The open-weight options worth knowing as of mid-2026:
- MedGemma 1.5 (Google, January 2026). The best-known purpose-built open medical model. The 4B variant handles x-ray, CT, MRI, and microscope slides on a single GPU or edge hardware. The 27B text variant scores higher on medical question-answering benchmarks. Released under Google's HAI-DEF terms, which you should read before assuming "open" means "do whatever."
- GLM-4.5V (Zhipu). A general-purpose open vision-language model, 106B mixture-of-experts with about 12B active. Not medically tuned. Useful when a small medical model runs out of reasoning on a complicated mixed image-and-text question.
- gpt-oss-120b and gpt-oss-20b (OpenAI, Apache 2.0). Text only, so they cover the clinical-notes side and none of the images. The 120B fits on a single 80 GB GPU.
- DeepSeek-R1 (MIT license). Text-only reasoning model, and a big one. A university hospital documented running it on-prem for radiology, which is the kind of evidence I was looking for.
- Research-grade radiology models. RadFound (8.1 million x-rays and scans across 10 image types), CheXmix (Stanford, chest x-rays plus reports, CVPR 2026), and OpenMedQ (fully open training data across microscope slides, x-rays, and scans). These are research artifacts. Evaluate them if you need domain-specific options, but nobody is on call for them.
The deployment that convinced me self-hosting is not theoretical is University Hospital Bonn's write-up of a self-hosted LLM service for its radiology department. They locked it onto its own network segment with the minimum permissions it needs, built it so the model underneath can be swapped out (vLLM does the serving), tested it in a real pilot with clinicians, and now run it as a permanent service for more than 10,000 employees processing PHI. A hospital did this, with the constraints a hospital has.
What you get: PHI never leaves your boundary, so there is no business associate at the model layer and no BAA to negotiate for it. Full control over retention, logging, network isolation, and model versioning, which matters for trials because a hosted model can be updated under you mid-study. Predictable cost once volume is high.
What it costs you: the entire security posture is yours, where a cloud BAA would have carried part of it. Open-weight models trail the frontier on general reasoning, and the medical-tuned small models are strong on narrow tasks and weaker on complex mixed reasoning. You need GPU ops expertise, meaning capacity planning, driver and CUDA patching, on-call, and an evaluation harness on your own data. And if "your own GPUs" are a cloud's managed GPU instances, that compute still has to be HIPAA-eligible under the cloud's BAA. Self-hosting removes the model vendor from the chain and leaves the infrastructure provider in it.
One more cost that is easy to miss on a whiteboard: your own hardware caps how much the model can do at once. A hosted API hides a fleet of GPUs behind one endpoint, so a burst of a few hundred requests gets absorbed without you noticing. On a self-hosted box, a single GPU can only hold one copy of the model and work through a limited number of requests at a time, and a vision model chewing on a large scan or a stack of slices ties it up longer than a short text prompt would. With one or two GPUs, that means requests queue instead of running in parallel, throughput is whatever that card can push, and a nightly batch job over a few thousand studies can turn into a multi-day run. Getting real concurrency means buying enough GPUs to run several model copies side by side, and that hardware sits idle most of the day. Budget for the capacity you need at peak, not the average, or accept that some workloads will wait. This ceiling is the reason I stopped pushing further on self-hosted models for anything beyond my own use. A local model for coding works fine, because one developer sends one request at a time. A department sending hundreds is a different machine.
Best fit: existing GPU infrastructure, strict rules about which country or building the data can live in, clinical trial rules that forbid external processing, or a need to freeze a model version for the life of a study.
Path B: AWS Bedrock
Bedrock is AWS's managed service that puts frontier models from Anthropic, Meta, Mistral, Amazon, and others behind one API inside your AWS account.
The paperwork here is the simplest of the three. Bedrock is on the list of services AWS will cover under its HIPAA agreement, and that includes AgentCore, the add-on for hosting agents. You sign one agreement with AWS and it covers every model you can reach through Bedrock, whether it comes from Anthropic, Meta, or anyone else. There is no separate contract with each model company. The agreement itself is a document you accept with a few clicks in the AWS console, and if your company already has an enterprise deal with AWS it is probably signed already. Compare that to going to a model vendor directly, where a BAA means a sales call and a negotiation. AWS also promises in writing not to use your data to train or improve its services, and the underlying infrastructure already carries the security certifications healthcare auditors ask about.
The caveats:
- Coverage is per feature, not per product. AWS's own terms say that if a specific Bedrock feature is not on the covered list, the agreement does not apply to it, and running patient data through it breaks the rules. Check the exact features you plan to use, plus everything around them (where files are stored, what orchestrates the calls, where logs go), in the region you plan to run in. AWS adds features faster than the list gets updated.
- You still have to lock it down yourself. The agreement covers AWS's side. Your side means keeping the model calls inside your private network instead of going over the public internet, encrypting prompts and outputs with your own keys, turning on audit logging, giving people and services the narrowest access that works and only for as long as they need it, keeping patient data in specific accounts and regions, using fake data in dev and test, and writing down where patient data flows. None of that comes turned on.
- AWS's other AI products are not covered. The chat assistants and coding copilots AWS sells are built on the same models but are not meant for patient data.
- Models change when the vendor says so. If you are running a study, pin the exact model version and write it down, because AWS retires old versions on the model vendor's schedule.
Net: as of this writing, Bedrock is the most defensible paper trail for running a hosted frontier model on text plus medical images.
Path C: Azure OpenAI and Microsoft Foundry
Microsoft's version: OpenAI's models hosted by Microsoft, plus a catalog of other companies' models (Foundry), running inside your Azure account.
The paperwork is even easier than AWS if your company already has a deal with Microsoft. The HIPAA agreement is built into Microsoft's standard terms, so there is nothing extra to sign, and Microsoft publishes a list of which services it covers. Microsoft also promises not to train on your data. One thing to keep straight: Azure OpenAI and OpenAI's own API are two different companies' products. A deal with Microsoft does not cover OpenAI's API, and the other way around.
Here is the catch for images. I could not find anything in Microsoft's official documents that says image inputs are covered by the HIPAA agreement. The clearest answers I found were on Microsoft's public Q&A forum, from community members rather than Microsoft itself. They say coverage applies to normal text use, and that images and preview features are not covered unless Microsoft says so explicitly. Two other forum questions raise the same doubt from different angles: one asks whether a partner vision model from the Foundry catalog is covered at all, and one asks whether Microsoft's abuse monitoring (it stores flagged prompts and can have a human look at them) falls inside the agreement. If you instead run a model's weights on your own Azure VMs, that is Path A on Azure hardware, and the responsibility is yours.
The honest summary: the answer is not written down anywhere I could find, and the people asking are healthcare organizations asking in public. Azure can be the right pick if you already live there. For image inputs, get Microsoft to confirm coverage in writing before any patient data moves.
Going direct to the model vendor
Anthropic and OpenAI both offer BAAs through sales-assisted enterprise processes. The consumer products and the standard API tiers do not include one and must not receive PHI. For most teams this route takes longer than Bedrock or Azure, because you are negotiating a contract instead of accepting one that already exists. It is a legitimate path, and rarely the fastest one.
Side by side
| Self-host | AWS Bedrock | Azure OpenAI / Foundry | |
|---|---|---|---|
| BAA for the model layer | Not needed (you are the host) | One AWS agreement covers every Bedrock model | Included in Microsoft's standard terms |
| Image input coverage | Your responsibility | Covered when the feature is HIPAA-eligible | Verify in writing; text coverage is clearer than image coverage |
| Model capability | Open-weight: medical-tuned small models or large generalists | Frontier models from multiple vendors | OpenAI models plus the Foundry catalog |
| Model version control | Full | Pin versions; vendor deprecation schedule | Pin versions; vendor deprecation schedule |
| Ops burden | High | Medium (configuration still required) | Medium (configuration still required) |
| Time to first PHI request | Weeks to months | Days if the AWS agreement is signed | Days if you already have a Microsoft enterprise deal; longer to confirm image scope |
The checklist I'd run before sending a byte of PHI
This is the order I would do it in, which is roughly the reverse of the order I researched it.
- Write down what you need. What goes in (single images, DICOM files, 3D scans, free text, records), what comes out, how accurate it has to be, how fast, how much, and who checks the answers. Half the hosting argument goes away once this is on paper.
- Figure out whether the data is PHI. Can you strip the identifiers, including text stamped on the image and the metadata in the file? Do the study rules or the ethics board forbid sending data outside? If you can strip it, do that and skip most of this post.
- Pick where the model runs before picking the model. Whether you already own GPUs, and where the data is allowed to live, decide this faster than benchmarks do.
- Get coverage confirmed in writing for every service you touch. That includes image inputs and any monitoring or logging the provider does on your traffic.
- Check coverage per feature and per region. A "HIPAA-eligible" label on the product tells you nothing about the specific feature you are about to call.
- Build your test set first, using fake or scrubbed data. No patient data touches a model until it has passed on that.
- Pin the model version and write it down. Hosted models change under you.
- Write down where patient data goes. Where it lives, how long it stays, who can read it, and what gets logged. Use fake data in dev and test.
- Keep a human checking the output, and treat the medical-device and trial rules as a separate project. Picking a host does not answer them.
The thing that shifted for me over the course of this research is how little of the decision is about the model. I started out comparing benchmarks. I ended up reading contract terms and forum threads about abuse monitoring, because that is where the risk lives. The model can read the x-ray. Whether it is allowed to is a question for your paperwork, and the paperwork is different on every path.