ASU researchers are working on VR glasses that walk users through medical procedures in real time

A project funded by NASA could help bring emergency medical care to rural Arizona.

Arizona State University researchers are developing augmented reality glasses that can help walk users through some procedures in real time — without needing to be online. That’s especially important in rural areas — or in space — which is where the NASA component comes in.

Pooyan Fazli is an assistant professor in the GAME School at ASU as well as graduate faculty in the School of Computing and Augmented Intelligence. He said he and his colleagues are currently running lab-based studies on the glasses, working with ASU’s nursing school to do simulated work on medical manikins. If that’s successful, he said, they can move to the next step — which would be to work with hospitals and community health centers.

Fazli spoke with The Show about the project, including about what these glasses look like.

Full conversation

POOYAN FAZLI: So imagine you are in a space mission and we have astronauts and other crew members and some medical emergency happens. So what you can do is use our device, which is basically AR glasses — or augmented reality glasses. And these glasses can look at the scene, process what is going on, and interpret the scene. And generate basically step-by-step instructions on what to do, and how to perform some medical procedure on the injured crew, basically.

And these devices are equipped with some AI models. And these AI models are the extension of these large language models or LLMs. Which may you have heard about it, the ChatGPT, for example. But these extensions of LLMs can process other modalities like images, videos, and audios. We call them multimodel large language models. They can look at the scene, you can ask questions on the scene and get like instructions on how to perform some medical procedure on the person.

MARK BRODIE: So you mentioned that they are sort of loaded with these AI platforms. And I think that's interesting because in doing some reading about this, these are meant to be used offline, right? Like you don't necessarily have Wi-Fi if you're on Mars, for example, or somewhere up in space. How exactly does that work?

FAZLI: Yeah, exactly. So there is no need for internet connection. And the whole thing can be run offline, basically. As long as your AR glasses can be attached to a computer or a laptop, the AI model can be run on the machine and process the scene. And you can do question answering without having any internet connection.

BRODIE: Does that mean that the AI model, though, doesn't get updated necessarily? Like if some new information comes along or new medical breakthrough, the glasses won't necessarily have that on them.

FAZLI: Yeah, that's a great question. So it can be updated, but not like during the procedure, you know, when it's basically at work and processing the scene. Of course, we can update the AI model outside the work and then bring it back to the mission. But when it is not connected to internet, it can still work, and it can still process the scene. But also, if it connects to internet, which is totally fine, it can update the AI model and new databases, new data can be added to the AI model.

BRODIE: OK. So obviously most of us will never be in space, but am I right that you are looking at this as potentially having applications for those of us on Earth as well?

FAZLI: Yeah, that's a great question. So I think like one of the main applications of these AR glasses is rural health care, basically. For example, in Arizona, there are like many counties that are designated as a term we call health professional shortage areas — or HPSAs. Meaning that, you know, there are a shortage of physicians, nurses, hospitals in these areas.

And when some medical emergency happens, it takes time for medical experts ... to get to the scene. So this device, can be helpful in those scenarios that. For example, family caregivers or one of the family members or community health workers, they can use these AR glasses and at least use some basic medical tasks like suturing wounds, for example. Or doing CPR or administering medication until the help arrives, the experts arrive, and do further work on the patient.

BRODIE: Well, let me ask you about that phrase you use, basic medical tasks here. I assume you're not looking at these as a way for a layperson to do surgery on a family member or on themselves if they need to. But how confident are you that people with no medical training can use these glasses and use the AI loaded on them to do some of the things that you're hoping they'll do?

FAZLI: So, of course, they're not going to do like, complex like operation on the patient. What we are hoping that at least they can start doing some basic tasks before the experts, the medical experts, arrive to the scene. Could be something like a CPR or wound like stitching, for example ... or administering medication to the patient.

There are some basic tasks that a layperson, family caregivers — and family caregivers or health professional aids, they have some backgrounds as well. It's not that they have no knowledge about patient care.

BRODIE: So in a situation like that, if I were wearing your device and somebody in front of me needed medical assistance, let's say they cut themselves and needed help suturing that up. If I looked in the glass, and I was looking at the wound that needed care, it would show me sort of in front of my eyes, OK, here's what it is, here's what you do, here's what you need. And sort of walk me through how to do that?

FAZLI: Yeah, exactly. So, you know, these AR glasses, they have visual overlays of the instructions or images which guide you, know, through the procedure. But also audio as well that, you know, it can generate audio depending on your preference.

BRODIE: How good is the information available through AI at this point to help people get through some of these things? Like if you're loading all this information onto the glasses, is that information where it needs to be to really be useful?

FAZLI: ... Like any other like AI model, you know, there are like limitations in terms of like accuracy of these models. You know, we are always trying to retrain these AI models to make them like more and more like accurate. The training data for these AI models are mostly medical videos that are mostly online. And these medical videos are also coming with question answers on the videos as well.

And we are using these data for to train like other models. ... As we collect like more and more data, more videos, more scenes from medical operations, you know, our models are also going to be more and more accurate. it can cover more scenarios, more procedures we can do with our AI model and AR glasses.

BRODIE: How big of a dent do you think this could put into the problem of people having access to emergency care in rural Arizona?

FAZLI: It could have a lot of impact, because there is a lot of like a discrepancy, like in terms of, you know, the coverage, health coverage, like in those areas. And, again, the shortages of staff, the nurses, physicians in those areas is significant. And in Arizona in particular, this problem is even more significant. So these devices can be impactful.