From 3M Health Information Systems
Podcast Episode Transcript: Data analytics: Extracting the signal from the noise
Gordon Moore: Welcome to Inside Angle. This is Gordon Moore, Senior Medical Director for 3M Health Information Systems.
Today we have a conversation with Dr. John Cromwell. Dr. Cromwell is a clinical professor of surgery at the University of Iowa where he serves as, wait for the list, it’s big, Associate Chief Medical Officer, Director of Surgical Quality and Safety, Director of the Division of Gastrointestinal Minimally Invasive and Bariatric Surgery, and Faculty in the Interdisciplinary Graduate Program in Health Informatics.
And that’s just part of his bio. The reason that I want to with John today is that I’ve had a chance to speak with him before about his work in natural language processing, machine learning, quality, safety. And what’s fascinating to me is that he sits I think in the intersection of technology and data and analytics and quality and safety in a way that speaks to the promise of what technology can do. We can talk about that, I hope, and then talk about the frustrations of what currently exists in sort of technology and how we interface with that, and figure out what’s the path forward that delights both the clinicians and improves outcomes for patients.
So, welcome, Dr. Cromwell.
John Cromwell: Thanks very much, it’s really a pleasure to be here, and thanks for having me on the podcast.
Gordon: What I’d love for you to do is just start with: what got you into this line of research?
John: This is a really interesting time in my career. Having been in academic surgery since I finished my training many years ago, I originally was doing cancer research straight out of my training, and at some point along the way, like many researchers do, I lost my cancer research funding due to the challenges of getting NIH funding nowadays. I, at that point, had a really big interest in extracting useful information from the medical devices and from the medical record systems that we are all now using, and putting those to good use for bigger issues in medicine. So I credit this gap in cancer research funding with my ability to move forward in this area in the intersection of machine learning and NLP and clinical care.
Since I’ve been here at the University of Iowa, I’ve been really trying to push the limits on what we can do at the front line of medicine using the tools that have become so ubiquitously available now in terms of these technology oriented things.
Gordon: So you got into computer science world I think in a way. What kind of training did you have in that?
John: Well, I had really none. A lot of what I did along the way was learned on my own. Now, as an undergraduate, I was a physical chemistry major and I’d done some stuff doing simulations of hemoglobin and things like that in my undergraduate training. But it was quite a long stretch, it took several years for me to start at a small level doing very simple things with machine learning, to the point where today, in terms of building platforms that manage machine learning at an enterprise level. So it’s been a long, steep slope for me.
Gordon: Unpack for our audience a little bit about what is machine learning.
John: I’ve heard numerous people talk about machine learning and I think I’ve heard just as many definitions of it as I’ve heard people speak about it. But to me, it’s essentially extracting the signal from the noise and large amounts of data. The signal can be whatever it is that you’re interested in. From my standpoint, it’s the application of computational methods to information systems within the hospital or within our organization.
Gordon: One of the things that I know about you is that, at a prior HIMSS conference, you wowed the audience with the dramatic reduction in surgical site infection, post-operative infections, because of the work that you’ve been doing. Could you describe how you got into that and a little bit about that work?
John: This work started many years ago, probably around 2010 or 2011. It was at that point that I took on this role as the Director of Surgical Quality and Safety at the University of Iowa. As I started looking into the data from our hospital and from lots of hospitals around the country, it’s pretty clear that hospital-acquired infections are one of the big things that drag down the value of the healthcare we deliver. And within that space of hospital-acquired infections, surgical infections and pneumonia are probably the largest in terms of their effect on patients and their expense to the system.
I had some ideas on how to deal with surgical infections being a surgeon myself, but a lot of the things that we wanted to be able to do really depended on being able to assess the risk, an individual patient’s risk of developing a surgical infection, and having that data available to me or to any surgeon at the time that they’re actually performing the surgery.
Most of what we do nowadays in preventing surgical site infection involves application of the same preventative methods to every patient who comes through the system. So that they do some surgical preparation of their skin before they come in. If they’re having intestinal surgery, they do a bowel prep. After surgery or just before surgery, they get intravenous antibiotics. I mean, there’s stuff that we do with the wounds after surgery. That’s just how we treat every patient to reduce surgical site infections.
What became pretty clear is that there are interventions that we as surgeons have the ability to do in real time at the time that we’re closing a patient’s wound that can address the risk in very profound ways. But it’s not things that you would want to do on every patient. It’s interventions that you’d want to be very selective with and use in those settings. So, things like delayed primary closure of a wound or the use of prophylactic negative pressure wound therapy are a couple of those things that can be applied very selectively.
My idea at that time was that, if we could use the information in the medical record system to really have an objective approach and a very systematic approach to quantifying that risk using data from the patient’s history, using real-time data from the operating room during that procedure, that we could have a much better idea of the risk involved for this individual patient. That’s where the idea came from.
So we started at a very small scale using me writing R code, a Python code, on a laptop computer, and modeling the data that we had available. Then eventually moving that into a system where we could apply that machine learning in the operating room and gather that data in real time. And what we learned was that, through a small pilot study in colorectal surgery patients, that if we use that information and we use these wound care strategies in a very selective way, that ultimately we were able to reduce the risk of surgical site infections by about 74 percent over the course of that study. That was a pretty profound moment for us in terms of understanding how we could apply machine learning to a very specific decision point in surgery.
Gordon: So what it sounds like to me, is that there are lots of things you can know about a patient at the time of wound closure but it’s more than a human is able to consume reliably consistently. That’s where you can both find all those different variables and analyze them and serve them up in a way that provides significant decision support, which then enables the work of the surgeon.
John: You’ve said it better than I could. I think surgeons are really focused on what they’re doing in an operation. And we have great surgeons and we have great support staff, but the ability for any individual or human to integrate all of the variables in the system into a, in an objective way, into a risk score is virtually impossible.
One of the things that we struggle with in medicine is something like hospital readmissions. There was a really nice study done out of the University of California at San Francisco several years ago that looked at an individual physician’s ability to predict if a patient was going to be readmitted to the hospital once they were discharged. They found that physicians were no better than the flip of a coin at really determining that risk.
And yet we have machine learning algorithms now that do very, very well on those types of predictions in a very general population. So that highlights sort of what physicians are and are not good at, and how we can use the technology to really augment very specific decisions that we’re making.
Gordon: So you had figured out this process of consuming a bunch of data and you do that by abstracting information automatically out of the record. During your study, did you have to have people fill out a bunch of forms? Was that the way you got the information initially?
John: We were able to use a data warehouse to house most of the historic data. There’s roughly 30 features or variables within the model that we use to score the risk of surgical site infection. Most of those were available to us that people didn’t have to enter. However, there were seven or eight variables from the operating room, things like how much blood loss was there and what was the lowest and highest blood pressure, what was lowest and highest temperature of the patient’s core temperature during the procedure. Those things were entered by our nursing staff in the operating room at specific points in the workflow prior to the surgeon’s closing the patient’s wound. So, workflow became a very important part of this whole thing.
Gordon: Yeah, that’s part that’s really interesting to me. Because when I think about this issue of decision support, it’s ubiquitous in healthcare. I think the information that we could potentially bring to bear to support a decision at any moment with any patient is almost beyond human comprehension. I think about the pharmacopeia, all the different drugs that could be used, all the different drug interactions, side effects, and things like that. It’s like asking a travel agent to try to remember all the different flights and connections. We never do that. We use computers to do that sort of thing now.
So that to me is the promise now of modern medicine that can be enabled through the smart use of computers. So that sounds good to me. When I think about the frustration of clinicians and practice right now is because they have too much time interfaced with computers trying to do their work, because the promise is so great, we’re pulled into that, but the technology isn’t as advanced as we would like it to be. But you’ve used other processes to extract information from the medical record, is that correct, using natural language processing or other tools like that?
John: We’ve used a number of methods and we’ve interfaced with our information systems in many ways. In our EMR we’ve used industry standard interfaces, things like FHIR and HL7 and other more vendor-specific interfaces. But then in terms of extracting features from the data, we have used natural language processing. We’ve used a number of other methods of extracting both structured and unstructured data out of the record.
Gordon: So that to me is just critical. We’re drowning in information, some of it is in this big block of what’s considered inaccessible because it’s in Notes and things like that. That’s where the natural language processing obviously goes. So that’s a huge advantage I think in technology most recently.
Let me pivot for a second. In prior discussions with you, you’ve also, you’ve gotten well beyond wound closure and surgical site infection and into other aspects. Do you want to talk about other kinds of things that are interesting to you and where you’re looking into automating information and bringing that to bear?
John: Sure. We’re really focused on staying at the front line. There’s lots of different initiatives going on in hospitals to improve value, but one of the areas that’s been really difficult to crack is at the front line. We’re really focused on looking at very specific decisions that we make at the front line that affect patient outcome and, of course, value that follows along with that. So as we’ve tracked down those very specific decision points, we’ve really worked on curating those points at which almost a binary decision in what people choose, what providers choose at that point, can make a huge impact in the patient’s outcome and in the value that we deliver.
It seems like they’re almost limitless. So our problem is prioritizing these areas. So we’ve looked at our ability to impact patient outcomes and prioritize those.
Some of the things we’ve come up with, one of these is blood conservation, perioperative blood conservation. We know that lots of blood gets transfused in the context of surgery, and much of that is very, very necessary. But we also understand that, if we’re able to prepare patients better for surgery, from the standpoint of, if we can identify anemia in a patient upfront and we can correct that anemia in time for their surgery, that the risk of requiring a blood transfusion and the costs and the risks and the physical risks associated with that, go down markedly.
We know that blood transfusions do more than just raise your blood count. They also are a non-specific immunosuppressant. Patients who get blood transfusions have worse outcomes in terms of higher risk of infections, higher risk of, for instance, anastomotic leakages in colorectal surgery. Virtually every outcome you look at can be worse in the setting of patients who’ve gotten blood transfusions. So, one of the frustrating things is the lack of a very systematic approach though to identifying those patients beforehand and getting them through the appropriate therapies to better prepare them for surgery.
So we’ve been working on models to identify those patients upfront. Through a combination of machine learning, technologies that allow for non-invasive screening of anemia, have been able to begin to reduce the number of blood transfusions that we give in the hospital. As you know, there’s risks of viral transmission with blood transfusions, in addition to the other things I talked about. They also turn out to be very, very expensive. The fully-loaded cost of a transfusion of a unit of blood is probably around $1000, and hospitals transfuse thousands and thousands of units of blood every year.
Gordon: One thing that I think about, as you describe the process of thinking about who needs blood products or not, that a care guideline addresses that on its face. That’s the intent. Yet what I’m hearing is that a machine learning approach may be better. What’s your take on that?
John: Yeah, I do. I do think it’s better. It’s pretty clear, after being through a number of these initiatives with machine learning, that it’s simply a high-tempo environment where people are trying to care for many patients with very complex conditions, that clinicians simply aren’t able to integrate all the information needed to make the best decision that they can make. Even though we have wonderful clinicians who make the decisions every day, we simply can do better by integrating more information.
The way that EMRs are constructed, there’s a lot of room for improvement, as I think most clinicians will agree with, and the information needed, just getting the information out of the EMR system to be able to integrate it can be quite a difficult challenge, looking in different parts of the record and looking at unstructured and structured data. So, machine learning is really built for that and it’s quite facile at doing that. Once we’ve built the pipelines in and out of the system, we can really apply it to so many different clinical conditions if we’re attentive to what the workflow looks like.
Gordon: Out of curiosity, I think about this as a naïve clinician in the machine learning world, when I hear of the work that you’ve done, I think about all the different hundreds and thousands of decision points that could be supported through this kind of work, how easy or difficult is it to build the decision support tool using machine learning? Is it minutes’ effort, is it hours, days, months?
John: Yeah. It really varies. I think we can separate this really into three real parts. One part is gaining access to the data. The second is actually creating the machine learning algorithm to provide the feedback that you’re looking for. Then the third is validating this in a way that makes us feel like it’s ready for deployment in an organizational enterprise patient care way.
Getting access to the data is usually the longest part of this, because from one project to the next, we may not be using the exact same data features. In some cases we are, and that makes it go much more quickly. But if we’re building a new pipeline to gather certain types of data out of the EMR, that part can take weeks to build those pipelines through the EMR interfaces if we’re needing it in real time.
The machine learning is actually the quickest part of it. Once you’ve got data that’s been cleaned, we can spend an afternoon building a model, and that part’s done. Then we spend weeks, if not months, validating the model, by applying it prospectively against numerous patient populations to make sure it performs in the way that we want it to, before we actually use it for clinical decision-making. So you’re talking about, from end to end, is probably several months of work.
Gordon: One thing that occurs to me with the machine learning idea, in the most stereotypic reductionist view, I’m thinking of a bunch of 20-somethings in Silicon Valley who are turning on massive machine learning engines and saying “I don’t need doctors or nurses, I can just run the data and say this is the best thing to do for a patient.” But then I’m also reminded of the occasional self-driving car that goes off the road. So I’m a little bit nervous about that. Now, I admit being a physician and feeling a little protective of the profession, but am I overblown in my concern?
John: No, I don’t think so. We’ve had experience with a number of those technologies that you’re referring to in terms of companies who build a tool. One example is readmissions. Several years ago, lots of companies were working on readmissions and helping the hospitals with readmissions. They would extract the data, they build the model, they build the dashboard, and give people access to the dashboard. But then people didn’t know what to do with the information. Well, once you know a patient is at higher risk of readmission, what do you do with it? You don’t really know why they’re going to be readmitted.
So you have people applying interventions on patients for potentially reasons that had nothing to do with why they might be readmitted back to the hospital. At the same time, you were potentially removing resources away from patients who you felt were at lower risk of being readmitted, without really a deep understanding of the underlying output of the model. So that’s one example of the way that technology companies have failed at bringing these types of things into the healthcare space.
I think what we’re doing differently is we’re really, really very focused on very specific decisions. We don’t want to know who’s going to be readmitted. What we want to know is: if I make this particular decision on my patient one way or the other, how is that going to affect their outcome? That requires a much deeper understanding of what are the decisions actually being made on a day-to-day basis. That’s where technology companies really need the input of clinicians and providers.
Gordon: When I think about the sweet spot, and again this is a little bit out of my depth not being trained as a computer scientist, but I’m thinking that there’s this machine model and an expert model, and putting those together may have an advantage over both. Is that what your validation does in that third stage of your work?
John: Yeah, exactly. If we failed to connect the dots in some way where we’ve not applied the tool to the right part of the physician workflow or we forced a physician or a provider out of their normal workflow, the tool won’t get used. That’s a real important spot where these fail, is that we fail to implement it in a way that’s useful or that’s actionable by the people using them every day. So, really, that validation step, as you said, it allows us to make sure that we’ve connected all the dots from one end to the other, to make sure that the actual outcome that we’re seeking is what we’re actually getting.
Gordon: So I’m thinking about, last parts of the conversation here, and I’m thinking that, as a person, as a surgeon, as a quality and safety expert researcher, what do you hope for the future? What obstacles do you see? And what are you hoping to see solved in the next year or so?
John: There’s so much promise for the technology to improve patient care. As we’re finding out, hospital care is the most expensive type of care that we deliver in the healthcare system, if you compare it to outpatient care and everything. As I look at the system, I see that there’s just enormous missed opportunities at the point of care to improve outcomes. At the same time, we have technologies that are very fragmented. We have companies who produce sort of one-trick ponies in terms of what their system does for hospitals.
So as a healthcare leader in my hospital, we have to figure out, how do we smartly bring those technologies into the hospital in a very unified and strategic way? At the same time, we have to be conscious of our ability to, as I said, curate where those decision points are that will advance patient care.
It becomes more and more challenging to do that. The doubling time for medical knowledge is three and a half years in 2010, and I think by 2020 the doubling time for medical knowledge is 73 days. So, for us to be able to understand and to curate all that knowledge and then apply it in a very systematic way requires a very big change in our information systems in hospitals and removing the silos. So there’s a lot of advancement across the industry that needs to be done in data governance, particularly with clinical data, not so much with financial data, but certainly with clinical data there’s a lot of work to be done.
Gordon: You mentioned data governance several times, and I’m curious what you mean by that, what you envision. Because people can have different takes on the meaning.
John: As an example, our children’s hospital has I think over 60 different registries where service lines, specialists submit data to allow themselves to compare themselves to other similar specialties around the country, to benchmark themselves to get better. As an example, neonatologists may submit to a registry or pediatric cardiothoracic surgeons may submit data to a registry, and they’ll be able to benchmark themselves on their performance against others.
But, unfortunately, none of these systems look anything alike. Whereas in the pediatric neonatology system they may report weights in grams, they may report it in kilograms or pounds in another one. There’s no consistent way of sending data through the system other than what the registry requires. And this data is very, very valuable because it’s been very carefully curated and reviewed from the medical records, and having access to that’s really important. But right now, because we don’t have a common data framework to combine elements out of these systems, the work to do that is a real heavy lifting job.
Even in our own medical record system, there may be silos who chart things differently in one part of the hospital than the way that they do it in another part of the hospital. So you’ve got empty data fields in one place and not in another. That creates big challenges from our ability to make the data useful.
And some work on the hospital side to come to a common data model of how we want to put data in the system and how we want to make that data available, and even how we define our organizational structure. For instance, if I want to pull data and look at performance of specific service line or something like that, our business office may have defined what our service lines look like based upon financial systems that makes no sense for us to [bide 23:15] ourselves up in that way for the purposes of using machine learning on the clinical side.
All of those things affect how useful these systems are. Again, we’ve had vendors bring tools into the hospital and connect and look at our performance in ways that would really be helpful to us if they understood how we think of ourselves clinically. Again, that’s work on our side, to make sure that we understand how we collect our data and how we report it and having a common data framework for that. That makes it much easier for us to plug in machine learning tools and have these really amazing ways to augment physician performance.
Gordon: I think about the doubling of medical information. When I was a family medicine resident, I remember reading at one point that if I wanted to keep up with study publication, I’d have to read about 40 journal articles a day. And that was a while ago. I think that it’s just beyond human comprehension. So the promise of technology is still huge compared to the delivery. We have lots we need to overcome. But I’m fascinated by the work that you’ve done at the University of Iowa in getting to very straightforward decision support around very specific aspects of clinical care. And I want to thank you so much for your time.
John: Thanks, Gordon. It’s been a great privilege to be on here and really for taking the time to understand what we’re doing. We’re really excited about what’s coming down the pike and hopefully have a chance to come back again.
Gordon: Great. Thanks so much.
John: You bet.