PUBLIC LECTURE
Medical and molecular image analysis
The Shine Dome, Canberra, 11 December 2007
Professor Sir Michael Brady
BP Professor of Information Engineering
Department of Engineering Science, University of Oxford, UK
Professor Brady discussed his team's work on image analysis applied to cancer. There is large variation in the clarity of medical images and Professor Brady and his team have been working on mathematical and physics-based models of image formation and cellular processes to improve diagnosis, therapy evaluation and the study of cancerous cells. He showed a number of his team's results in colorectal and liver cancer, and in tumour hypoxia (when tumour cells are starved for oxygen) as well as acidosis (when abnormal amounts of acid build up in body fluids).
(Click on image for a larger version)
Those of you who knew me about 40 years ago would understand what a poignant moment it is for me to come back here.
(Click on image for a larger version)
Over the past 20 years, we have developed entirely new ways to see inside the body to image anatomy, and we can do so non-invasively. Images such as these showing my brain (shown at the top left), my heart (at the top right) and these breasts – which I confess are not mine – are now absolutely de rigueur in clinical medicine.
But actually we can now begin to watch the body, to highlight aberrant physiology; we can begin to study the functioning of the brain – for example, in this case I show a putamen-caudate nucleus as part of the project we have been doing for looking at Parkinson's disease and its progressions.
More recently, we are now beginning to combine the work that we have done on image analysis, together with the work that has been done over the last 50 years in molecular biology, to begin to understand and to image disease and normal processes at the cellular and molecular level. I hope to have time to come back to that towards the end of my talk.
(Click on image for a larger version)
This slide illustrates one of the typical challenges we run into. Here at the top left and the bottom left I show some work of cardiac MR – this is MRI. You can see that these are spatially beautiful images, but it turns out that they have relatively poor temporal resolution. So if one wants to understand, for example, either tachycardias or bradycardias, or abnormalities of the beating of the heart, then the spatial resolution is awful.
On the other hand, at the top right, in the cardiac ultrasound you get wonderful temporal resolution – near real-time imaging – but we have very, very poor spatial resolution.
(Click on image for a larger version)
So one of the games that we have to play is to bring those two different kinds of images together so that we can build models of the normal beating heart and the abnormal beating heart, and we can begin to understand, for example, the kinds of problems that there are within myocardy.
(Click on image for a larger version)
To get a little closer to the topic of my talk tonight: here is a CT scan – a CAT scan – of a region of interest, in this particular case around the lower abdomen. You can see the very, very high precision of this image, which is to do with 3D X-ray, but of course the information is often ambiguous. If, on the other hand, we take a positron emission tomography (PET) scan of the whole body, now we can see (as highlighted here by the arrows) two possible tumours. But it is hard to know, because of the poor spatial resolution, exactly where those tumours are.
So now what we are faced with is taking the image at the top left, which has essentially an image of integrated attenuation of X-rays, and the image at the bottom, which is showing us essentially about metabolism – in this particular case, glycolysis within the body – and pulling them together into an information fusion which provides more information to the clinician than would be available from either of those two images singly.
(Click on image for a larger version)
Indeed, that has led to the development – and I am delighted to have had Nick Cerneaz along the way, one of my graduate students and former Chief Operating Officer of Mirada –from what was essentially a piece of technology developed in a university laboratory to something that became a commercial product and is now in use in thousands of hospitals around the world. That actually gives me a tremendous buzz, for while it is a huge pleasure and privilege to write papers, there is nothing in my life that compares to the thrill of watching doctors actually use the fruits of my applied mathematics to cure people – something I hope to come back to continually throughout the lecture.
(Click on image for a larger version)
There is a fundamental problem that we run into, and this is what informs my talk tonight.
Medical image analysis is intrinsically tough, and it is intrinsically tough for a number of reasons. First of all, the signal-to-noise ratio – that is to say, the quality of the images – is very poor, typically 10–15 times worse than standard CCD cameras. There is enormous variation of those images across the population. The shapes that we have to deal with for organs inside the body are extraordinarily diverse and complex. (It turns out that polyhedral livers are quite rare.) And disease signs are often quite subtle, so even clinicians find it quite difficult to see disease, particularly in the early stages.
Yet, despite all of those difficulties, in order to be taken seriously we need to be able to build or to do science that can lead to the construction of systems that work 24/7, 99.9 per cent of the time, with very, very few false calls. How on earth can we reconcile this extraordinary, exquisite difficulty of the images that we deal with, and the diversity, and at the same time get that kind of level of performance, unless as scientists we basically mobilise knowledge?
That knowledge turns out to be, first of all, knowledge of physics – for example, I spent 10 years through the 1990s (again with Nick Cerneaz) building models of the passage of X-rays through the breast, in order to build more reliable mammography systems. We have worked on other things, for example for dynamic reconstruction of PET images and contrast enhanced MRI, using pharmacokinetics. I will maybe mention a little bit about that later on.
I mentioned about anatomy and, increasingly, physiology. And then recently, as I entered my geriatric stage of life, I began to learn a little bit of biology and chemistry, and in particular inter- and intracellular processes. That is what I would like to come back to at the end.
(Click on image for a larger version)
My primary driver in life is cancer, and the figures are truly shocking. In developing countries, around 1 in 3 people – imagine it as one-third (and probably 20 years from now half) of the people in this room – would be diagnosed with cancer at some point during their lifetime. You can look at the incidence and, for a whole range of reasons which I won't go into now, that number is expected to double over the next 20 years.
To give you some idea of the futility of government targets – although I see that you have recently done something about this – 130,000 people died of cancer in the UK in 2006, and 50 per cent of those were under the age of 75. The government target is 20 per cent to be under 75 by 2010. There isn't a snowball's chance in hell that they are going to meet that target.
In the figure at the bottom of this slide I have shown you some of the incidence and also the death figures. You can see that for women, breast cancer dominates, and for men, prostate cancer dominates. The imaging of prostate is actually quite difficult.
The one thing which is common to both, which is second, is in fact the colon and the rectum, and that is what I thought I would talk about tonight to give you an idea of the kind of work we do.
(Click on image for a larger version)
Colorectal cancer is very much a disease of the developed world. It is not really encountered much in the underdeveloped world. It is the second most common cancer in the UK, and also in Australia, with about 14.5 per cent of the cases for men and women.
The survival rate at five years is about 50 per cent – actually the survival rate at one year is only about 75 per cent. Most people die of metastases, and they die of metastases to the liver or to the pelvis. That is one of the reasons why we do both colorectal and liver cancer, because of the flow of metastases through the portal veins to the liver.
At the moment, surgery is the only curative therapy, but – just so that you are aware – the amount that is removed is typically the size of this water tumbler I am holding. This is not a pleasant proposition. It usually ends up with a colostomy.
Chemotherapy is used in 65 per cent of cases, and that is a figure that is worth thinking about and coming back to at the end. And patient management decisions are largely based on the analysis of MRI.
So those are the figures, and that is why, about three years ago, we were approached by a group of clinicians at the Churchill Hospital, in Oxford, to see if we could help them to work on this.
(Click on image for a larger version)
Just to worry you a little bit, I show you a couple of pictures here. In the top left-hand panel you can see, for example, the hip joint, you can see the bladder; in the lower left-hand panel you can draw a line that bisects the hips; returning to the top panel you can see the coccyx, which is the base of the spine, you can see the rectum, and around it you can see the mesorectum. If you watch the little video loop in the right-hand panel you will probably see that at a certain point the tissue becomes light, and that is the cancer that we are trying to find. We have to be able to measure that, we have to figure out how far up it is, and we have to be able to give advice on the surgical procedure – whether surgery is in fact an option.
(Click on image for a larger version)
The first thing we can do, using standard technology from computer vision these days, is to track a contour – I am not going to go into that; I talked to a conference last week and explained how we do this – through these various slices and build a three-dimensional representation, in this case, of the lower rectum and around it the beginnings of the mesorectum. (The mesorectum is the fatty layer shown as a white ring in the top right-hand panel.) The importance of that is that is where the lymph nodes live, and that is what really determines, to a very large extent, the decision to go to surgery or not.
(Click on image for a larger version)
At the moment in the UK, all cancer patients are treated through a team which is known as the Multidisciplinary Decision Team. This comprises a couple of radiologists, at least one but usually two surgeons, a pathologist and a medical oncologist, a senior nurse and so forth. I will come back to that a little bit later, because one of the things that we have been developing is the decision support tool that is currently deployed in the John Radcliffe Hospital, in Oxford, and which we are now commercialising – that is something I have no qualms about.
When a patient presents, on his first image, usually with CT and MRI, the team then do a first assessment and there will be one of three outcomes from that. The patient will normally – as I showed you, 65 per cent – will be sent for downstaging chemotherapy, or will be sent immediately for surgery, for total mesorectal resection. Or, if surgery is deemed not to be an option, will be essentially sent for surveillance and eventually for palliative care. After the downstaging chemotherapy there will be a second image taken, and a second assessment. (The second image is taken at the point numbered 2 here.)
The kinds of questions that doctors want to understand are, first of all: can we assess the lymph nodes, can we assess the circumferential resection margin (CRM)? I will show you that in a second. These require that we accurately segment that mesorectal fascia, the mesorectum I showed you, that fatty stuff.
Second of all, they ask: can we evaluate the response to the downstaging chemotherapy? Did it work? Has the patient now progressed to the point where surgery becomes an option? That turns out mathematically to be quite a hell of a challenge, because this is a large displacement, complex non-rigid registration.
I should say at this point that there is something of a dilemma that faces me in giving this talk tonight. If I were giving a talk to an engineering or mathematical audience there would be quite a lot of maths that I would spell out, because it is necessary to understand the story. On the other hand, a lot of people here tonight perhaps do not have that mathematics. And so I want to show the pictures but at the same time I have left some of the mathematics in so that those people who have that mathematical background can at least get a flavour of the kind of stuff that we are doing. But you don't need to understand the maths to understand the general flow of the argument.
(Click on image for a larger version)
The top left panel here contains a picture of the rectum and the mesorectum around it, showing a big black blob which is an infected lymph node. What we are really interested in is what is the clearance between that infected lymph node and the mesorectum boundary around it. The clinical trials strongly suggest that that CRM, the circumferential resection margin, will be at least 1mm if surgery is to be an option. That means making that measurement. So the National Institutes of Health require that you make a measurement, typically in 2D.
Actually, many of us realised that that is a pretty stupid thing to do, first of all because the sampling planes for the MRI are not necessarily going to be orthogonal to the axis of the rectum, and that inevitably will lead to a foreshortening or forelengthening of that distance. So that distance by itself is not terribly useful. Okay, one can resample there.
(Click on image for a larger version)
The next thing we want to do, though, is to segment this mesorectum, and that is where we begin to have to deal with a little bit of mathematics. What we need to do is to be able to find, in the end, the blue contour shown in the top right-hand panel, towards the outside and the inside. You see here the lumen, you see air inside here, the tumour and the mesorectum. We have to start from lots and lots of places where we know exactly what the tissue is – in one place we know it is air, in another we know it is tumour and so forth – and then grow them and merge them. And it actually turns out that about 15 years ago there was a very beautiful piece of work developed at UCLA by Stanley Osher and James Sethian for embedding active contours within an implicit surface (I have written it down for the mathematicians here) and so the desired solution, the blue contour I referred to, is the zero level set.
It is easy to show, in fact, as a result of that, that you end up with an equation of the particular form I have set out here, where we have the 'speed function', the gradient and the rate of change over time. The speed function typically has a term which attracts the level set to intensity changes, in order to try and find the boundaries, and it needs to have a term which represents what it believes is inside the mesorectum.
(Click on image for a larger version)
Basically we have to do those two things, but the devil actually turns out to be in the detail. We first of all have to figure out how we can represent those signal changes, and we have to figure out what the inside of this particular mesorectum is.
Having got through those pretty quickly, I have thrown up, just for completeness, what the actual thing that we solve is. There is a term related to the curvature rate, there is one that is related to the region, basically based around a representation of the region/probability mass function, and one that has got potentially volume. That is mathematically what we actually end up solving.
(Click on image for a larger version)
The next thing to ask is: how the heck can we find these intensity changes? Here we run into a problem, because if you go and ask 99.9 per cent of people in computer vision how they would find those intensity changes, they will say they would take the gradient of the image. And that turns out to be a pretty dumb thing to do. So what we have done here is to take a whole series of points around the mesorectum which have been labelled by the surgeon, and at each point we have drawn a cross-section. In each one of these we have taken a cross-section of the signal and then a cross-section of what I call phase congruency and a cross-section of the image gradient.
The most important lesson to take home from this picture is that actually the cross-sections that correspond to the magnitude of the image gradient, which is what most people use, convey almost no information.
(Click on image for a larger version)
Fortunately, there is a different way of doing that, which is to do with local phase. This is an idea that was developed by Professor Robyn Owens, at the University of Western Australia, around 20 years ago.
(Click on image for a larger version)
The idea is that you take an image like the one at the top left of this slide and you can try to do what most people in computer vision do, which is to compute the local energy or the gradient. Or you can treat the alternative representation shown below that, which is known as the local phase representation. In order to do that, we need to compute what electrical engineers call quadrature phases. A key part of that is to use a bandpass filtered image – for those of you who know any engineering or filtering.
(Click on image for a larger version)
The particular way we have to work with this – just to give you an idea, again for some of my mathematical colleagues here – is to take an image, bandpass filter it and compute three representations with a couple of convolutions. From those we can compute a local amplitude, a local phase and a local orientation.
And the important thing is that the second and third of those re
