![[Go to Home page]](http://www.science.org.au/images/smallseg.gif){kind=small} |
Australian Academy of Science |
| About the Academy |
Home > Events > Lectures and speeches
Medical and molecular image analysis
Those of you who knew me about 40 years ago would understand what a poignant moment it is for me to come back here.
Over the past 20 years, we have developed entirely new ways to see inside the body to image anatomy, and we can do so non-invasively. Images such as these showing my brain (shown at the top left), my heart (at the top right) and these breasts – which I confess are not mine – are now absolutely de rigueur in clinical medicine. But actually we can now begin to watch the body, to highlight aberrant physiology; we can begin to study the functioning of the brain – for example, in this case I show a putamen-caudate nucleus as part of the project we have been doing for looking at Parkinson's disease and its progressions. More recently, we are now beginning to combine the work that we have done on image analysis, together with the work that has been done over the last 50 years in molecular biology, to begin to understand and to image disease and normal processes at the cellular and molecular level. I hope to have time to come back to that towards the end of my talk.
This slide illustrates one of the typical challenges we run into. Here at the top left and the bottom left I show some work of cardiac MR – this is MRI. You can see that these are spatially beautiful images, but it turns out that they have relatively poor temporal resolution. So if one wants to understand, for example, either tachycardias or bradycardias, or abnormalities of the beating of the heart, then the spatial resolution is awful. On the other hand, at the top right, in the cardiac ultrasound you get wonderful temporal resolution – near real-time imaging – but we have very, very poor spatial resolution.
So one of the games that we have to play is to bring those two different kinds of images together so that we can build models of the normal beating heart and the abnormal beating heart, and we can begin to understand, for example, the kinds of problems that there are within myocardy.
To get a little closer to the topic of my talk tonight: here is a CT scan – a CAT scan – of a region of interest, in this particular case around the lower abdomen. You can see the very, very high precision of this image, which is to do with 3D X-ray, but of course the information is often ambiguous. If, on the other hand, we take a positron emission tomography (PET) scan of the whole body, now we can see (as highlighted here by the arrows) two possible tumours. But it is hard to know, because of the poor spatial resolution, exactly where those tumours are. So now what we are faced with is taking the image at the top left, which has essentially an image of integrated attenuation of X-rays, and the image at the bottom, which is showing us essentially about metabolism – in this particular case, glycolysis within the body – and pulling them together into an information fusion which provides more information to the clinician than would be available from either of those two images singly.
Indeed, that has led to the development – and I am delighted to have had Nick Cerneaz along the way, one of my graduate students and former Chief Operating Officer of Mirada –from what was essentially a piece of technology developed in a university laboratory to something that became a commercial product and is now in use in thousands of hospitals around the world. That actually gives me a tremendous buzz, for while it is a huge pleasure and privilege to write papers, there is nothing in my life that compares to the thrill of watching doctors actually use the fruits of my applied mathematics to cure people – something I hope to come back to continually throughout the lecture.
There is a fundamental problem that we run into, and this is what informs my talk tonight. Medical image analysis is intrinsically tough, and it is intrinsically tough for a number of reasons. First of all, the signal-to-noise ratio – that is to say, the quality of the images – is very poor, typically 10–15 times worse than standard CCD cameras. There is enormous variation of those images across the population. The shapes that we have to deal with for organs inside the body are extraordinarily diverse and complex. (It turns out that polyhedral livers are quite rare.) And disease signs are often quite subtle, so even clinicians find it quite difficult to see disease, particularly in the early stages. Yet, despite all of those difficulties, in order to be taken seriously we need to be able to build or to do science that can lead to the construction of systems that work 24/7, 99.9 per cent of the time, with very, very few false calls. How on earth can we reconcile this extraordinary, exquisite difficulty of the images that we deal with, and the diversity, and at the same time get that kind of level of performance, unless as scientists we basically mobilise knowledge? That knowledge turns out to be, first of all, knowledge of physics – for example, I spent 10 years through the 1990s (again with Nick Cerneaz) building models of the passage of X-rays through the breast, in order to build more reliable mammography systems. We have worked on other things, for example for dynamic reconstruction of PET images and contrast enhanced MRI, using pharmacokinetics. I will maybe mention a little bit about that later on. I mentioned about anatomy and, increasingly, physiology. And then recently, as I entered my geriatric stage of life, I began to learn a little bit of biology and chemistry, and in particular inter- and intracellular processes. That is what I would like to come back to at the end.
My primary driver in life is cancer, and the figures are truly shocking. In developing countries, around 1 in 3 people – imagine it as one-third (and probably 20 years from now half) of the people in this room – would be diagnosed with cancer at some point during their lifetime. You can look at the incidence and, for a whole range of reasons which I won't go into now, that number is expected to double over the next 20 years. To give you some idea of the futility of government targets – although I see that you have recently done something about this – 130,000 people died of cancer in the UK in 2006, and 50 per cent of those were under the age of 75. The government target is 20 per cent to be under 75 by 2010. There isn't a snowball's chance in hell that they are going to meet that target. In the figure at the bottom of this slide I have shown you some of the incidence and also the death figures. You can see that for women, breast cancer dominates, and for men, prostate cancer dominates. The imaging of prostate is actually quite difficult. The one thing which is common to both, which is second, is in fact the colon and the rectum, and that is what I thought I would talk about tonight to give you an idea of the kind of work we do.
Colorectal cancer is very much a disease of the developed world. It is not really encountered much in the underdeveloped world. It is the second most common cancer in the UK, and also in Australia, with about 14.5 per cent of the cases for men and women. The survival rate at five years is about 50 per cent – actually the survival rate at one year is only about 75 per cent. Most people die of metastases, and they die of metastases to the liver or to the pelvis. That is one of the reasons why we do both colorectal and liver cancer, because of the flow of metastases through the portal veins to the liver. At the moment, surgery is the only curative therapy, but – just so that you are aware – the amount that is removed is typically the size of this water tumbler I am holding. This is not a pleasant proposition. It usually ends up with a colostomy. Chemotherapy is used in 65 per cent of cases, and that is a figure that is worth thinking about and coming back to at the end. And patient management decisions are largely based on the analysis of MRI. So those are the figures, and that is why, about three years ago, we were approached by a group of clinicians at the Churchill Hospital, in Oxford, to see if we could help them to work on this.
Just to worry you a little bit, I show you a couple of pictures here. In the top left-hand panel you can see, for example, the hip joint, you can see the bladder; in the lower left-hand panel you can draw a line that bisects the hips; returning to the top panel you can see the coccyx, which is the base of the spine, you can see the rectum, and around it you can see the mesorectum. If you watch the little video loop in the right-hand panel you will probably see that at a certain point the tissue becomes light, and that is the cancer that we are trying to find. We have to be able to measure that, we have to figure out how far up it is, and we have to be able to give advice on the surgical procedure – whether surgery is in fact an option.
The first thing we can do, using standard technology from computer vision these days, is to track a contour – I am not going to go into that; I talked to a conference last week and explained how we do this – through these various slices and build a three-dimensional representation, in this case, of the lower rectum and around it the beginnings of the mesorectum. (The mesorectum is the fatty layer shown as a white ring in the top right-hand panel.) The importance of that is that is where the lymph nodes live, and that is what really determines, to a very large extent, the decision to go to surgery or not.
At the moment in the UK, all cancer patients are treated through a team which is known as the Multidisciplinary Decision Team. This comprises a couple of radiologists, at least one but usually two surgeons, a pathologist and a medical oncologist, a senior nurse and so forth. I will come back to that a little bit later, because one of the things that we have been developing is the decision support tool that is currently deployed in the John Radcliffe Hospital, in Oxford, and which we are now commercialising – that is something I have no qualms about. When a patient presents, on his first image, usually with CT and MRI, the team then do a first assessment and there will be one of three outcomes from that. The patient will normally – as I showed you, 65 per cent – will be sent for downstaging chemotherapy, or will be sent immediately for surgery, for total mesorectal resection. Or, if surgery is deemed not to be an option, will be essentially sent for surveillance and eventually for palliative care. After the downstaging chemotherapy there will be a second image taken, and a second assessment. (The second image is taken at the point numbered 2 here.) The kinds of questions that doctors want to understand are, first of all: can we assess the lymph nodes, can we assess the circumferential resection margin (CRM)? I will show you that in a second. These require that we accurately segment that mesorectal fascia, the mesorectum I showed you, that fatty stuff. Second of all, they ask: can we evaluate the response to the downstaging chemotherapy? Did it work? Has the patient now progressed to the point where surgery becomes an option? That turns out mathematically to be quite a hell of a challenge, because this is a large displacement, complex non-rigid registration. I should say at this point that there is something of a dilemma that faces me in giving this talk tonight. If I were giving a talk to an engineering or mathematical audience there would be quite a lot of maths that I would spell out, because it is necessary to understand the story. On the other hand, a lot of people here tonight perhaps do not have that mathematics. And so I want to show the pictures but at the same time I have left some of the mathematics in so that those people who have that mathematical background can at least get a flavour of the kind of stuff that we are doing. But you don't need to understand the maths to understand the general flow of the argument.
The top left panel here contains a picture of the rectum and the mesorectum around it, showing a big black blob which is an infected lymph node. What we are really interested in is what is the clearance between that infected lymph node and the mesorectum boundary around it. The clinical trials strongly suggest that that CRM, the circumferential resection margin, will be at least 1mm if surgery is to be an option. That means making that measurement. So the National Institutes of Health require that you make a measurement, typically in 2D. Actually, many of us realised that that is a pretty stupid thing to do, first of all because the sampling planes for the MRI are not necessarily going to be orthogonal to the axis of the rectum, and that inevitably will lead to a foreshortening or forelengthening of that distance. So that distance by itself is not terribly useful. Okay, one can resample there.
The next thing we want to do, though, is to segment this mesorectum, and that is where we begin to have to deal with a little bit of mathematics. What we need to do is to be able to find, in the end, the blue contour shown in the top right-hand panel, towards the outside and the inside. You see here the lumen, you see air inside here, the tumour and the mesorectum. We have to start from lots and lots of places where we know exactly what the tissue is – in one place we know it is air, in another we know it is tumour and so forth – and then grow them and merge them. And it actually turns out that about 15 years ago there was a very beautiful piece of work developed at UCLA by Stanley Osher and James Sethian for embedding active contours within an implicit surface (I have written it down for the mathematicians here) and so the desired solution, the blue contour I referred to, is the zero level set. It is easy to show, in fact, as a result of that, that you end up with an equation of the particular form I have set out here, where we have the 'speed function', the gradient and the rate of change over time. The speed function typically has a term which attracts the level set to intensity changes, in order to try and find the boundaries, and it needs to have a term which represents what it believes is inside the mesorectum.
Basically we have to do those two things, but the devil actually turns out to be in the detail. We first of all have to figure out how we can represent those signal changes, and we have to figure out what the inside of this particular mesorectum is. Having got through those pretty quickly, I have thrown up, just for completeness, what the actual thing that we solve is. There is a term related to the curvature rate, there is one that is related to the region, basically based around a representation of the region/probability mass function, and one that has got potentially volume. That is mathematically what we actually end up solving.
The next thing to ask is: how the heck can we find these intensity changes? Here we run into a problem, because if you go and ask 99.9 per cent of people in computer vision how they would find those intensity changes, they will say they would take the gradient of the image. And that turns out to be a pretty dumb thing to do. So what we have done here is to take a whole series of points around the mesorectum which have been labelled by the surgeon, and at each point we have drawn a cross-section. In each one of these we have taken a cross-section of the signal and then a cross-section of what I call phase congruency and a cross-section of the image gradient. The most important lesson to take home from this picture is that actually the cross-sections that correspond to the magnitude of the image gradient, which is what most people use, convey almost no information.
Fortunately, there is a different way of doing that, which is to do with local phase. This is an idea that was developed by Professor Robyn Owens, at the University of Western Australia, around 20 years ago.
The idea is that you take an image like the one at the top left of this slide and you can try to do what most people in computer vision do, which is to compute the local energy or the gradient. Or you can treat the alternative representation shown below that, which is known as the local phase representation. In order to do that, we need to compute what electrical engineers call quadrature phases. A key part of that is to use a bandpass filtered image – for those of you who know any engineering or filtering.
The particular way we have to work with this – just to give you an idea, again for some of my mathematical colleagues here – is to take an image, bandpass filter it and compute three representations with a couple of convolutions. From those we can compute a local amplitude, a local phase and a local orientation. And the important thing is that the second and third of those representations are what are known as the Reisz transform, and they provide precisely the kind of cosine and sine within the Fourier domain. So we can actually now begin to compute the quadrature that we need in order to find the filters.
Actually, all that is going on here when you strip out all the mathematics is that we end up something that looks like a nice, even filter, the kind of stuff that you have in the first stage of the retina, and then we have this pair of odd filters and these guys together. It turns out that choice of bandpass filter is pretty critical, and not surprisingly people fight the usual academic death-grip struggles as to which should be the appropriate one. And we've got the right one! (But, of course, nobody believes it.)
Secondly – just to give you an idea, just a taste of what we are doing here – one of the things we need to do is to compute what are known as probability mass functions, or density functions. These are used right the way through the whole of computer vision and signal image analysis, for a whole range of reasons. In fact, at NICTA (National ICT Australia) here in Canberra this afternoon I saw about four different applications of this idea of writing down probability density functions, and I have just written down some here. There are two fundamental requirements that we have. The first one is that we actually end up generating an estimate of the density function from a relatively small number of samples – for example, nine or 25 – but this has to be evaluated many thousands of times (typically, many hundreds of thousands of times) and so it needs to run quickly, or doctors will not use it.
Just to give you an idea: here is an image at the top left. If we take the histogram, which was introduced to us by statisticians many years ago and simply records the number of instances of any particular grey value and brightness value in the test image, it can be seen that what we get is very, very jagged. People have recognised that for years, and so they have developed kernel estimators, which are probably, in control theory, the most widely used. The problem with that is that it depends on a thing known as the kernel. If you find the optimal value of that, the good news is that you can get a pretty good estimate of the shape of the probability mass function, but the problem is that it takes 35 seconds. That is to say, these things will work fine but the algorithm will basically take till hell freezes before you get a decent answer for a whole image. So what we want is this: right now we know for an optimal kernel we get very poor computational efficiency but we get a great estimate, and for the others we get very high computational efficiency but a poor estimate. If you compute garbage in a picosecond, the result is still garbage. We really want to have something fast, but good. How can we do that? People have been working on statistics for tens of years. How can we possibly have anything new to say on this story?
Actually, it turns out that statisticians have typically not looked at signals, and we have a number of things. For example, we know that samples are ordered, that signals are critically sampled, that they are band-limited. This is what undergraduate electrical engineering tells us. Those criteria are almost never taken into account by statisticians in techniques for estimating PDF. What we have done is to try to understand how we can link those together, and I am very quickly going to show you the results of doing that.
For that same image, we have here basically what we can think of as being ground truth, and then the outcome of our system that we have been developing – and which I was talking about in Adelaide last week. Notice that we can do this in 15 milliseconds. So, by using some ideas about signals and some of the properties of signals, we can actually end up by finding really good density functions.
I won't go through the details of the previous comparison, except to say that what we end up with is that on this segmentation slide we can very quickly find a plausible estimate. (You can't get the hidden leg, of course – the system doesn't know that zebras have four legs, particularly where there is no texture because it is entirely white.) And in the bottom right panel we have a reasonable approximation for finding the texture of the wooded region in this image.
So now we can begin to pull those together, and essentially, using that idea which we called Non-parametric Windows Estimator, we can begin to make an estimate of the probabilities of the various grey values. The actual overall value is shown in red and ours is shown in blue, and they are almost identical. So you can get a pretty good estimate of what is in those things.
What is the upshot of all of this? What is the point of doing this mathematics? You would remember that I said we wanted to start out by finding something that had the circumferential resection margin, in order to show it to clinicians, in order to help clinicians make a judgment whether surgery was an option or not. So what we have in the top image here – and this is the take-home message from doing that maths – is firstly, in cyan, a very painstakingly labelled contour on that image that has been written down by Professor Neil Mortensen, who is one of the top colorectal surgeons in the UK, and secondly, in yellow, the result of doing this using the maths that I have just shown you, using the active contour plus the intensity changes and so forth.
In essence, I wanted to make the case that we could take some of these ideas of mathematical analysis, mathematical modelling, and apply them to something which enables us, in the end, to take, for example, this representation of the lower rectum, the mesorectum around it, the blood vessels, the lymph nodes, and in the end produce colour coding of the circumferential resection margin. Blue is large and safe, and it is fine for the surgery to proceed, and red is worrisome. This system now is running day-in, day-out at the Churchill Hospital, in Oxford, and we have been working on that – it took us one and a half years to build that from scratch.
To move on to the second question, though: remember that the patient is sent for downstaging chemotherapy. Here is an image taken before that chemotherapy: the body, when there is a tumour, will typically locally generate swelling, adenosis. And after we have had eight weeks of this we would like to understand whether the chemotherapy was effective. If so, by how much? Is surgery now an option? Can we downstage this patient? In order to do that we have to be able to align the post-chemo image here to the pre-chemo one, despite the fact that we know that the image afterwards will need to be shrunk because it has expanded in reaction to the tumour, and the effect of the chemotherapy is to warp it and change it down. So we now have to pull those two things together.
I never wanted to do this. I have to tell you that I just wanted to get this stuff working in a hospital – and God, there's been hundreds of papers written about non-rigid warping of one image into another; I sure as hell didn't want to do that, I didn't want to add yet another one. And so I went around and asked a bunch of my mates around the world whether we could have their algorithms and try them out. We found good news and bad news. The good news was that when they worked, they were pretty good. The bad news was that they didn't work very much; they worked about 70 per cent of the time. For 30 per cent of the time these algorithms failed, and failed miserably. So how could we measure that? What we did was to take a bunch of points from the pre and the post images, shown here as red dots – this person has not come up with some strange internal measles – that we could use to make to measure the accuracy of any of the registration algorithms. We wanted to have something that was accurate, with at most 5mm error in the placement of these points, and that was robust. It had to give a useful result every single time, without fail. (We had 100 trial cases from the Churchill Hospital.)
This is us running these algorithms that we had got from a number of people. B-splines & Mutual Information is a particularly brilliant piece work done at Imperial College, which is a small university in London, but you can see that it was making an error of, typically, 3cm. We wanted to get down to 4.5 or 5mm, so 3cm is simply not good enough. There was a bunch of others from Michigan, from University College, London, there was ours from Mirada Solutions Ltd – it was all right, but we would expect that – and a publicly available toolkit, but again at a centimetre out. Why is that?
Basically, it turns out that the reason that happens is that all of those algorithms that I showed you are based upon a similarity criterion, and that similarity criterion gets absolutely fascinated by signal complexity. The regions I have highlighted in the right-hand image here with the two arrows are where the signal is most complex. Unfortunately, those points which are muffled have got absolutely zero clinical significance. So the algorithm is being distracted by stuff that has no clinical significance. The point, the take-home message of this, is that the algorithms themselves are not deficient in the mass that they embody; they just don't know anything about the colon, the rectum and kinds of stuff we have. That is the bottom line of this. So what we wanted to do was to direct the algorithms to be only interested in the region of interest, and I have just told you how to find that region of interest.
We had the idea that maybe what we could do is to add to these algorithms a small amount of knowledge, first of all, which is anatomical: 'The hip bones are connected to the bladder, and the bladder is connected' and so on – I won't sing it – 'From the hip bones you can find the coccyx, from that you can find the colorectum,' and so forth. That actually defines the search region; it restricts where the algorithms are interested in.
Second of all, we need to be able to put in some physiological knowledge. In this particular case, we don't want our algorithms to be baffled or surprised when chemotherapy acts the way it ought to do – namely, to reduce swelling around either lymph nodes or the tumour. The best way to do that, probably, is to look at the Jacobian of a vector field. (This is standard second-year undergraduate mathematics. There is nothing particularly complex about that.)
If you supply those two pieces of knowledge to the algorithm, what you end up with is that you are now guiding exactly those same algorithms to the point where not only are they sufficiently accurate but they now work all the time – for 99 out of the 100 they work exactly right.
There is a next piece of the talk but I am going to skip through it, just to show some of my former colleagues in maths what we did. Essentially, what we need to do is to take the pre and the post images, we need to be able to compute both the transform from one to the other and the one back, in order to guarantee a smooth inverse. That is known as a diffeomorphism. In order to do that, we have to find some way of representing the diffeomorphism, and we use a polyaffine transform together with the local phase that I showed you a little earlier.
If Brian Anderson were here, he would be chuckling because this is exactly the kind of Lie group maths that Brian has known and used for years, essentially using a differential equation to solve the transform from the one image to the other.
Let me just skip over this stuff and get to the bottom line of it.
Here you can take a pre-treatment image, you can then warp the post-treatment image, you can subtract them after normalising the images, and basically you can find out where there is a difference in a lymph node and where there is a difference in a tumour. You can actually measure the extent to which a tumour has been reduced in size and a lymph node has been reduced in size, and so you can now downstage the cancer. Again this is stuff which is running day-in, day-out at the hospital.
You may recall that when I first started talking about this I said that there was a thing called a multidisciplinary team meeting which is now mandated by law in the UK. You have a group of specialists that come together for an hour, and in that time they study in depth the cases of about 40 patients in order to figure out what the next step in management of each particular patient is. It actually turns out that in about 35 cases it is pretty obvious, but in five cases it is not and that is where they spend their time. Amazingly, this meeting, with all of these specialists brought together, has almost no IT support. It is almost zero, which strikes me as crazy. The second thing is this: suppose that a patient, who has come up to this MDT meeting and has been sent away for downstaging chemotherapy, comes back eight weeks later with another image. You find out that the radiologist that was at the first meeting is now away giving a seminar (which is what they call skiing trips), the pathologist has been called away, the surgeon is in the operating theatre – there is a complete change of personnel. And they can't remember why the decision was taken last time. There is no memory in the system that records not just the decision that was taken but the contingency, the information that the decision was contingent upon. So that was the situation that we set out to develop a response to, and we have implemented a system, over the last 18 months, which is in fact being commercialised right now by GE Healthcare.
This is a typical example of the use of the system, which has been running in a clinical trial at the John Radcliffe Hospital for the last eight weeks. Basically, it first presents a page in which it says, in this particular case, 'Mr Peters presented with rectal bleeding and weight loss … He is a father of two, and a successful businessman. He has undergone radiochemotherapy and is now being considered for surgery.' At the top of the page are shown the age, the name – by the way, these names are fictitious, because this is sensitive information – and the current TNM status, which is the way in which tumours are scored.
The next thing the committee can do is to pull up the history, which says that this patient went for downstaging radiochemotherapy the last time round.
And then basically they can look at the investigations that have been done – in this particular case, a series of images: MRI, colonoscopy, CT – and, next, the current understanding of the tumour size, the nodal involvement, the level of metastasis et cetera.
Then basically the system can look at whether or not this patient could tolerate chemotherapy. The standard thing that is given is 5-FU and oxaliplatin or cisplatin, but there are now far more powerful, far more toxic drugs available.
It can also give an estimate of the so-called predicted operative mortality. This is something that we are doing with Professor David Kerr, who is the clinical pharmacologist in Oxford.
On the basis of that, the committee can pull up the images and take a number of the measurements that I have been showing you for the last 35 or 40 minutes.
Finally, the system can make a recommendation as a suggestion – we are not usurping the clinicians at this point – of what the system believes to be the only candidate around at this particular point, surgery. But, if the doctors want to commit to that, the system says, in effect, 'Well, hang on a minute, there is a piece of information on which that decision is contingent and which I don't believe is available.' So it can prompt to ask for that.
On the basis of that, the system will then generate a representation, not only of the decisions that have been taken but of why – the arguments for and against various alternative decisions that have been put in place. So where is this system up to now? Just to give you an idea: we started the clinical trial in Oxford in the first week of October this year. Of the entire MDT suite, we did the last 10 minutes and we ran three patients through it. That was just to see if it worked – toe in the water. And the committee might have said, 'Sorry, that's pretty flaky. We don't want to do that any more.' Actually, what they said was, 'Next time we'd like to spend half an hour doing it,' and so we ran 20 patients through the system. And after week 3 they have dispensed with their previous system and they are only use this. Such is the thirst which clinicians have for this kind of information technology support in their decision making. By the way, as you can see along the bottom of this screen shot, the system can generate automatically a referral letter and a letter for the primary care physician, the patient's GP.
Working on the liver cancers led us to another thing which is intriguing, especially for me. It turns out that in an early day, particularly in places which are homogeneous, such as the liver, tumours will typically just grow, rather like a sphere. But now and again what you will find is that they will generate a second clonal centre. And that second clonal centre is quite often associated with a mutation in the DNA of the tumour. What we are interested in is: can we model the growth of tumours, and on the basis of that, can we use the shape of those tumours to give us an idea how serious that cancer might be?
So does that occur? Here are five examples which I took over a three-month period from our local liver cancer system. You can see quite large tumours beginning to be visible up and down here. This is a real problem in many, many cases that we see in practice.
Here we have images of a post mortem liver, just to show you exactly the kinds of shapes that we are seeing in the operative field. By the way, when my students work on this stuff they have to understand and appreciate the sheer impossibility of the task that clinicians do. So I require the students who work with me on this stuff to go and spend time in the operating theatre, require them to spend time shadowing clinicians, so that they understand what it is that they do.
Here is a particular case to get that point across. These are three views of a patient in 2004 – you can see this is pretty well approximated by a sphere.
This person then had nine months of chemotherapy, after which the shape had changed absolutely, totally, as you can see here.
The patient was then left for three months, prior to high-intensity focused ultrasound ablation, and the shape changed yet again. So what we had to do was to build a system that would optimally fit groups of these spheres – and growing spheres – to the data.
What we find is basically that in certain cases, if you applied chemotherapy, some of these spheres would reduce in size but others would grow in size. Well, so what?
It turns out that pretty much all clinical practice at the moment believes that size is everything. It measures the volume and says, 'If a tumour has shrunk in size, then it must have gotten better.' Actually, there is no evidence at all to support that. In fact, it can be the case that – because each one of these clonal centres is associated with a significant mutation in the DNA – the post-chemotherapy tumour can actually, despite having shrunk overall, be a damn sight more serious for the patient than previously. So one of the things we have just been doing is an experiment – in fact, I received the preliminary results just last night, but because they are preliminary I am not going to talk about them – with a set of 10 knockout mice in Oxford, growing particular tumours. We have just got the first results back from pathology. But we have also taken the ones in the photographs I showed you a couple of minutes ago, from a patient, and shipped them over for a full DNA analysis of each of what we have marked as the spheres on the samples. I am really seriously beginning to hope that we can relate morphology and changes in morphology to the underlying disease process and the growth of the cancer. There is some fantastic mathematics that we are trying to work on, in modelling this particular growth.
I said I would finish off by coming back and talking a little bit about the fantastic idea called molecular imaging.
Basically, what we are interested in is taking images over time and modelling time activity curves at each of the points that we have got. To me, this idea of pulling together image analysis and molecular biology – which I think have been two of the most exciting technologies of the last 50 years – to understand disease at the level of cellular processes and pathways, is one of the most scientifically demanding but interesting topics that we are going to see in the next 20 years or so. I really wish I was 18 again (as does my wife, I can tell you!)
The kind of thing we are interested in – this is stuff we are doing with Siemens – is looking at the take-up of a drug that was introduced into the tail. It was actually supposed to be targeting the kidneys in this particular case. You can see that eventually it will be picked up by the heart, as it is pumped around the body and so forth. This is a particular drug that is labelled with a sugar. But the problem with this one is that it has got a huge build-up in the brain. So the good news for this particular candidate drug turned out to be that it would cure kidney cancer; the bad news was that it would turn the subject into a vegetable. That is not a great trade-off. But image analysis, and in particular dynamic image analysis, is giving us an absolutely new way of understanding the behaviour, the action, and evaluating the potential, of new drugs, and understanding the side effects that they may generate.
Let me just give you one taste of a fundamental challenge in understanding cancer: hypoxia. One of the things that we have known for years is that many tumours can grow extraordinarily without oxygen. You can get a really quite marked imbalance in the supply of oxygen from vessels versus the demand from the various cells which are shown here. (The structure at the top of the image is the vasculature.) So one of the issues we wanted to understand is whether we could model hypoxia. This is probably one of the top two or three targets for cancer chemotherapy research in the world at the moment. Can we model hypoxia? Can we estimate the degree of hypoxia from a bunch of images, and if so, which images? It turns out that there are two components for hypoxia. One is intercellular, so you get oxygen being transported from the blood vessels to a bunch of cells, and the second is an intracellular component: basically, you have a number of pathways which are regulated by hypoxia-inducible factor, HIF. That is a complex of proteins and is typically mediated by another protein, Akt/PKB.
The first thing we do is to spend a lot of time modelling – using, effectively, reaction diffusion equations – the way in which we can diffuse and metabolise oxygen in the steady state from vessels. It turns out there that we have to have a particular representation of the vasculature which caters for the very poor resolution of the kinds of images that we have got. Basically, the vasculature itself, particularly the vasculature that is grown by a tumour, has got sizes of the order of about a micron. We can't image less than about a 3mm cube. So you are out by a factor of 1000 x 3 (that is, a billion). All you can ever do is to try to come out with an approximation to the vasculature, by doing a particular representation of the vasculature which gives us pictures of the hypoxic islands, which are known – shown in the top right-hand panel in black – and you end up with tissue activity curves that we can predict. (In this particular case I am showing the concentration of a fluorine-based misonidazole.) And we look at the concentration in terms of kilobequerels as a function of time. This is typically what is done in nuclear medicine in your hospitals.
That is the intercellular. What about the intracellular? I am showing you this because I wanted to tell you a story, which has been one of the most fascinating stories that I have had in the last two or three years. In a normal cell, when there is an insufficiency of oxygen – hypoxia (and I apologise to biologists, who would realise that this is a rather cartoon simplified version) – typically one of two things happens. The first is that a particular pathway governed by, essentially, the watchdog of the cell, a particular protein p53, springs into action and programs the cell for death. It basically programs the apoptosis of the cell. The cell, essentially, programs itself for suicide. Or, second, it activates the HIF complex, to which I will come back in a moment. Those are the two ways, by and large, that normal cells deal with an insufficient supply of oxygen. What happens in a cancer cell? Well, the first thing that happens is that p53 is taken out of the running. Basically, tumours turn off that particular escape mechanism. They don't want to see the cell killing itself; they want to keep the cell going. So what they do is to go back onto HIF, which you would remember is regulated rather like an amplifier by Akt, which essentially acts as a game control on the HIF. There are three outputs. One is quiescence, where the cell basically goes to sleep and hopes to God it will be better in the morning. In the second one, it essentially begins to do what is known as the Warburg effect, glycolysis. And the third one is to generate angiogenesis. This, by the way, is the target of chemotherapy agents like Herceptin. You may have seen some of the press recently about such agents.
It turns out that there is a huge number of clinical findings which are absolutely contradictory about whether or not you can image the state of hypoxia by using a sugar compound, fluorodeoxyglucose (FDG), or whether you can use that as a surrogate for misonidazole, which we know is a direct correlate to hypoxia but which is a damn sight more expensive and harder to make. The question is: can we use FDG, and under what circumstances? There are people who write papers and say yes, you can, or no, you can't – yes, you can, no, you can't, and so forth – with no theoretical underpinnings. So I persuaded a young student, Catherine Kelly, to come and work with me on this. She is a biologist. I promised I would teach her a bit of maths if she would teach me some biology. I think it's great fun.
Cat would come in and start showing me things like the diagram on the left here, saying 'This is upregulated, this is downregulated.' I would respond, 'Goddammit, woman, I don't know about upregulated, downregulated, side-regulated. Come on! This mathematics on the right is reality; it's the only reality of the Universe. What are you giving me all this gumph over there for?' So we would fight like cats for about two years, and in the end she understood what I was talking about and I understood what she was talking about, and we actually built a model that we could both understand and from which we could make a series of predictions. And, surprise, surprise, it turns out – it doesn't matter what these graphs at the bottom of the slide are – that you can explain all of the current clinical findings that were apparently contradictory, just in terms of the setting of a single game control of one protein, one pathway, on the HIF complex. That is the power of mathematical modelling and it shows you how, with using computational models and mathematical models together with an understanding of biology, you can begin to understand cellular and pathway models.
We have done a load of other stuff along these lines.
We have predicted that there would be an acellular gap in human neck carcinomas, and that is what you find, typically of a size around 100 microns.
We have predicted that you would have cells that would switch between a proliferated state and a quiescent state, and that is exactly what we find. We can even predict what the cycle time of that is.
And that all comes about, again, by writing down a remarkably simple differential equation.
Finally, you can understand how to relate model fitting of that mathematics to the images that we can now begin to generate within medical research. (This was a collaboration with Arizona, with the Mathematics Department, the Engineering and Computing Lab, Manchester University in Arizona, which is increasingly the way things are done.)
So what have I tried to do? I have wanted to show you, and I hope I have succeeded, that cancer provides a very challenging application for image analysis, because it really pushes us way to the limits of what we understand about feature detection; about the segmentation (the case I showed you, with the mesorectum); the estimation of probability densities, and again I showed you the mesorectum; the analysis, the alignment of two images; and various other things as well. But the real purpose of my talk today is that I hope to convince you that image analysis and IT can very, very substantially impact upon the diagnosis, the monitoring, the therapy options, and also to provide theoretical insights into the biochemical basis of cancer and the way in which this awful disease can grow and develop. Thanks very much indeed. If there are any questions, I would be absolutely delighted to answer them.
Discussion Question 1: You mentioned earlier in your talk about how image analysis might relate to tachycardia. Can you explain a little bit more about that, please? Michael Brady: Sure. What I was trying to get at was the following.
I wanted to make a very simple point. On the left here is actually a very healthy heart. It's mine. (My liver might be in bad shape, my heart is in great shape.) I wanted to show you that you could take snapshots of the heart and they are beautiful images – really stunning quality images. But it is very hard to get those images at good temporal resolution. The game has just recently changed, but at the time we were doing this work it was about five times a second, one-fifth of a second. We really needed to have a factor of 10 better than that in terms of temporal resolution. What I was trying to get at was that if you look on the right-hand side images, you see that although we can get a factor of 10 in temporal resolution – we can get 50 images a second – we sacrifice the spatial resolution. In fact, we end up with these very noisy images. These images are bedevilled by what is known as speckled noise because of the statistics of local scatters within the tissue.
So the name of the game was: could we pull the best properties – the spatial excellence of the images on the left and the temporal excellence of the images on the right – into a composite story? We wanted a story that would enable us not only to see that there were fluctuations in the beating, whether too fast or too slow. Or, much more to the point, typically what we are interested in is looking at the myocardium, and to find out whether any part of that myocardium is not moving at all. That occurs typically where, for example, you have an ischaemia which eventually, because of the pressure on the rest of the muscle in the heart, will lead to coronary heart disease. So what I was trying to point out was that we have to pull together two images, each of which has got one positive quality but counterbalanced with one negative problem. And I wanted to pull those together. All I was trying to show you was this particular pair of images. What I didn't do was to go on and say: how can we take that and make measurements, for example, of regional function within the heart to understand the state of the myocardium and so forth? That would have been the subject of a completely different talk, and I just wanted to use this as a transition to the idea of fusion before I got into the main theme, which was about cancer. But, if you are interested in this, we have done a hell of a lot of work on that subject and I would be very happy to give you any references and demonstrations of the kinds of stuff that we do. Question 2: Thanks very much for the inspiring talk. I have a question regarding the pre-chemo and post-chemo registration of the images. You said that for improving the non-rigid registration, [anatomical] atlas information is thrown in. My question is how the atlas information is integrated with the pre-chemo and post-chemo. Doesn't that require non-rigid registration in itself? Michael Brady: No, but that is a pretty shrewd question.
In showing this slide, the arguments I was making were very simple. What I was trying to get across was the idea that, if we took those two images, what we really needed to do to overcome the fascination of algorithms with points where we have got signal complexity – as indicated in the image at the right – but which have got no clinical significance, was to draw their attention to where we would get clinical significance.
Your question is: how on earth can you do that unless you have already solved the non-rigid registration problem? Well, it is not as quite as bad as that, to be honest. It is actually very easy to find the hip bones, and if you draw a line between the hip bones and you take a perpendicular down, it is pretty easy to find the coccyx. So once you have got that local coordinate frame, then because you have got three points in the pre image you can do an affine transform and now you are already in the patient. Now you are already not allowing these points basically to persuade the algorithm to concentrate on complexity. The stuff I skipped over because I thought it was probably too much mathematical stuff about the polyaffine transform showed that very clearly. Again I can give you more details about that later. Question 3: I am not a scientist so this might be a silly question. Please bear with me. About three months ago the Melbourne synchrotron was heralded in the ABC news as modelling a switch that might turn off cancer. I think what they were talking about was modelling the molecule telemerase. Michael Brady: Yes, that's exactly right. Question 3 (cont.): You have described very articulately how your measurements only go down to a certain level… Michael Brady: Yes. Question 3 (cont.): And of course the synchrotron is going at an extremely detailed level. Is there any prospect of combining the information you are getting on telemerase from the synchrotron and applying that to techniques for cancelling telemerase, and seeing what effect this has on tumours in real time, for example? Michael Brady: First and foremost, if you are not a scientist I thank you enormously for coming tonight. That makes me very pleased. My work – I should have made this clear right from the start – apart from the stuff I talked about right at the end in the molecular biology, my stuff is driven totally by clinical reality. I am interested in seeing the work we do being used with patients. So I spend a hell of a lot of time in the hospital. Lagging behind that, though, is a fundamental lack of understanding of a whole range of cancer processes. If you write down the full, staggering, awesome complexity of pathway models within the cell, we actually still understand relatively few. We don't understand very many, for example, to the level of the HIF pathway that I was showing here tonight. And that poses an immense mathematical challenge. So we have a limit on how accurately we can see what is going on at the level of within the cellular processes, and we have a limit on the way in which we can model mathematically the pathway and the interaction within the cell. What the synchrotron gives us is a fantastic microscope onto some of these cellular processes. It provides us with a whole new generation of models that we can use. Whether they will ever be used in a therapeutic way, whether they will ever be used in regular clinical practice – or whether, as is perhaps more likely, they will be used to give us the increased understanding so that we develop better chemotherapy agents – time will tell. But I do think that things like the synchrotron (and we have a very similar one at the Diamond project in the UK) are going to bring with them an enormous increase in our understanding of the cellular processes. What they are, we will find out. Question 4: You spoke about hypoxia and explained the normal and abnormal difference between cells in their uptake. Recently also people have found out that there is a difference in a cell's uptake of nanoparticles. Michael Brady: Yes. Question 4 (cont.): Can your models also show that, and find out exactly how much difference there would be between the two kinds of cells, normal and abnormal? It would be really interesting to know that. Michael Brady: A very nice question. In fact, a bunch of my colleagues in Radiation Oncology and Biology, in Oxford, directed by Gillies McKenna, are precisely working on that issue right now, and at the moment we don't know the answer to the question. Question 4 (cont.): [inaudible] Michael Brady: Absolutely so. So at the moment, with the HIF model, we are actually trying to do two things at the same time. One is to combine it with the stuff I flicked over at the end because of time: glycolysis, the acidosis of the tumour. The other is that the HIF complex is also very heavily involved in angiogenesis, as I have tried to show. It turns out that we can both image and model the angiogenesis in the same way as I showed you with the equations. So we can pull these things together. In fact, a postdoc and another of my students are pulling those together at the moment, but in order to do that you have to move up the food chain. We can't get as far back as Ras but we can get as far back as CPR2. So, as the French say, petit pas – we're just making little steps: understand a bit, pull them together, understand how far we have to push it back. So I don't know, but we have a lot of interactions with Gillies McKenna and his amazing group. He has brought 270 scientists in radiation oncology and biology to Oxford. It absolutely wonderful. Chair: We are very honoured to have had Mike Brady here to talk to us, and I am very pleased that he has been able to come back to the scene of his postgraduate education. We are very proud that he is a graduate from the ANU. Let us thank Mike once again for his talk. Michael Brady: Thank you. If any of you do want to get some more detail on this stuff, please, I'd be very happy to talk with you.
|
||||||||||||||||||
© Australian Academy of Science | aas@science.org.au