Making Images at the Speed of Light

Today, the capabilities of modern technologies are constantly increasing, and instruments are becoming smaller, faster, cheaper, more portable, and more easily interconnected. This is true for many analytical spectroscopy techniques as well as for a wide range of other technologies that have the potential to intersect with the field of spectroscopy and expand its boundaries. To explore these developments, Spectroscopy is launching an article series about new technologies and new applications of existing technologies that are based on or related to light. We kick off the series with this interview with Andreas Velten about his work as a postdoctoral associate at the Massachusetts Institute of Technology (MIT) Media Lab in Cambridge, Massachusetts. (Velten has since taken a position as associate scientist at the Morgridge Institute for Research at the University of Wisconsin in Madison).

Velten and his colleagues in Professor Ramesh Raskar's "Camera Culture" group at the MIT Media Lab, in collaboration with the spectroscopy laboratory of MIT Professor Moungi Bawendi, developed a technique they called "femto photography." The technique uses a titanium–sapphire laser that emits pulses every ~13 ns, picosecond-accurate detectors, and complex mathematical reconstruction techniques. By combining hundreds of "streak" images (one-dimensional movies of a line), captured with this high-speed camera, they have created moving pictures (never perhaps was there a more apt use of the term) that show the movement of light (groups of photons). Examples of their use of the technique include combined images of light traveling through a soda bottle and, in a separate application, over a piece of fruit.

Spectroscopy: How did the femto photography project get started?

Velten: About two years ago, I joined Ramesh Raskar's group at the MIT Media Lab to do a post-doc. Ramesh had been thinking for a long time about combining ultrafast optics and computational photography to build an imaging system that can look around corners. He and his group had taken some initial steps in implementing the idea. It's kind of an unusual match, because my background is in ultrafast optics and this group is doing computer vision and computational photography. But it's very interesting to combine the two fields. People in ultrafast optics are trying to push the envelope of the hardware — to see how short we can make the pulses, to improve ranging. For example, with light detection and ranging (LIDAR) we send a laser pulse to a target and wait until the light comes back, and from the time that has passed, we can measure the distance to the target. It's used in traffic control these days. But I was thinking about imaging and what could be done with imaging data in signal processing.

On the other hand, with computational photography people basically take consumer cameras and make small modifications to them, and do amazing things by processing the data and looking at the data in a new way. Our project is kind of a combination of the two fields. We use nonstandard hardware — hardware you can set to the time of flight of the light that you are using it for imaging. We wanted to develop new capabilities of this method by further processing the data. The initial goal was to build a camera that could image around a corner (a special application).

Spectroscopy: So how did you end up photographing visible light photons — in other words, doing photography at the speed of light?

Velten: Once you have the time-of-flight imaging, you can get a lot more information from the light by post-processing the data. Professor Raskar and our whole Camera Culture group is very interested in computational photography and were inspired by the "bullet through an apple" strobe photos by Doc Edgerton. I had taken some of our streak camera images and created one-dimensional movies. Professor Raskar challenged us to think about ways to convert the one-dimensional streak tube to create visually meaningful ultrafast two-dimensional movies. I realized at some point, from playing with the camera, that you could actually compose movies — that you could stitch the data together in a way that would allow you to reconstruct a complete movie out of the data that you capture. Making these movies was really a side project. Our team, especially Everett Lawson and I, started to put together a mirror based system. Then a set of collaborators, Diego Gutierez, Di Wu, and members of Diegos group, Adrian Jarabo, Elisa Amoros Galindo, and Belen Masia, got excited and worked on visualizing the results better in the videos and doing things like generating single pictures from them.

Spectroscopy: How do you create such clear moving images from multiple still images?

Velten: We use a titanium–sapphire laser that gives very regular pulses, and the camera is synchronized to that. The camera takes lots of images of the scene, but because the camera and the laser are very well synchronized and everything is very regular, the images all look the same. So we can just stitch them together to get the final moving image of the scene.

Spectroscopy: Does it take a lot of time or work to put them together?

Velten: The image capturing is what takes hours, actually. To get a movie that looks really good, you need quite a few shots. And it takes much longer to set everything up.

Then there's the post-processing. What you see is actually a color photo in the background, and the light is put on top of that. The raw, straight camera images are all black and white; the color is added for visualization. That's all post-processing. If you watch the videos, the plain data — the pulsed elimination only, which shows how the plain black-and-white image from the laser looks — is always there in the videos.

Spectroscopy: The photons moving through the soda bottle look like a fluid in motion; there is more velocity at the center than at the outside, and then it seems to bunch up at the spout. What do you think is going on?

Velten: The view of what you see is a little distorted, because you don't see an event when it's actually happening; rather, you see it when the light from the event has traveled to the camera. I think that's why it looks like things are moving slower far away from the pulse than close to the pulse, because there is a distortion that comes from the light having to travel to the camera.

Spectroscopy: What potential applications do you see for using this technology?

Velten: One idea is to use this technology to look into materials, because some of the light always travels into the material and some scatters back out. You can see this very nicely in our images of the tomato: The tomato actually keeps glowing after it has been exposed, because light travels inside and then slowly comes back out. From that light, you could actually get information about what is going on inside that material, if you developed a proper way of probing and analyzing the materials that you are looking at. There is currently some interest in this. Many people have tried to image living tissue, for medical applications, like doing an ultrasound or X-ray with light, which potentially has a lot of advantages over the common X-ray, because it would be practically harmless. That is one thing people are trying.

For industrial imaging, you could try to detect cracks inside material, if some light is actually transmitted through. You don't need a lot, you just need a little bit of light coming through and coming back out. By analyzing how the light scatters off the materials, we can learn a lot about the material.

Spectroscopy: For potential applications like medical imaging, to what level of detail or size can you see things?

Velten: That's an open research question, because really, you are posed with a data collection and processing problem. The information that the light gets — the wavelengths, the resolution of the light — is very high, so you should be able to get quite detailed information out. But the scattering inside the material destroys a lot of that information by bouncing the light around. And then the question is, How well can you detect the intensity of what is coming out (not only the time it takes for the light to come back out) and how well can you computationally reconstruct what is inside the material? It depends a lot on how deep the material is, obviously.

Spectroscopy: Can you envision using different wavelengths so that you could actually make vibrational spectroscopy measurements?

Velten: Our limitation right now is that we only have one light source at 800 nm. So the movie, the moving part of it, is actually in the near infrared. But if we had three different light sources, red, green, and blue, or a tunable light source, like if you used an optical parametric oscillator (OPO) instead of a titanium–sapphire laser, then we could scan different wavelengths or even try to send a broad spectrum in and learn more about the spectral properties of the material. That is something we are looking into.

The vibrational modes in some materials are slow enough that you could actually see something happening. You wouldn't just see a spectral signature; I think you might be able to see the vibration happening inside, the frequency. That would be very interesting.

For this camera, or at this point for this camera, it would need to be something macroscopic — something we could put in front of the camera and in which we could excite enough molecules or enough material simultaneously so we could actually see something.

Spectroscopy: What do you envision this high-speed camera technology may be able to do in terms of measuring and studying the interaction of light with various media, such as liquids, powders, and large and small particles?

Velten: We have done some of that. It's always an interesting game to put something in front of the camera to see if the camera can detect something interesting, because the amount of information you get in one of these exposures is actually quite big, and you may, by chance, discover something that has been missed before. Visually analyzing something is an incredibly powerful technique, compared to looking at things with a computer or looking at plots, or just numbers.

Applying this technology to spectroscopy techniques is a very interesting direction to take this work. There are, of course, a lot of interesting effects that occur when you hit something with a very powerful laser pulse and evaporate material. Of course, usually that is a very small effect, and you would need a microscope in front of our camera to see something there. But all these things are interesting. You could trace plasmas in the air; light filaments would also be interesting to look at. At this stage, we are just brainstorming about all the things that could be done by looking at things with this speed.

Right now our limitation is that we need repeatable events. We have to be able to repeat the same thing over and over again many times to collect enough data to capture the movie. That limits some of the applications that you can actually do in terms of looking at interactions with matter, because often you destroy your sample the first time you shoot light at it, and then you're done.

Spectroscopy: There has been so much work done using simulations to generate models for how light interacts with multiple particles of different sizes, depending on their scattering properties or absorptive properties. Is that a potential field of use?

Velten: We have done a little bit of that sort of thing, such as taking tissue samples or materials like wax, and trying to analyze them. So far we have been looking more in the direction of computer graphics, which is more about how things look than what's actually happening. That's the background of the group, and that's why we have done some work in that direction. But the other direction, of looking at scattering models, is also very interesting and a worthwhile direction for this research.

Spectroscopy: This work brings to mind the dramatic effects of high-speed cameras, such as in "The Matrix," when you see a bullet moving very slowly and interacting with the images of the people and the scene around it. At the high speed of your cameras, you could actually do the same thing with light moving through a scene, right?

Velten: It's very interesting. Actually, if you were to capture a bullet with this camera, that is, if you could shoot the bullet several times and actually make a movie of the bullet, it would take about three years to watch the film of the bullet going from one side of the scene to the other. That's true for almost any piece of matter. So I don't think you will ever look at solid or atomic matter with this camera, unless you have a very violent explosion, because there is just not enough happening in a time frame that is interesting to watch.

The other question people ask is, Don't you capture a lot of data? Well, we capture 512 frames, so it's not a lot, about half a megabyte per frame. If you were actually capturing a whole second, of course there would be a lot of data, but it would also take tens of thousands of years to watch. So there is no point in capturing a full second's worth of data.

Spectroscopy: What are your next steps with this work?

Velten: We have a publication that is under review right now, using an application of this time-of-flight imaging. We have another one we just submitted that is about further processing and visualizing the data — visualizing the movies, essentially.

Beyond that, many other things that have been done in computer vision and computational photography can also be done using this system and using this camera. We would like to go through those and demonstrate them one by one, to demonstrate the capabilities, including some things that were not possible before, like looking inside materials from a distance by analyzing the scattering and the time of flight. We also have received some interest from people who would like to investigate scientific phenomena, to lead to new discoveries. So that's an interesting direction that will also be explored.

The other thing that comes to mind is improving the system. Right now, it uses a lot of high-end equipment that is bulky and heavy. It's really a laboratory setup. For many applications, it would be better to make it much more compact and much more portable. So we are working on building a dedicated compact system.

Spectroscopy: What plans do you have to partner with other scientists at MIT or other universities?

Velten: We don't have any thing concrete yet. We have chatted with a number of people who are interested, but we are still exploring what kind of concrete collaborations could come out of it.

Here at MIT, for example, Professor Moungi Bawendi (of the spectroscopy laboratory) is providing the equipment for us, and is also a member of this project. He usually works on spectroscopy and quantum dots, so he is an interesting partner for the spectroscopy and nanocrystal applications.

We are also interested in working with the people at the MIT Edgerton Center, because we see our work as being kind of in line with the work that Doc Edgerton did about 50 years ago. I don't know if you've ever seen those art photography pictures of a bullet going through an apple. That was done at MIT. I think it was kind of a similar situation, that he had this equipment, and wondered if he could just take still pictures with it, and he came up with that stunning photography. That is still on display in the Edgerton Center, and I think it's mentioned on our web page.

Another person is Nils Abramson in Sweden who did light-in-flight holography back in the 1970s, also trying to capture light moving, with holographic film. He had some very interesting short movies of moving light and things like that. He is retired, but we are in contact with him.

Most of these things are in the early stages of just talking to people and seeing what can be done.

More Online:

To see the moving images produced at the MIT Media Laboratory, and more about their work, visit:

http://web.media.mit.edu/~raskar//trillionfps

http://www.mit.edu/~velten/press/content/

http://www.mit.edu/~velten

http://cameraculture.media.mit.edu/