Stanford Light Field Microscope Project

Click on a thumbnail to watch the corresponding image sequence (MOV file).

(also available as AVI)
Note that both sequences were computed from a single snapshot by the camera.

(also available as AVI)
Our first prototype light field microscope (LFM). The eyepiece has been removed and replaced with a microlens array (circled in red). Above the array is a camera and 1:1 relay lens. Click here to see some more recent, portable versions. A light field as recorded by the optical arrangement at left. The specimen is a 200-micron-tall "tower" of fluorescent crayon wax. The objective is 20x/0.5 (dry), used without a cover slip. Perspective pan. Since microscopes are telecentric, they are normally only capable of producing orthographic views. Computed in real time. Focal stack. Using deconvolution, this 3D stack can be converted into a 3D (cross-sectional) volume. See below for more examples.


News flashes:
  • 4/4/10 - We have a public release of LFDisplay: a software package for viewing microscope light fields, with instructions on how to build an LFM.


    Overview

    Light field photography is an image-based technique for recording scene appearance. Unlike conventional photography, light fields permit manipulation of viewpoint and focus after the imagery has been recorded. Devices for capturing light fields range in scale from room-size arrays of cameras to a handheld camera in which a microlens array has been inserted between the main lens and sensor plane.

    In this project, we are exploring light field photography at the microscopic scale. Specifically, by inserting a microlens array into the optical train of a conventional optical microscope, we can capture light fields of biological specimens in a single snapshot. Although diffraction places a limit on the product of spatial and angular resolution in these light fields, we can nevertheless produce useful perspective flyarounds and 3D focal stacks from them. Applying standard deconvolution algorithms to these focal stacks, we can reconstruct 3D volumes. Since microscope optics produce orthographic views, perspective flyarounds represent a new way to look at microscopic specimens. Focal stacks are not new, but manual techniques for capturing them are time-consuming and hence not applicable to moving or light-sensitive specimens.

    Like any new scientific imaging instrument, we expect the light field microscope to have many applications in science, medicine, and industry. For example, since the light field microscope separates image acquisition from the selection of viewpoint and focus. it can be used as a "digital viewfinder" for a conventional microscope. Applying computer vision algorithms to these selection procedures may lead to greater automation of microscope operation when analyzing large numbers of specimens, such as in clinical pathology.

    People:

    Collaborators:

    Recent papers in this area:

    Enhancing the performance of the light field microscope using wavefront coding
    Noy Cohen, Samuel Yang, Aaron Andalman, Michael Broxton, Logan Grosenick, Karl Deisseroth, Marc Levoy
    Stanford Computer Graphics aboratory Technical report , September 2014.
    Wave Optics Theory and 3-D Deconvolution for the Light Field Microscope
    Michael Broxton, Logan Grosenick, Samuel Yang, Noy Cohen, Aaron Andalman, Karl Deisseroth, Marc Levoy
    Optics Express, Vol. 21, Issue 21, pp. 25418-25439 (2013).
    Elastic source selection for in vivo imaging of neuronal ensembles
    Logan Grosenick, Todd Anderson, Stephen J. Smith
    IEEE Symposium on Biomedical Imaging: From Nano to Macro, June 28 - July 1, 2009. Also available via IEEE Xplore.
    Recording and controlling the 4D light field in a microscope
    Marc Levoy, Zhengyun Zhang, Ian McDowall
    Journal of Microscopy, Volume 235, Part 2, 2009, pp. 144-162. Cover article.
    Click here for a 2-minute video (.mov file, H.264, 16 MB) showing digital refocusing of illumination and observation in our microscope
    Click here for a 2-minute video (.mov file, H.264, 14 MB) showing focused illumination passing through a fluorescein-filled chamber
    Light Field Microscopy
    Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, Mark Horowitz
    ACM Transactions on Graphics 25(3), Proc. SIGGRAPH 2006
    Click here for a 4-minute video describing the idea and showing some of our results.
    See below for additional results and animations.
    The Light Field Microscope (extended abstract)
    Marc Levoy, Zhengyun Zhang
    Proc. Focus on Microscopy 2007.

    Background papers:

    Light Field Photography with a Hand-Held Plenoptic Camera
    Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz, Pat Hanrahan
    Stanford University Computer Science Tech Report CSTR 2005-02
    Light Field Rendering
    Marc Levoy and Pat Hanrahan,
    Proc. SIGGRAPH '96
    See also the papers produced by the Light fields and computational photography and Stanford Multi-Camera Array projects

    Archives of images and animations:

    2005 - our first light field micrographs, including insect legs, silkworm mandibles, and mouse lungs. Also included are volumetric reconstructions using 3D deconvolution, and some experimental all-focus images.
    2006 - light field micrographs captured at the Marine Biological Laboratory, including onion skin (under DIC illumination), squid skin (under grazing illumination), a light field scatterometry experiment, and mouse embryos.
    2007 - light field micrographs captured at the Hopkins Marine Station, including fern spores and cone snails.
    Also included is a description of our portable light field microscope and real-time software viewer.
    2008 - our light field illuminator (LFI) is used to examine a human hair fiber under changing illumination.
    Also, functional neural imaging of larval zebrafish, recorded in Florian Engert's lab at Harvard University.

    Downloadable software:

    LFDisplay: a software package for real-time viewing of microscope light fields. Several sample light fields are included.
    Also included is a "protocol" (i.e. step-by-step instructions) on how to build a Light Field Microscope (LFM).


    Breaking news: a high-resolution light field microscope

    February 14, 2008

    Microscope design: Marc Levoy
    Specimen: Shinya Inoue (Marine Biological Laboratory)
    Compact light field microscope Prototype high-resolution light field microscope

    At left above is our most compact light field microscope (LFM). As introduced on this web page, it consists of an ordinary research microscope (Nikon Eclipse 80i) and cooled scientific camera (Retiga 4000R, 2048 x 2048 pixels) with a microlens array inserted between the two (red circle). The image formed by the microlens array is conveyed to the sensor in the camera by a 1:1 relay lens system consisting of two nose-to-nose Nikon 50mm/1.4 photographic lenses. With this arrangement, instead of capturing an ordinary 2048 x 2048 pixel image, the camera captures a 2048 x 2048 sample light field, capable of producing 17 x 17 different oblique views, each about 120 x 120 pixels in size. We have fabricated several copies of this arrangement, which have been used by microscopists in several departments at Stanford University, at the Hopkins Marine Institute (Monterey, CA), the Marine Biological Laboratory (Woods Hole, MA), and ETH (Zurich).

    At right above is a prototype high-resolution light field microscope. A 45-degree mirror (A) sends the microscope image through a 2:1 telecentric relay lens (B), our microlens array (C), a 1:1 telecentric relay lens (D), and a Canon 5D full-frame digital SLR (E). Since doubling the lateral magnification using a 2:1 relay lens roughly halves the angular aperture (this is called the LaGrangian invariant), we use an f/30 microlens array rather than the f/20 array we more commonly use. The camera was manually microstepped to place a green pixel beneath each pixel site, thereby avoiding color demosaicing and the consequent loss in spatial resolution. The light fields captured using this arrangement (see below) contain approximately 4500 x 3000 samples, capable of producing 15 x 15 oblique views, each about 300 x 200 pixels in size. The field of view is approximately 1.8mm horizontally for the 20x light field and 0.9mm for the 40x. For the objectives we used and the pixel resolution of our camera, these light fields are also diffraction-limited, meaning that no further spatial or angular resolution can be squeezed from the wavefront propagating through the microscope, at least not using conventional image capture protocols.

     
    Light field (click on image for full size)
    20x/0.75NA (dry)


    Flyaround (click on image to watch movie)
    (or reduced to CIF size in this .mp4 file)

    Light field (click on image for full size)
    40x/1.3NA (oil)



    Flyaround (click on image to watch movie)
    (or reduced to CIF size in this .mp4 file)
    Click here to view these light fields interactively in your browser, using a Flash-based light field viewer,
    and click here to view them using LFDisplay, our hardware-accelerated viewer for microscope light fields.

    The two columns above represent two light fields captured using this high-resolution microscope. The specimen is a Golgi-stained slice of rat brain (courtesy of Shinya Inoue, Marine Biological Laboratory). The light field at left was photographed using a 20x/0.75NA objective, and the light field at right using a 40x/1.3NA oil immersion objective. Note the spatial detail in both movies; you can easily see the fine dendrites. Note also the large amount of parallax visible in the 40x movie, especially on the looping capillaries in the lower-left part of the field of view. This much parallax is available because a 1.3NA objective can capture rays leaving the specimen at angles up to 59 degrees on either side of the optical axis. For photographers, this would correspond to a lens having a relative aperture of f/0.38 - far faster than any commercially available photographic lens.


    © 2005-2008,2013 Marc Levoy
    Last update: December 7, 2015 07:19:44 PM