Speeding Scientific Discovery with AI-Powered Microscopy
User-Friendly Image Processing and Insights from Massive Lightsheet Data Sets
Optical microscopes are becoming increasingly important for interrogating biological samples, both live and fixed. Lightsheet technology lets scientists rapidly capture microscopic images and videos of living cells and organisms by imaging an entire cross-section in a single frame. This reduces laser photodamage to the sample, as compared with other optical imaging techniques, allowing for long-term imaging of sensitive samples such as cells, embryos, and tissues. 3D volumes are acquired through rapid serial scanning.
Researchers increasingly turn to this advanced imaging technique to visualize intricate structures within cells and track the development secrets hidden within embryonic origins. But these microscopes generate massive terabyte-sized volumes of complex imagery, creating bottlenecks in image analysis for scientists seeking insights.
A team building a new tool working with Dr. Abhishek Kumar, an investigator and Chan Zuckerberg Initiative Imaging Scientist at the Marine Biological Laboratory (MBL) in Woods Hole, Massachusetts, is guided by these concerns. They are building the new tool in MATLAB®.
The project, led by MBL Research Assistant William Ramos with help from MBL Imaging Specialist Anthony Mautino, is an end-to-end lightsheet microscopy workflow. From microscope control and image acquisition to reconstruction, visualization, analysis, and quantification, the tool helps researchers focus on their imagery instead of requiring them to learn to code.
The custom-built lightsheet microscope at the Marine Biological Laboratory. (Image credit: Emily Greenhalgh, Marine Biological Laboratory)
“MATLAB is an obvious choice for us because images are stored and operated on as matrices and arrays, and for us MATLAB is the cleanest and most efficient way to code and package these tools for both our laboratory and dissemination,” Dr. Kumar said. “And we know that it comes with outstanding support and undergoes rigorous quality assurance with each new release.”
Automating Algorithms
The lightsheet microscopy tool consists of multiple user interfaces built in MATLAB App Designer with various options for preprocessing, segmenting, and modifying images. Often, when biologists come to Dr. Kumar’s team with a project, their samples have fundamentally new image processing requirements. This leaves the team figuring out which method works best and what algorithms to choose. This manual trial-and-error process takes hours, something the tool speeds up significantly.
“Having a tool like this where you can just push buttons and test the results immediately saves considerable time.”
“Will [Ramos] is putting all the things that we typically throw at these projects in one tool,” Mautino said. “A tool like this, where you can just push buttons and test the results immediately, saves considerable time.”
There’s also a variety of preprocessing steps that Ramos performs with Image Processing Toolbox™ before employing various image transformations. This incorporates classical image processing algorithms and convolutional neural networks (CNNs) trained with Deep Learning Toolbox™.
“Once he has a bunch of settings that work on a particular project’s data, we can batch process based on the settings,” Mautino said.
Because some lightsheet microscopy projects take images every 15 minutes or more, the parts of the sample the researcher wants to track move quite a bit, possibly out of the field of view. The tool developed by the lab can provide real-time feedback and help position the sample so that it does not drift out of the field of view.
The lightsheet microscopy tool also helps the CNN identify portions of the image that the researcher isn’t interested in studying and indeed obscures the portion they are interested in analyzing. For example, when researchers study the cytoskeleton in fly ovaries, the oocyte can occlude important features in other cells during visualization. Researchers often manually remove the oocyte from every image. The tool lets Ramos select the area in a few frames and then train a neural network to remove it from the rest of the time series of volumetric data.
Another application Ramos foresees the lightsheet microscopy tool being helpful for is data reduction. To enable quicker computational processing and analysis, researchers reduce the full raw data set to just the relevant regions of interest. This involves detecting sample motion, shifting the data to recenter, and cropping out uninformative background regions. Such data reduction techniques discard surplus pixels, allowing more efficient compute resource usage.
“Data reduction with this tool is very useful, especially in our setup where we collect several terabytes of data daily,” Mautino said.
Ramos also built custom tools to align multichannel data, fix artifacts such as bad pixels, and visualize and filter data on the fly during acquisition. Depending on the project, he has also used Optimization Toolbox™, Image Acquisition Toolbox™, and Statistics and Machine Learning Toolbox™.
The front panel of the lightsheet microscopy tool. (Image credit: Marine Biological Laboratory)
“Our application is very modular in that respect,” Dr. Kumar said. “It has many features, and many of those tools are common and used frequently. Then there are more unique steps you can add, like a specific denoising model or a specific neural network, depending on what outcome the researcher wants.”
Imaging Butterfly Ovaries
This end-to-end workflow has already had successes, including our collaborator Dr. Nipam Patel and his lab members, who study how butterfly ovaries develop. Butterflies are delicate, and their ovaries are even more so. The lightsheet microscopy setup the researchers used was carefully optimized to minimize the photodamage of the sample over 40 hours of imaging. Often, a single time point alone can be over half a gigabyte. Besides grappling with the massive amounts of data, the biologists needed help removing noise from the images, which had more than usual because of the imaging conditions. The segmentation also had to shift as the embryo developed and changed.
Left: The maximum image intensity projection of a three-dimensional stack of a butterfly ovary taken on a single-view lightsheet microscope. Right: The maximum image intensity of a computationally cleaned version of the original image stack. (Image credit: Kyle Demarr and Sophia Kelly, Marine Biological Laboratory)
After Ramos manually segmented 162 images, the tool processed the remaining 4,000 images within minutes. The manual segmentation would have taken up to 10 hours alone, but Ramos’ process had it to under half an hour.
Ramos used another CNN trained to identify and segment specific features in the data, like removing the ovary’s yolk, which was obstructing part of the image. After Ramos manually segmented 162 images, the tool processed the remaining 4,000 images within minutes. The manual segmentation would have taken up to 10 hours alone, according to Ramos, but his process had it to under half an hour.
“Our collaborators were quite happy with that processing time and image quality,” Ramos said.
Benefits of End-to-End MATLAB
From the software used with the microscopes to the visualization, the team built everything in MATLAB, finding the image processing and analysis capabilities more flexible than proprietary microscope software.
“The reason we use MATLAB is both because of interfacing with hardware and because the math is so much nicer looking,” Mautino said.
“What the lightsheet microscopy tool does is allow people without experience and knowledge of coding to still benefit from all these sophisticated tools.”
“And then you can imagine many of our microscopes contain different hardware pieces, and each one needs its own code,” Mautino said. “Then that itself will become a long-term project. In that case, MATLAB comes in handy since those manufacturers already work with it and they develop these drivers for us.” Some of the cameras they use do not have endorsed control software written in Python.
The goal is to empower as many researchers without coding expertise as possible to be able to use advanced analysis techniques. Many different biological disciplines could use such image analysis tools, according to Dr. Kumar.
The tools have saved collaborators days of manual processing or waiting for other researchers to do the image processing. The simplicity also opens access for nonexperts such as high school students. Ramos said they will soon share the user interface on GitHub® for wider adoption, get feedback, and spur growth in tools for biological image analysis.
“Nowadays, it’s hard to imagine any imaging project without sophisticated image analysis and image processing,” Dr. Kumar said. “What the lightsheet microscopy analysis tool does is allow people without experience and knowledge of coding to still benefit from all these sophisticated tools. Having all these tools readily accessible and easy to use is critical.”
Read Other Stories
ACADEMIA
Building a Microscope That Sees with Helium
Shooting Small Atoms at Samples Doesn’t Damage Delicate Specimens
