Implement Digital Camera Processing Pipeline

This example shows how to implement a camera processing pipeline that renders an RGB image from a RAW Bayer-pattern color filter array (CFA) image.

Digital single-lens reflex (DSLR) cameras, and many modern phone cameras, can save data collected from the camera sensor directly to a RAW file. Each pixel of RAW data is the amount of light captured by the corresponding camera photosensor. The data depends on fixed characteristics of the camera hardware, such as the sensitivity of each photosensor to a particular range of wavelengths of the electromagnetic spectrum. The data also depends on camera acquisition settings, such as exposure time, and factors of the scene, such as the light source.

Depending on the information available in the metadata for your image, there can be multiple ways to implement a pipeline that yields aesthetically pleasing results. The example shows one sequence of operations in a traditional camera processing pipeline using a subset of the metadata associated with the RAW data.

1. Import the RAW file contents

2. Linearize the CFA image

3. Scale the CFA data to a suitable range

4. Apply white-balance adjustment

5. Demosaic the Bayer pattern

6. Convert the demosaiced image to the sRGB color space

The example also shows how to create an RGB image from a RAW image when you do not have the RAW file metadata.

Read RAW File Contents

The RAW files created by a digital camera contain:

• A CFA image recorded by the photosensor of the camera

• Metadata, which contains all information needed to render an RGB image

Read a Bayer-pattern CFA image from a RAW file using the rawread function.

fileName = "colorCheckerTestImage.NEF";
cfaImage = rawread(fileName);

List information about the cfaImage variable and display the image.

whos cfaImage

  Name             Size                 Bytes  Class     Attributes

  cfaImage      4012x6034            48416816  uint16              

imshow(cfaImage,[])
title("RAW CFA Image")

Read the RAW file metadata using the rawinfo function.

cfaInfo = rawinfo(fileName);

The metadata contains several fields with information used to process the RAW image. For example, display the contents of the ImageSizeInfo field. The metadata indicates that there are more columns in the CFA photosensor array than in the visible image. The difference typically arises when a camera masks a portion of the photosensor to prevent those sections from capturing any light. This enables an accurate measure of the black level of the sensor.

disp(cfaInfo.ImageSizeInfo)

                 CFAImageSize: [4012 6080]
             VisibleImageSize: [4012 6034]
    VisibleImageStartLocation: [1 1]
             PixelAspectRatio: 1
                ImageRotation: 0
            RenderedImageSize: [4012 6034]

The RAW file metadata also includes information that enable the linearization, black level correction, white balance, and other processing operations needed to convert the RAW data to RGB.

colorInfo = cfaInfo.ColorInfo

colorInfo = struct with fields:
           LinearizationTable: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 ... ] (1x65536 uint16)
                   BlackLevel: [0 0 0 0]
                   WhiteLevel: [3827 3827 3827 3827]
                  CameraToXYZ: [3x3 double]
                 CameraTosRGB: [3x3 double]
    CameraAsTakenWhiteBalance: [495 256 256 324]
              D65WhiteBalance: [2.1900 0.9286 0.9286 1.0595]
                   ICCProfile: []

Linearize CFA Image

Many cameras apply nonlinear range compression to acquired signals before storing them in RAW files. Cameras typically store this range compression as a lookup table.

Plot a representative subset of the values in the LinearizationTable field of the image metadata. Values above the maxLinValue continue to increase linearly.

maxLinValue = 10^4;
linTable = colorInfo.LinearizationTable;
plot(0:maxLinValue-1,linTable(1:maxLinValue)) 
title("Linearization Table")

Processing operations on the digital camera processing pipeline are typically performed on linear data. To generate linear data, you must reverse the nonlinear range compression. The rawread function automatically performs this operation and returns linearized light values.

Scale Pixel Values to Suitable Range

Perform Black Level Correction

RAW images do not have a true black value. Even with the shutter closed, electricity flowing through the sensors causes nonzero photon counts. Cameras use the value of the masked pixels to compute the black level of the CFA image. To scale the image, subtract the measured black level from the CFA image data.

RAW file formats can report this black level in different formats. The RAW file metadata for the image in this example specifies black level as a vector, with one element per channel of the CFA image. Other RAW file formats, such as DNG, specify black level as a repeated m-by-n matrix that starts at the top left corner of the visible portion of the CFA.

Get the black level value of the RAW data from the BlackLevel metadata field.

blackLevel = colorInfo.BlackLevel;

To perform black level correction, first convert the black level vector to a 2-by-2 matrix. Omit this step for RAW images that specify black level as a matrix.

blackLevel = reshape(blackLevel,[1 1 numel(blackLevel)]);
blackLevel = planar2raw(blackLevel);

Replicate the black level matrix to be the size of the visible image.

repeatDims = cfaInfo.ImageSizeInfo.VisibleImageSize ./ size(blackLevel);
blackLevel = repmat(blackLevel,repeatDims);

Subtract the black level matrix from the CFA image matrix.

cfaImage = cfaImage - blackLevel;

Clamp Negative Pixel Values

To correct for CFA data values less than the black-level value, clamp the values to 0.

cfaImage = max(0,cfaImage);

Scale Pixel Values

RAW file metadata often represent the white level as the maximum value allowed by the data type. If this white level value is much higher than the highest intensity value in the image, then using this white level value for scaling results in an image that is darker than it should be. To avoid this, scale the CFA image using the maximum pixel value found in the image.

cfaImage = double(cfaImage);
maxValue = max(cfaImage(:))

maxValue = 
3366

cfaImage = cfaImage ./ maxValue;

Adjust White Balance

White balance is the process of removing unrealistic color casts from a rendered image, such that it appears closer to how human eyes would see the subject.

Get the white balance values from the metadata. There are two types of white balance metadata available. This step of the example uses the CameraAsTakenWhiteBalance field scales the color channels to balance the linear pixel values. The example uses the D65WhiteBalance field later in the pipeline to adjust the colors to a D65 white point.

whiteBalance = colorInfo.CameraAsTakenWhiteBalance

whiteBalance = 1×4

   495   256   256   324

Scale the multipliers so that the values of the green color channels are 1.

gLoc = strfind(cfaInfo.CFALayout,"G"); 
gLoc = gLoc(1);
whiteBalance = whiteBalance/whiteBalance(gLoc);

whiteBalance = reshape(whiteBalance,[1 1 numel(whiteBalance)]);
whiteBalance = planar2raw(whiteBalance);

Replicate the white balance matrix to be the size of the visible image.

whiteBalance = repmat(whiteBalance,repeatDims);
cfaWB = cfaImage .* whiteBalance;

Convert the CFA image to a 16-bit image.

cfaWB = im2uint16(cfaWB);

Demosaic

Convert the Bayer-encoded CFA image into a truecolor image by demosaicing. The truecolor image is in linear camera space.

cfaLayout = cfaInfo.CFALayout;
imDebayered = demosaic(cfaWB,cfaLayout);
imshow(imDebayered)
title("Demosaiced RGB Image in Linear Camera Space")

Convert from Camera Color Space to RGB Color Space

The metadata enables two options for converting the image from the linear camera space to a gamma-corrected RGB color space. You can use the CameraToXYZ metadata field to convert the data from the linear camera space to the RGB color space through the XYZ profile connection space (PCS), or you can use the CameraTosRGB metadata field to convert the image from the linear camera space to the RGB color space directly.

Use Profile Connection Space Conversion

Get the transformation matrix between the linear camera space and the XYZ profile connection space from the CameraToXYZ metadata field. This matrix imposes an RGB order. In other words:

[X,Y,Z]' = CameraToXYZ.*[R,G,B]'

cam2xyzMat = colorInfo.CameraToXYZ

cam2xyzMat = 3×3

    1.5573    0.1527    0.0794
    0.6206    0.9012   -0.2690
    0.0747   -0.3117    1.4731

Normalize the cam2xyzMat matrix according to a D65 white point. Get the XYZ normalization values from the D65WhiteBalance metadata field.

whiteBalanceD65 = colorInfo.D65WhiteBalance

whiteBalanceD65 = 1×4

    2.1900    0.9286    0.9286    1.0595

The white balance multipliers are ordered according to the CFALayout metadata field. Reorder the multipliers to match the row ordering of the cam2xyzMat matrix.

cfaLayout = cfaInfo.CFALayout;
wbIdx(1) = strfind(cfaLayout,"R");
gidx = strfind(cfaLayout,"G");
wbIdx(2) = gidx(1); 
wbIdx(3) = strfind(cfaLayout,"B");

wbCoeffs = whiteBalanceD65(wbIdx);
cam2xyzMat = cam2xyzMat ./ wbCoeffs;

Convert the image from the linear camera space to the XYZ color space using the imapplymatrix function. Then, convert the image to the sRGB color space and apply gamma correction using the xyz2rgb function.

imXYZ = imapplymatrix(cam2xyzMat,im2double(imDebayered));
srgbPCS = xyz2rgb(imXYZ,OutputType="uint16");
imshow(srgbPCS)
title("sRGB Image Using PCS")

Use Conversion Matrix from RAW File Metadata

Convert the image from the linear camera space to the linear RGB color space using the transformation matrix in the CameraTosRGB metadata field.

cam2srgbMat = colorInfo.CameraTosRGB;
imTransform = imapplymatrix(cam2srgbMat,imDebayered,"uint16");

Apply gamma correction to bring the image from the linear sRGB color space to the sRGB color space.

srgbTransform = lin2rgb(imTransform);
imshow(srgbTransform)
title("sRGB Image Using Transformation Matrix")

Convert RAW to RGB Without Metadata

You can convert RAW data directly to an RGB image using the raw2rgb function. The raw2rgb function provides comparable results as a custom processing pipeline tailored to your data and acquisition settings. However, the raw2rgb function does not provide the same fine-tuned precision and flexibility as a custom pipeline.

Convert the image data in the RAW file to the sRGB color space using the raw2rgb function.

srgbAuto = raw2rgb(fileName);
imshow(srgbAuto)
title("sRGB Image Using raw2rgb Function")

Compare the result of the raw2rgb conversion to that obtained using the full camera processing pipeline using PCS and a transformation matrix.

montage({srgbPCS,srgbTransform,srgbAuto},Size=[1 3]);
title("sRGB Image Using PCS, raw2rgb Function, and Transformation Matrix (Left to Right)")