bundleAdjustment

Adjust collection of 3-D points and camera poses

Syntax

[xyzRefinedPoints,refinedPoses] = bundleAdjustment(xyzPoints,pointTracks,cameraPoses,intrinsics)

[wpSetRefined,vSetRefined,pointIndex] = bundleAdjustment(wpSet,vSet,viewIDs,intrinsics)

[___,reprojectionErrors] = bundleAdjustment(___)

[___] = bundleAdjustment(___,Name=Value)

Description

[xyzRefinedPoints,refinedPoses] = bundleAdjustment(xyzPoints,pointTracks,cameraPoses,intrinsics) refines 3-D points and camera poses to minimize reprojection errors. The refinement procedure is a variant of the Levenberg-Marquardt algorithm. The function uses the same global reference coordinate system to return both the 3-D points and camera poses.

[wpSetRefined,vSetRefined,pointIndex] = bundleAdjustment(wpSet,vSet,viewIDs,intrinsics) refines 3-D points from the world point set, wpSet, and refines camera poses from the image view set, vSet. viewIDs specify the camera poses in vSet to refine.

[___,reprojectionErrors] = bundleAdjustment(___) returns the mean reprojection error for each 3-D world point, in addition to the arguments from the previous syntax.

[___] = bundleAdjustment(___,Name=Value) specifies options using one or more name-value arguments in addition to any combination of arguments from previous syntaxes. For example, MaxIterations=50 sets the number of iterations to 50. Unspecified arguments have default values.

Examples

Refine 3-D Points and Camera Poses From Single Camera

Load data for initialization.

data = load("globeBA.mat");

Refine the camera poses and points.

[xyzRefinedPoints,refinedPoses] = ...
    bundleAdjustment(data.xyzPoints,data.pointTracks,data.cameraPoses,data.intrinsics);

Display the 3-D points and camera poses before and after the refinement.

pcshowpair(pointCloud(data.xyzPoints), pointCloud(xyzRefinedPoints), ...
    AxesVisibility="on", VerticalAxis="y", VerticalAxisDir="down", MarkerSize=40);
hold on
plotCamera(data.cameraPoses, Size=0.1, Color="m");
plotCamera(refinedPoses, Size=0.1, Color="g");
legend("Before refinement", "After refinement", color="w");

Input Arguments

xyzPoints — Unrefined 3-D points

M-by-3 matrix

Unrefined 3-D points, specified as an M-by-3 matrix of [x y z] locations.

pointTracks — Matching points across multiple images

N-element array of pointTrack objects

Matching points across multiple images, specified as an N-element array of pointTrack objects. Each element contains two or more matching points across multiple images.

cameraPoses — Camera pose information

two-column table

Camera pose information, specified as a two-column table with columns ViewId and AbsolutePose. The view IDs relate to the IDs of the objects in the pointTracks argument. You can use the poses object function to obtain the cameraPoses table.

intrinsics — Camera intrinsics

cameraIntrinsics object | N-element array of cameraIntrinsics objects

Camera intrinsics, specified as a cameraIntrinsics object or an N-element array of cameraIntrinsics objects. N is the number of camera poses or the number of IDs in viewIDs. Use a single cameraIntrinsics object when images are captured using the same camera. Use a vector cameraIntrinsics objects when images are captured by different cameras.

wpSet — 3-D world points

worldpointset object

3-D world points, specified as a worldpointset object.

vSet — Camera poses

imageviewset object

Camera poses, specified as an imageviewset object.

viewIDs — View identifiers

N-element array

View identifiers, specified as an N-element array. The viewIDs represent which camera poses to refine specifying their related views in imageviewset.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: MaxIterations=50 sets the number of iterations to 50.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

MaxIterations — Maximum number of iterations

50 (default) | positive integer

Maximum number of iterations before the Levenberg-Marquardt algorithm stops, specified as a positive integer.

AbsoluteTolerance — Absolute termination tolerance

1.0 (default) | positive scalar

Absolute termination tolerance of the mean squared reprojection error in pixels, specified as positive scalar.

RelativeTolerance — Relative termination tolerance

1e-5 (default) | positive scalar

Relative termination tolerance of the reduction in reprojection error between iterations, specified as positive scalar.

PointsUndistorted — Flag to indicate lens distortion

false (default) | true

Flag to indicate lens distortion, specified as false or true. When you set PointsUndistorted to false, the 2-D points in pointTracks or in vSet must be from images with lens distortion. To use undistorted points, first use the undistortImage function to remove distortions from the images, then set PointsUndistorted.

FixedViewIDs — View IDs for fixed camera pose

[] (default) | vector of nonnegative integers

View IDs for fixed camera pose, specified as a vector of nonnegative integers. Each ID corresponds to the ViewId of a fixed camera pose in cameraPoses. An empty value for FixedViewIDs means that all camera poses are optimized.

Solver — Solver

"sparse-linear-algebra" (default) | "preconditioned-conjugate-gradient"

Solver, specified as "sparse-linear-algebra" or "preconditioned-conjugate-gradient". Use the "sparse-linear-algebra" solver for low sparsity images. Low sparsity indicates that many camera views observe some of the same world points. Use the "preconditioned-conjugate-gradient" (PCG) solver, from the general graphic optimization (g2o) library, for high sparsity images. High sparsity indicates that each camera view observes, only a small portion of the world points, specified by xyzPoints.

Verbose — Display progress information

false (default) | true

Display progress information, specified as false or true.

Output Arguments

xyzRefinedPoints — 3-D locations of refined world points

M-by-3 matrix

3-D locations of refined world points, returned as an M-by-3 matrix of [x y z] locations.

Data Types: single | double

refinedPoses — Refined camera poses

two-column table

Refined camera poses, returned as a two-column table. The table contains columns for ViewId and AbsolutePose.

wpSetRefined — Refined 3-D world points

worldpointset object

Refined 3-D world points, returned as a worldpointset object.

vSetRefined — Refined camera poses

imageviewset object

Refined camera poses, specified as an imageviewset object.

pointIndex — Indices of refined 3-D world points

M-element array

Indices to wpSetRefined of refined 3-D world points, returned as an M-element array.

reprojectionErrors — Reprojection errors

M-element vector

Reprojection errors, returned as an M-element vector. The function projects each world point back into each camera. Then, in each image, the function calculates the reprojection error as the distance between the detected and the reprojected point. The reprojectionErrors vector contains the average reprojection error for each world point.

References

[1] Lourakis, Manolis I. A., and Antonis A. Argyros. "SBA: A Software Package for Generic Sparse Bundle Adjustment." ACM Transactions on Mathematical Software 36, no. 1 (March 2009): 2:1–2:30.

[2] Hartley, Richard, and Andrew Zisserman. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge, UK ; New York: Cambridge University Press, 2003.

[3] Triggs, Bill, Philip F. McLauchlan, Richard I. Hartley, and Andrew W. Fitzgibbon. "Bundle Adjustment — A Modern Synthesis." In Proceedings of the International Workshop on Vision Algorithms, 298–372. Springer-Verlag, 1999.

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

Supports code generation in only host target platforms.

Version History

Introduced in R2016a

R2022b: Supports premultiply geometric transformation convention

Starting in R2022b, most Computer Vision Toolbox™ functions create and perform geometric transformations using the premultiply convention. The bundleAdjustment function now supports both the premultiply and the postmultiply convention.

Regardless of the convention, the cameraPoses argument must be a two-column table with variables ViewId and AbsolutePose. You can no longer specify the cameraPoses argument as a three-column table with variables ViewID, Orientation, and Location. The bundleAdjustment function chooses the convention according to the value of the AbsolutePose variable:

• When you specify AbsolutePose as a rigidtform3d object, the bundleAdjustment function uses the premultiply convention and returns refinedPoses with the AbsolutePose variable as a rigidtform3d object.

• When you specify AbsolutePose as a rigid3d object, the bundleAdjustment function uses the postmultiply convention and returns refinedPoses with the AbsolutePose variable as a rigid3d object.

Before, the function returned refinedPoses as a three-column table with variables ViewID, Orientation, and Location. If you need to obtain the values of Orientation and Location, which were in the postmultiply convention, then you can query properties of the geometric transformation object stored in the AbsolutePose variable:

• When AbsolutePose returns a rigidtform3d object, you can query the R and Translation properties of the object. The value of R is the transpose of Orientation.

• When AbsolutePose returns a rigid3d object, you can query the Rotation and Translation properties of the object. The value of Rotation is identical to Orientation.

You can streamline your geometric transformation workflows by switching to the premultiply geometric transformation convention. For more information, see Migrate Geometric Transformations to Premultiply Convention.