Depth Estimation Using Stereo Imaging in Matlab Object Tracking in 3D
Disparity Equation P(X,Y,Z)
Stereo system with parallel optical axes
Depth
Disparity: dx = xr - xl
B Z D f dx Image plane f = focal length
Image plane pl(xl,yl)
pr(xr,yr)
Optical Center Ol
f = focal length
Optical Center Or B = Baseline
LEFT CAMERA
RIGHT CAMERA
Disparity vs. Baseline P(X,Y,Z)
Stereo system with parallel optical axes
Depth
Disparity dx = xr - xl
B Z D f dx Image plane f = focal length
Image plane pl(xl,yl)
Optical Center Ol
pr(xr,yr)
f = focal length
Optical Center Or B = Baseline
LEFT CAMERA
RIGHT CAMERA
Depth Accuracy › Given the same image localization error – Angle of cones in the figure › Depth Accuracy (Depth Resolution) vs. Baseline – Depth Error 1/B (Baseline length) – PROS of Longer baseline, › better depth estimation
Ol
Two viewpoints Or
Z1 Z2
Z1
– CONS › smaller common FOV
› Depth Accuracy (Depth Resolution) vs. Depth – Disparity (>0) 1/ Depth – Depth Error Depth2 – Nearer the point, better the depth estimation
› An Example – f = 16 x 512/8 pixels, B = 0.5 m – Depth error vs. depth
Z2>Z1
Absolute error
Z2 Z (dx) fB Relative error Z Z (dx) Z fB
Depth from disparity image I(x,y)
Disparity map D(x,y)
image I´(x´,y´)
(x´,y´)=(x+D(x,y), y) So if we could find the corresponding points in two images, we could estimate relative depth…
Find the corners. points1 = detectHarrisFeatures(I1); points2 = detectHarrisFeatures(I2);
Extract the neighborhood features. [features1,valid_points1] = extractFeatures(I1,points1); [features2,valid_points2] = extractFeatures(I2,points2);
Match the features. indexPairs = matchFeatures(features1,features2);
Retrieve the locations of the corresponding points for each image. matchedPoints1 = valid_points1(indexPairs(:,1),:); matchedPoints2 = valid_points2(indexPairs(:,2),:);
Visualize the corresponding points. You can see the effect of translation between the two images despite several erroneous matches. figure; showMatchedFeatures(I1,I2,matchedPoints1,matchedPoints2);
Camera parameters Camera frame 2
Extrinsic parameters: Camera frame 1 Camera frame 2
Camera frame 1
Intrinsic parameters: Image coordinates relative to camera Pixel coordinates
• Extrinsic parameters: rotation matrix and translation vector • Intrinsic parameters: focal length, pixel sizes (mm), image center point, radial distortion parameters
load('webcamsSceneReconstruction.mat');
Read in the stereo pair of images. I1 = imread('sceneReconstructionLeft.jpg'); I2 = imread('sceneReconstructionRight.jpg');
Rectify the images. [J1, J2] = rectifyStereoImages(I1,I2,stereoParams);
Display the images after rectification. figure imshow(stereoAnaglyph(I1,I2));
Compute the disparity. disparityMap = disparity(rgb2gray(J1), rgb2gray(J2)); figure imshow(disparityMap,[0,64],'InitialMagnification',50);
xyzPoints = reconstructScene(disparityMap,stereoParams); points3D = xyzPoints; points3D = points3D ./ 1000; ptCloud = pointCloud(points3D, 'Color', J1); % Create a streaming point cloud viewer player3D = pcplayer([-3, 3], [-3, 3], [0, 8], 'VerticalAxis', 'y', ... 'VerticalAxisDir', 'down'); % Visualize the point cloud view(player3D, ptCloud);
%Undistort the images. I1 = undistortImage(I1,stereoParams.CameraParameters1); I2 = undistortImage(I2,stereoParams.CameraParameters2); [nBW1,maskedImage1] = createMask3(I1); [nBW2,maskedImage2] = createMask3(I2); BW1 = bwareaopen(nBW1,600); BW2 = bwareaopen(nBW2,600); stats1 = regionprops(BW1,'Centroid','BoundingBox'); stats2 = regionprops(BW2,'Centroid','BoundingBox'); bc1 = stats1.Centroid; bc2 = stats2.Centroid; bb1 = stats1.BoundingBox; bb2 = stats2.BoundingBox;
%Compute the distance from camera 1 to the face. point3d = triangulate(bc1, bc2, stereoParams); distanceInMeters = norm(point3d)/1000;
colorDevice = imaq.VideoDevice('kinect',1); depthDevice = imaq.VideoDevice('kinect',2); step(colorDevice); step(depthDevice); colorImage = step(colorDevice); depthImage = step(depthDevice); ptCloud = pcfromkinect(depthDevice, depthImage, colorImage); % Initialize a player to visualize 3-D point cloud data. The axis is % set appropriately to visualize the point cloud from Kinect. player = pcplayer(ptCloud.XLimits, ptCloud.YLimits, ptCloud.ZLimits,... 'VerticalAxis', 'y', 'VerticalAxisDir', 'down'); xlabel(player.Axes, 'X (m)'); ylabel(player.Axes, 'Y (m)'); zlabel(player.Axes, 'Z (m)');
view(player, ptCloud);
ptCloud = pcfromkinect(depthDevice, depthImage, colorImage);
XYZPoints
