In this tutorial we will see how to warp a single triangle in an image to another triangle in a different image.

In computer graphics people deal with warping triangles all the time because any 3D surface can approximated by triangles. Images can be broken down into triangles and warped. However, in OpenCV there is no out of the box method that warps pixels inside a triangle to pixels inside another triangle.

This tutorial will explain step by step how to transform the triangle in the left image in Figure 1 to the right image.

Before we dive into code we need to understand what an affine transform is.

## What is an Affine Transform ?

An ** Affine Transform ** is the simplest way to transform a set of 3 points ( i.e. a triangle ) to another set of arbitrary 3 points. It encodes translation ( move ), scale, rotation and shear. The image below illustrates how an affine transform can be used to change the shape of a square. Note that using an affine transform you can change the shape of a square to a parallelogram at any orientation and scale. However, the affine transform is not flexible enough to transform a square to an arbitrary quadrilateral. In other words, after an affine transform parallel lines continue to be parallel.

In OpenCV an affine transform is a 2×3 matrix. The first two columns of this matrix encode rotation, scale and shear, and the last column encodes translation ( i.e. shift ).

Given a point , the above affine transform, moves it to point using the equation given below

## Triangle Warping using OpenCV

__download__the C++ and Python code and images by subscribing to our newsletter

__here__.

We now know that to warp a triangle to another triangle we will need to use the affine transform. In OpenCV, *warpAffine* allows you to apply an affine transform to an image, but not a triangular region inside the image. To overcome this limitation we find a bounding box around the source triangle and crop the rectangular region from the source image. We then apply an affine transform to the cropped image to obtain the output image. The previous step is crucial because it allows us to apply the affine transform to a small region of the image thus improving computational performance. Finally, we create a triangular mask by filling pixels inside the output triangle with white. This mask when multiplied with the output image turns all pixels outside the triangle black while preserving the color of all pixels inside the triangle.

Before we go into the details, let us read in input and output images, and define input and output triangles. For this tutorial our output image is just white, but you could read in another image if you wish.

**C++**

// Read input image and convert to float Mat img1 = imread("robot.jpg"); img1.convertTo(img1, CV_32FC3, 1/255.0); // Output image is set to white Mat imgOut = Mat::ones(imgIn.size(), imgIn.type()); imgOut = Scalar(1.0,1.0,1.0); // Input triangle vector <Point2f> tri1; tri1.push_back(Point2f(360,200)); tri1.push_back(Point2d(60,250)); tri1.push_back(Point2f(450,400)); // Output triangle vector <Point2f> triOut; tri2.push_back(Point2f(400,200)); tri2.push_back(Point2f(160,270)); tri2.push_back(Point2f(400,400));

**Python**

# Read input image and convert to float img1 = cv2.imread("robot.jpg") # Output image is set to white img2 = 255 * np.ones(img_in.shape, dtype = img_in.dtype) # Define input and output triangles tri1 = np.float32([[[360,200], [60,250], [450,400]]]) tri2 = np.float32([[[400,200], [160,270], [400,400]]])

Our inputs and outputs are now defined and we are ready to go through the steps needed to transform all pixels inside the input triangle to output triangle.

**Calculate the bounding boxes**

In this step we calculate bounding boxes around triangles. The idea is to warp only a small part of the image and not the entire image for efficiency.**C++**// Find bounding rectangle for each triangle Rect r1 = boundingRect(tri1); Rect r2 = boundingRect(tri2);

**Python**# Find bounding box. r1 = cv2.boundingRect(tri1) r2 = cv2.boundingRect(tri2)

**Crop images & change coordinates**

To efficiently apply affine transform to a piece of the image and not the entire image, we crop the input image based on the bounding box calculated in the previous step. The coordinates of the triangles also need to be modified so as to reflect their location in the new cropped images. This is done by subtracting the x and y coordinates of the top left corner of the bounding box from the x and y coordinates of the triangle.**C++**// Offset points by left top corner of the respective rectangles vector<Point2f> tri1Cropped, tri2Cropped; vector<Point> tri2CroppedInt; for(int i = 0; i < 3; i++) { tri1Cropped.push_back( Point2f( tri1[i].x - r1.x, tri1[i].y - r1.y) ); tri2Cropped.push_back( Point2f( tri2[i].x - r2.x, tri2[i].y - r2.y) ); // fillConvexPoly needs a vector of Point and not Point2f tri2CroppedInt.push_back( Point((int)(tri2[i].x - r2.x), (int)(tri2[i].y - r2.y)) ); } // Apply warpImage to small rectangular patches Mat img1Cropped; img1(r1).copyTo(img1Cropped);

**Python**# Offset points by left top corner of the # respective rectangles tri1Cropped = [] tri2Cropped = [] for i in xrange(0, 3): tri1Cropped.append(((tri1[0][i][0] - r1[0]),(tri1[0][i][1] - r1[1]))) tri2Cropped.append(((tri2[0][i][0] - r2[0]),(tri2[0][i][1] - r2[1]))) # Apply warpImage to small rectangular patches img1Cropped = img1[r1[1]:r1[1] + r1[3], r1[0]:r1[0] + r1[2]]

**Estimate the affine transform**: We have just obtained the coordinates of input and output triangles in the cropped input and output images. Using these two triangles we can find the affine transform that will transform the input triangle to the output triangle in the cropped images using the following code.**C++**// Given a pair of triangles, find the affine transform. Mat warpMat = getAffineTransform( tri1Cropped, tri2Cropped );

**Python**# Given a pair of triangles, find the affine transform. warpMat = cv2.getAffineTransform( np.float32(tri1Cropped), np.float32(tri2Cropped) )

**Warp pixels inside bounding box**

The affine transform found in the previous step is applied to the cropped input image to obtain the cropped output image. In OpenCV you can apply an affine transform to an image using*warpAffine*.**C++**// Apply the Affine Transform just found to the src image Mat img2Cropped = Mat::zeros(r2.height, r2.width, img1Cropped.type()); warpAffine( img1Cropped, img2Cropped, warpMat, img2Cropped.size(), INTER_LINEAR, BORDER_REFLECT_101);

**Python**# Apply the Affine Transform just found to the src image img2Cropped = cv2.warpAffine( img1Cropped, warpMat, (r2[2], r2[3]), None, flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT_101 )

**Mask pixels outside the triangle**

In the previous step we obtained the output rectangular image. However, we are interested in a triangle inside the rectangular region. So we create a mask using*fillConvexPoly*that is used to black out all pixels outside the triangle. This new cropped image can finally be put in the right location in the output image using top left corner of the output bounding rectangle.**C++**// Get mask by filling triangle Mat mask = Mat::zeros(r2.height, r2.width, CV_32FC3); fillConvexPoly(mask, tri2CroppedInt, Scalar(1.0, 1.0, 1.0), 16, 0); // Copy triangular region of the rectangular patch to the output image multiply(img2Cropped,mask, img2Cropped); multiply(img2(r2), Scalar(1.0,1.0,1.0) - mask, img2(r2)); img2(r2) = img2(r2) + img2Cropped;

**Python**# Get mask by filling triangle mask = np.zeros((r2[3], r2[2], 3), dtype = np.float32) cv2.fillConvexPoly(mask, np.int32(tri2Cropped), (1.0, 1.0, 1.0), 16, 0); # Apply mask to cropped region img2Cropped = img2Cropped * mask # Copy triangular region of the rectangular patch to the output image img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] * ( (1.0, 1.0, 1.0) - mask ) img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3], r2[0]:r2[0]+r2[2]] + img2Cropped

This brings us to the end of this tutorial. I hope it was a good learning experience. I recommend you download the code and give it a try.

## Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource guide. In our newsletter we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

** Image Credits **

The robot image is in public domain. Many thanks to artist bamenny.

李凯 says

Great tutorial all the time! ^_^

Satya Mallick says

Thank you!

thinh says

I am implementing face swap function on iphone, so that I need to warp triangle to another. I used this algorithm, but it is quite slow when running on iphone. Is there any other algorithm faster than this? Thanks.

Adriano Rodrigues says

Hi, Satya!

I would like to download the code, but I can’t! I already subscribe me but I can’t do the download….

I don’t have received the guide….

Louis-P. Bourret says

nice tutorial Satya yet again 🙂

One small optimization I noticed could be done: your mask could be filled with CV_8UC1 instead CV_32FC3

Bahramudin Adil says

This does not need to be such a complex process, you can do it very easy, just by few lines of code. Just make a mask and copy the source to the destination using the mask.