 |
 |
 |
 |
 |
|---|
Most of the prior image manipulation techniques that I've tinkered with were done through filtering. In filtering, it is the range of the image that is changed. We now shift to point processing, image warping, and image morphing, which are processes with which we can change the domain of an image.
On one of the most popular, illustrative, and fun applications of this is face morphing. Face morphing is creating an animation such that one face smoothly morphs into another face.
Before we can dive straight in to the morphing, we need a way of deciding how a face should morph in to the other. This can be accomplished by defining correspondence points. That is, we can use points to identify the same features on the faces. For example, we can have points surrounding the eyes of the faces on both images. This tells us that we should transform the eye one on face to match the eye that we marked on the other face.
Now the points alone are not enough, since the points are only a pixel, we can not only make the corresponding points match and expect the entire image to match. What we really want is for the pixels around each point to match as well. In order to accomplish this, we turn to triangulation.
After choosing our correspondence points, we can create a triangulation. Once we create the triangulation, we now have corresponding triangular sections of the image. Now our task is simply to transform each triangle such that the corresponding triangles match.
| Triangulations Example | |
|---|---|
 |
 |
Before we compute all of the frames required for the morphing animation, let's examine just one frame in particular: the frame at the exact halfway point when morphing between face 1 and face 2. This will give us a concrete example and allow us to examine the process in more detail, afterwhich we can generalize the process to explain how we generate all of the frames.
As previously mentioned, once we have the corresponding triangles, the problem boils down to computing the transformations to transform each triangle in face 1 to its corresponding triangle in face 2. Since our regions are defined by 3 points, we can use an affine transformation to do this. However, in order to minimize distortion, instead of morphing one face into the other, it is better to morph both images to some mean shape, which is easily found by averaging the corresponding points and calculating an average triangulation.
This is a good time to discuss inverse warping and interpolation.
We know that we are going to use an affine transformation to get from
one set of triangles to the other. The problem is, if we simply look at
each of our original points and then apply our transformation and move
the value to our new point we may have holes since we are not guaranteed
that every value will get mapped to. It is instead better to
look at each of our new points and find which value should go there using
the inverse transformation. This ensures every pixel in our new image
has a value. However, what if the pixel we are choosing our color
from is not an exact location? There are several methods to handle
this such as choosing the nearest neighbor or interpolation.
As an example, here is my face warped to the average shape of mine
and President Obama's face with foward and inverse warping:
| Forward Warping | Inverse Warping |
|---|---|
 |
 |
Now, we have found our affine transformation to transform the set of triangles in both faces to the corresponding triangles in the average shape, we have our inverse warping and interpolation to prevent empty areas, but how do we combine the two images? Well this is simply a matter of cross dissolving, meaning we we take the values at each pixel from both images and, in the case of the midway face, average them.
Here are the results of finding the midway face:
| Face 1 | Face 2 | Midway Face |
|---|---|---|
 |
 |
 |
Now let's examine how we can generalize the above process to create a sequence of frames that, when combined, produce a smooth animation from face 1 to face 2.
In the case of the midway face, we morphed each image to an average shape and then took the pixel values of each of the warped images and averaged them. In the case of the mid-way face, using the true average makes sense. So how should we adjust if we are not producing the frame at the exact middle of the animation? Simple, we just take use a weighted average instead of the true average. Since we are producing an animation, it only makes sense that our weight is a function of time.
Here is the final result:
| Face 1 | Face 2 | Morph Animation |
|---|---|---|
|
|
 |
Merging from one face to another is not the only application of this technique. We can also use this process to calculate the average face of a population.
As an example we will use this data set of Danish engineers:
| Danish Engineers |
|---|
 |
Thankfully, the set of images also comes pre annotated with correspondence points. So in order to calculate the mean face, we simply:
Here are a few examples of some of the engineers warped to the average shape:
 |
 |
 |
|---|
And here is the final result of the computed average face:
| The Average Male Danish Engineer |
|---|
 |
Here is an example of two of the engineers morphing into the average:
 |
|---|
Here are the scary results of warping my face into the shape of the average Danish male engineer and vice versa:
 |
 |
|---|
And here is an animation of me morphing into the average and then the average morphing in to me:
 |
|---|
In addition to interpolating to create new faces as we have been doing, we can also extrapolate. For example, I can create new versions of my face by extrapolating from the mean face that we have been working with. That is, instead of morphing my face to some points between the corresponding points on my face and the corresponding points on the average face, we can morph my face to some points beyond the corresponding points but along the correct direction.
Recall that when calculating points in between the corresponding points, we used: alpha*points_1 + (1-alpha)*points_2 to get our points where alpha was between 0 and 1. In order to extrapolate, all we do is let alpha grow beyond 1.
Here are the results:
| alpha = 0 | alpha = 0.25 | alpha = 0.5 | alpha = 0.75 | alpha = 1 | alpha = 1.25 | alpha = 1.50 |
|---|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
| alpha = 0 | alpha = -0.25 | alpha = -0.5 | alpha = -0.75 | alpha = - 1 | alpha = -1.25 | alpha = -1.50 |
|---|---|---|---|---|---|---|
|
 |
 |
 |
 |
 |
 |
Note that alpha can not grow infinitely large since the transformations of our triangulations will break down.
Notice in the extrapolation section above that as alpha grows in the positive direction, we begin to emphasize the features that make the Danish face distinct. Similarly, as alpha grows in the negative direction, we begin to emphasize the features that make my face distinct. This ability to selectively emphasize certain features allows us to change the attributes of a face to match those of another face type. For example, we can take my face, and make it look more feminine by looking at the average face of a Mexican woman.
We will use the following image:
| Average Mexican Female |
|---|
 |
We can then morph the color, shape, or both:
| Original |  |
 |
Just Shape |
|---|---|---|---|
| Just Color |  |
 |
Both |
We can also do the same with ethnicity. We can use the following image as an example:
| Average African-American Male |
|---|
 |
| Original |  |
 |
Just Shape |
|---|---|---|---|
| Just Color |  |
 |
Both |