9.2 High-level overview (part 2)

Replacement

The correlation of each pair of boxes is calculated in the same way as in the identification process, by taking the sum of products of each pixel. Once every pair of boxes has had its correlation calculated, the best match in the replacement bank is determined, and is placed in the position the detected red cup occupies. The best match does not necessarily have the most boxes with the highest correlation, or have the highest total correlation. Instead, some boxes are given a greater weight based on results from running an edge detector on each box. If the box contains more edges, it is given a greater weight. This is because it is most important to match edges when replacing the red solo cup, particularly hands. It is more obvious when edges do not match, and therefore edges are given a higher priority when selecting a replacement image. The replacement bank contains pictures of objects being held in hands in different orientations, so this methodology of weighting the boxes differently should lead to the most visually pleasing replacement possible.

The surrounding areas whose correlation was calculated are blurred and combined with the area surrounding the hole using a simple linear intensity blend to further facilitate blending of the replacement image with the image being filtered.