On Face Segmentation, Face Swapping, and Face Perception

On Face Segmentation, Face Swapping, and Face Perception

2017, Apr 22    

What happens to machine face recognition when you perform state-of-the-art face swapping? We show how we achieve state-of-the-art face swapping and face segmentation under unprecedented conditions using simple methods. Unlike previous work, our swapping is robust enough to allow for extensive quantitative tests.

Yuval Nirkin, Iacopo Masi, Anh Tuan Tran, Tal Hassner, and Gerard Medioni, On Face Segmentation, Face Swapping, and Face Perception, IEEE Conference on Automatic Face and Gesture Recognition (FG), Xi’an, China on May 2018

We show that even when face images are unconstrained and arbitrarily paired, face swapping between them is actually quite simple. To this end, we make the following contributions. (a) Instead of tailoring systems for face segmentation, as others previously proposed, we show that a standard fully convolutional network (FCN) can achieve remarkably fast and accurate segmentations, provided that it is trained on a rich enough example set. For this purpose, we describe novel data collection and generation routines which provide challenging segmented face examples. (b) We use our segmentations to enable robust face swapping under unprecedented conditions. (c) Unlike previous work, our swapping is robust enough to allow for extensive quantitative tests. To this end, we use the Labeled Faces in the Wild (LFW) benchmark and measure the effect of intra- and inter-subject face swapping on recognition. We show that our intra-subject swapped faces remain as recognizable as their sources, testifying to the effectiveness of our method. In line with well known perceptual studies, we show that better face swapping produces less recognizable inter-subject results. This is the first time this effect was quantitatively demonstrated for machine vision systems.

PDF arXiv