Implementing a virtual try-on network using deep generative models

My projects mainly revolved around potential use cases for generative networks in fashion. Broadly speaking, the end goal of my project here was to implement a virtual try-on network — essentially taking in-shop clothing and a person image as input to give output as an image of a person wearing those clothes. The model implemented has been done with a focus on tops, with complete apparel transfer being potential future work.

The Segmentation Algorithm

For this, we initially needed to be able to implement a segmentation algorithm. Even though open-source state-of-the-art models could have been used to implement this, we stuck with robust image processing techniques for segmentation. With the idea being to localize the face and understand the skin color of the model from the face image to be able to divide an image into hair, clothes, skin, and background.

Geometric Matching Module

Once we have the clothing segment, we can now geometrically compare this clothing segment to the in-shop clothing. Our goal is now to be able to learn transforms on the in-shop clothing to make it as geometrically similar to the model clothing. To visually describe this one can refer to the image below. The example is a grid of six images. Top left is the in-shop clothes, top right being the clothing segment of the model, the top middle being the transform (bottom left) applied on the in-shop clothes.

The above examples are generated during training and hence the in-shop clothing and the model clothing are the same. This can also lead to an easier qualitative assessment. The network architecture to learn this transform is briefly described below.

We call the learning of this transformation as the Geometric Matching Module, as it matches the in-shop clothing to the current clothing trying to get them to match geometrically. Some of the results after training are shown below.

Try-On Module

The instinctive approach to imposing the new clothing now is to simply paste it over the image, but as one can see this will cause problems due to overlap with hair and hands, and the previous clothing stays, making it look very unrealistic. The solution to this was the try-on module, where we implement an encoder-decoder network to smoothen out the image.

This gives a smoothened image that looks much more realistic than the results we would have if we were to paste the image over the model. The article has avoided any in-depth description of the work done and for a thorough description of the model and training strategy, it is advisable to read the paper here.