Photo Based Clothing Measurements

Data science begins not with data but with questions. And sometimes getting the data necessary to answer those questions requires some ingenuity.

At Stitch Fix, we have many questions pertaining to clothing measurements. Construction tolerances on clothing are less tight than outsiders to the industry might expect; there is considerable variation in the measurements for garments both within particular manufacturing runs and between different runs. So in an ideal world, we would like to measure every garment that comes in, both for the sake of quality assurance (flagging items that are too far from spec) and to fine-tune our recommendations at the item level. For example, we can pair a shirt that is within tolerances but still slightly longer than spec in the sleeves with a client that prefers slightly long arm lengths.

We also want to capture much more than just the standard points of measure. For example, we want detailed numerical descriptions of sleeves and pant leg profiles for use both in our recommendations and in our new clothing design.

So we cannot rely solely on measurements from manufacturers. And it is infeasible to have a human measure every garment entering our warehouses by hand. To solve this problem, we decided to look to photos.

Photo Based Measurements

The motivation behind extracting measurements from photographs is simple: speed. Simply lay out a garment on a demarcated board and use our app to snap a photo with the overhead camera and voila! The app captures many of the standard points of measure and also some detailed profiles that humans wouldn’t dream of measuring on their own.

So how do we measure clothing with an image? The basic strategy is to first detect key-points of interest on the clothing item and then use known measurements from demarcations on the backdrop to infer distances between those points. In practice the basic steps go something like this:

1. Detect the bounding box (which has dimensions we have previously measured) and apply an affine transformation to morph and crop the image into a rectangular shape.

2. Apply edge detection over the clothing item and clean up the resultant output to gain a well-defined outline of the clothing item.

3. Calculate the outermost contour of the clothing with the marching squares algorithm.

4. Calculate the corners of the outside contour by following along a sequence of vectors tangent to the contour and finding the angles between them, defining key-points as those with large angle differences between successive vectors.

5. Identify which key-points correspond to which parts of the clothing by fitting a predefined clothing template using gradient descent and an expectation-maximization procedure.

6. Interpolate physical distance from the predefined bounding box measurements and calculate physical separation of the clothing key-points by either integrating distance along the outlining contour or simply calculating straight line distances between specific pairs.

Using this process we can – in a matter of seconds – readily determine things like inseam, waist, or even the thickness profile of the leg flare in a pair of pants with a simple overhead view from a common webcam.

Humans and Machines

Regular readers of this blog will know that a lot of what we do at Stitch Fix combines algorithms and human expertise to produce systems that perform better than the sum of their parts. We are still working out some of the details of their interaction in this case, but there are some promising overlaps. For example, if the algorithm shows an uncertainty in a particular measurement that is greater than some threshold, then the human user can be asked to (a) take the measurement themselves, or (b) lay markers on key points and retake the photo, or (c) move key points or lines around in the user interface.

This can help not only to get the right measurement for that particular point on that particular garment, but it can also be used to improve the algorithm via machine learning, so that over time the human-machine system can get more efficient.

Ouroboros

Data science has grown up in the tech industry, with many of the prime initial movers being companies that deal only with the virtual. Collecting data through customer interactions with websites and apps has turned out to be both cheap and, with the help of data science, tremendously useful. But as data science expands into other industries – Stitch Fix being an example of a crossover between tech and retail – the data of interest are now more often about the physical world instead of just the virtual, and capturing the data is not always as straightforward.

While this effort is still a work in progress, it’s becoming clear that our algorithmic efforts need to leverage automated processes to pull in data from the physical world as well as the digital. We expect that the example discussed here – adapting techniques of computer vision and machine learning to quickly measure clothing – will see a growing number of correlates in other fields, as data science techniques are used to make instruments and apps to collect more and different data. The discipline keeps evolving, driven not just by the deluge of existing data but also by its own extension of the variety of ways that one can address questions via data capture and analysis.