Diffbot is a powerful scraping tool that can be used to automatically parse a wide variety of web pages using computer vision. Developers are using the product to power reading services (Instapaper, Digg, Reverb, Onswipe, Longform), price comparison engines, media monitoring tools, and many other apps and systems.

When analyzing consumer products and platforms, I am eager to explore novel ways to find data that can help inform my decisions. This fall, I was looking at a handful of fashion resale marketplaces that had seen rapid growth. The specific marketplace we were analyzing provided us with lots of data, but we wanted a more thorough understanding of their competitors. I spun up Diffbot’s Crawlbot tool to quickly run crawls on the three main marketplaces and automatically extract pricing data using Diffbot’s Product API.

The first set of analysis helped us understand the relative size of each of the marketplaces and price of the products. The prices are important as it may inform the potential profile of customers and also the basket size from which the platform will earn a percentage. Four months later we ran the same analysis on two of the marketplaces to compare how they had evolved over time. Out of respect, I chose to anonymize the marketplaces to MPA, MPB, MPC.


Number of Items – Diffbot’s Crawlbot found 61,240 items on MPB with offer prices vs. 18,992 items on MPA[1].

List Price - Items on MPA have a noticeably higher list price than those on MPB. The mean list price on MPA is 277% that of MPB ($521 vs. $188), and 75% of the items on MPC are less than $120 vs. $388 on MPA. This may indicate that MPA is attracting higher quality items for resale.

Offer Price - MPA’s mean offer price is 219% that of MPB ($234 vs. $107). MPC sells significantly higher priced items, but discounts steeply, so the mean offer price ($275) is only 117% that of MPA but 257% that of MPB. 75% of items on MPB are less than $50, vs. $143 on MPA and $235 on MPC. MPB clearly skews significantly cheaper.

Average Discount – The distribution of percent discount is roughly equal for MPA and MPB. MPC, however, offers a much steeper discount from list price.

The figure below clearly indicates two important details: 1) MPB has significantly more products in their marketplace and 2) MPA has a much healthier long tail of higher priced products. The long tail MPA has captured will help increase AOV and may attract and retain higher value consumers.


  • In December MPA had ~19k items for sale vs. ~63k on MPB
  • In March MPA now has ~264k items for sale (14.6x growth) vs. ~278k on MPB (4.4x growth)
  • In December MPA had a much healthier long tail of higher priced items (the 75th percentile was $149 vs. $49 on MPB)
  • In March MPA has an almost equal number of products but has maintained a healthier long tail of higher priced items (the 75th percentile was $87 vs. $45) which will drive higher AOV and help attract higher value customers

Analyzing Consumer Marketplaces Using Diffbot’s Product API was published on (revised: ) Miles Grimshaw

A Personal Map of NYC

Our digital activities create a personal data exhaust that traces a unique map of the world.Here's my personal trace of NYC based on [GPS...… Continue reading

MindBody's Market Penetration

Published on March 17, 2015

Replacing Email With Slack

Published on March 08, 2015