DEV Community

Discussion on: Help gathering data

Collapse
 
theaccordance profile image
Joe Mainwaring

I've had ideas wither away because they lacked access to the data sources required to drive their purpose. It's frustrating at times, even more so when you know the datasets exist.

PageScraping might work, but I do wonder how complex that might get. It's one thing to scrape a product catalog for a vendor, and another to parse independent websites for possibilities of data.

A couple alternative approaches:

  • Outsource it to an offshore resource to compile
  • Create a means for community contributions, and run a campaign to get an initial dataset populated by reaching out directly to the orgs and asking them to fill out a survey or something to capture the data