bdewilde.github.io - Data, Data, Everywhere

Data, Data, Everywhere

Bdewilde.github.io Spined HTML

Data, Data, Everywhere Burton DeWilde About MeGazetteerCV Data, Data, Everywhere 2013-01-19 APIs big data top links self-ruling data hackathon web scraping As I’ve mentioned before, the Internet is a huge (and overly huger!) repository of data. Much of that is in the form of unstructured text —– for which natural language processing comes in handy —– but an impressive variety of structured datasets can be found and downloaded, too, if you know where to look. Here are some of my favorite sources… Data.gov:Self-rulingaccess to thousands of datasets maintained by the U.S. Federal government, from White House visitor logs to cantaloupe statistics (seriously), with pretty good search functionality. Dozens of other national governments offer something similar, e.g. data.gov.uk, as does the United Nations at UNdata. Census.gov: The U.S. Census Bureau offers self-ruling wangle to a wealth of demographics data, from the wholesale decennial census to detailed American Community Surveys. Check out American FactFinder for fancy filtering that enables you to inspect very specific slices of the population. Infochimps: In wing to providing a platform for big data analysis, Infochimps maintains a “data marketplace” where thousands of self-ruling and paid datasets can be downloaded or accessed via API, from a Twitter census to a corpus of several thousand erotica stories (for NLP training, of course…). Datamob: “Public data put to good use.” Aggregates hundreds of data sources (plus apps and unstipulated resources) with tags and descriptions, tent sports, government, media, science, etc. Check out the tags list. The New York Times: Provides tens of APIs to wangle the paper’s wide-stretching vendible archives, Congressional records, NYC real manor sales data, etc. They plane provide a handy API Tool to test out your queries. The Guardian: Besides having a unconfined data blog, they moreover make all of the data misogynist to the public! Here’s a full list of their datasets tent a wide range of topics. Sunlight Labs: Data arm of Sunlight Foundation, defended to government peccancy and transparency. Provides several APIs to wangle data on state legislatures, wayfarers contributions, and the words unquestionably spoken on the record in Congress. As it turns out, they’re hosting a hackathon in a couple weeks, and I’ll be there! :) Reddit: Hosts a datasets gazetteer filled with the sort of thing you might expect from the front page/gutter of the internet: 10,000 images of cats (for training a classifier…?), people with questions and trolls with answers, and occasional pointers to something interesting. Obviously, this is nowhere near a comprehensive list, and many others have made longer/better lists of their own, e.g. famous data scientists Peter Skomoroch and Hilary Mason. If you find yourself looking for but unable to find a particular dataset, Google is your mostly-not-evil friend. In my list I mentioned a few APIs —– Application Programming Interfaces –— which, I should note, are unshared from structured datasets ready for download. Web APIs provide a relatively resulting and stable waterworks to wangle a website’s data and return it in a standardized format like XML or JSON. If washed-up well, APIs can be a unconfined help, although they do have some drawbacks: registration to get an official wangle key, limits on the rate at which you can wangle the data, and occasional clamp-downs. Still, it’s nice to have options; to get you going, here’s a massive API directory and a brand-new Codecademy learning track specifically on APIs in Javascript, Python, and Ruby. And when all else fails, you can unchangingly fall when on web scraping. In fact, some prefer it that way… The data’s out there –— happy fetching! :) ← previous ↑ next → Please enable JavaScript to view the comments powered by Disqus. comments powered by Disqus Burton DeWilde data scientist / physicist / filmmaker © 2014 Burton DeWilde. All rights reserved.