Fork me on GitHub

Day Scale

January 1, 2012

tags: scraping day-scale

First conceived in December 2010, dayscale was a data aggregation and analysis website. The server is written in Java for the Google App Engine platform. The system collects weather, stock, and sport (MLB/NBA) information throughout the day and runs a faux competition between imaginary characters based on the collected data.

DOM traversal, statistical analysis, and data summarization are a few of the cornerstones of this project.

For several months the scrapers were running smoothly with little or no upkeep. Now it has fallen into disrepair though historical data is still available.

A typical day of the app (pacific time):

The Day Before:

Scraping the Day:

Finalizing the Day:

You can see the code I used for the scrapers. There is also an old wiki with more info. And some blog commentary.

Discussion, links, and tweets

a small avatar image of Tom

I am a software developer in Los Angeles.