Rediscovering Old Music: C89.5 in 2010 and 2011

I used to listen to C89.5 quite a bit in 2010 and 2011. I even attended C89.5’s Listener Appreciation Party 5 back on August 14, 2010. I discovered a ton of music, to the point where it was difficult keeping track of all it. I remember trying to memorize names of songs that came on while I was driving so I could note them later, only to have trouble recalling it afterward. I found it easier to keep track of the time the song had played and then resort to the playlist on C89.5’s website. However, C89.5 only kept the past 5 or so songs that played.Eventually the entire current day was kept once they redid their website, however that wasn’t until near the end of 2011.1 That meant there was a short amount of time I could look it up before it was too late.

I managed to find a site that was scraping the playlist on C89.5’s website and it kept a history. However, if I recall correctly, the site had problems collecting data at times and going back through pages sometimes didn’t work. I decided to create my own scraper and host publicly history of what C89.5 played. I collected data from the first track being on 2010-07-28 18:55:33 to the last track being on 2011-12-06 10:13:29. These timestamps are in Pacific Time.

I would have kept this scraper running, but on December 6, 2011, around when they redid their website, I found out my scraper was no longer collecting data because it appeared that C89.5 had blocked the server my scraper was on from accessing their website. After contacting them, it unfortunately was the case that they wanted me to cease scraping their site. Respecting their wishes, I dismantled the scraper and eventually I forgot about it. While they do now have a calendar at the bottom of their playlist page to go through historic data, they did not have this when this scraper existed.

Recently, I have been wanting to rediscover the music that I listened in the past. It finally hit me that I still have this data and I could aggregate it in a way where I can find the total amount of plays in this date range. Chances are I’ve heard the song if it was played often enough, and I could find songs that I missed in the past that weren’t played so frequently. It was all in a MySQL table so I was able to easily aggregate it all in such a manner.

I have exported the data as I feel this may be useful to others that may be trying to rediscover music during this time period. I hope you enjoy rediscovering all this music as much as I did. 🙂

The Data

I have compiled the total plays in a Google Sheet.
It also contains the raw data, which is just the SQL table with formatted timestamps.

The CSVs and SQL dumps are available in this folder on Google Drive.

Continue reading “Rediscovering Old Music: C89.5 in 2010 and 2011”

Deploying to S3 upon Git Push

With a simple post-receive hook and using s3cmd, you can have Git deploy to S3 after a pushing to your remote repository. If you’re simply interested in the hook code, I have provided it at the bottom of this post.

Setting up s3cmd

To get started, you’ll want to configure s3cmd on the user account that is holding the bare repository with your either security credentials of your AWS account or security credentials of an IAM user. I highly recommend creating a dedicated IAM user for s3cmd with an user policy that grants it full control to S3 and use its security credentials rather than giving it unlimited permissions by using your AWS account security credentials.

$ s3cmd --configure

You will be prompted for the access key and secret key:

Access key and Secret key are your identifiers for Amazon S3
Access Key: ACCESSKEY
Secret Key: SECRETKEY

Next, you’ll be prompted for a GPG encryption key and the path to GPG that will be used when transferring files to S3. You can leave these blank to not use GPG when transferring.

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

Then, you’ll be prompted if you want to use HTTPS when transferring files:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

If you said no to HTTPS, you will be able to provide a proxy. Leave the proxy name blank if you do not wish to provide a proxy.

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:
HTTP Proxy server port [0]:

You will then have a chance to review what you have provided and to test access with the supplied credentials.

New settings:
  Access Key: ACCESSKEY
  Secret Key: SECRETKEY
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]

If all goes well, you will be provided with the following:

Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Encryption will also be tested. Finally, you will be prompted whether to save the configuration.

Save settings? [y/N] Y
Configuration saved to '/home/git/.s3cfg'

Setting up the hook

Navigate into the working directory of your bare Git repository. Then, open up hooks/post-receive in your favorite text editor. Let’s start with the following:

#!/bin/sh

S3_BUCKET=yourbucket
TEMP_DEPLOY_DIR=/tmp/$S3_BUCKET/

These are variables we will be working within the hook. You’ll want to set S3_BUCKET to the actual name of your S3 bucket. Currently, we’ll be writing to a directory named after the bucket name in /tmp/, however you can change this if necessary.

We will want to ensure the temporary directory is clean and any Git environment variables aren’t going to conflict, so we’ll add the following to the hook:

# Ensure that the temporary directory is clean and unset potential conflicting
# environment variables
rm -rf $TEMP_DEPLOY_DIR
unset GIT_DIR
unset GIT_WORK_TREE

Now we will want to set up populating the working tree. If you have no submodules in your repository, we will use the following:

# Create a working tree with a bare repo that does not have submodules
mkdir -p $TEMP_DEPLOY_DIR
export GIT_DIR=$(pwd)
export GIT_WORK_TREE=$TEMP_DEPLOY_DIR
git checkout -f
cd $TEMP_DEPLOY_DIR

If you do have submodules, dealing with them using the above method is problematic. I found the best solution is to make an entire clone of the repository in order to get the submodules to initialize and update properly:

# Create a working tree with a bare repo that has submodules
git clone $(pwd) $TEMP_DEPLOY_DIR
cd $TEMP_DEPLOY_DIR
git submodule update --init --recursive

Then, we can now sync the repository with S3:

# Sync with S3
s3cmd sync --delete-removed --acl-public --exclude '.git/*' ./ s3://$S3_BUCKET/

If anything should be preprocessed before syncing with S3, say a Jekyll site, we can build the site and sync only the _site directory:

# Build and sync
jekyll build
s3cmd sync --delete-removed --acl-public --exclude '.git/*' _site/ s3://$S3_BUCKET/

You will want to ensure anything ran from the hook is set up on the remote server, otherwise it will fail.

Finally, we clean up the temporary directory we were using to sync with S3.

# Clean up
cd ..
rm -rf $TEMP_DEPLOY_DIR

That’s all there is to it. Git will now deploy to your S3 bucket each time you push to your remote repository.

Example post-receive hook

Here is the complete post-receive hook code.

#!/bin/sh
# post-receive hook that syncs with S3 upon a push

S3_BUCKET=yourbucket
TEMP_DEPLOY_DIR=/tmp/$S3_BUCKET/

# Ensure that the temporary directory is clean and unset potential conflicting
# environment variables
rm -rf $TEMP_DEPLOY_DIR
unset GIT_DIR
unset GIT_WORK_TREE

# Create a working tree with a bare repo that does not have submodules
mkdir -p $TEMP_DEPLOY_DIR
export GIT_DIR=$(pwd)
export GIT_WORK_TREE=$TEMP_DEPLOY_DIR
git checkout -f
cd $TEMP_DEPLOY_DIR

# If the repo has submodules, comment out ore remove the above and uncomment the below:
#
# git clone $(pwd) $TEMP_DEPLOY_DIR
# cd $TEMP_DEPLOY_DIR
# git submodule update --init --recursive

# Sync with S3
s3cmd sync --delete-removed --acl-public --exclude '.git/*' ./ s3://$S3_BUCKET/

# If you use Jekyll, comment out or remove the above line and uncomment the below:
#
# jekyll build
# s3cmd sync --delete-removed --acl-public --exclude '.git/*' _site/ s3://$S3_BUCKET/

RMagick on Windows

RMagick on Windows is tricky. Recently, I wrote up an answer on Stack Overflow on ways to get it to work under Windows with Rails. Unfortunately, my answer hasn’t received much attention.

I have managed to find two solutions to install RMagick and getting it to work with Rails back when I was on Windows. These solutions aren’t specific to Rails.

The Simple Solution

The easiest solution is to install the ancient rmagick-2.12.0-x86-mswin32 gem. To get this specific gem version to work with Bundler, you will need to add the following to your Gemfile.

if RUBY_PLATFORM =~ /(win|w)32$/
  gem 'rmagick', '2.12.0', :path => 'vendor/gems/rmagick-2.12.0-x86-mswin32', :require => 'RMagick'
else
  gem 'rmagick', :require => 'RMagick'
end

Note the path argument there. You will need to place the gem in a location where Bundler can find it. This example uses vendor/gems/ for its location. You will need to unpack the .gem file to this location.

gem unpack rmagick-2.12.0-x86-mswin32 vendor/gems/

A Better Solution

The provided Windows gem is heavily outdated and it is intended for Ruby 1.8.6, meaning there’s no guarantee that it will work with future Ruby versions. It is possible to compile a newer version of the RMagick gem on Windows using DevKit. You will need a 32-bit version of ImageMagick installed with development headers.

I have created a batch file that maps the directory of ImageMagick to X:\ and it gives the parameters to RubyGems on where to find the required files to build RMagick. This sort of mapping is necessary as the configuration options don’t know how to handle spaces in the paths. Alternatively, you can install ImageMagick to a location that has no spaces in its path and avoid binding it to a drive letter altogether.

The following commands map the directory of ImageMagick to X:\ and have RubyGems compile and install RMagick.

subst X: "C:\Program Files (x86)\ImageMagick-6.7.6-Q16"
gem install rmagick --platform=ruby -- --with-opt-lib="X:\lib" --with-opt-include="X:\include"
subst X: /D

The path in this example will need to be modified if you have a version other than 6.7.6-Q16 installed or if you are not on 64-bit Windows.

If you are using Bundler, a much nicer one liner in Gemfile is all that is needed with this solution.

gem 'rmagick', :require => 'RMagick'