KramerC

Deploying to S3 upon Git Push

October 23, 2013 at 4:00 PM PDT

With a simple post-receive hook and using s3cmd, you can have Git deploy to S3 after a pushing to your remote repository. If you're simply interested in the hook code, I have provided it at the bottom of this post.

Setting up s3cmd

To get started, you'll want to configure s3cmd on the user account that is holding the bare repository with your either security credentials of your AWS account or security credentials of an IAM user. I highly recommend creating a dedicated IAM user for s3cmd with an user policy that grants it full control to S3 and use its security credentials rather than giving it unlimited permissions by using your AWS account security credentials.

$ s3cmd --configure

You will be prompted for the access key and secret key:

Access key and Secret key are your identifiers for Amazon S3
Access Key: ACCESSKEY
Secret Key: SECRETKEY

Next, you'll be prompted for a GPG encryption key and the path to GPG that will be used when transferring files to S3. You can leave these blank to not use GPG when transferring.

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

Then, you'll be prompted if you want to use HTTPS when transferring files:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

If you said no to HTTPS, you will be able to provide a proxy. Leave the proxy name blank if you do not wish to provide a proxy.

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:
HTTP Proxy server port [0]:

You will then have a chance to review what you have provided and to test access with the supplied credentials.

New settings:
#!/bin/sh
  Access Key: ACCESSKEY
  Secret Key: SECRETKEY
  Encryption password:
  Path to GPG program: /usr/bin/gpg
#!/bin/sh
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]

If all goes well, you will be provided with the following:

Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Encryption will also be tested. Finally, you will be prompted whether to save the configuration.

Save settings? [y/N] Y
Configuration saved to '/home/git/.s3cfg'

Setting up the hook

Navigate into the working directory of your bare Git repository. Then, open up hooks/post-receive in your favorite text editor. Let's start with the following:

#!/bin/sh

S3_BUCKET=yourbucket
TEMP_DEPLOY_DIR=/tmp/$S3_BUCKET/

These are variables we will be working within the hook. You'll want to set S3_BUCKET to the actual name of your S3 bucket. Currently, we'll be writing to a directory named after the bucket name in /tmp/, however you can change this if necessary.

We will want to ensure the temporary directory is clean and any Git environment variables aren't going to conflict, so we'll add the following to the hook:

# Ensure that the temporary directory is clean and unset potential conflicting
# environment variables
rm -rf $TEMP_DEPLOY_DIR
unset GIT_DIR
unset GIT_WORK_TREE

Now we will want to set up populating the working tree. If you have no submodules in your repository, we will use the following:

# Create a working tree with a bare repo that does not have submodules
mkdir -p $TEMP_DEPLOY_DIR
export GIT_DIR=$(pwd)
export GIT_WORK_TREE=$TEMP_DEPLOY_DIR
git checkout -f
cd $TEMP_DEPLOY_DIR

If you do have submodules, dealing with them using the above method is problematic. I found the best solution is to make an entire clone of the repository in order to get the submodules to initialize and update properly:

# Create a working tree with a bare repo that has submodules
git clone $(pwd) $TEMP_DEPLOY_DIR
cd $TEMP_DEPLOY_DIR
git submodule update --init --recursive

Then, we can now sync the repository with S3:

# Sync with S3
s3cmd sync --delete-removed --acl-public --exclude '.git/*' ./ s3://$S3_BUCKET/

If anything should be preprocessed before syncing with S3, say a Jekyll site, we can build the site and sync only the _site directory:

# Build and sync
jekyll build
s3cmd sync --delete-removed --acl-public --exclude '.git/*' _site/ s3://$S3_BUCKET/

You will want to ensure anything ran from the hook is set up on the remote server, otherwise it will fail.

Finally, we clean up the temporary directory we were using to sync with S3.

# Clean up
cd ..
rm -rf $TEMP_DEPLOY_DIR

That's all there is to it. Git will now deploy to your S3 bucket each time you push to your remote repository.

Example post-receive hook

Here is the complete post-receive hook code.

#!/bin/sh
# post-receive hook that syncs with S3 upon a push

S3_BUCKET=yourbucket
TEMP_DEPLOY_DIR=/tmp/$S3_BUCKET/

# Ensure that the temporary directory is clean and unset potential conflicting
# environment variables
rm -rf $TEMP_DEPLOY_DIR
unset GIT_DIR
unset GIT_WORK_TREE

# Create a working tree with a bare repo that does not have submodules
mkdir -p $TEMP_DEPLOY_DIR
export GIT_DIR=$(pwd)
export GIT_WORK_TREE=$TEMP_DEPLOY_DIR
git checkout -f
cd $TEMP_DEPLOY_DIR

# If the repo has submodules, comment out ore remove the above and uncomment the below:
#
# git clone $(pwd) $TEMP_DEPLOY_DIR
# cd $TEMP_DEPLOY_DIR
# git submodule update --init --recursive

# Sync with S3
s3cmd sync --delete-removed --acl-public --exclude '.git/*' ./ s3://$S3_BUCKET/

# If you use Jekyll, comment out or remove the above line and uncomment the below:
#
# jekyll build
# s3cmd sync --delete-removed --acl-public --exclude '.git/*' _site/ s3://$S3_BUCKET/

RMagick on Windows

March 18, 2013 at 8:51 AM PDT

RMagick on Windows is tricky. Recently, I wrote up an answer on Stack Overflow on ways to get it to work under Windows with Rails. Unfortunately, my answer hasn’t received much attention.

I have managed to find two solutions to install RMagick and getting it to work with Rails back when I was on Windows. These solutions aren’t specific to Rails.

The Simple Solution

The easiest solution is to install the ancient rmagick-2.12.0-x86-mswin32 gem. To get this specific gem version to work with Bundler, you will need to add the following to your Gemfile.

if RUBY_PLATFORM =~ /(win|w)32$/
  gem 'rmagick', '2.12.0', :path => 'vendor/gems/rmagick-2.12.0-x86-mswin32', :require => 'RMagick'
else
  gem 'rmagick', :require => 'RMagick'
end

Note the path argument there. You will need to place the gem in a location where Bundler can find it. This example uses vendor/gems/ for its location. You will need to unpack the .gem file to this location.

gem unpack rmagick-2.12.0-x86-mswin32 vendor/gems/

A Better Solution

The provided Windows gem is heavily outdated and it is intended for Ruby 1.8.6, meaning there’s no guarantee that it will work with future Ruby versions. It is possible to compile a newer version of the RMagick gem on Windows using DevKit. You will need a 32-bit version of ImageMagick installed with development headers.

I have created a batch file that maps the directory of ImageMagick to X:\ and it gives the parameters to RubyGems on where to find the required files to build RMagick. This sort of mapping is necessary as the configuration options don’t know how to handle spaces in the paths. Alternatively, you can install ImageMagick to a location that has no spaces in its path and avoid binding it to a drive letter altogether.

The following commands map the directory of ImageMagick to X:\ and have RubyGems compile and install RMagick.

subst X: "C:\Program Files (x86)\ImageMagick-6.7.6-Q16"
gem install rmagick --platform=ruby -- --with-opt-lib="X:\lib" --with-opt-include="X:\include"
subst X: /D

The path in this example will need to be modified if you have a version other than 6.7.6-Q16 installed or if you are not on 64-bit Windows.

If you are using Bundler, a much nicer one liner in Gemfile is all that is needed with this solution.

gem 'rmagick', :require => 'RMagick'

Converting DateTime.Ticks into a Unix timestamp in PHP

November 30, 2010 at 11:46 AM PST

I needed a way to convert the number of ticks from the DateTime.Ticks property in .NET into a Unix timestamp in PHP. So I wrote up this solution that makes this possible. The solution works by taking the number of ticks that I want to convert into a Unix timestamp and subtracting it with the number of ticks of the Unix epoch.

To figure out the number of ticks of the Unix epoch, I used C# (it is possible to do this in a different .NET language) using these two lines of code to find the number of ticks of the Unix epoch in a simple Console Application:

DateTime unix = new DateTime(1970, 1, 1, 0, 0, 0);
System.Console.WriteLine(unix.Ticks);

This ends up printing out: 621355968000000000

I then took this number, which is in one hundred nanoseconds, and created a PHP function which subtracts the number of ticks to be converted by the number of ticks of the Unix epoch and then divides it by 10000000 to convert the now Unix timestamp from one hundred nanoseconds to seconds as Unix timestamps in PHP are handled in seconds.

<?php
function ticks_to_time($ticks) {
    return floor(($ticks - 621355968000000000) / 10000000);
}
?>

This is an example of this function in use:

<?php
// 11/24/2010 9:20:45 PM UTC in DateTime.Ticks
$time = ticks_to_time(634262304450000000);
echo "$time\n";
echo date("F j Y g:i:s A T", $time) . "\n";
?>

Which should output:

1290633645
November 24 2010 1:20:45 PM PST

So there we have it, a function that makes it easy to convert the number of ticks from the DateTime.Ticks property in .NET into a Unix timestamp in PHP.

How to Revert Your SVN Repository on Assembla

July 24, 2010 at 11:36 AM PDT

There is no simple way to revert your repository to a previous revision on Assembla. However, a reversion is possible by following these steps.

First, export the SVN repository in your space. This can be done under the Import/Export section of your repository. It’ll take a minute for the dump to be created. Once that has completed, download the repository dump.

Then, extract the contents of the ZIP file to a temporary directory, then run the following commands in the temporary directory:

svnadmin create REPO_NAME
svnadmin load REPO_NAME &lt; rXX.dump
svnadmin dump -r 1:YY REPO_NAME --incremental &gt; rYY.dump
gzip rYY.dump

Replace XX with the current revision of your repository, YY with the revision you wish to revert to, and REPO_NAME with any name, such as your repository’s name. This name will not be carried over later.

Afterwards, delete the repository tool on Assembla by going to Admin -> Tools, then clicking Delete next to it which is located on the right. Now re-add the Source/SVN repository tool.

Finally, import the dump to the newly created repository by going to the Import/Export section and uploading the gzipped SVN dump. The process will take from a few to several minutes depending on how large your repository is.

Your repository should now be reverted back to the revision you specified.