mikegriffin.ie

Octokit and SSL

2017-09-29T11:54:00+01:00

I do a lot of work using self signed certificates and sometimes I want to use octokit.rb. By default, octokit tries to verify the SSL cert and fails with an error like:

SSL_connect returned=1 errno=0 state=error: certificate verify failed

To get past this, you can disable this setting by adding:

client.connection_options[:ssl] = { :verify => false }

I’m adding this here because I keep having to look it up and it takes me a while to find it each time.

Speaking at the Waterford Tech Meetup

2017-02-03T22:50:00+00:00

I was asked to speak at the January Waterford Tech Meetup on January 25th at Boxworks. I spoke about a project that I’ve been working on for quite some time for my Dad. He takes a lot of photographs and wanted to update the site he uses to show odd his photos, it was running an old version of ZenPhoto. Now it’s a static site hosted on GitHub Pages and using AWS S3 to store the photos.

I talked about how to use Dropbox and webhooks to automatically update the GitHub Pages site and upload the photos to S3. At the moment it’s not very sophisticated, it lacks a lot of checks and tests to ensure that things aren’t overwritten, but as it’s backed by a Git repository, I figure that I’ll always be able to manually fix things.

My Dad’s site is https://www.photosofblacklion.net. Go check it out and take a look at the relevant repositories at https://github.com/PhotosOfBlacklion.

Pull Requests and questions welcome!

Here’s a long form version of the talk I gave too.

This project started out because my Dad likes to takes photos and wanted to share photographs of our local village in Cavan, including any activities that were happening around the place. So I set up a PHP based gallery called ZenPhoto. This was running on PHP4 on a Dreamhost box. I upgraded it a few times over the years, but each time something would change and there would be some weirdness about not being able to log in for a while. Or else my Dad would lose the password from the browser auto fill. Anything out of the ordinary would throw him off, so I gave up upgrading it and it’s been running OK for a few years. It wasn’t fast due to being hosted in the US, and most visitors being in Ireland. and it wasn’t particularly pretty (extra clicks to see the full size photo, clunky paginated interface to view 700 photos in an album), but he kept putting photos up and people kept looking at them. Then as more people emigrated, there were more international hits, Australia, Canada, the UK.

He started asking was there anything better available and of course there was. I got thinking about it and my criteria were:

Cheap
Fast
Easy to change the layout
As simple as possible to upload photos

After lots of messing around in the name of research, and some failed experiments, I decided on:

AWS S3 to hold the photos (should be pretty cheap, can make use of a CDN easily)
GitHub Pages to host the HTML (free, I know it pretty well)
Dropbox to upload photos so they could be processed (don’t have to write a multi file resumable file upload app)
DigitalOcean to host the Sinatra server and process images

So that’s the list of things. How do we go from that to an online gallery that is available for anyone to look at with a minimum of visible fuss.

First, a little diversion. Webhooks. I see that there three ways to know if something has changed somewhere:

Stand looking at it until it changes. This doesn’t work all that well, think of watching a kettle boil but you’re also waiting for it to be switched on. You’ll either need a lot of patience or some way of seeing into the future.
Check periodically to see if anything has changed. This is the equivalent of someone coming up behind you every five minutes, tapping you on the shoulder and asking, “Are you finished yet?”. It might be tolerable if one person does it, but it gets annoying quickly if multiple people do it even if it’s a different times. This is called polling and is the simplest way to set up most CI systems. Use a cron job, or schedule it in the CI system itself.
Get the application to tell you. Like Archimedes jumping up and shouting Eureka when he stepped into the bath, webhooks notify any subscribers when something has happened. The subscriber can be as simple as an HTTP server waiting to be be hit or a full blown CI system.

To get started, I used a Sinatra app on a DigitalOcean Droplet.This listens for a POST request with the correct parameters, does some parsing to check that the request comes from Dropbox, and stores any new photo paths into a MySQL database.

Just like it’s important to respond to a regular webpage request as quick as possible, you have to respond to the fired webhook in as short space of time to acknowledge it. For Dropbox, this is 10 seconds. You can do a lot in ten seconds but it’s recommended that any heavy lifting be done in an asynchronous process.

My first thought to get past this was a cron job to poll the database…but there’s polling again, which I want to avoid.

Secondly, I could have used a job queue instead of the database and a worker like resque to process the queue. But this is essentially polling again, just at a much faster rate.

I decided to go with forking out to another script, which first checks if there’s another instance running and if so, exits.

Because this webhook is also fired when files are removed, it checks if there are any new photos to process before looping over each one to do the work.

Then it runs through these steps:

download the file from dropbox
delete it from dropbox
create a thumbnail
copy the photo and thumbnail to S3
delete the local files
add the S3 paths to an array
and repeat for each photo

At the end of the process, it creates a markdown file in a predefined repository directory. The photograph information is stored in the YAML front matter of a GitHub Pages post file. This is parsed by Jekyll and ends up with a gallery page showing all of the photos.

To get the site to build, the new file has to be committed and pushed to GitHub. These steps are done using the libgit2/rugged gem, before the script exits. And that’s it.

This code is messy, has minimal error checking and to be honest a bit of a hack job, but it works and that’s the important bit at the end of the day. There is logging in place to trace back anything that happened and at worst, it’s a Git repository which can be rolled back to a known good state.

Other things that can be added to it are a webhook on the GitHub side that fires on a page build success to send an email or other notification saying that the new gallery is available.

There was also a large data migration project involved to move over 8 years of photographs, 2129 directories and 125702 files to give 63GB of photos. This involved bash scripting, some ruby, some judicious MySQL and a lot of late nights.

To compare the costs, Zenphoto on Dreamhost was costing me $120 a year in hosting with free storage. The new system is costing:

$5 a month for a digitalocean droplet (that I use for my own website anyway)
$0 for GitHub Pages
$0 for Dropbox because the files get removed after they are copied down
$1.48 + tax for S3, so round it up to $2
for a grand total of $7 a month, or $84 dollars a year, a saving of $36!

Autocomplete SSH Hosts

2016-10-12T22:22:00+01:00

Once upon a long time ago, I had autocomplete set up in ZSH to complete hosts that I connected to through SSH. Only in the last few days have I set it up again in bash.

I was reading through Luciano Fiandesio’s blog on how to configure Mutt and I saw that he has shared his dotfiles. I took a look at his bash configuration and saw this line:

[ -e "$HOME/.ssh/config" ] && complete -o "default" -o "nospace" -W "$(grep "^Host" ~/.ssh/config | grep -v "[?*]" | cut -d " " -f2 | tr ' ' '\n')" scp sftp ssh

complete is a bash builtin function, which you can read more about in this TheGeekStuff article. This line will add the hosts in your ~/.ssh/config file to the autocomplete list for scp, sftp, and ssh.

Pretty handy!

Drupalcon Dublin

2016-10-04T21:16:00+01:00

Drupalcon Dublin took place last week and I went to the sessions on Tuesday, Wednesday and Thursday (27-29 September). Here are some of my thoughts on the conference and the talks that I went to.

Tuesday

Ride the Whale! Docker for Drupalists

Really enjoyed this as an overview of how to use Docker. Even though it targetted Drupal, there are lots of bits that can be used elsewhere. The slides are available too

Weather.com (The Weather Company) as an Inspiration to Extend TMGMT

Disappointed with this talk as it wasn’t what I expected. It turned into a sales pitch for their product.

Configuration Management: theory and practice

Drupal specific, but a very good overview of the challenges of creating development and production environments that mirror each other but still allow small differences. For example, having verbose logging enabled on a development site.

Leading Healthy Teams

A good reminder on not burning out. He wasn’t a huge fan of remote teams though, advised meeting up on site at least once a week.

Wednesday

A Cautionary Tale for Defensive Programmers

Lots of tips on returning early and using exceptions. How to handle and bubble up exceptions in your application.

Building a co-lingual website - lessons learned from ireland.ie

An enjoyable look at how difficult it is to get content in different languages to match up and how hard it is to get it translated. Reminded me a lot of the work that went in to http://www.museum.ie too.

HTTP for Web Developers

I’ve been looking forward to seeing Lorna talk in person for a long time. This was a very approachable introduction to HTTP and APIs. Also included a run through some HTTP clients and how Lorna tests things. I also picked up a signed copy of “PHP Web Services” before the talk.

Imposter Syndrome and Individual Competence

Beautiful slides and a good talk. Everyone in the room had experienced imposter syndrome in some form or another.

Thursday

Guzzle: Extraordinary HTTP Client

Rob is one of the developers of Guzzle and the Slim framework and showed some really useful ways that Guzzle wraps around curl and makes even multiple curl calls easy.

Search and beyond with Elasticsearch

How to use Elasticsearch to make searching a Drupal site better, faster and more useful. Drupal specific again but with information on Elasticsearch that can be used everywhere. Also showed some sites that use it and one that doesn’t. The room was packed for this one, lots of interest.

Summary

It was hard to go to a conference not knowing anyone beforehand, I’m not one for going up to someone and saying hi. Despite this, I found some people to talk to and had good chats outside of the talks themselves.

I was surprised that there were so many Drupal specific developers and companies. I expected there to be mostly PHP developers that used Drupal, but it was the other way round. I had used Drupal a few years ago and it’s great to see that it’s come on a long way since then. The automations possible now would have made life a lot easier before.

Saying that, the tracks that I enjoyed the most were the PHP and human specific ones. I guess they were easier for me to drop in to despite not having done any PHP in the last two years.

I’m not sure I’d go to another Drupal specific conference, but there’s enough going on in PHP to drag me back and keep me interested. It’s not at all the same as it was back in PHP3 and 4!

I got rid of a lot of my stash of stickers too, if you missed out on them or want any, send me an email and I’ll see what I can do.

Using Pipe With Ruby Scripts

2016-05-25T11:50:00+01:00

I’ve been writing some Ruby scripts recently to parse some text files and I was looking for a way to pass in only part of a file instead of the full 1500 lines. If I was bash scripting, I could use head to extract part of the text file and then pipe it in to my script. But that wasn’t going to work in this case because I was passing the filename as a command line argument like this:

input = File.open(ARGV[0])

A quick search and I found this blog post describing exactly what I wanted to achieve.

Instead of using File.open(ARGV[0]), I could use ARGF.read and it would work in both scenarios. Now I can run the Ruby script in two ways:

./script.rb filename

head filename | ./script.rb

Very handy.

Don't Be That Guy

2015-11-02T16:00:00+00:00

I’ve played canoe polo for a long time, longer than I care to admit to myself. Not long after I started playing in the highest national division, I came up against a guy that had once played at the highest level. The only problem was that he hadn’t played recently and his brain was moving faster than his body. To compensate, or maybe because he was that little bit slower to react, he played dirty, cynical things like obstructing and chopping. I used to hate playing against him because you finished up the game with cut fingers and sore arms.

He used to give out to referees that he thought were picking on him too. He just couldn’t accept that he was doing something wrong and thought everyone was out to get him because he used to be good.

Over the weekend, I realised that I’m in danger of turning into that guy. - I once played at the highest level. - I haven’t trained ot played consistently in a few years - I give out to referees when things don’t go my way

This is a note to myself so that I realise that I can still turn this around. Enjoy the games you get to play, they won’t always be there. Don’t get upset or angry if you’re beaten in what you think is an unfair way, it’s probably that you’ve slowed down more than you realise. But most of all…

“Don’t be that guy…”

Moving From LastPass to 1Password

2015-10-19T21:23:00+01:00

Last weekend I moved my passwords from LastPass to 1Password. This was partly due to the notice that LogMeIn bought LastPass and partly to test out if it was possible.

There were some manual steps involved and some caveats, which I’ll document here.

Export your passwords

The first thing to do is to export everything from LastPass to a CSV file. If you use the browser plugin, you can do this by going to:

Tools -> Advanced Tools -> Export To -> LastPass CSV File

You’ll be prompted for your password and then a location to save the file. I chose to save it to the Desktop.

Modify the CSV file

I removed some old passwords that I don’t need any more before fixing up and reordering the columns. The original headers are:

url,username,password,extra,name,grouping,fav

but the 1Password headers are:

title,location,username,password,notes

I moved them around so that url became location and name became title. I also deleted the extra, grouping and fav columns.

I removed any Secure Notes and saved them into another file for later.

Import into 1Password

Importing was easy. Open 1Password, got to File -> Import, select the file and hey presto, all done. A quick check to make sure everything was there and I was done.

Caveats

Make sure to remove the header line from the CSV file before you import or else you will be left with a new entry with the headers in it.
Don’t try to import Secure Notes, it just made some more Logins for me.

Moving Email to Fastmail

2015-04-11T11:15:00+01:00

I’ve hosted my own email for the past few years, going through setting it up, moving the config to puppet, having it crash and burn, recreating it and reimporting all the old mails. I don’t have the time to do this any more so a short while ago I moved it all, lock stock and barrel to Fastmail.

I could have moved it back to Gmail, but I wanted to try out something different and I’d heard a lot of good things about Fastmail.

They have a nice easy method to import mails from an IMAP account as well as from some of the large common email hosters. After a few hours, everything was imported and I had to change over the DNS entries. This was the worst bit. What would happen if I did something wrong here? Would I even know that mail was falling through the cracks?

After a few days and seeing mails still being delivered, I called it a success. I set up the new account details on my phone and laptop and I was all moved.

Overall, it was a simple process with very little that could go wrong once the excellent instructions were followed. I plumped up for the personal enhanced package for $40 per year so that I could use a personal domain and get 15GB storage.

If you feel like giving it a go, try it out using this referral link

LVAC XC Club Championships

2015-02-15T00:00:00+00:00

This is just a short report on how the race went. The weather was as good as we could expect, with a bit of a headwind down the final straight.

There was a new course for the LVAC XC Club Championships this year. We were spared some of the roots and a large hill, but we had to cross the Munichs instead.

Peter won the race, Damien was second and Chris was third. I finished in fifth, behind Bren and just ahead of Gavin, who almost caught me on the line.

The race went out as expected and I tried to hold on to the lead bunch for as long as I could. I dropped off about half way through the first lap and shortly after Chris went past me. I tried to stay with him, but he dropped me after a few hundred metres. Shortly after that, I caught Gavin and went past him as strong as I could. I didn’t quite drop him as much as I wanted to though.

Bren came through on the third lap and passed me. I stayed with him until the end of the Munichs but then totally died. I felt like I was walking at this stage and had to dig really deep to keep going.

I wasn’t strong enough on the day, but I hadn’t slept well in the week leading up to it, Éabha was teething and keeping us all awake. This is no excuse, but more of a reason, if that makes sense to anyone else!

Rock and Roll New Orleans 10k

2015-01-25T00:00:00+00:00

I went to New Orleans for a week to meet up with work mates and found out a week before that there was a 10k taking place on the day after I arrived. Without thinking too much about it, I entered, thinking it would be a nice way to see the city without having to worry about getting lost. I didn’t have a lot of time to spare to collect my number after arriving in New Orleans, but I missed the connecting flight in Newark and arrived too late to register. I didn’t think there would be any chanve to register on the day as the race was starting at 7am, so I went out for a few drinks with the guys from work. Sometime after midnight, I got a message saying I could register before the race, not what you want to hear with about 6 hours to go before you have to get up.

Just before 6am, I got up and went to register, I had paid for it and I was awake, it seemed like the right thing to do. Registration was quick and easy and by 6:30, I was off warming up after handing over my spare clothes for the end of the race. After some long strides and a few chats to other runners, I went to the start area. As usual, I spent some time looking around, checking out the other runners, sussing out the competition. I was a little surprised, it looked like there were only a small number of “proper” runners.

Just before the race, the national anthem was sung and then we were off. After about 200m, I was in the leading bunch of three, but 100m later, I was talking to the only other person still in the group. It turned out she was a runner from Manchester who had moved to Colorado to work as a physiotherapist. She was looking to run around 38 minutes, I said I was hopeing for somewhere between 35 and 36.

I dropped her after another couple of minutes and from then on I was on my own. The roads were closed, there were two police motorbikes, two lead cars and a cyclist keeping me company but it was a surreal experience being the only runner on large city streets. I don’t think the bands were expecting someone to come along so soon, most of the time they only started to play after I had gone past them.

The finish was another surreal experience, I didn’t know how to celebrate going over the line and breaking the tape. I was presented with my finisher medal, some water, gatorade, chocolate milk, banana and recovery bar before realising I should go back and cheer in the other runners. The girl in second place finished 3 minutes after me, with the second man another 3 minutes back.

My finish time was 36 minutes 22 seconds, after going through half way in just over 18 minutes. I was pretty happy overall, coming to a large city and winning a road race was something I never thought I would do. It would have been nice to come in under 36 minutes, I know I had it in me, but seeing as I wasn’t being pushed and the cyclist kept talking to me, I’ll live with my time.

I ran back very slowly to the hotel watching the marathon runners as they wound their way to the half way mark. They were still going past the hotel two hours after I got back and showered. More than 13,000 took part in the three races overall, great to see so many people getting out early in the day.