my recent reads..

Atomic Accidents: A History of Nuclear Meltdowns and Disasters; From the Ozark Mountains to Fukushima
Power Sources and Supplies: World Class Designs
Red Storm Rising
Locked On
Analog Circuits Cookbook
The Teeth Of The Tiger
Sharpe's Gold
Without Remorse
Practical Oscillator Handbook
Red Rabbit

Thursday, February 11, 2016

LittleArduinoProjects#183 Polarity Testing

A simple polarity test for uses a series of inverters. The input signal is pumped into two parallel inverter chains:

  • a single inverter
  • a series of two inverters

So, regardless of input polarity, one chain output will be high and the other low.

I used a venerable CD4069 for a quick test, although any inverter (matched to the voltage of the signal) will do. The result of the polarity test is displayed with a pair of LEDs reversed in parallel.

As always, all notes, schematics and code are in the Little Arduino Projects repo on GitHub.

Wednesday, February 10, 2016

LittleArduinoProjects#182 Building a Bench Power Supply

I've wanted a variable mains-powered power supply for a while, so when I found this kit for a reasonable price I decided to give it a go. Some things that attracted me:

  • 220V/110V mains-powered
  • continuously adjustable output voltage
  • isolated output
  • nice acrylic case
  • built-in LED voltmeter

The kit and PCB comes with a few "valued-added features" unrelated to the power supply function (CD4069 square-wave generator, externally-triggered piezo buzzer, externally-triggered polarity tester), but I decided to leave those out of the build.

I also enjoyed investigating and old-school transformer-based power supply. These are getting rare .. it's hard to even find a 220/12 transformer for less than the price of the kit these days, and then they are mostly used/refurbished.

How does it perform? Nice! No smoke on power-up, but a few things to note and/or improve. And I'm sure it wouldn't pass a safety certification, so don't go building one like this for friends.

As always, all notes, schematics and code are in the Little Arduino Projects repo on GitHub.

LittleArduinoProjects#181 Voltmeter Modules

Voltmeter modules are a very convenient way of adding voltage display to any project, because they require no supporting circuitry or microcontrollers.

There are two and three wire modules in the market. Two-wire modules are the simplest (and generally cheapest). For most applications where a simple readout of a power supply is required, they are most convenient since a separate power supply connection is not required. It does mean of course that the meter draws current from the circuit under test, and will only work down to a certain voltage (usually ~4.5V).

For more precision measurement, the three-wire modules are best as they present relatively high input impedance (> 100kΩ), and can measure voltages all the way down to 0. As an example, Adafruit have this 3-wire module in their catalogue, but they are widely available from most online sellers.

I recently received a 3-wire module as part of a kit and put it to the test. Using my DMM as a reference, it appears accuracy is within 40mV, good enough for general purposes.

As always, all notes, schematics and code are in the Little Arduino Projects repo on GitHub.

Tuesday, February 09, 2016

LittleArduinoProjects#180 Colpitts Oscillator

A Colpitts oscillator uses a combination of inductors and capacitors to produce an oscillation at the resonant frequency of LC circuit.

To see that in action, I built one on a protoboard and it delivers an almost perfect 22.9kHz .. compared to the theoretical 22.5kHz.

As always, all notes, schematics and code are in the Little Arduino Projects repo on GitHub.

Here's a trace of the output signal on CH1, and the mid-point of the capacitor pair on CH2:

Monday, February 08, 2016

LittleArduinoProjects#174 USB LED Notifiers

So four of these USB Webmail Notifier devices turned up in a dusty cupboard
in the office.

A quick tear-down shows they contain a super-simple circuit - just a
SONiX Technology SN8P2203SB 8-Bit microcontroller that handles the USB protocol and drives an RGB LED. The SN8P2203SB is an old chip phased out 2010/04/30, superseded by the SN8P2240. They have a supremely primitive USB implementation - basically mimicking a very basic USB 1.0 HID device.

A quick google reveals quite a bit of old code lying around for various projects using devices like this. Most seem to use libusb for convenience - and often 0.1 legacy libusb that. As I'm mainly on MacOSX, the code is not much use since Apple no longer allows claiming of HID devices
and the libusb team decided not to try to get around that.

So to bring things up-to-date, I wrote a simple demo using hidapi
and things all work fine - see the video below.

Now I just need to ponder on good ideas for what to do with these things!

As always, all all notes, schematics and code are in the Little Arduino Projects repo on GitHub.

Sunday, February 07, 2016

LittleArduinoProjects#173 Mini 64-LED Cube

LED cubes were a "thing" a few years back maybe ... but I've never built one. Time to fix that...

Here's my "mini" 4x4x4 cube - 3cm per side with 3mm clear blue LEDs. Pretty compact, and delivers nice effects. The clear blue LEDs work really well - very bright, even when driven with minimal current.

It's encased in a Ferrero Rocher cube box. During the build, that raised some challenges - most of the effort in building the project concerned squeezing all the electronics into the space in the lid (which becomes the base of the cube). Not quite as small as HariFun's "World's Tiniest RGB LED Cube" but about as small as you can get without resorting to SMD LEDs!

As always, all notes, schematics and code are in the Little Arduino Projects repo on GitHub. Here's a quick demo:

Sunday, June 28, 2015

LittleArduinoProjects#100 Retrogaming on an Arduino/OLED "console"

(blogarhythm ~ invaders must die - The Prodigy)
Tiny 128x64 monochrome OLED screens are cheap and easy to come by, and quite popular for adding visual display to a microcontroller project.

My first experiments in driving them with raw SPI commands had me feeling distinctly old school, as the last time remember programming a bitmap screen display was probably about 30 years ago!

So while in a retro mood, what better than to attempt an arcade classic? At first I wasn't sure it was going to be possible to make a playable game due to the limited Arduino memory and relative slow screen communication protocol.

But after a few tweaks of the low-level SPI implementation, I was surprised myself at how well it can run. Even had enough clock cycles left to throw in a sound track and effects.

Here's a quick video on YouTube of the latest version. ArdWinVaders! .. in full lo-rez monochrome glory, packed into 14kb and speeding along at 8MHz.

As always, all notes, schematics and code are in the Little Arduino Projects repo on GitHub.

Full source and schematics are in the LittleArduinoProjects collection on Github.

Sunday, February 01, 2015

LittleArduinoProjects#018 The Fretboard - a multi-project build status monitor

(blogarhythm ~ Diablo - Don't Fret)

The Fretboard is a pretty simple Arduino project that visualizes the build status of up to 24 projects with an addressable LED array. The latest incarnation of the project is housed in an old classical guitar … hence the name ;-)

All the code and design details for The Fretboard are open-source and available at Feel free to fork or borrow any ideas for your own build. If you build anything similar, I'd love to hear about it.

Wednesday, January 14, 2015

cancannible role-based access control gets an update for Rails 4

Can You Keep a Secret? / 宇多田ヒカル

cancannible is a gem that has been kicking around in a few large-scale production deployments for years. It still gets loving attention - most recently an official update for Rails 4 (thanks to the push from @zwippie).

And now also some demo sites - one for Rails 3.2.x and another for Rails 4.3.x so that anyone can see it in action.

So what exactly does cancannible do? In a nutshell, it is a gem that extends CanCan with a range of capabilities:

  • permissions inheritance (so that, for example, a User can inherit permissions from Roles and/or Groups)
  • general-purpose access refinements (to automatically enforce multi-tenant or other security restrictions)
  • automatically stores and loads permissions from a database
  • optional caching of abilities (so that they don't need to be recalculated on each web request)
  • export CanCan methods to the model layer (so that permissions can be applied in model methods, and easily set in a test case)

Friday, January 09, 2015

555 Timer simulator with HTML5 Canvas

The 555 timer chip has been around since the '70s, so does the world really need another website for calculating the circuit values?

No! But I made one anyway. It's really an excuse to play around with HTML5 canvas and demonstrate a grunt & coffeescript toolchain.

See this running live at, where you can find more info and links to projects and source code on GitHub.

(blogarhythm ~ Time's Up / Living Colour)

Tuesday, January 28, 2014

Learning Devise for Rails

(blogarhythm ~ Points of Authority / Linkin Park)

I recently got my hands on a review copy of Learning Devise for Rails from Packt and was quite interested to see if it was worth a recommendation (tldr: yes).

A book like this has to be current. Happily this edition covers Rails 4 and Devise 3, and code examples worked fine for me with the latest point releases.

The book is structured as a primer and tutorial, perfect for those who are new to devise, and requires only basic familiarity with Rails. Tutorials are best when they go beyond the standard trivial examples, and the book does well on this score. It covers a range of topics that will quickly become relevant when actually trying to use devise in real life. Beyond the basic steps needed to add devise in a Rails project, it demonstrates:
  • customizing devise views
  • using external authentication providers with Omniauth
  • using NoSQL storage (MongoDB) instead of ActiveRecord (SQLite)
  • integrating access control with CanCan
  • how to test with Test::Unit and RSpec

I particularly appreciate the fact that the chapter on testing is even there in the first place! These days, "how do I test it?" should really be one of the first questions we ask when learning something new.

The topics are clearly demarcated so after the first run-through the book can also be used quite well as a cookbook. It does however suffer from a few cryptic back-references in the narrative, so to dive in cookbook-style you may find yourself having to flip back to previous sections to connect the dots. A little extra effort on the editing front could have improved this (along with some of the phraseology, which is a bit stilted in parts).

Authentication has always been a critical part of Rails development, but since Rails 3 in particular it is fair to say that devise has emerged as the mature, conventional solution (for now!). So I can see this book being the ideal resource for developers just starting to get serious about building Rails applications.

Learning Devise for Rails would be a good choice if you are looking for that shot of knowledge to fast-track the most common authentication requirements, but also want to learn devise in a little more depth than a copy/paste from the README and wiki will get you! It will give enough foundation for moving to more advanced topics not covered in the book (such as developing custom strategies, understanding devise and warden internals).

Tuesday, November 12, 2013

Punching firewalls with Mandrill webhooks

(blogarhythm ~ Fire Cracker - Ellegarden)

Mandrill is the transactional email service by the same folks who do MailChimp. I've written about it before, in particular how to use the mandrill-rails gem to simplify inbound webhook processing.

Mandrill webhooks are a neat, easy way for your application to respond to various events, from recording when users open email, to handling inbound mail delivery.

That all works fine if your web application lives on the public internet i.e. Mandrill can reach it to post the webhook. But that's not always possible: your development/test/staging environments for example; or perhaps production servers that IT have told you must be "locked down to the max".

Mandrill currently doesn't offer an official IP whitelist, so it's not possible to use firewall rules to just let Mandrill servers in. Mandrill does provide webhook authentication (supported by the mandrill-rails gem), but that solves a different problem: anyone can reach your server, but you can distinguish the legitimate webhook requests.

I thought I'd share a couple of techniques I've used to get Mandrill happily posting webhooks to my dev machine and servers behind firewalls.

Using HAProxy to reverse-proxy Mandrill Webhooks

If you have at least one internet-visible address, HAProxy is excellent for setting up a reverse-proxy to forward inbound Mandrill Webhooks to the correct machine inside the firewall. I'm currently using this for some staging servers so we can run real inbound mail scenarios.

Here's a simple scenario:
  • - your publicly-visible server, with HAProxy installed and running OK
  • internal/ - a machine on an internal network that you want to receive webooks posted to

Say the gateway machine already hosts, but we want to be able to tell Mandrill to post it's webhooks to, and these (and only these) requests will be forwarded to

Here are the important parts of the /etc/haproxy/haproxy.cfg used on the gateway machine:
  mode http                                # enable http mode which gives of layer 7 filtering

# this is HAProxy listening on
frontend app *:80                          
  default_backend webapp                   # set the default server for all requests
  # next we define a rule that will send requests to the internal_mandrill backend instead if the path starts with /inbox_internal
  acl req_mandrill_inbox_path path_beg /inbox_internal 
  use_backend internal_mandrill if req_mandrill_inbox_path 

# define a group of backend servers for the main web app
backend webapp                             
  server app1             
# this is where we will send the Mandrill webhooks
backend internal_mandrill                  
  reqirep     ^([^\ ]*)\ /inbox_internal(.*) \1\ /inbox\2 
  server int1  # add a server to this backend
Obviously the path mapping is optional (but neat to demonstrate), and I've left out all the normal HAProxy options like balancing, checks and SSL option forwarding that you might require in practice, but are not relevant to the issue at hand.

Job done! Our internal server remains hidden behind the firewall, but Mandrill can get through to it by posting webhooks to

Tunneling to dev machines with ngrok

For development, we usually don't want anything so permanent. There are quite a few services for tunneling to localhost, mainly with developers in mind. Lately I've been using ngrok which is living up to it's name - it rocks! Trival to setup and works like a dream. Say I'm developing a Rails app:
# run app locally (port 3000)
rails s
# run ngrok tunnel to port 3000
ngrok 3000

Once started, ngrok will give you http and https addresses that will tunnel to port 3000 on your machine. You can use these addresses in the Mandrill webhook and inbound domains configuration, and they'll work as long as you keep your app and ngrok running.

Sunday, June 23, 2013

Design thinking is not rocket science

OH on the ABC Radio National By Design podcast (00:56): In the field: Paul Bennet
For us the idea of small ideas that people can actually connect together and actually implement are very big ideas.

And I'm sure you've heard people describe design thinking as sort of a combination of rocket science, string theory and calculus.

It isn't. It's not rocket science at all. It's actually very very straight-forward.

It's looking in the world, being inspired by people, co-creating with them, prototyping and then iterating. And it has to be impactful. It has to work.

Writing simple ruby utilities for Google IMAP + OAuth 2.0

(blogarhythm ~ Unpretty/Fanmail: TLC)

There are some good ruby gems available for dealing with OAuth 2.0 and talking to Google APIs, for example:
  • google-api-client is the official Google API Ruby Client makes it trivial to discover and access supported APIs.
  • oauth2-client provides generic OAuth 2.0 support that works not just with Google
  • gmail_xoauth implements XAUTH2 for use with Ruby Net::IMAP and Net::SMTP
  • gmail provides a rich Ruby-esque interface to GMail but you need to pair it with gmail_xoauth for OAuth 2 support (also seems that it's in need of a new release to merge in various updates and extensions people have been working on)

For the task I had at hand, I just wanted something simple: connect to a mailbox, look for certain messages, download and do something with the attachments and exit. It was going to be a simple utility to put on a cron job.

No big deal. The first version simple used gmail_xoauth to enable OAuth 2.0 support for IMAP, and I added some supporting routines to handle access_token refreshing.

It worked fine as a quick and dirty solution, but had a few code smells. Firstly, too much plumbing code. But most heinously - you might seen this yourself if you've done any client utilities with OAuth - it used the widely-recommended Python script to orchestrate the initial authorization. For a ruby tool!

Enter the GmailCli gem

So I refactored the plumbing into a new gem called gmail_cli and it is intended for one thing: a super-simple way to whip up utilities that talk to Google IMAP and providing all the OAuth 2.0 support you need. It actually uses google-api-client and gmail_xoauth under the covers for the heavy lifting, but wraps them up in a neat package with the simplest interface possible. Feel free to go use and fork it!

With gmail_cli in your project, there are just 3 things to do:

  1. If you haven't already, create your API project credentials in the Google APIs console (on the "API Access" tab)
  2. Use the built-in rake task or command-line to do the initial authorization. You would normally need to do this only once for each deployment:
    $ rake gmail_cli:authorize client_id='id' client_secret='secret'
    $ gmail_cli authorize --client_id 'id' --client_secret 'secret'
  3. Use the access and refresh tokens generated in step 2 to get an IMAP connection in your code. This interface takes care of refreshing the access token for you as required each time you use it:
    # how you store or set the credentials Hash is up to you, but it should have the following keys:
    credentials = {
      client_id:     'xxxx',
      client_secret: 'yyyy',
      access_token:  'aaaa',
      refresh_token: 'rrrr',
      username:      ''
    imap = GmailCli.imap_connection(credentials)

A Better Way?

Polling a mailbox is a terrible thing to have to do, but sometimes network restrictions or the architecture of your solution makes it the best viable option. Much better is to be reactive to mail that gets pushed to you as it is delivered.

I've written before about Mandrill, which is the transactional email service from the same folks who do MailChimp. I kinda love it;-) It is perfect if you want to get inbound mail pushed to your application instead of polling for it. And if you run Rails, I really would encourage you to checkout the mandrill-rails gem - it adds Mandrill inbound mail processing to my Rails apps with just a couple of lines of code.

Tuesday, June 18, 2013

Ruby Tuesday

(blogarhythm ~ Ruby - Kaiser Chiefs)
@a_matsuda convinced us to dive into Ruby 2.0 at RedDotRubyConf, so I guess this must be the perfect day of the week for it!

Ruby 2.0.0 is currently at p195, and we heard at the conference how stable and compatible it is.

One change we learned that may catch us if we do much multilingual work that's not already unicode is the change that Ruby now assumes UTF-8 encoding for source files. So the special "encoding: utf-8" marker becomes redundant, but if we don't include it the behaviour in 2.0.0 can differ from earlier versions:
$ cat encoding_binary.rb 
s = "\xE3\x81\x82"
p str: s, size: s.size
$ ruby -v encoding_binary.rb 
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin11.4.2]
{:str=>"あ", :size=>1}
$ ruby -v encoding_binary.rb 
ruby 1.9.3p429 (2013-05-15 revision 40747) [x86_64-darwin11.4.2]
{:str=>"\xE3\x81\x82", :size=>3}

Quickstart on MacOSX with RVM

I use rvm to help manage various Ruby installs on my Mac, and trying out new releases is exactly the time you want it's assistance to prevent screwing up your machine. There were only two main things I needed to take care of to get Ruby 2 installed and running smoothly:
  1. Update rvm so it knows about the latest Ruby releases
  2. Update my OpenSSL installation (it seems 1.0.1e is required although I haven't found that specifically documented anywhere)
Here's a rundown of the procedure I used in case it helps (note, I am running MacOSX 10.7.5 with Xcode 4.6.2). First I updated rvm and attempted to install 2.0.0:
$ rvm get stable
# => updated ok
$ rvm install ruby-2.0.0
Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.7/x86_64/ruby-2.0.0-p195.
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.
Installing requirements for osx, might require sudo password.
-bash: /usr/local/Cellar/openssl/1.0.1e/bin/openssl: No such file or directory
Updating certificates in ''.
mkdir: : No such file or directory
mkdir: : No such file or directory
Can not create directory '' for certificates.
Not good!!! What's all that about? Turns out to be just a very clumsy way of telling me I don't have OpenSSL 1.0.1e installed.

I already have OpenSSL 1.0.1c installed using brew (so it doesn't mess with the MacOSX system-installed OpenSSL), so updating is simply:
$ brew upgrade openssl
==> Summary
 /usr/local/Cellar/openssl/1.0.1e: 429 files, 15M, built in 5.0 minutes
So then I can try the Ruby 2 install again, starting with the "rvm requirements" command to first make sure all pre-requisites are installed:
$ rvm requirements
Installing requirements for osx, might require sudo password.
Tapped 41 formula
Installing required packages: apple-gcc42.................
Updating certificates in '/usr/local/etc/openssl/cert.pem'.
$ rvm install ruby-2.0.0
Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.7/x86_64/ruby-2.0.0-p195.
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.
Installing requirements for osx, might require sudo password.
Certificates in '/usr/local/etc/openssl/cert.pem' already are up to date.
Installing Ruby from source to: /Users/paulgallagher/.rvm/rubies/ruby-2.0.0-p195, this may take a while depending on your cpu(s)
OK, this time it installed cleanly as I can quickly verify:
$ rvm use ruby-2.0.0
$ ruby -v
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin11.4.2]
$ irb -r openssl
2.0.0p195 :001 > OpenSSL::VERSION
 => "1.1.0"
2.0.0p195 :002 > OpenSSL::OPENSSL_VERSION
 => "OpenSSL 1.0.1e 11 Feb 2013"

Saturday, June 15, 2013

Optimising presence in Rails with PostgreSQL

(blogarhythm ~ Can't Happen Here - Rainbow)
It is a pretty common pattern to branch depending on whether a query returns any data - for example to render a quite different view. In Rails we might do something like this:
query = User.where(deleted_at: nil).and_maybe_some_other_scopes
if results = query.presence
  results.each {|row| ... }
  # do something else
When this code executes, we raise at least 2 database requests: one to check presence, and another to retrieve the data. Running this at the Rails console, we can see the queries logged as they execute, for example:
(0.9ms)  SELECT COUNT(*) FROM "users" WHERE "users"."deleted_at" IS NULL
 User Load (15.2ms)  SELECT "users".* FROM "users" WHERE "users"."deleted_at" IS NULL
This is not surprising since under the covers, presence (or present?) end up calling count which must do the database query (unless you have already accessed/loaded the results set). And 0.9ms doesn't seem too high a price to pay to determine if you should even try to load the data, does it?

But when we are running on PostgreSQL in particular, we've learned to be leery of COUNT(*) due to it's well known performance problems. In fact I first started digging into this question when I started seeing expensive COUNT(*) queries show up in NewRelic slow transaction traces. How expensive COUNT(*) actually is depends on many factors including the complexity of the query, availability of indexes, size of the table, and size of the results set.

So can we improve things by avoiding the COUNT(*) query? Assuming we are going to use all the results anyway, and we haven't injected any calculated columns in the query, we could simply to_a the query before testing presence i.e.:
query = User.where(deleted_at: nil).and_maybe_some_other_scopes
if results = query.to_a.presence
  results.each {|row| ... }
  # do something else

I ran some benchmarks comparing the two approaches with different kinds of queries on a pretty well-tuned system and here are some of the results:
Query Using present? Using to_a Faster By
10k indexed queries returning 1 / 1716 rows 17.511s 10.938s 38%
4k complex un-indexed queries returning 12 / 1716 rows 23.603s 15.221s 36%
4k indexed queries returning 1 / 1763218 rows 22.943s 20.924s 9%
10 complex un-indexed queries returning 15 / 1763218 rows 23.196s 14.072s 40%

Clearly, depending on the type of query we can gain up to 40% performance improvement by restructuring our code a little. While my aggregate results were fairly consistent over many runs, the performance of individual queries did vary quite widely.

I should note that the numbers were *not* consistent or proportional across development, staging, test and production environments (mainly due to differences in data volumes, latent activity and hardware) - so you can't benchmark on development and assume the same applies in production.

Things get murky with ActiveRecord add-ons

So far we've talked about the standard ActiveRecord situation. But there are various gems we might also be using to add features like pagination and search magic. MetaSearch is an example: a pretty awesome gem for building complex and flexible search features. But (at least with version 1.1.3) present? has a little surprise in store for you:
irb> User.where(id: '0').class
=> ActiveRecord::Relation
irb> User.where(id: 0).present?
   (0.8ms)  SELECT COUNT(*) FROM "users" WHERE "users"."id" = 0
=> false
irb> 0).class
=> MetaSearch::Searches::User
irb> 0).present?
=> true

Any Guidelines?

So, always to_a my query results? Well, no, it's not that simple. Here are some things to consider:
  • First, don't assume that <my_scoped_query>.present? means what you think it might mean - test or play it safe
  • If you are going to need all result rows anyway, consider calling to_a or similar before testing presence
  • Avoid this kind of optimisation except at the point of use. One of the beauties of ActiveRecord::Relation is the chainability - something we'll kill as soon as we hydrate to a result set Array for example.
  • While I got a nice 40% performance bonus in some cases with a minor code fiddle, mileage varies and much depends on the actual query. You probably want to benchmark in the actual environment that matters and not make any assumptions.