Punching firewalls with Mandrill webhooks
(blogarhythm ~ Fire Cracker - Ellegarden)
Mandrill is the transactional email service by the same folks who do MailChimp. I've written about it before, in particular how to use the mandrill-rails gem to simplify inbound webhook processing.
Mandrill webhooks are a neat, easy way for your application to respond to various events, from recording when users open email, to handling inbound mail delivery.
That all works fine if your web application lives on the public internet i.e. Mandrill can reach it to post the webhook. But that's not always possible: your development/test/staging environments for example; or perhaps production servers that IT have told you must be "locked down to the max".
Mandrill currently doesn't offer an official IP whitelist, so it's not possible to use firewall rules to just let Mandrill servers in. Mandrill does provide webhook authentication (supported by the mandrill-rails gem), but that solves a different problem: anyone can reach your server, but you can distinguish the legitimate webhook requests.
I thought I'd share a couple of techniques I've used to get Mandrill happily posting webhooks to my dev machine and servers behind firewalls.
Using HAProxy to reverse-proxy Mandrill Webhooks
If you have at least one internet-visible address, HAProxy is excellent for setting up a reverse-proxy to forward inbound Mandrill Webhooks to the correct machine inside the firewall. I'm currently using this for some staging servers so we can run real inbound mail scenarios.Here's a simple scenario:
- gateway.mydomain.net - your publicly-visible server, with HAProxy installed and running OK
- internal/192.168.0.1 - a machine on an internal network that you want to receive webooks posted to 192.168.0.1:8080/inbox
Say the gateway machine already hosts http://gateway.mydomain.net, but we want to be able to tell Mandrill to post it's webhooks to http://gateway.mydomain.net/inbox_internal, and these (and only these) requests will be forwarded to http://192.168.0.1:8080/inbox.
Here are the important parts of the /etc/haproxy/haproxy.cfg used on the gateway machine:
globalObviously the path mapping is optional (but neat to demonstrate), and I've left out all the normal HAProxy options like balancing, checks and SSL option forwarding that you might require in practice, but are not relevant to the issue at hand.
#...
defaults
mode http # enable http mode which gives of layer 7 filtering
#...
# this is HAProxy listening on http://gateway.mydomain.net
frontend app *:80
default_backend webapp # set the default server for all requests
# next we define a rule that will send requests to the internal_mandrill backend instead if the path starts with /inbox_internal
acl req_mandrill_inbox_path path_beg /inbox_internal
use_backend internal_mandrill if req_mandrill_inbox_path
# define a group of backend servers for the main web app
backend webapp
server app1 127.0.0.1:8001
# this is where we will send the Mandrill webhooks
backend internal_mandrill
reqirep ^([^\ ]*)\ /inbox_internal(.*) \1\ /inbox\2
server int1 192.168.0.1:8080 # add a server to this backend
Job done! Our internal server remains hidden behind the firewall, but Mandrill can get through to it by posting webhooks to http://gateway.mydomain.net/inbox_internal.
Tunneling to dev machines with ngrok
For development, we usually don't want anything so permanent. There are quite a few services for tunneling to localhost, mainly with developers in mind. Lately I've been using ngrok which is living up to it's name - it rocks! Trival to setup and works like a dream. Say I'm developing a Rails app:# run app locally (port 3000)
rails s
# run ngrok tunnel to port 3000
ngrok 3000
Once started, ngrok will give you http and https addresses that will tunnel to port 3000 on your machine. You can use these addresses in the Mandrill webhook and inbound domains configuration, and they'll work as long as you keep your app and ngrok running.
read more and comment..
Modern Art
Blogarhythm: Modern Life ~ Regurgitator
A great quote from Craig Damrauer that I found in "Steal Like an Artist"..
read more and comment..
Design thinking is not rocket science
OH on the ABC Radio National By Design podcast (00:56): In the field: Paul Bennet
For us the idea of small ideas that people can actually connect together and actually implement are very big ideas.
And I'm sure you've heard people describe design thinking as sort of a combination of rocket science, string theory and calculus.
It isn't. It's not rocket science at all. It's actually very very straight-forward.
It's looking in the world, being inspired by people, co-creating with them, prototyping and then iterating. And it has to be impactful. It has to work.
read more and comment..
Writing simple ruby utilities for Google IMAP + OAuth 2.0
(blogarhythm ~ Unpretty/Fanmail: TLC)
There are some good ruby gems available for dealing with OAuth 2.0 and talking to Google APIs, for example:
- google-api-client is the official Google API Ruby Client makes it trivial to discover and access supported APIs.
- oauth2-client provides generic OAuth 2.0 support that works not just with Google
- gmail_xoauth implements XAUTH2 for use with Ruby Net::IMAP and Net::SMTP
- gmail provides a rich Ruby-esque interface to GMail but you need to pair it with gmail_xoauth for OAuth 2 support (also seems that it's in need of a new release to merge in various updates and extensions people have been working on)
For the task I had at hand, I just wanted something simple: connect to a mailbox, look for certain messages, download and do something with the attachments and exit. It was going to be a simple utility to put on a cron job.
No big deal. The first version simple used gmail_xoauth to enable OAuth 2.0 support for IMAP, and I added some supporting routines to handle access_token refreshing.
It worked fine as a quick and dirty solution, but had a few code smells. Firstly, too much plumbing code. But most heinously - you might seen this yourself if you've done any client utilities with OAuth - it used the widely-recommended oauth2.py Python script to orchestrate the initial authorization. For a ruby tool!
Enter the GmailCli gem
So I refactored the plumbing into a new gem called gmail_cli and it is intended for one thing: a super-simple way to whip up utilities that talk to Google IMAP and providing all the OAuth 2.0 support you need. It actually uses google-api-client and gmail_xoauth under the covers for the heavy lifting, but wraps them up in a neat package with the simplest interface possible. Feel free to go use and fork it!With gmail_cli in your project, there are just 3 things to do:
- If you haven't already, create your API project credentials in the Google APIs console (on the "API Access" tab)
- Use the built-in rake task or command-line to do the initial authorization. You would normally need to do this only once for each deployment:
$ rake gmail_cli:authorize client_id='id' client_secret='secret'
$ gmail_cli authorize --client_id 'id' --client_secret 'secret' - Use the access and refresh tokens generated in step 2 to get an IMAP connection in your code. This interface takes care of refreshing the access token for you as required each time you use it:
# how you store or set the credentials Hash is up to you, but it should have the following keys:
credentials = {
client_id: 'xxxx',
client_secret: 'yyyy',
access_token: 'aaaa',
refresh_token: 'rrrr',
username: 'name@gmail.com'
}
imap = GmailCli.imap_connection(credentials)
A Better Way?
Polling a mailbox is a terrible thing to have to do, but sometimes network restrictions or the architecture of your solution makes it the best viable option. Much better is to be reactive to mail that gets pushed to you as it is delivered.I've written before about Mandrill, which is the transactional email service from the same folks who do MailChimp. I kinda love it;-) It is perfect if you want to get inbound mail pushed to your application instead of polling for it. And if you run Rails, I really would encourage you to checkout the mandrill-rails gem - it adds Mandrill inbound mail processing to my Rails apps with just a couple of lines of code.
read more and comment..