Last Tuesday I attended a talk/workshop organised by IrishDev.com about StormMQ and the Advanced Message Queuing Protocol (AMQP) standard that's cooking up. I was very impressed by how polished and well prepared the presentation was, very enjoyable. Showing "time slices" beside each topic from the beginning, giving a rough idea of how long each section would last was great to keep things interesting and disarmed the 30 minutes attention span limit I often hit during meetings or long talks. Here are some incomplete notes from the evening.

Raph Cohn started by giving us a history of message queuing, from the origins to the evolution to where we're at today. Message queuing has been very expensive, and very vendor specific/locked in so far. However over the last few years, various companies and bodies have been getting together to define an open standard, AMQP, and enable interoperable solutions.

The presentation continued with a welcome refresher introduction to message queues. Message queuing works like glue and can be the solution to many integration problems, both within and between systems. It keeps interacting systems loosely coupled and thus avoid many pitfalls of other solutions (such as RPC, database sharing, file transfers...): it's still possible to automate, scale, deploy easily, keep configuration information isolated, and overall keep the administration burden lower. (Though to work best it should be designed in from the beginning.)

This leaves us at a point where there are MQ solutions out there, however they are either proprietary, and/or platform specific, or come with other sorts of strings attached. Note the speaker included Redis in the proprietary systems which surprised me a little. Thinking about it though, the point is that once you develop your application to work with Redis, you can't easily switch Redis for another provider, you're tied to it.

This is a problem. <Cue slide covered in fake fire -- the animations were awesome! A video of the presentation should be posted sometime soon actually, I'll link to it. (Update: Here)>

And the solution to this is AMQP, a lightweight protocol aiming to smooth or resolve these issues.

Some common Message Queuing architectures:

• Store and forward, or buffering. If one system is down, the queue buffers the messages and the systems still up don't see any difference (as opposed to tightly coupled systems where everything would be brought down in parallel).
• Fire and forget. You send many messages, e.g. orders and don't think about them once they're sent away...
• One to many, or "fan out". You publish a message that is broadcast to every interested system.
• Round robin. This is good for batch jobs and for scaling large jobs, you divide it into smaller tasks on the queue and a free system takes on the next task.
• Publisher/Subscriber, with "Topics".  Subscribers listens for specific messages on the queue, without the publisher having or needing any knowledge of their existence.

A message queue is a basic FIFO queue of messages. A message consists of the environment, a header and a payload. The payload is defined in AMQP and can be binary, text or anything.

AMQP works with connections and channels. Every channel is independent, logically partitioned and TLS protected. First you open a connection, then a channel to send and receive. You close the channel after sending, and close the connection if you're completely done.

AMQP is an ACK-less protocol, and therefore very efficient: it will only tell you when things go wrong.

AMQP uses 'exchanges' for the routing and binding. Exchanges can be direct (either with the specific routing key, or ideally more general to avoid needing special knowledge), fan out (any key bound to X will get the message, good for debugging), topics. Any MQ only receives a message once. It's in spec to use the the default definitions for the exchanges.

The night ended with a demo of using StormMQ with Java. I found it more confusing than helpful (for instance we went through sending back acknowledgements, but I thought one of the main points of the presentation was that AMQP was a ACK-less protocol?), though it could be that it was getting late and I was too tired.