iMatix Corporation

Introduction

This article describes AMQP (Advanced Message Queuing Protocol). AMQP represents an attempt by some large businesses and some smaller suppliers to alter the business messaging market; in many respects they seem to have succeeded. IBM and Tibco dominate this market with two dominant suppliers and few open standards or open source competitors; it is bad both for clients as well as vendors; when there’s no genuine competition in an area of technology nothing new emerges to drive innovation forward. At its core, technology creates wealth through layering on layers. Sterility benefits nobody; eventually the old money runs out. Think HTTP, SQL and Linux and how much wealth has grown on top of these basic technologies over the past 25-30 years; compare that growth against messaging-only growth seen with Java but even this doesn’t cut it!

Let me briefly outline how standards operate. While you might believe they exist to bring revolutionary new concepts and innovations to the public, in reality standards serve as more of a way to block innovation than facilitate it; by freezing innovation at one level we allow more of it at higher ones.

So standardization works from the bottom up: from fundamental, widely-used technologies to increasingly specialized ones with narrower usage patterns. Over time, standards tend to consolidate, with unnecessary ones eventually dropping away altogether – something AMQP stands out by starting low down the stack using TCP rather than HTTP as its basis and being ambitious in going from data on the wire directly through to application semantics in one step – more on this below!

Economic principles underpinning setting standards rest upon what economists refer to as natural monopoly: an attraction-and-retention theory which means, at least in this instance, that any successful standard will attract and keep all users. One such natural monopoly occurs in currency trading: when states mandate that we all utilize one particular currency unit exclusively – becoming its monopoly and penalizing anyone possessing any other unit for trading activities or penalizing anyone holding any alternative currencies if possible.

Similar natural monopolies would include rail transport, electricity, phones and the Internet Protocol. Just as we want our toasters to plug into any socket we want our phones to reach anyone anywhere; so too should they reach any person they could.

People reading the AMQP specifications today often express concern that its protocol is too complicated, which is an understandable response; complexity is fatal: its effects can range from imprisoning Sleeping Beauty in her garden of thorns, minefields and high walls of technology that disengage people, to simply being impossible for small teams or projects to implement and use successfully; failure will come quickly wherever AMQP fails in FOSS projects if left to its own devices.

Pain Is Never an Obvious Sign

In previous sections, I outlined why this article exists as well as making some obvious yet annoying recommendations to try to salvage some of the work done so far. AMQP could move to a Safe Place by adopting its 0-9SP1 text standard as version 1.0 and pressuring all implementations to support it – acknowledging mistakes is human; admitting them and being prepared to backtrack requires divine grace or at least an excellent parser.

However, securing the work that has already been accomplished – which only partially pertains to AMQP itself but more significantly on deployment – is only half the battle. Although closure would be nice at least here, moving forward AMQP should aim towards having releases 2.x 3.x 4.x over 50 years (how?).

As I don’t consider myself to be an expert in messaging, but do have an undiplomatic and accurate eye for seeing things that don’t work, working on hundreds of teams from professional to voluntary has taught me invaluable lessons about effective ways of organizing things – AMQP Working Group being better but worse in comparison.

My primary indicator for whether an organizational model is effective or not is pain level: my own and that of those I work with – including my colleague at New York face-to-face meetings. If its use causes discomfort for participants in any capacity whatsoever then that model likely has something amiss; pain should never be an indicator that something’s going wrong!

One key difference among AMQP Working Group participants lies in how they perceive pain. Some members, for instance, who are large and wealthy may become used to the complex processes required of larger organizations; as a result they probably feel it regularly but remain relatively intolerant of it. But others, typically small in scale but well funded like agile organizational models tend to feel it all day every day more acutely and remain relatively intolerant compared to these larger players; small participants often act as canaries in coal mines–I hope AMQP appeals more so FOSS developers because the more sensitive smaller players become for problems which ultimately impact all others involved–thus I want AMQP to attract FOSS developers as soon as possible as these smaller players tend to spot issues sooner!

Presumably everyone involved with AMQP Working Group holds their own opinions on this matter; for iMatix there have been three real causes that have made us retreat and treat AMQP more like an obstacle or risk than as an opportunity.

AMQP was designed as one document, in terms of written text and an XML file, to ease interoperability issues more efficiently than having multiple specifications layered upon one another. Unfortunately, that single spec now exceeds 300 pages – perhaps normal when compared with certain RFC documents; but here it clearly seems excessive in size compared with its peers like IEFT RFC documents!

Before we released AMQP/0.8, its XML specifications consisted of one top-level file and then separate files per class, making editing individual classes simple; expanding or changing classes was an effortless experience as was managing AMQP itself. Unfortunately this structure proved too challenging for parsers so this structure was flattened into just one file (despite my efforts with Perl scripts to achieve flattening as code generators needed a uniform representation in AMQP itself!). I argued against it being made One File instead — no such concession was allowed when designing AMQP!

Large single documents (XML or text), not only present an editing nightmare for editors but are often incongruent with standards like AMQP which seek to address multiple levels. Compatibility with Java Messaging System (JMS), another Enterprise Technology check box item that does not pertain directly to FOSS developers but seems to affect some AMQP participants like an evil spirit possessing an innocent priest in an Nollywood horror flick, is another item from that Enterprise Tech checklist that no FOSS developer cares much for. Why does JMS compatibility, an outdated high-level application language-specific feature, come out of nowhere in a text like wire-level encoding of 4-byte integers – another low-level feature? Or take another example, why would AMQP specifications on connection negotiation appear alongside distributed transaction specifications in one document?

Mixing concepts like JMS, basic data types, connections and transactions together in one document makes it possible for people to propose changes that span multiple levels. People might propose drastic alterations without regard to architecture – say making connections part of transactions that use JMS headers compatible with Java! Sounds great right? Soon thereafter however basic notions of architecture and structure disappear, leaving what remains nothing more than format changes with more pleasing fonts or page layout.

AMQP’s functional scope far outstrips what can be captured within one document, so its existence as one document doesn’t make much sense to me. At present there are multiple other specifications (remote administration, federation, message bundling etc) which don’t fall within its confines and cannot therefore exist on paper as documents.

AMQP must be designed as an interoperable stack of protocols defining individual layers that allows competition and experimentation at each tier, which brings me back to another source of pain: ownership or rather its lack.

Premature optimization can lead to disaster and I won’t go into too much detail here on AMQP (partially because of how difficult it has become for me to keep up with changes) but some design decisions that affect everything can have far reaching effects if they go awry.

AMQP is defined as a binary protocol. This means that methods like Queue. Create are defined as binary frames in which every field sits in its proper place and encoded correctly to optimize space usage. Binary numbers encodings tend to be much shorter than string encodings: “123” requires one octet while three strings would suffice in string form; because AMQP uses binary encoding it is very fast for parsing since string data need not be parsed first before being added as text parsed afterwards compared with string. AMQP software also means no buffer overflow attacks can happen quickly while it also easily generate encoding/decoding routines as AMQP defines easy-to-parse XML documentations!

Overall, AMQP’s binary encoding was once thought of as a significant advantage; that claim has since been disproven by evidence and I admit full responsibility. Evidence proves it to be fundamentally mistaken; an error I take full ownership for since this design was my creation and made proudly under my care; an expert error. To understand my admission of responsibility let’s review some advantages and costs associated with binary encoding along with deconstructing its base assumptions that now seem false before finally comparing this approach with one with more accurate assumptions (or another approach with more accurate assumptions).

Advantages associated with binary encoding include:

* Compactness

* It’s faster parse time when parsing strings through to parse strings without issues.

* The codecs can fully generated making processing silicon easy.

Costs associated with binary Encoding include

* You need codecs up front when processing silicon is considered.

* String Encoding creates incompatible versions of AMQP that cannot interoperate, is more complex to comprehend and utilize, and places an increased emphasis on data types.

* Even the simplest client API can be quite complex. But in practice, creating fast, compact wire-level encodings should certainly make up for its complexity; after all AMQP was designed specifically to maximize speed; its Enterprise Technology status means complex APIs shouldn’t pose too many difficulties either; text-based protocols like HTTP may offer similar options but real HTTP clients often use existing libraries instead of writing their own clients to interact.

Standard AMQP wisdom dictates that binary encoding costs must be covered if one wishes for a fast, reliable protocol; we believed otherwise because text-encoded protocols like HTTP simply couldn’t provide sufficient performance in comparison.

Underlying AMQP’s binary encoding scheme lies the assumption that its necessity lies with performance issues (speed of parsing and compactness of data). By disproving this assumption I will have successfully dismantled its main justification – depriving binary encoding its main justification and rendering its use obsolete.

Here’s a pop quiz designed to test your knowledge of protocols: What is the fastest messaging protocol that comes built-in with every modern operating system, can saturate ordinary networks, and can even run faster than HTTP and AMQP? In fact, implementations of this protocol would likely score as the fastest messaging application ever devised!

FTP (file transfer protocol ), is often employed by network engineers when testing whether an interface supports 100Mbps or 1Gbps: its high data transfer rates ensure it can provide proof of how fast or slow any given link really is.

Now for my question. What sets FTP apart and provides so quickly transfer of data? And can this serve as an inspiration to AMQP developers?

My beliefs regarding AMQP performance stem from our work on ZeroMQ, an ultrafast messaging fabric which transmits millions of messages per second using FTP techniques instead of those found within AMQP.

FTP stands out by using one connection for control commands and another connection for message transfer – something other protocols like HTTP do not do; additionally, FTP tends to be faster and simpler than its rival in many aspects – all features considered desirable qualities in any protocol.