RabbitMQ is a messaging application. You can relate it to similar solutions like ActiveMQ or MSMQ.
Essentially RabbitMQ is commonly-used, light, flexible and relatively fast AMQP solution. The Advanced Message Queuing Protocol (AMQP) is two layered (Functional and Transport) binary protocol specially tunned for message sending 😀 so to say – more information about AMQP and AMQ Architecture here. Furthermore RabbitMQ provides additional plugnins for STOMP, MQTT and bridge for ØMQ
ActiveMQ is more sophisticated, highly configurable and multi wire level protocol solution – currently possible protocols are AMQP, MQTT, OpenWire, REST, RSS and Atom, Stomp, WSIF, WS Notification, XMPP – more information here. The default protocol for ActiveMQ is OpenWire you can see/edit it in /apache-activemq-version/conf/activemq.xml