JMSTester online

The last couple of weeks I have spent some time on site working with ActiveMQ users helping them to get a decent benchmark in place for their JMS layer. I already had some code for a JMS benchmarking framework from earlier assignments and used the chance to get that a bit polished and documented.

Essentially it is designed to run a set of distributed JMS clients with a defined load profile. The framework will then measure the throughput and also allows to measure key performance indicators via JMX or on the OS level during the benchmark. This gives nice hints, whether the JMS server machines are CPU bound, lack IO capacity or have to less memory etc.

For all metrics graphs are created and if you want to run even more sophisticated analysises, all metrics are pushed into a CSV file that can be used in your favorite spreadsheet calculator.

The JMSTest framework is now a public project at FUSE Source.

Have fun trying it out if you want, any feedback is more than welcome.

Enabling Security for ActiveMQ web apps

Recently I was working with for a company to harden their set up of Active MQ. Though not the ultimate monitoring solution for Avtive MQ, they wanted to use the web console for providing some kind of ad hoc visibility on key properties. The ActiveMQ web page describes the set up that is pretty straight forward when an external servlet engine is used, but configuring the embedded jetty is a bit different.

From the ActiveMQ web console documentation we could see that we need 3 things:

1. An User Realm that can be plugged into the Jetty engine.


2. Credentials definitions.


3. A security aware web application.

Let's have a look at each of these things:

We start with rewriting the jetty bean definition in the default activemq.xml with the following activemq.xml:

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:jetty="http://mortbay.com/schemas/jetty/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd"> <!-- Allows us to use system properties as variables in this configuration file --> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="locations"> <value>file:///${activemq.base}/conf/credentials.properties</value> </property> </bean> <broker xmlns="http://activemq.apache.org/schema/core" brokerName="TRT_INT" dataDirectory="${activemq.base}/data"> <!-- Destination specific policies using destination names or wildcards --> <destinationPolicy> <policyMap> <policyEntries> <policyEntry queue=">" producerFlowControl="true"/> <policyEntry topic=">" producerFlowControl="true"/> </policyEntries> </policyMap> </destinationPolicy> <!-- Define Virtual Destinations for Topics --> <!-- All topics will have a corresponding queue corresponding to the name pattern below where consumers can connect to in order to act as subscription groups. --> <destinationInterceptors> <virtualDestinationInterceptor> <virtualDestinations> <virtualTopic name=">" prefix="VTopic.*." /> </virtualDestinations> </virtualDestinationInterceptor> </destinationInterceptors> <!-- predefine Queues corresponding to Virtual topics so that no messages get lost due to unregistered consumers. --> <destinations> <!-- The Subscriber group App1 is the name of a queue that ALL instances of App1 would use to consume messages from the topic "X". All consumers using the same queue name act as ONE subscriber and the messages will be load balanced across the group. --> <!-- queue physicalName="VTopic.App1.X" / --> </destinations> <!-- Use the following to configure how ActiveMQ is exposed in JMX --> <managementContext> <managementContext createConnector="true"/> </managementContext> <!-- The store and forward broker networks ActiveMQ will listen to --> <networkConnectors> <!-- by default just auto discover the other brokers --> <!--<networkConnector name="default-nc" uri="multicast://default"/>--> <!-- Example of a static configuration: <networkConnector name="host1 and host2" uri="static://(tcp://host1:61616,tcp://host2:61616)"/> --> </networkConnectors> <!-- Using the Kaha DB store for persistence now --> <persistenceAdapter> <kahaDB directory="${activemq.base}/data/kahadb" /> </persistenceAdapter> <!-- Don't have ssl <sslContext> <sslContext keyStore="file:${activemq.base}/conf/broker.ks" keyStorePassword="password" trustStore="file:${activemq.base}/conf/broker.ts" trustStorePassword="password"/> </sslContext --> <!-- The maximum about of space the broker will use before slowing down producers --> <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="350 mb"/> </memoryUsage> <storeUsage> <storeUsage limit="20 gb" name="foo"/> <!-- used for persistent messages --> </storeUsage> <tempUsage> <tempUsage limit="100 mb"/> <!-- used for non persistent messages --> </tempUsage> </systemUsage> </systemUsage> <!-- The transport connectors ActiveMQ will listen to --> <transportConnectors> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/> </transportConnectors> </broker> <jetty:jetty> <connectors> <jetty:nioConnector port="8161"/> </connectors> <handlers> <jetty:webAppContext contextPath="/admin" resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/> </handlers> <userRealms> <bean class="org.mortbay.jetty.security.HashUserRealm"> <property name="name" value="ActiveMQ Realm" /> <property name="config" value="${activemq.base}/conf/webconsole.properties" /> </bean> </userRealms> </jetty:jetty> </beans>

Next you need to create the file webconsole.properties in the conf directory of ActiveMQ.The file contains one line per user with the format

useName: password, [roleName]*

For example:

myAdmin: THESECRET, amqAdmin

Finally you need to amend the web.xml files of the ActiveMQ web applications (${activemq.base}/webapps/*/WEB-INF/web.xml) with the following xml fragment:

<security-constraint> <web-resource-collection> <web-resource-name>A Protected Page</web-resource-name> <url-pattern>/*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>amqAdminrole-name> </auth-constraint> </security-constraint> <login-config> <auth-method>BASIC</auth-method> <realm-name>ActiveMQ Realm</realm-name> </login-config>

Make sure, the Realm name matches the one you have set in the Realm definition and the role name match roles you have named in the properties file.

Live from Apache Con in Amsterdam

I am writing this entry as I am sitting at Apache Con in Amsterdam. It is a really good conference with good and plenty information around evrything that is currently going on in the Apache community. In the middle of the conference Ade and I had 6 sessions in total where we discussed the activities of the FUSE team in the Apache community, especially focussing on the FUSE ESB (Apache ServiceMix), the FUSE Message Broker(Apache ActiveMQ) and the FUSE Services Framework (Apache CXF).

It was surprising to me the it was relatively unknown in the community that the FUSE of PROGRESS employs so many active committers and the conference was a god place to explain what we are doing. I have made my presentations available at the ApacheCon web site and we will put them on the FUSE web site as well.

Enjoy reading...

Better Diagnosis for ActiveMQ

Over the last couple of weeks I was involved in a number of projects where we had to understand how messages are being handled within an ActiveMQ broker. Browsing the ActiveMQ code I found the LoggingBrokerPlugin that did some rudimentary logging upon message sends.

That seemed to be a good point to have a little enhancement to ActiveMQ itself. ActiveMQ basically provides a framework for writing plugins that can hook into the message processing within the broker. Usually such a plugin extends the BrokerFilter class.

As there is a whole lot of events it made sense to categorize the events into certain classes to prevent the log files being flooded with unnecessary information. I have finally contributed a patch to ActiveMQ that provides this kind of functionality.

In a nutshell, the plugin can be attached to the broker via the normal ActiveMQ XML configuration :

<beans> <broker usejmx="true" deleteallmessagesonstartup="true" xmlns="http://activemq.org/config/1.0"> <transportconnectors> <transportconnector uri="tcp://localhost:61616" /> </transportconnectors> <plugins> <loggingbrokerplugin logconnectionevents="true" /> </plugins> </broker> </beans>

With the configuration above and a logger assigned to org.apache.activemq.broker.util.LoggingBrokerPlugin in the log4j settings you will
see log entries whenever a JMS client connects to or disconnects from the broker and when a session is created/closed by these clients. Other
event classes include transaction events, consumer events and producer events. The internal event group is some kind of a catch all group
that produces log entries for methods that are only of interest for debugging.

As a side note, the methods in the plugin also provide excellent entry points for debugging in to the broker behavior.

Limiting disk store usage with Active MQ

This week I have been onsite with Gary Tully, one of my colleagues from our Dublin office. We were visiting a customer having a very simple sounding requirement for Active MQ. His company contributes contents to various web pages and portals all over the internet and
he has a Tomcat infrastructure in place for providing the content. The next step is to note
whether a visitor to a site has recently visited other pages where he also provided some content.

Getting the information is quite simple by placing a cookie on the machine of the visitor so that the page visitor can be recognized again whenever he visits another page. Now whenever the visitor hits another page and we find the cookie, we also want to link that cookie with the newly hit URL. At the end of the day we would have a cookie related to a set of web pages. To keep the performance of the backend as high as possible, the web application shall only fire a notification of the correlation between cookie and web page and then deliver the content as fast as possible. The database maintenance should be done asynchronously.

The asynchronus link between the web application and the database is realized using the Fuse release of Active MQ 5.2. One requirement that sounded very odd in the beginning was to use persistent messages, but at the same time allow to drop messages in cases of heavy load. Translated that basically means to gather as much statistical data as possible, but it doesn't matter if we can't keep up with the load entirely.

Now what has this to do with messaging and ActiveMQ? - Well, we have a quite typical slow consumer problem here. We are producing messages much faster than we can consume them, because creating a message is more or less a string concatenation while consuming the message involves a database insert. As a result any buffer - on disk or in memory - might be filled up during longer periods of high traffic. The normal messaging behavior - also with ActiveMQ - is to slow down the message producer so that the consumers get a chance to catch up and ensure that no message is lost. In our case that would mean that the response times for delivering the content would decline, which should not happen. The solution we are looking for is to give the message broker a well defined amount of space for buffering messages and simply throw back an exception when the buffer is full.

That sounds easy enough and we were thinking along the lines of putting in place a proper system usage section in the broker configuration. We started looking at the default activemq.xml that is provided with an ActiveMQ download and in there we found the following section:

<systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="20 mb"/> </memoryUsage> <storeUsage> <storeUsage limit="1 gb" name="foo"/> </storeUsage> <tempUsage> <tempUsage limit="100 mb"/> </tempUsage> </systemUsage> </systemUsage>

The memory usage is easy enough to understand and from Gary I learned, that the temporary storage is used for spooling out messages that have been sent NON_PERSISTENT while the storage is used for those messages that have been sent PERSISTENT. As we were sending only persistent messages to keep as much of the statistical data as possible even in case of broker restarts we simply modified the storage section.

To turn the behavior in case of reaching the buffer limits, the systemUsage xbean understands a boolean property sendFailIfNoSpace. Setting this to true should simply throw a JMSException instead of blocking the send call.

So we made those changes and they were happily ignored by our ActiveMQ broker. It took us a bit to understand why that was the case. The bottom line is that the system usage as seen above is not associated with a persistence adapter and therefore it is not associated with a store either. When the ActiveMQ broker is started, we saw a persistence adapter and a store being initialized, but we learned that the broker creates a default persistence adapter if none is in the configuration. This default one is not referenced by the storage settings.

As a result, it is important to explicitly specify a persistence adapter within the configuration file and reference it from the broker and the storage definition as shown in the configuration below.

<beans> xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd"> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/> <bean id="store" class="org.apache.activemq.store.amq.AMQPersistenceAdapter" > <property name="directory" value="target/amqdata" /> <property name="maxFileLength" value="1000000" /> <property name="checkpointInterval" value="5000" /> <property name="cleanupInterval" value="5000" /> </bean> <broker xmlns="http://activemq.apache.org/schema/core" persistent="true" advisorySupport="false" dataDirectory="target/amqdata" deleteAllMessagesOnStartup="true" useJmx="true" brokerName="localhost" monitorConnectionSplits="false" splitSystemUsageForProducersConsumers="false" start="false" persistenceAdapter="#store"> <!-- Use the following to configure how ActiveMQ is exposed in JMX --> <managementContext> <managementContext createConnector="false"/> </managementContext> <!-- The maximum about of space the broker will use before slowing down producers --> <systemUsage> <systemUsage sendFailIfNoSpace="true" > <memoryUsage> <memoryUsage limit="400kb" /> </memoryUsage> <storeUsage> <storeUsage limit="10mb" store="#store" /> </storeUsage> <tempUsage> <tempUsage limit="64mb" /> </tempUsage> </systemUsage> </systemUsage> <!-- The transport connectors ActiveMQ will listen to --> <transportConnectors> <transportConnector name="openwire" uri="tcp://localhost:0" /> </transportConnectors> </broker> </beans>

One caveat in the set up is to make sure that the maximumFileLength defined in the persistence adapter is less than half the maximum storage size. Otherwise only one data file is used by the underlying store and that will never get cleaned up. Having maximumFileLength set correctly ensures that we will have at least 2 data files.

When we finally were on our way home we decided that we need to get our findings in writing so that we could reference it in the future. I am working with Gary to get the test cases we produced and the configuration files we created into the ActiveMQ project.

Going Live at Apache Con EU 2009

After I have settled in in my new role at Progress within the Open Source Center of Competence I will "go live" in that new role at ApacheCon Europe 2009 in Amsterdam with 3 presentations.

In a session called Distributed Team Building we will have a closer look at developing a distributed application based on an Enterprise Service Bus. As the components and services are developed quite independently from each other,the specification for each component and it's documentation must be fit for retrieval and reuse. New challenges for development teams include knowledge sharing and knowledge reuse as well as shorter and more agile development iterations. Teams also have to overcome their fears when it comes down to changing their way of working. We will have a look at some strategies to address those challenges and overcome the fears in order to compose a team working successfully in an agile and distributed environment.

Another session called Servicemix Topologies has been inspired by the work I have done for a large retail chain in Europe. For that company I had to deploy round about 2000 containers collaborating with each other across eastern Europe. So in this talk we will look at some deployment topologies for ESB applications based on Apache ServiceMix. To illustrate the topologies we will deploy a sample application into various constellations and discuss the pro's and con's of each approach. Especially we will look at fail over and load balancing capabilities, throughput, deployment complexity and discuss the packaging of an application feasible for distributed deployment. Finally we will come up with a potential approach for deploying such a distributed retail application with Apache ServiceMix.

Finally we will turn to the topic of Tooling for ServiceMix where we will walk through the development and deployment of a sample application based on Apache Service Mix 4. After a short overview of the Service Mix 4 architecture and principles we will use the Eclipse based FUSE Integration Designer to design and implement a sample workflow. We will then package this workflow up into OSGi bundles and demonstrate how to deploy those via a central Maven repository into a given Service Mix instance instance.

Hope to see you in Amsterdam.

Start of the adventure

Hello,

after having read Blogs on the web for a while I am now down to writing my own. I thought it might be nice to introduce myself first before I get into the topic of Open Source adventures.

My name is Andreas Gies and I currently work as a Principal Architect for PROGRESS Software. As some of you may know, PROGRESS recently acquired IONA Technologies. With that acquisition a large portion of open source technologies came to the footsteps of PROGRESS Software and not surprisingly a section of our consulting force will concentrate on those technologies.

Before the acquisistion I was working mainly with the PROGRESS ESB and a specialist for large scale deployments. The tool of choice to manage project complexity for me was Maven and over the last 3 years I have spent a lot of time writing a suite of Maven plugins addressing various topics in that area: packaging applications fit for distributed deployments and deploying the applications to their distributed runtimes using a "Model driven deployment" approach.

As Maven has brought me to the side of Open Source and I knew all the IONA guys from my past (I was at IONA before joining PROGRESS) it was only natural to become a member of the team dedicated to the Open Source Technologies.

Given my background on ESB based architectures, middleware and deployment I will concentraIn my Blog foremost on Apache ServiceMix, ActiveMQ with it's most interesting Camel subproject and deployment strategies. The start will be slow as I need to get my head around those projects first and I hope readers new to those technologies will benefit from that. As ServiceMix and I get along better and better, may be we can address more advanced topics as well.