The objective of this article is to explain the ESB Performance Test Framework and the tests carried out in Round 4 in detail. Understanding this article will help develop the necessary configurations to compare the performance of any ESB using this test framework.
Update: This document and results were based on the 1.0-beta-2(b3) release of the UltraESB from February 2010. After the beta user feedback was received, AdroitLogic has updated the configuration syntax slightly. We will provide an updated performance benchmark round, along with a pre-built AMI and incorporate the performance configuration into the release itself to enable smoother performance testing rounds in future.
Update: This document is now superseded by http://esbperformance.org and will remain only for historical purposes
Introduction
The ESB Performance Test Framework [PTF] has been used since June 2007 to compare the performance of Proprietary ESBs and Open Source ESBs - and has been used by vendors such as Mule, WSO2 and BEA in the past, to publish benchmark results against their implementations. However, as different vendors used different hardware configurations, the test results couldn't be really considered on an equal plane. The PTF used in this Round 4 has been developed to be executed on the Amazon EC2 environment - so that end users could request for the configurations for the ESB of their choice from the vendors/development teams, and run the tests themselves in a fair environment.
The Performance Test Framework [PTF]
The PTF is based on a simple deployment diagram, where a load generation client is used to simulate 20, 40, 80, 160, 320, 640, 1280 and 2560 concurrent users making requests of 512 bytes, 1K, 5K, 10K and 100K requests from the ESB. The ESB will perform processing as described below - for each of the scenarios - and forward the request to a backend service, that will respond by echoing back the same request. The ESB will then perform any processing of the response message, and forward it to the client. The load generation client, ESB and the backend service are all hosted on the same instance - to minimize any effect of actual network performance in a cloud computing environment.
The request messages used are of the form shown below:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<m:buyStocks xmlns:m="http://services.samples/xsd">
<order><symbol>IBM</symbol><buyerID>asankha</buyerID><price>140.34</price><volume>2000</volume></order>
<order><symbol>MSFT</symbol><buyerID>ruwan</buyerID><price>23.56</price><volume>8030</volume></order>
....
</m:buyStocks>
</soapenv:Body>
</soapenv:Envelope>
The PTF includes a set of shell scripts to start the Load Generation client and these could be found at [2]. The script "loop.sh" invokes the Java client, passing the parameters from the main script "r3.sh". The main script should be passed the root URL where the test services are exposed over the ESB. For the UltraESB this would be of the form "./r3.sh http://localhost:8280/service > ue-r3.txt" so that the output is captured into a text file. Note that the script expects the UltraESB 1.0-beta2-b3 to be installed at the same level with these scripts made available in a directory named "LoadGenerator"
e.g.
LoadGenerator/loop.sh
LoadGenerator/r3.sh
ultraesb-1.0-beta-2/bin/toolbox.sh
ultraesb-1.0-beta-2/bin/...
.....
The PTF includes a Java NIO based Echo service - capable of handling 2560 or more concurrent users with just a few threads. This service is exposed by default at the following URL: http://localhost:9000/service/EchoService. To start this service, execute the "./toolbox.sh -echo" from the bin directory of the UltraESB.
The PTF and the sample requests are included in the UltraESB download (of approximately ~25MB) from http://adroitlogic.org
Test Scenarios
The PTF currently includes 4 scenarios as follows:
1. Direct Proxy Service
This scenario will demonstrate the ability of the selected ESB to act as a virtualization layer for backend web services - similarly to how the Apache Web Servers or Hardware Load balancers were used in front of web backends to virtualize and balance load and provide failo-ver capabilities. Although this scenario does not include load balancing, fail-over, security verification or schema validation etc, most of the ESBs does support these features.
This scenario assumes the service to be a SOAP Web Service, and thus expects the ESB to present the WSDL for the service to its clients. The WSDL used for the Proxy service maybe found at "UltraESB/conf/resources" as "ProxyWSDL-embedded.wsdl" within the configuration and resource bundle [1]
Requirements:
- The Proxy service MUST be available over http at a URL of the form: http://localhost:<port>/<path>/DirectProxy
- The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost:<port>/<path>/DirectProxy?wsdl
- The Proxy service MAY present a custom error reply, if there was an error
2. The Content Based Routing (CBR) Proxy
This scenario will demonstrate the abilty of the selected ESB to route an XML payload based on the evaluation of an XPath expression over the payload body. Typically this pattern is used in practice to route request messages to different endpoints, or handle them differently at the ESB based on the message attributes - such as the payload, or transport headers etc.
This test scenario expects the first order elements' symbol to be equal to "IBM" to forward the request to the backend service. On a failure, its expected to return a SOAP fault to the client with an error message. The WSDL used is the same as per scenario #1
Requirements:
- The Proxy service MUST be available over http at a URL of the form: http://localhost:<port>/<path>/CBRProxy
- The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost:<port>/<path>/CBRProxy?wsdl
- The Proxy service MUST present a custom error reply if the routing condition is not satisfied by the request
3. The XSL Transformation Proxy
This scenario will demonstrate the ability of the selected ESB to transform XML payload messages (such as SOAP) between different schemas. In practice transformations are used to convert messages from XML to CSV or Text etc, or to change a request adhering to one version of a schema to a different version (for backwards compatibility of services/clients), or to create messages that are enriched with some attributes from the original request, and other attributes picked up during mediation of the message - for example from a Database.
This test expects the transformation to convert the original message to the format shown below - by reversing the element names. Thus, the Echo service backend will echo back this reverse message, which is transformed again to the original format and sent to the client. The XSLT files used could be found at "UltraESB/conf/resources" as "transform_env_reverse.xslt" and "transform_env.xslt" within the configuration and resource bundle [1]. The WSDL used is the same as per scenario #1
<soapenv:Envelope xmlns:m0="http://services.samples/xsd" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<m:skcotSyub xmlns:m="http://services.samples/xsd">
<redro><lobmys>IBM</lobmys><DIreyub>asankha</DIreyub><ecirp>140.34</ecirp><emulov>200000</emulov></redro>
<redro><lobmys>MSFT</lobmys><DIreyub>ruwan</DIreyub><ecirp>23.56</ecirp><emulov>803000</emulov></redro>
...
</m:skcotSyub></soapenv:Body></soapenv:Envelope>
Requirements:
- The Proxy service MUST be available over http at a URL of the form: http://localhost:<port>/<path>/XSLTProxy
- The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost:<port>/<path>/XSLTProxy?wsdl
- The Proxy service MAY present a custom error reply, if there was an error
4. WS-Security Proxy
This scenario will demonstrate the ability of the selected ESB to act as a Security Gateway. Typically an ESB is used to validate and verify security credentials, and establish the authenticity of messages, before passing them through mediation and to internal services within a corporate environment. Although this test case considers WS-Security, many SOA deployments rely on SSL, HTTP Basic and Digest authentication and other mechanisms as well.
This test expects the client to send a timestamped, digitally signed and encrypted request, and verifies the message and forwards the actual payload without the WS-Security header to the backend service. Once the Echo service responds back with the same message, the ESB timestamps, digitally signs and encrypts the response and sends it back to the client.
The test suite has bundled a set of requests that wrap the original requests with a timestamp valid for 10 years. These requests are found with file names of the form "<size>_buyStocks_secure.xml". Thus, the Apache Bench style load generator can very efficiently push a load of secured messages for a valid performance test of the ESB. The messages are secured using the standard WS-Security interoperability test suite keystore found at "UltraESB/conf/resources" within the configuration and resource bundle [1]. The requests are signed as the user "alice" and encrypted for user "bob", and the passwords for the keystore as well as "alice" and "bob" is "password" - as per the standard WS-Security interoperability test cases.
Requirements:
- The Proxy service MUST be available over http at a URL of the form: http://localhost:<port>/<path>/SecureTProxy
- The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost:<port>/<path>/SecureProxy?wsdl
- The Proxy service MAY present a custom error reply, if there was an error
Developing a configuration to test an ESB
To develop a configuration that will be able to load test an ESB of your choice, request your ESB vendor or the open source project team for suitable and optimized configuration files that will satisfy the above scenarios. Usually the configuration will be made available as a bundle of files similar to the UltraESB bundle [1]. Then setup your ESB on the Amazon EC2 node or your local test environment - with the necessary JDK etc, and adjust the heap memory to be consistent across the ESBs you select. For Round 4 on EC2, we selected an "m1.large" instance of the AMI "ami-eef61587" with 7.5G of RAM, and allocated a heap size of 2G to the ESB's tested. The system was running Ubunu server 9.04 64 bit, and the JDK was the Sun JDK 1.6.0_18 64 bit.
The UltraESB configuration files for Round 4
This section of the document lists the UltraESB configuration file and explains the different components making up the configuration. The UltraESB is driven from a standard Spring configuration, and includes custom extension elements as shown below.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:u="http://www.adroitlogic.org/ultraesb"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-2.0.1.xsd
http://www.adroitlogic.org/ultraesb http://schemas.ultraesb.org/ultraesb-1.0.xsd">
The above lines sets up the Spring configuration, by declaring the namespaces and schemas, and the UltraESB specific extension elements - so that an intelligent IDE could help with auto completion and popup of available configuration options. (See the Getting Started guides for more details)
<u:endpoint id="RealService">
<u:address>http://localhost:9000/service/EchoService</u:address>
</u:endpoint>
The above defines the external Echo service endpoint with a friendly name "RealService" which is used in the subsequent elements
<u:proxy id="DirectProxy">
<u:transport id="http-8280">
<u:property name="wsdlURL" value="file:conf/resources/ProxyWSDL-embedded.wsdl"/>
</u:transport>
<u:target inDestination="RealService">
<u:outDestination>
<u:address type="response"/>
</u:outDestination>
<u:errorSequence>
<u:java><![CDATA[
Mediation.setPayloadToSOAP11Fault(msg, null, "Error", null);
Mediation.sendResponse(msg, 500);
]]></u:java>
</u:errorSequence>
</u:target>
</u:proxy>
This is the full configuration of the Direct Proxy when using the UltraESB. The 'errorSequence' element is optional - and is used to demonstrate how an error handler could be specified in-case things go wrong during message processing. The Proxy Service is identified as "DirectProxy" and the WSDL the user wishes the UltraESB to expose is specified as the "wsdlURL" property of the transport selected. The transport definition for "http-8280" could be found in the "ultra-root.xml" configuration file - which holds other definitions. The inDestination specifies that the incoming message should be sent to the default target destination "RealService". The response received from that service is then processed by the "outDestionation" - which will send the message back to the default response destination - i.e. the original client of the ESB.
<u:proxy id="CBRProxy">
<u:transport id="http-8280">
<u:property name="wsdlURL" value="file:conf/resources/ProxyWSDL-embedded.wsdl"/>
</u:transport>
<u:target>
<u:inSequence>
<u:java><![CDATA[
if (Mediation.filter(msg, "//order[1]/symbol", null, "IBM")) {
Mediation.sendToEndpoint(msg, "RealService");
} else {
Mediation.setPayloadToSOAP11Fault(msg, null, "First order must be for the symbol IBM", null);
Mediation.sendResponse(msg, 500);
}
]]></u:java>
</u:inSequence>
<u:outDestination>
<u:address type="response"/>
</u:outDestination>
</u:target>
</u:proxy>
This is the definition of the CBR Proxy service "CBRProxy". When a message is received, the 'inSequence' processes this message, and the UltraESB public API methods could be used to easily processes the message in any supported language - such as Java, Javascript, Ruby, Groovy or any other JSR-223 scripting language. The sequence shown above uses the Java syntax, and filters the message by evaluating the XPath result against the regular expression "IBM". If the message matches the condition, it is sent to the "RealEndpoint", and if not, a SOAP fault is returned with a message of choice.
<u:proxy id="XSLTProxy">
<u:transport id="http-8280">
<u:property name="wsdlURL" value="file:conf/resources/ProxyWSDL-embedded.wsdl"/>
</u:transport>
<u:target inDestination="RealService">
<u:inSequence>
<u:java><![CDATA[
Mediation.transform(msg, "conf/resources/transform_env_reverse.xslt", null, null);
]]></u:java>
</u:inSequence>
<u:outSequence>
<u:java><![CDATA[
Mediation.transform(msg, "conf/resources/transform_env.xslt", null, null);
]]></u:java>
</u:outSequence>
<u:outDestination>
<u:address type="response"/>
</u:outDestination>
</u:target>
</u:proxy>
The "XSLTProxy" service definition above specifies an 'inSequence' and 'outSequence' that uses the two XSLT transformations over the message before being sent to the "RealService" endpoint, and before the response is being returned to the original client.
<u:proxy id="SecureProxy">
<u:transport id="http-8280">
<u:property name="wsdlURL" value="file:conf/resources/ProxyWSDL-embedded.wsdl"/>
</u:transport>
<u:target inDestination="RealService">
<u:inSequence>
<u:java import="org.adroitlogic.soapbox.*;"><![CDATA[
try {
WSSecurityManager wssecMgr = (WSSecurityManager) Mediation.getSpringBean("wssecMgr");
wssecMgr.verifyTimestampedEncryptedAndSignedMessage(msg, true);
} catch (Exception e) {
Mediation.setPayloadToSOAP11Fault(msg, null, "Security validation failed : " + e.getMessage(), null);
Mediation.sendResponse(msg, 500);
}
]]></u:java>
</u:inSequence>
<u:outSequence>
<u:java import="org.adroitlogic.soapbox.*;"><![CDATA[
WSSecurityManager wssecMgr = (WSSecurityManager) Mediation.getSpringBean("wssecMgr");
wssecMgr.timestampSignAndEncryptMessage(msg, "bob", "alice");
]]></u:java>
</u:outSequence>
<u:outDestination>
<u:address type="response"/>
</u:outDestination>
</u:target>
</u:proxy>
The "SecureProxy" service makes use of the WSSecurityManager instance declared below - and verifies the security aspects of the message that is timestamped, signed and encrypted. On successful processing of the security header, it is specified that the security header be removed. If there is a failure, a custom SOAP fault is returned to the client - possibly hiding the real cause since the error is a security exception. The unsecured payload is then forwarded to the "RealService" and the response received is then timestamped, signed and encrypted using the credentials of "bob" and "alice" and sent back to the client.
The WSSecurityManager instance defines the security trust keystore and an optional identity keystore (in this case we use the same ksystore as both), and specifies the password for the keystore and the other credentials used. Note that the password shown here in plain text can be easily encrypted for a production deployment as per this article.
<bean id="wssecMgr" class="org.adroitlogic.soapbox.WSSecurityManager">
<constructor-arg value="conf/resources/store.jks"/>
<constructor-arg value="password"/>
<constructor-arg>
<map>
<entry key="alice" value="password"/>
<entry key="bob" value="password"/>
</map>
</constructor-arg>
</bean>
</beans>
Resources
[1] The configuration files and resources - http://downloads.adroitlogic.com/performance/round4/ue.tar
[2] The Load Generator scripts -http://downloads.adroitlogic.com/performance/round4/scripts.tar
[3] ESB Performance Testing Round 4 - http://adroitlogic.org/samples-articles-and-tutorials/15-tutorials/48-esb-performance.html