Cometd Cluster with LoadBalancing

Cometd cluster

Motivation

When we run Stress Testing Cometd (Jetty implementation) http://docs.codehaus.org/display/JETTY/Stress+Testing+Cometd we see that if we connect many clients, deliver message latency tends to grow up. This situation described in article "20,000 Reasons Why Comet Scales". So, in order to support a lot of concurrent cometd connections we need a mean to horizontally scale eXo cometd support.

eXo cometd clustering

1- Getting base URL for cometd connection (one of the node in cometd cluster)

2 - Getting userToken at cometd server (it need for subscribing on channel)

3 - Sending message from Exo-server

LoadBalancer - component that give base URL of one of free cometd server.

ContinuationServiceDelegate - component that send message from Exo-server to client via cometd server there client is registered

RESTContinuationService - component that receive message from ContinuationServiceDelegate and delegate it to ContinuationServer and send userToken generated by ContinuationServer for user.

ContinuationService - component that provide cometd connection.

How it works

To start work with cometd service the client should send request to Exo-server with the username, receive with response URL one of cometd servers (1 - on the scheme). This URL gives out LoadBalancer using the information set in a configuration

<init-params>
  <object-param>
  <name>cometd.lb.configuration</name>
  <description>cometd lb nodes</description>
  <object type="org.exoplatform.ws.frameworks.cometd.loadbalancer.LoadBalancerImpl$LoadBalancerConf">
   <field name="nodes">
    <collection type="java.util.ArrayList">
     <value>
      <object type="org.exoplatform.ws.frameworks.cometd.loadbalancer.Node">
       <field name="id">
        <string>1</string>
       </field>
       <field name="url">
        <string>http://localhost:8080</string>
       </field>
       <field name="maxConnection">
        <int>10</int>
       </field>
      </object>
     </value>
     <value>
      <object type="org.exoplatform.ws.frameworks.cometd.loadbalancer.Node">
       <field name="id">
        <string>2</string>
       </field>
       <field name="url">
        <string>http://localhost:8081</string>
       </field>
       <field name="maxConnection">
        <int>15</int>
       </field>
      </object>
     </value>
    </collection>
   </field>
  </object>
 </object-param>
</init-params>

In the above configuration we described two cometd-cluseter nodes of. Let's consider one of them.

1 - are unique the identifier of node (id) http://localhost:8080 - base URL 15 - the maximum number of the connected clients. LoadBalancer chooses one of node in cometd cluster (on which connection is possible) and associate with a name of the user!
Now the client know on that cometd server connection is possible, for this purpose it is necessary to request userToken on cometd server(2). As for this need do request on other domain client must use framework that allowed cross-domain-ajax. We have framework that can do this task, how use this framework describe in article - Framework for cross-domain AJAX. After client receive userToken the client can do cometd-registration.
Example:
the Client with a name exo1 wishes to be connected to cometd to service. It is necessary to make HTTP request (GET) on Exo-server URL = "http://localhost:8080/rest/cometdurl/exo1" with the answer receives a base URL of cometd server, it can be as a remote server or local. We will assume that we use cluster and receive some thing like this "http://192.168.0.21:8080". Further it is necessary get userToken already from cometd server for this purpose do request (GET) URL ="http://192.168.0.22:8081/rest/gettoken/exo1", after that we do standard procedure for cometd connections.

Then it is necessary to send the message to client ContinuationServiceDelegate requests at LoadBalancer the cometd server address on which the client is registered sends the message in format JSON on cometd server should be RESTContinuationService which to accept the message and to transfer them to ContinuationService (3)
Example:
we will want to send the message to the client exo1. For this purpose ContinuationServiceDelegate requests to LoadBalancer the information on that to what server the given user is connected, we will receive necessary URL http://192.168.0.22:8081) and do HTTP request (POST) on comets a server, URL = "http://192.168.0.22:8080/rest/sendprivatemessage/" in a body the message in format JSON is transferred, RESTContinuationService having received the given message transfers it ContinuationService which already in turn delivers it to the client.

<configuration>
 <component>
  <type>org.exoplatform.ws.frameworks.cometd.ContinuationService</type>
 </component>
 <component>
  <key>org.mortbay.cometd.continuation.AbstractBayeux</key>
  <type>org.mortbay.cometd.continuation.EXoContinuationBayeux</type>
 </component>
 <component>
  <key>org.exoplatform.ws.frameworks.cometd.transport.ContinuationServiceDelegate</key>
  <type>org.exoplatform.ws.frameworks.cometd.transport.ContinuationServiceRemoteDelegate</type>
 </component>
 <component>
  <type>org.exoplatform.ws.frameworks.cometd.transport.RESTContinuationService</type>
 </component>
 <component>
  <type>org.exoplatform.ws.frameworks.cometd.loadbalancer.RESTLoadBalancerService</type>
 </component>
 <component> 
  <key>org.exoplatform.ws.frameworks.cometd.loadbalancer.LoadBalancer</key>
  <type>org.exoplatform.ws.frameworks.cometd.loadbalancer.LoadBalancerImpl</type>
  <init-params>
  <object-param>
  <name>cometd.lb.configuration</name>
  <description>cometd lb nodes</description>
  <object type="org.exoplatform.ws.frameworks.cometd.loadbalancer.LoadBalancerImpl$LoadBalancerConf">
   <field name="nodes">
    <collection type="java.util.ArrayList">
     <value>
      <object type="org.exoplatform.ws.frameworks.cometd.loadbalancer.Node">
       <field name="id">
        <string>1</string>
       </field>
       <field name="url">
        <string>http://localhost:8080</string>
       </field>
       <field name="maxConnection">
        <int>10</int>
       </field>
      </object>
     </value>
     <value>
      <object type="org.exoplatform.ws.frameworks.cometd.loadbalancer.Node">
       <field name="id">
        <string>2</string>
       </field>
       <field name="url">
        <string>http://localhost:8081</string>
       </field>
       <field name="maxConnection">
        <int>15</int>
       </field>
      </object>
     </value>
    </collection>
   </field>
  </object>
 </object-param>
</init-params>
 </component>
</configuration>

The test:
At testing running two servlet-containers Tomcat (in role Exo - server'a) and Jetty (cometd server).
A configuration example:

<configuration clients="12" repeat="1" sleep-connection="500" sleep-sending="200">
  <container containerStart="false" port="8080" home=""/>
  <messages>
    <message broadcast="false" id="1">hello</message>
    <message broadcast="true" id="2">hello!!!</message>
  </messages>
  <cometd-url>http://localhost:8080/cometd/cometd</cometd-url>
  <base-url>http://localhost:8080/rest/</base-url>
  <channels>
    <channel>/eXo/comedt/test</channel>
  </channels>
</configuration>

configuration describe that we create 12 cometd connections with sleep-connection = "500" (мс), will be subscribe on the channel "/eXo/comedt/test". Then it will be sent two message "hello" "hello!!!" the First individually to each of the clients, the second broadcast on the channel. Messages delivered with an interval sleep-sending = "200".
http://localhost:8080/rest/ - base URL of Exo - server. For start test execute a command:

mvn clean install -f pom-test.xml -Dexo.test.skip=false -Djetty.home = "./target/jetty"

Bench

See Cometd Cluster Bench