This tutorial will show you how to configure and initialize a JCR cluster in persistent mode for any eXo product using the automatic remote initializer mechanism.
1 Introduction
Since version 1.11.1 there is a new mechanism available to automate initialization of participants nodes in a JCR cluster.
Used terms:
- master node : the node which is used to initialize other nodes.
- initialized node : the node which initializes itself form master node.
The mechanism is based on two key components :
- RemoteWorkspaceInitializationService : allows other nodes to initialize themselves against the master node.
- RemoteWorkspaceInitializer: uses the RemoteWorkspaceInitializationService to retrieve workspace data from master node.
1.1 Configuring RemoteWorkspaceInitializationService
The
RemoteWorkspaceInitializationService should be configured on both
master node and
initialized node. It uses a JGroups channel to send the initialization data between
master and
initialized nodes.
On master node
You need to add component
org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializationService to services configuration and define the channel name and JGroups configuration for that channel.
<component>
<type>org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializationService</type>
<init-params>
<properties-param>
<name>remote-initializer-properties</name>
<property name="remote-source-url" value="http://root:exo@192.168.0.15:8080"/>
<property name="bind-ip-address" value="192.168.0.15"/>
<property name="channel-config" value="TCP(start_port=7700;oob_thread_pool.queue_max_size=100;thread_naming_pattern=cl;use_concurrent_stack=true;oob_thread_pool.rejection_policy=Run;discard_incompatible_packets=true;thread_pool.max_threads=40;oob_thread_pool.enabled=true;oob_thread_pool.max_threads=20;loopback=false;oob_thread_pool.keep_alive_time=5000;thread_pool.queue_enabled=false;oob_thread_pool.queue_enabled=false;max_bundle_size=64000;thread_pool.queue_max_size=100;thread_pool.enabled=true;enable_diagnostics=true;max_bundle_timeout=30;oob_thread_pool.min_threads=8;use_incoming_packet_handler=true;thread_pool.rejection_policy=Run;bind_addr=$bind-ip-address;thread_pool.min_threads=8;thread_pool.keep_alive_time=5000;enable_bundling=true):MPING(timeout=2000;num_initial_members=8;mcast_port=35526;mcast_addr=224.0.0.1):FD(timeout=2000;max_tries=5;shun=true):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(max_xmit_size=60000;print_stability_history_on_failed_xmit=true;use_mcast_xmit=false;gc_lag=0;discard_delivered_msgs=true;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=8000000):pbcast.GMS(print_local_addr=true;join_timeout=3000;view_bundling=true;join_retry_timeout=2000;shun=true;merge_leader=true;reject_join_from_existing_member=true)"/>
<property name="channel-name" value="Remote_Initializer"/>
<property name="temp-dir" value="../temp/remote-initializer"/>
</properties-param>
</init-params>
</component>
Where :
- bind-ip-address - the IP address on local network interface.
- channel-name - prefix the channel name, should be the same on master node and initialized node
- temp-dir - the path to temporary folder.
On initialized node
You also need to configure the
org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializationService component and reference the URL of master node.
<component>
<type>org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializationService</type>
<init-params>
<properties-param>
<name>remote-initializer-properties</name>
<property name="remote-source-url" value="http://root:exo@192.168.0.15:8080"/>
<property name="bind-ip-address" value="192.168.0.135"/>
<property name="channel-config" value="TCP(start_port=7700;oob_thread_pool.queue_max_size=100;thread_naming_pattern=cl;use_concurrent_stack=true;oob_thread_pool.rejection_policy=Run;discard_incompatible_packets=true;thread_pool.max_threads=40;oob_thread_pool.enabled=true;oob_thread_pool.max_threads=20;loopback=false;oob_thread_pool.keep_alive_time=5000;thread_pool.queue_enabled=false;oob_thread_pool.queue_enabled=false;max_bundle_size=64000;thread_pool.queue_max_size=100;thread_pool.enabled=true;enable_diagnostics=true;max_bundle_timeout=30;oob_thread_pool.min_threads=8;use_incoming_packet_handler=true;thread_pool.rejection_policy=Run;bind_addr=$bind-ip-address;thread_pool.min_threads=8;thread_pool.keep_alive_time=5000;enable_bundling=true):MPING(timeout=2000;num_initial_members=8;mcast_port=35526;mcast_addr=224.0.0.1):FD(timeout=2000;max_tries=5;shun=true):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(max_xmit_size=60000;print_stability_history_on_failed_xmit=true;use_mcast_xmit=false;gc_lag=0;discard_delivered_msgs=true;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=8000000):pbcast.GMS(print_local_addr=true;join_timeout=3000;view_bundling=true;join_retry_timeout=2000;shun=true;merge_leader=true;reject_join_from_existing_member=true)"/>
<property name="channel-name" value="Remote_Initializer"/>
<property name="temp-dir" value="../temp/remote-initializer"/>
</properties-param>
</init-params>
</component>
Where :
- remote-source-url - the url to master node, in format http(s)//login:password@host:port/ .
- bind-ip-address - the IP address on local network interface.
- channel-name - prefix the channel name, should be the same on master node and initialized node.
- temp-dir - the path to temporary folder.
Additionally, you must configure the
RemoteWorkspaceInitializer in the
block of all workspaces of the JCR configuration.
For example, (repository-configuration.xml):
...
<workspaces>
<workspace name="system" ... >
<container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
...
</container>
<initializer class="org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializer" />
...
</workspace>
<workspace name="collaboration" ... >
<container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
...
</container>
<initializer class="org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializer" />
...
</workspace>
<workspace name="backup" ... >
<container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
...
</container>
<initializer class="org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializer" />
...
</workspace>
</workspaces>
2 Tutorial environment
2.1 Database
We will use an Oracle database. The JDBC URL is : jdbc:oracle:thin:@tornado.exoua-int:1523:orcl
- exocluster1
- exocluster2
- exocluster3
- exocluster4
Learn more about Database Configuration.
2.2 JCR settings
This tutorial was written and tested using JCR 1.11.1.
2.3 Cluster shape
The cluster has 4 participant nodes each is bound to its own IP address:
- cluster_node1 : 192.168.0.15
- cluster_node2 : 192.168.0.135
- cluster_node3 : 192.168.0.5
- cluster_node4 : 192.168.0.3
Warning: Ensure that all participants are time-synchronized with the same NTP-server for example. This is a common requirement for most clustering solutions
2.4 Get an eXo tomcat product bundle
Get and install the eXo tomcat product bundle from the download area.
Alternatively, you can always build the product you want from sources using eXoBuild tool?.
Extract the the eXo tomcat product bundle to D:\java\exo-working\exo-tomcat.
3 First cluster node configuration and startup.
3.1 Configuration of replication
The following contents add to services configuration:
<component>
<type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
<init-params>
<value-param>
<name>force-xml-configuration</name>
<value>true</value>
</value-param>
<values-param>
<name>repositories</name>
<value>repository</value>
</values-param>
<properties-param>
<name>replication-properties</name>
<property name="enabled" value="true"/>
<property name="mode" value="persistent"/>
<property name="bind-ip-address" value="192.168.0.15"/>
<property name="channel-config" value="TCP(oob_thread_pool.queue_max_size=100;thread_naming_pattern=cl;use_concurrent_stack=true;oob_thread_pool.rejection_policy=Run;discard_incompatible_packets=true;thread_pool.max_threads=40;oob_thread_pool.enabled=false;oob_thread_pool.max_threads=20;loopback=false;oob_thread_pool.keep_alive_time=5000;thread_pool.queue_enabled=false;oob_thread_pool.queue_enabled=false;max_bundle_size=64000;thread_pool.queue_max_size=100;thread_pool.enabled=false;enable_diagnostics=true;max_bundle_timeout=30;oob_thread_pool.min_threads=8;use_incoming_packet_handler=true;thread_pool.rejection_policy=Run;bind_addr=$bind-ip-address;thread_pool.min_threads=8;thread_pool.keep_alive_time=5000;enable_bundling=true):MPING(timeout=2000;num_initial_members=8;mcast_port=34526;mcast_addr=224.0.0.1):FD(timeout=2000;max_tries=5;shun=true):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(max_xmit_size=60000;print_stability_history_on_failed_xmit=true;use_mcast_xmit=false;gc_lag=0;discard_delivered_msgs=true;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=8000000):pbcast.GMS(print_local_addr=true;join_timeout=3000;view_bundling=true;join_retry_timeout=2000;shun=true;merge_leader=true;reject_join_from_existing_member=true)"/>
<property name="recovery-dir" value="../temp/replication/recovery"/>
<property name="node-name" value="cluster_node1"/>
<property name="other-participants" value="cluster_node2;cluster_node3;cluster_node4"/>
<property name="wait-confirmation" value="2000"/>
</properties-param>
<properties-param>
<name>replication-priority-properties</name>
<property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
<property name="node-priority" value="100"/> <!-- max == 100 -->
</properties-param>
</init-params>
</component>
<component>
<type>org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializationService</type>
<init-params>
<properties-param>
<name>remote-initializer-properties</name>
<property name="remote-source-url" value="http://root:exo@192.168.0.15:8080"/>
<property name="bind-ip-address" value="192.168.0.15"/>
<property name="channel-config" value="TCP(start_port=7700;oob_thread_pool.queue_max_size=100;thread_naming_pattern=cl;use_concurrent_stack=true;oob_thread_pool.rejection_policy=Run;discard_incompatible_packets=true;thread_pool.max_threads=40;oob_thread_pool.enabled=true;oob_thread_pool.max_threads=20;loopback=false;oob_thread_pool.keep_alive_time=5000;thread_pool.queue_enabled=false;oob_thread_pool.queue_enabled=false;max_bundle_size=64000;thread_pool.queue_max_size=100;thread_pool.enabled=true;enable_diagnostics=true;max_bundle_timeout=30;oob_thread_pool.min_threads=8;use_incoming_packet_handler=true;thread_pool.rejection_policy=Run;bind_addr=$bind-ip-address;thread_pool.min_threads=8;thread_pool.keep_alive_time=5000;enable_bundling=true):MPING(timeout=2000;num_initial_members=8;mcast_port=35526;mcast_addr=224.0.0.1):FD(timeout=2000;max_tries=5;shun=true):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(max_xmit_size=60000;print_stability_history_on_failed_xmit=true;use_mcast_xmit=false;gc_lag=0;discard_delivered_msgs=true;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=8000000):pbcast.GMS(print_local_addr=true;join_timeout=3000;view_bundling=true;join_retry_timeout=2000;shun=true;merge_leader=true;reject_join_from_existing_member=true)"/>
<property name="channel-name" value="Remote_Initializer"/>
<property name="temp-dir" value="../temp/remote-initializer"/>
</properties-param>
</init-params>
</component>
<component>
<key>org.exoplatform.services.jcr.ext.backup.BackupManager</key>
<type>org.exoplatform.services.jcr.ext.backup.impl.BackupManagerImpl</type>
<init-params>
<properties-param>
<name>backup-properties</name>
<property name="default-incremental-job-period" value="3600" /> <!-- set default incremental periond = 60 minutes -->
<property name="full-backup-type" value="org.exoplatform.services.jcr.ext.backup.impl.fs.FullBackupJob" />
<property name="incremental-backup-type" value="org.exoplatform.services.jcr.ext.backup.impl.fs.IncrementalBackupJob" />
<property name="backup-dir" value="../temp/backup" />
</properties-param>
</init-params>
</component>3.2 Set the actual name of a repositories.
In configuration the ReplicationService:
...
<values-param>
<name>repositories</name>
<value>repository</value>
</values-param>
...
- Set the URL to the server database
- Specify the user/password for database in database configuration
In services configuration will be configured the database configuration (see component org.exoplatform.services.naming.InitialContextInitializer):
<external-component-plugins>
<target-component>org.exoplatform.services.naming.InitialContextInitializer</target-component>
<component-plugin>
...
<properties-param>
<name>ref-addresses</name>
<description>ref-addresses</description>
<property name="driverClassName" value="oracle.jdbc.OracleDriver" />
<property name="url" value="jdbc:oracle:thin:@tornado.exoua-int:1523:orcl" />
<property name="username" value="exocluster1" />
<property name="password" value="exo12321" />
<property name="maxActive" value="100" />
<property name="maxIdle" value="2" />
<property name="initialSize" value="2" />
</properties-param>
...
</component-plugin>
</external-component-plugins>
Set the JCR database dialect
The JCR dialect must be set to oracle for all workspaces in repository configuration :
<property name="dialect" value="oracle"/>
Set the JDBC persister database dialect
The JDBCConfigurationPersister also requires oracleto be specified as dialect.
In configuration of component org.exoplatform.services.jcr.config.RepositoryServiceConfiguration
<component>
<key>org.exoplatform.services.jcr.config.RepositoryServiceConfiguration</key>
<type>org.exoplatform.services.jcr.impl.config.RepositoryServiceConfigurationImpl</type>
<init-params>
<value-param>
<name>conf-path</name>
<description>JCR configuration file</description>
<value>war:/conf/jcr/repository-configuration.xml</value>
</value-param>
<properties-param>
<name>working-conf</name>
<description>working-conf</description>
<property name="persisterClassName" value="org.exoplatform.services.jcr.impl.config.JDBCConfigurationPersister"/>
<property name="sourceName" value="jdbcexo"/>
<property name="dialect" value="oracle"/>
</properties-param>
</init-params>
</component>
3.4 Copy tomcat to other nodes
Copy the contents of the folder D:\java\exo-working\exo-tomcat in three folders:
- D:\javaexo-working\exo-tomcat_2
- D:\javaexo-working\exo-tomcat_3
- D:\javaexo-working\exo-tomcat_4
Which are needed for other cluster nodes.
3.5 Start the tomcat.
The first node of cluster is started. Now we need to initialize the other nodes.
4 2nd cluster node configuration and startup.
To component ReplicationService :
| Parameter | Value |
|---|
| ip-address | 192.168.0.135 |
| node-name | cluster_node2 |
| other-participants | cluster_node1;cluster_node3;cluster_node4 |
To component RemoteWorkspaceInitializationService:
| Parameter | Value |
|---|
| ip-address | 192.168.0.135 |
In services configuration the component ReplicationService and RemoteWorkspaceInitializationService:
<configuration>
<component>
<type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
<init-params>
...
<property name="enabled" value="true"/>
...
<property name="bind-ip-address" value="192.168.0.135"/>
...
<property name="node-name" value="cluster_node2"/>
<property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4"/>
...
<property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
<property name="node-priority" value="50"/> <!-- max == 100 -->
...
</init-params>
</component>
...
<component>
<type>org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializationService</type>
<init-params>
...
<property name="bind-ip-address" value="192.168.0.135"/>
...
</init-params>
</component>| Parameter | Value |
|---|
| user name | exocluster2 |
| password | exo12321 |
In services configuration will be configured the database configuration (see component org.exoplatform.services.naming.InitialContextInitializer):
org.exoplatform.services.naming.InitialContextInitializer
…
ref-addresses
ref-addresses
1.1.1 Initialize the workspaces using RemoteWorkspaceInitializer.
Set the <tt>RemoteWorkspaceInitializer</tt> for all workspaces in repository configuration.
We use the <tt>RemoteWorkspaceInitializer</tt> to initialize workspace from first node of cluster.
For example:
In repository configuration:
{code:xml}
...
<workspaces>
<workspace name="system" ... >
<container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
...
</container>
<initializer class="org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializer" />
...
</workspace>
<workspace name="collaboration" ... >
<container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
...
</container>
<initializer class="org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializer" />
...
</workspace>
<workspace name="backup" ... >
<container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
...
</container>
<initializer class="org.exoplatform.services.jcr.ext.initializer.RemoteWorkspaceInitializer" />
...
</workspace>
</workspaces>4.4 Run the tomcat on 2nd node
2nd node has now joined the cluster!
5 Configuration of 3rd and 4th cluster nodes
Proceed similarly as for 2nd cluster node.
5.1 Values for 3rd cluster node:
To component ReplicationService :
| Parameter | Value |
|---|
| ip-address | 192.168.0.5 |
| node-name | cluster_node3 |
| other-participants | cluster_node1;cluster_node2;cluster_node4 |
| priority-type | dynamic |
| node-priority | 30 |
| db user | exocluster3 |
| db password | exo12321 |
| tomcat home | D:\java\exo-working\exo-tomcat_3 |
To component RemoteWorkspaceInitializationService:
| Parameter | Value |
|---|
| ip-address | 192.168.0.5 |
5.2 Values for 4th cluster node:
To component ReplicationService :
| Parameter | Value |
|---|
| ip-address | 192.168.0.4 |
| node-name | cluster_node4 |
| other-participants | cluster_node1;cluster_node2;cluster_node3 |
| priority-type | dynamic |
| node-priority | 20 |
| db user | exocluster4 |
| db password | exo12321 |
| tomcat home | D:\java\exo-working\exo-tomcat_4 |
To component RemoteWorkspaceInitializationService:
| Parameter | Value |
|---|
| ip-address | 192.168.0.4 |
