Configure a persistent JCR cluster

Warning: If you use JCR 1.11.1, this procedure is obsolete and has been simplified by the RemoteWorkspaceInitializer

This tutorial will show you how to configure and initialize a JCR cluster in persistent mode for eXo ECM. The tutorial uses 4 tomcat nodes and 4 oracle database schemas.

1 Tutorial environment

1.1 Database

We will use an Oracle database. The JDBC URL is : 

jdbc:oracle:thin:@tornado.exoua-int:1523:orcl

Although there is a single database server distinct users are needed - one for each cluster node:

  • exocluster1
  • exocluster2
  • exocluster3
  • exocluster4
Learn more about Database Configuration.

1.2 JCR settings

This tutorial was written and tested using JCR 1.10 but should also work properly for version 1.9.3

1.3 Cluster shape

The cluster has 4 participant nodes each is bound to its own IP address:

  • cluster_node1 : 192.168.0.15
  • cluster_node2 : 192.168.0.135
  • cluster_node3 : 192.168.0.5
  • cluster_node4 : 192.168.0.3
Warning: Ensure that all participants are time-synchronized with the same NTP-server for example. This is a common requirement for most clustering solutions

1.4 Get an eXo tomcat bundle

The instructions of this tutorial should be compatible with eXo Portal 2.2.
Get and install eXo ECM 2.2 tomcat bundle from the download area.

Alternatively, you can always build the product you want from sources using eXoBuild tool. For exemple : 

exobuild --product=portal --version=2.2 --build --deploy

2 First cluster node configuration and startup.

2.1 Configuration of replication

On a following path D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\jcr\, create a file replication-config.xml with contents: 

<?xml version="1.0" encoding="ISO-8859-1"?>
<configuration>
 <component>
 <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
 <init-params>
  <value-param>
     <name>force-xml-configuration</name>
     <value>true</value>      
  </value-param>
  <values-param>
   <name>repositories</name>
   <value>repository</value>
  </values-param>
  <properties-param>
   <name>replication-properties</name>
   <property name="enabled" value="false"/>
   <property name="mode" value="persistent"/>
   <property name="bind-ip-address" value="192.168.0.15"/>
   <property name="channel-config" value="TCP(oob_thread_pool.queue_max_size=100;thread_naming_pattern=cl;use_concurrent_stack=true;oob_thread_pool.rejection_policy=Run;discard_incompatible_packets=true;thread_pool.max_threads=40;oob_thread_pool.enabled=false;oob_thread_pool.max_threads=20;loopback=false;oob_thread_pool.keep_alive_time=5000;thread_pool.queue_enabled=false;oob_thread_pool.queue_enabled=false;max_bundle_size=64000;thread_pool.queue_max_size=100;thread_pool.enabled=false;enable_diagnostics=true;max_bundle_timeout=30;oob_thread_pool.min_threads=8;use_incoming_packet_handler=true;thread_pool.rejection_policy=Run;bind_addr=$bind-ip-address;thread_pool.min_threads=8;thread_pool.keep_alive_time=5000;enable_bundling=true):MPING(timeout=2000;num_initial_members=8;mcast_port=34526;mcast_addr=224.0.0.1):FD(timeout=2000;max_tries=5;shun=true):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(max_xmit_size=60000;print_stability_history_on_failed_xmit=true;use_mcast_xmit=false;gc_lag=0;discard_delivered_msgs=true;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=8000000):pbcast.GMS(print_local_addr=true;join_timeout=3000;view_bundling=true;join_retry_timeout=2000;shun=true;merge_leader=true;reject_join_from_existing_member=true)"/>
   <property name="recovery-dir" value="../temp/replication/recovery"/>
   <property name="node-name" value="cluster_node1"/>
   <property name="other-participants" value="cluster_node2;cluster_node3;cluster_node4"/>
   <property name="wait-confirmation" value="2000"/>
  </properties-param>
  <properties-param>
   <name>replication-priority-properties</name>
   <property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
   <property name="node-priority" value="100"/> <!-- max == 100 -->
  </properties-param>
 </init-params>
</component>


<component>
 <type>org.exoplatform.services.jcr.ext.replication.test.ReplicationTestService</type>
</component>


<component>
    <key>org.exoplatform.services.jcr.ext.backup.BackupManager</key>
    <type>org.exoplatform.services.jcr.ext.backup.impl.BackupManagerImpl</type>
    <init-params>
      <properties-param>
        <name>backup-properties</name>
        <property name="default-incremental-job-period" value="3600" /> <!-- set default incremental periond = 60 minutes -->
        <property name="full-backup-type" value="org.exoplatform.services.jcr.ext.backup.impl.fs.FullBackupJob" />
        <property name="incremental-backup-type" value="org.exoplatform.services.jcr.ext.backup.impl.fs.IncrementalBackupJob" />
        <property name="backup-dir" value="../temp/backup" />
      </properties-param>
    </init-params>
  </component>
</configuration>

2.2 Set the actual name of a repositories.

D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\jcr\replication-config.xml: 

...
<values-param>
 <name>repositories</name>
 <value>repository</value>
</values-param>
...

For actual name of a repositories see D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\jcr\repository-configuration.xml.

2.3 Add the import of replication config

configuration.xml 

...
<import>war:/conf/ecm/cms-configuration.xml</import>
<import>war:/conf/ecm/ecm-templates-configuration.xml</import>
<import>war:/conf/ecm/ecm-scripts-configuration.xml</import>
<import>war:/conf/ecm/ecm-actions-configuration.xml</import>
<import>war:/conf/ecm/ecm-categories-configuration.xml</import>
<import>war:/conf/ecm/ecm-views-configuration.xml</import>
<import>war:/conf/ecm/ecm-drives-configuration.xml</import>
<import>war:/conf/ecm/ecm-metadata-configuration.xml</import>
<import>war:/conf/ecm/ecm-folksonomy-configuration.xml</import>
<import>war:/conf/ecm/ecm-records-configuration.xml</import>
<import>war:/conf/ecm/ecm-queries-configuration.xml</import>
<import>war:/conf/ecm/organization-component-plugins-configuration.xml</import>
<import>war:/conf/jcr/replication-config.xml</import>
</configuration>

2.4 Configure database

  • Set the URL to the server database
  • Specify the user/password for database in database configuration
D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\database\database-configuration.xml: 
<external-component-plugins>
 <target-component>org.exoplatform.services.naming.InitialContextInitializer</target-component>
 <component-plugin>
 ...
  <properties-param>
   <name>ref-addresses</name>
   <description>ref-addresses</description>
   <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
   <property name="url" value="jdbc:oracle:thin:@tornado.exoua-int:1523:orcl" />
   <property name="username" value="exocluster1" />
   <property name="password" value="exo12321" />
   <property name="maxActive" value="100" />
   <property name="maxIdle" value="2" />
   <property name="initialSize" value="2" />
  </properties-param>
 ...
 </component-plugin>
</external-component-plugins>

Will be used user exocluster1 of database for first cluster node

Set the JCR database dialect

The JCR dialect must be set to oracle for all workspaces in repository configuration :

D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\jcr\repository-configuration.xml 

<property name="dialect" value="oracle"/>

Set the JDBC persister database dialect

The JDBCConfigurationPersister also requires oracleto be specified as dialect.

D:\java\exo-working\exo-tomcat\webapps\portal.war\WEB-INF\conf\jcr\jcr-configuration.xml 

<component>
 <key>org.exoplatform.services.jcr.config.RepositoryServiceConfiguration</key>
 <type>org.exoplatform.services.jcr.impl.config.RepositoryServiceConfigurationImpl</type>
 <init-params>
  <value-param>
   <name>conf-path</name>
   <description>JCR configuration file</description>
   <value>war:/conf/jcr/repository-configuration.xml</value>
  </value-param>
  <properties-param>
   <name>working-conf</name>
   <description>working-conf</description>
   <property name="persisterClassName" value="org.exoplatform.services.jcr.impl.config.JDBCConfigurationPersister"/>
   <property name="sourceName" value="jdbcexo"/>
   <property name="dialect" value="oracle"/>
  </properties-param>
 </init-params>
</component>

2.5 Copy tomcat to other nodes

Copy the contents of the folder D:\java\exo-working\exo-tomcat in three folders:

  • D:\javaexo-working\exo-tomcat_2
  • D:\javaexo-working\exo-tomcat_3
  • D:\javaexo-working\exo-tomcat_4
Which are needed for other cluster nodes.

2.6 Run the tomcat once.

./eXo.bat run

2.7 Stop tomcat after the full launch.

The JCR is now initialized, but replication is not yet configured.

./eXo.bat stop

2.8 Activate replication

D:\java\exo-working\exo-tomcat\webapps\portal\WEB-INF\conf\jcr\replication-config.xml: 

<property name="enabled" value="true"/>

2.9 Restart the tomcat.

./eXo.bat run

The cluster is started and replication activated. Now we need to initialize the other nodes.

2.10 Create a backup.

Backups of each workspace are needed to initialize other cluster nodes. Enter the following URI in your browser :

  • Until WS 2.0 and JCR 1.11
http://<ip-address>:<port>/rest/replication-test/<repository name>/<workspace name>/<user name>/<password>/<incremental period(set in ms)>/?operation=startBackup

In our case it is needed perform of three requests (one for eachworkspaces): 

http://192.168.0.15:8080/rest/replication-test/repository/collaboration/root/exo/50000/?operation=startBackup
http://192.168.0.15:8080/rest/replication-test/repository/backup/root/exo/50000/?operation=startBackup
http://192.168.0.15:8080/rest/replication-test/repository/system/root/exo/50000/?operation=startBackup

  • Since WS 2.0 and JCR 1.11
http://<ip-address>:<port>/rest/replication-test/<repository name>/<workspace name>/<user name>/<password>/<incremental period(set in ms)>/startBackup

In our case it is needed perform of three requests (one for eachworkspaces): 

http://192.168.0.15:8080/rest/replication-test/repository/collaboration/root/exo/50000/startBackup
http://192.168.0.15:8080/rest/replication-test/repository/backup/root/exo/50000/startBackup
http://192.168.0.15:8080/rest/replication-test/repository/system/root/exo/50000/startBackup

Some files have been generated at D:\java\exo-working\temp\backup

3 2nd cluster node configuration and startup.

3.1 Configure replication

D:\java\exo-working\exo-tomcat_2\webapps\portal.war\WEB-INF\conf\jcr\replication-config.xml :

ParameterValue
ip-address192.168.0.135
node-namecluster_node2
other-participantscluster_node1;cluster_node3;cluster_node4

D:\java\exo-working\exo-tomcat_2\webapps\portal.war\WEB-INF\conf\jcr\replication-config.xml: 

<configuration>
 <component>
 <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
 <init-params>
 ...
  <property name="enabled" value="false"/>
  ...
  <property name="bind-ip-address" value="192.168.0.135"/>
  ...
  <property name="node-name" value="cluster_node2"/>
  <property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4"/>
  ...
  <property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
  <property name="node-priority" value="50"/> <!-- max == 100 -->
  ...
 </init-params>
</component>

3.2 Configure database

ParameterValue
user nameexocluster2
passwordexo12321

D:\java\exo-working\exo-tomcat_2\webapps\portal.war\WEB-INF\conf\database\database-configuration.xml 

<external-component-plugins>
 <target-component>org.exoplatform.services.naming.InitialContextInitializer</target-component>
 <component-plugin>
 ...
   <properties-param>
    <name>ref-addresses</name>
    <description>ref-addresses</description>
    <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
    <property name="url" value="jdbc:oracle:thin:@tornado.exoua-int:1523:orcl" />
    <property name="username" value="exocluster2" />
    <property name="password" value="exo12321" />
    <property name="maxActive" value="100" />
    <property name="maxIdle" value="2" />
    <property name="initialSize" value="2" />
   </properties-param>
  </init-params>
 </component-plugin>
</external-component-plugins>

3.3 Move backup files from 1st cluster node.

The files of backup have been created in D:\java\exo-working\temp\backup on first cluster node. On 2nd cluster node, copy backup files at D:\java\exo-working\backup .

3.4 Initialize from backup

Set the BackupWorkspaceInitializer for all workspaces in repository config We use the BackupWorkspaceInitializer to initialize workspace from the backup files that were created on first node.

For example:

D:\java\exo-working\exo-tomcat_2\webapps\portal.war\WEB-INF\conf\repository-configuration.xml : 

...
<workspaces>
 <workspace name="system" ... >
  <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
  ...
  </container>
  <initializer class="org.exoplatform.services.jcr.impl.core.BackupWorkspaceInitializer">
   <properties>
    <property name="restore-path" value="D:\java\exo-working\backup\repository_system-20080703_111009"/>
   </properties>
  </initializer>
  ...
 </workspace>

 <workspace name="collaboration" ... >
   <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
   ...
  </container>
  <initializer class="org.exoplatform.services.jcr.impl.core.BackupWorkspaceInitializer">
   <properties>
    <property name="restore-path" value="D:\java\exo-working\backup\repository_collaboration-20080703_111009"/>
   </properties>
  </initializer>
  ...
 </workspace>

 <workspace name="backup" ... >
  <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
  ...
  </container>

  <initializer class="org.exoplatform.services.jcr.impl.core.BackupWorkspaceInitializer">
   <properties>
    <property name="restore-path" value="D:\java\exo-working\backup\repository_backup-20080703_111009"/>
   </properties>
  </initializer>
  ...
  </workspace>
</workspaces>

3.5 Run the tomcat on 2nd node

./eXo.bat run

3.6 Stop tomcat after the full launch.

./eXo.bat stop

3.7 Activate replication

D:\java\exo-working\exo-tomcat_2\webapps\portal\WEB-INF\conf\jcr\replication-config.xml : 

<property name="enabled" value="true"/>

3.8 Run the tomcat.

./eXo.bat run

2nd node has now joined the cluster!

4 Configuration of 3rd and 4th cluster nodes

Proceed similarly as for 2nd cluster node.

4.1 Values for 3rd cluster node:

ParameterValue
ip-address192.168.0.5
node-namecluster_node3
other-participantscluster_node1;cluster_node2;cluster_node4
priority-typedynamic
node-priority30
db userexocluster3
db passwordexo12321
tomcat homeD:\java\exo-working\exo-tomcat_3

4.2 Values for 4th cluster node:

ParameterValue
ip-address192.168.0.4
node-namecluster_node4
other-participantscluster_node1;cluster_node2;cluster_node3
priority-typedynamic
node-priority20
db userexocluster4
db passwordexo12321
tomcat homeD:\java\exo-working\exo-tomcat_4

5 Adding a new node to the cluster

Our cluster is now running with 4 participants. Below is ow to add a new node to the running cluster.

Values for 5th cluster node:

Parametervalue
ip-address192.168.0.77
node-namecluster_node5
other-participantscluster_node1;cluster_node2;cluster_node3;cluster_node4
priority-typedynamic
node-priority10
db userexocluster5
db passwordexo12321
JDBC URLjdbc:oracle:thin:@tornado.exoua-int:1523:orcl

5.1 Change the replication config for first cluster node.

D:\java\exo-working\exo-tomcat\webapps\portal\WEB-INF\conf\jcr\replication-config.xml: 

<component>
 <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
  ...
  <property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4;cluster_node5"/>
  ...
</component>

5.2 Change the replication config for 2nd cluster node.

D:\java\exo-working\exo-tomcat2\webapps\portal\WEB-INF\conf\jcr\replication-config.xml: 

<component>
 <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
 ...
 <property name="other-participants" value="cluster_node2;cluster_node3;cluster_node4;cluster_node5"/>
 ...
 </component>

5.3 Change the replication config for 3rd cluster node.

D:\java\exo-working\exo-tomcat3\webapps\portal\WEB-INF\conf\jcr\replication-config.xml: 

<component>
 <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
 ...
 <property name="other-participants" value="cluster_node1;cluster_node2;cluster_node4;cluster_node5"/>
 ...
</component>

5.4 Change the replication config for 4th cluster node.

D:\java\exo-working\exo-tomcat3\webapps\portal\WEB-INF\conf\jcr\replication-config.xml: 

<component>
 <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
 ...
 <property name="other-participants" value="cluster_node1;cluster_node2;cluster_node3;cluster_node5"/>
 ...
</component>

5.5 Restart the tomcat on already working cluster nodes.

Restart nodes 1,2,3 and 4. The new configuration should be reloaded and the cluster would be fully working.

5.6 5th cluster node configuration and startup.

Prepare tomcat for 5th cluster node.

Copy from folder D:\java\exo-working\exo-tomcat_2 to the folder D:\java\exo-working\exo-tomcat_5 all files and folders, except folders logs, temp, work.

In the folder D:\java\exo-working\exo-tomcat_5\webapps remove expanded directories but keep the corresponding .war files.

Create new files backup

It is necessary to initiate the workspaces for a 5-node cluster.

Move the new files of backup from 1st cluster node

Take files from D:\java\exo-working\exo-tomcat\temp\backup, on 5th cluster node and copy them in D:\java\exo-working\backup_for_5_node.

Configure replication

Edit necessary ip-address, node-name, other-participants for the 5 cluster node in D:\java\exo-working\exo-tomcat_5\webapps\portal.war\WEB-INF\conf\jcr\replication-config.xml

D:javaexo-workingexo-tomcat_5webappsportal.warWEB-INFconfjcrreplication-config.xml: 

<configuration>
  <component>
  <type>org.exoplatform.services.jcr.ext.replication.ReplicationService</type>
  <init-params>
    ...
    <property name="enabled" value="false"/>
    ...
    <property name="bind-ip-address" value="192.168.0.77"/>
    ...
    <property name="node-name" value="cluster_node5"/>
    <property name="other-participants" value="cluster_node1;cluster_node3;cluster_node4;cluster_node5"/>
    ...
    <property name="priority-type" value ="dynamic"/> <!-- {static, dynamic} -->
    <property name="node-priority" value="10"/> <!-- max == 100 -->
    ...
  </init-params>
</component>

Configure database

Set the following parameters in database configuration for 5th cluster node:

ParameterValue
JDBC URLjdbc:oracle:thin:@tornado.exoua-int:1523:orcl
usernameexocluster5
passwordexo12321

D:javaexo-workingexo-tomcat_5webappsportal.warWEB-INFconfdatabasedatabase-configuration.xml: 

<external-component-plugins>
  <target-component>org.exoplatform.services.naming.InitialContextInitializer</target-component>
  <component-plugin>
  ...
    <properties-param>
      <name>ref-addresses</name>
      <description>ref-addresses</description>
      <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
      <property name="url" value="jdbc:oracle:thin:@tornado.exoua-int:1523:orcl" />
      <property name="username" value="exocluster5" />
      <property name="password" value="exo12321" />
      <property name="maxActive" value="100" />
      <property name="maxIdle" value="2" />
      <property name="initialSize" value="2" />
    </properties-param>
   </init-params>
  </component-plugin>
</external-component-plugins>

Configure BackupWorkspaceInitializer for all workspaces

Set the BackupWorkspaceInitializer for all workspaces in repository config D:\java\exo-working\exo-tomcat_5\webapps\portal.war\WEB-INF\conf\jcr\repository-configuration.xml. The BackupWorkspaceInitializer needed for initialize workspace from backup files are created on first node cluster. Prescribe backup files for consistent workspace.

For example : D:\javaexo-working\exo-tomcat_5\webapps\portal.war\WEB-INF\conf\jcr\repository-configuration.xml: 

...
  <workspaces>
    <workspace name="system" ... >
      <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
        ...
      </container>
      <initializer class="org.exoplatform.services.jcr.impl.core.BackupWorkspaceInitializer">
        <properties>
          <property name="restore-path" value="D:\java\exo-working\backup_for_5_node\repository_system-20080703_151009"/>
        </properties>
      </initializer>
      ...
    </workspace>

    <workspace name="collaboration" ... >
      <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
        ...
      </container>
      <initializer class="org.exoplatform.services.jcr.impl.core.BackupWorkspaceInitializer">
        <properties>
          <property name="restore-path" value="D:\java\exo-working\backup_for_5_node\repository_collaboration-20080703_151009"/>
        </properties>
      </initializer>
      ...
    </workspace>

    <workspace name="backup" ... >
      <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
        ...
      </container>
      <initializer class="org.exoplatform.services.jcr.impl.core.BackupWorkspaceInitializer">
        <properties>
          <property name="restore-path" value="D:\java\exo-working\backup_for_5_node\repository_backup-20080703_151009"/>
        </properties>
      </initializer>
      ...
    </workspace>
</workspaces>

Start tomcat.

eXo.bat run

Stop tomcat after the full launch.

eXo.bat stop

Activate replication

Set the follow parameter in replication config D:\java\exo-working\exo-tomcat_5\webapps\portal\WEB-INF\conf\jcr\replication-config.xml: 

<property name="enabled" value="true"/>

Restart the tomcat.

eXo.bat run

The 5th cluster node was fully initialized.

 
Products
generated on Sun Mar 14 03:52:17 UTC 2010
eXo Optional Modules
eXo Core Foundations
Recently Modified
almompalyboah | koravvinauj | oribranuali | uncoosota | intevy
Creator: Patrice Lamarque on 11/03/2008
Copyright (c) 2000-2009. Allright reserved - eXo platform SAS
1.6.13286