External Value Storages

1 Introduction

By default JCR Values are stored in the Workspace Data container along with the JCR structure (i.e. Nodes and Properties). eXo JCR offers an additional option of storing JCR Values separately from Workspace Data container, which can be extremely helpful to keep Binary Large Objects (BLOBs) for example (see Binary values processing).

Value storage configuration is a part of Repository configuration, find more details there.

Tree-based storage is recommended for most of cases. If you run an application on Amazon EC2 - the S3 option may be interesting for architecture. Simple 'flat' storage is good in speed of creation/deletion of values, it might be a compromise for a small storages.

2 Tree File Value Storage

Holds Values in tree-like FileSystem files. path property points to the root directory to store the files.

This is a recommended type of external storage, it can contain large amount of files limited only by disk/volume free space.

A disadvantage it's a higher time on Value deletion due to unused tree-nodes remove.

<value-storage id="Storage #1" class="org.exoplatform.services.jcr.impl.storage.value.fs.TreeFileValueStorage">
     <properties>
       <property name="path" value="data/values"/>
     </properties>
     <filters>
       <filter property-type="Binary" min-value-size="1M"/>
     </filters>

Where :

id - the value storage unique identifier, used for linking with properties stored in workspace container

path - a location where value files will be stored

Each file value storage can have the filter(s) for incoming values. A filter can match values by property type (property-type), property name (property-name), ancestor path (ancestor-path) and/or size of values stored (min-value-size, in bytes). In code sample we use a filter with property-type and min-value-size only. I.e. storage for binary values with size greater of 1MB. It's recommended to store properties with large values in file value storage only.

Another example shows a value storage with different locations for large files (min-value-size a 20Mb-sized filter). A value storage uses ORed logic in the process of filter selection. That means the first filter in the list will be asked first and if not matched the next will be called etc. Here a value matches the 20 MB-sized filter min-value-size and will be stored in the path "data/20Mvalues", all other in "data/values".

<value-storages>
  <value-storage id="Storage #1" class="org.exoplatform.services.jcr.impl.storage.value.fs.TreeFileValueStorage">
    <properties>
      <property name="path" value="data/20Mvalues"/>
    </properties>
    <filters>
      <filter property-type="Binary" min-value-size="20M"/>
    </filters>
  <value-storage>
  <value-storage id="Storage #2" class="org.exoplatform.services.jcr.impl.storage.value.fs.TreeFileValueStorage">
    <properties>
      <property name="path" value="data/values"/>
    </properties>
    <filters>
      <filter property-type="Binary" min-value-size="1M"/>
    </filters>
  <value-storage>
<value-storages>

3 S3 File Value Storage

Holds Values at Amazon S3 storage. For more about S3 see http://www.amazon.com/S3-AWS-home-page-Money/b/ref=sc_fe_l_2/103-7720231-3235021?ie=UTF8&node=16427261&no=3435361&me=A36L942TSJ2AJA.

This type of storage saves all matching Values on Amazon S3 service. That is very useful for cloud computing (like Amazon EC2 hosted repositories). But can be used in any environment and storages combinations. It's often used in combined with Workspace Simple DB storage.

It's networked storage with RESTbased access (via HTTP) that makes a footprint on performance of the Repository.

<value-storage id="Storage #1" class="org.exoplatform.services.jcr.impl.storage.value.s3.SimpleS3ValueStorage">
    <properties>
      <property name="bucket" value="BUCKET NAME HERE"/>
      <property name="aws-access-key" value="INSERT YOUR AWS ACCESS KEY ID HERE"/>
      <property name="aws-secret-access-key" value="INSERT YOUR AWS SECRET ACCESS KEY HERE"/>
      <property name="s3-swap-directory" value="s3swap_directory_name"/>
    </properties>
    <filters>
      <filter property-type="Binary"/>
    </filters>
</value-storage>

4 Simple File Value Storage

Not recommended to use in production due to low capacity capabilities on most file systems

But if you're sure in your file-system or data amount is small it may be useful for you as haves a faster speed of Value removal

Holds Values in flat FileSystem files. path property points to root directory in order to store files

<value-storage id="Storage #1" class="org.exoplatform.services.jcr.impl.storage.value.fs.SimpleFileValueStorage">
     <properties>
       <property name="path" value="data/values"/>
     </properties>
     <filters>
       <filter property-type="Binary" min-value-size="1M"/>
     </filters>

5 Content Addressable Value storage (CAS) support

Available from version 1.9.3

eXo JCR supports Content-addressable storage feature for Values storing.

Content Addressable Value storage stores unique content once. Different properties (values) with same content will be stored as one data file shared between those values. We can tell the Value content will be shared across some Values in storage and will be stored on one physical file.

Storage size will be decreased for application which governs potentially same data in the content.

If property Value changes it is stored in an additional file. Alternatively the file is shared with other values, pointing to the same content.

The storage calculates Value content address each time the property was changed. CAS write operations are much more expensive compared to the non-CAS storages.

Content address calculation based on java.security.MessageDigest hash computation and tested with MD5 and SHA1 algorithms.

CAS support can be enabled for Tree and Simple File Value Storage types.

To enable CAS support just configure it in JCR Repositories configuration like we do for other Value Storages.

<workspaces>
        <workspace name="ws">
          <container class="org.exoplatform.services.jcr.impl.storage.jdbc.JDBCWorkspaceDataContainer">
            <properties>
              <property name="source-name" value="jdbcjcr"/>
              <property name="dialect" value="oracle"/>
              <property name="multi-db" value="false"/>
              <property name="update-storage" value="false"/>
              <property name="max-buffer-size" value="200k"/>
              <property name="swap-directory" value="target/temp/swap/ws"/>
            </properties>
            <value-storages>
<!------------------- here ----------------------->
              <value-storage id="ws" class="org.exoplatform.services.jcr.impl.storage.value.fs.CASableTreeFileValueStorage">
                <properties>
                  <property name="path" value="target/temp/values/ws"/>
                  <property name="digest-algo" value="MD5"/>
                  <property name="vcas-type" value="org.exoplatform.services.jcr.impl.storage.value.cas.JDBCValueContentAddressStorageImpl"/>
                  <property name="jdbc-source-name" value="jdbcjcr"/>
                  <property name="jdbc-dialect" value="oracle"/>
                </properties>
                <filters>
                  <filter property-type="Binary"/>
                </filters>
              </value-storage>
            </value-storages>

Properties:

digest-algo - digest hash algorithm (MD5 and SHA1 were tested)

vcas-type - Value CAS internal data type, JDBC backed is currently implemented org.exoplatform.services.jcr.impl.storage.value.cas.JDBCValueContentAddressStorageImpl

jdbc-source-name - JDBCValueContentAddressStorageImpl specific parameter, database will be used to save CAS metadata. It's simple to use same as in workspace container.

jdbc-dialect - JDBCValueContentAddressStorageImpl specific parameter, database dialect. It's simple to use same as in workspace container.