High Availability - Resource Synchronization

High Availability - Resource Synchronization

Introduction

This Primary-Replica resource synchronization configuration is an alternative to the standard HA configuration with a shared database. The two setups can also be used in combination.

The Primary-Replica configuration is primarily intended for very large installations with multiple physical locations. The setup consists of one InterFormNG2 installation that acts as the primary instance, as well as one or more replica instances that synchronize their resources from the primary.

The purposes of the Primary-Replica configuration is:

  • To get almost infinite scalability by static distribution of the load across many installations.

  • To avoid having a single point of failure that can bring down everything.

  • To be robust against network issues and certain hacker attacks such as DDOS.

  • To provide a simpler way of being able to print on printers in different physical locations.


The Primary instance

The InterFormNG2 Primary is the single point where all common resources have to be maintained.
Since availability of the Primary instance itself is important, this instance can optionally be set up as a standard high-availability configuration with two InterFormNG2 servers and a shared database.

The purpose of the Primary is only to be a central repository for resources, i.e. it is not the intention for the Primary to be used for workflow execution, except for testing and maintenance purposes.

When a change is made to a resource on the Primary, then the Primary will register the changed resource in a change journal that keeps a sequential record of all changes to resources.

The Replica instances

Any number of InterFormNG2 replica instances can be connected to the single primary instance.

Each Replica maintains a copy of the common resources that exist on the Primary.
Any resource that is modified on the Primary, will be synchronized to each of the replicas. Synchronization is of course not instant, but happens individually for each replica when the replica requests an update. Synchronization is done by a pull model where the replica requests updates from the primary.

Only modified resources are synchronized so initially you will need to copy all primary resources to a new replica in order to ensure that all referenced resources (including unchanged resources) are available on the replica.

Each replica maintains an index to its current update in the primary’s change journal. For instance if the primary’s latest update is number 150 and a replica is at update 147, then the replica will ask the primary for all changes since update 147, so it gets 148, 149 and 150. The replica must apply each update individually in sequential order and only update its change index once the update is applied (saved) successfully.

If a replica is unable to contact the primary, then the replica will continue operating using the older version of the resources.

The replica is where workflows are run. If the input for the workflows come from a central common source, such as an ERP system, then these inputs somehow need to be distributed to the replicas. If all replicas can handle any input, then a simple load balancer can be used.

If inputs need to be sent to a specific replica, then some form of routing has to be done, for instance by using workflows with web service or message queue input. If a message queue is used, the workflows should be split into a main workflow that only has the input component, and a sub-workflow with the rest of the functionality. The sub-workflow must be defined on the primary instance, while the input workflow must be defined on the individual replicas, so each replica can read input from a different message queue topic.

Potentially the replicas can also have some resources that are local to the replica and not known by the primary, for instance a logo image for the specific branch that the replica belongs to. However, we recommend that all resources (except input workflows) are maintained only on the primary. The input XMLs for the workflows should contain the necessary information to select the correct variant of a resource.

Way of Working

If a user makes a mistake when modifying something on the Primary, the effect can potentially be quite large as the mistake will be synchronized to all replicas. Also, some changes may be dependent on each other, so it would cause problems if they are not activated at the same time.
Consequently the recommended way of working with this type of setup should be one of:

  1. Use versioning (optionally with approval process). Changes will only be synchronized when they are approved and published.

  2. Only do design changes on a test system and then import the changes to the Primary once they are ready.

Configuration of the Primary instance

On the Primary instance, it is necessary to set one or more properties in the application-default.properties file.

If the Primary is a single InterFormNG2 instance using the embedded Derby database, then the following configuration property has to be set:

resourceChangeJournal=file

If the Primary is configured to use an external database, then it is recommended to also use the external database for the change journal. These configuration properties must then be set:

resourceChangeJournal=database
resourceChangeJournal.defaultResources=ng2db.dbdatasource

Synchronization user account

One or more synchronization user account(s) must be created on the Primary. This user account will be used by the replicas to connect to the Primary instance when they pull for updates.
The synchronization user must have access to one tenant, but it does not need access to all tenants in a multi-tenant system. Access to a single tenant is sufficient for synchronization of all tenants. In a multi-tenant environment, a security-wise good option is to create a separate tenant for the sole purpose of holding this special user account.
The user must be assigned the role “Web service”. No other roles are required.

Configuration of the Replica instances

When configuring a multi-tenant Replica instance, first make sure to create all of the tenants that should exist on this replica. Only tenants that exist on both Primary and Replica are synchronized. The replica does not need to have all of the tenants that the Primary has, it just needs to have at least one tenant that also exists on the Primary.

For the Replica instances, the configuration is done through the web interface. Log in as a system admin user and navigate to the settings page. The configuration for the replica is found in the section “Resource Replication”:



Flip the switch, Enable Resource Replication from Primary to configure the instance as a replica.
As URL of primary specify the HTTPS URL and port of the Primary instance, in the format “https://primary.company.com:8086”.
As username set the username of the user that was created as synchronization user on the Primary instance.
As Tenant ID set the ID of a tenant that the synchronization user has access to on the Primary instance. (Please notice that all tenants found on both machines are replicated - this is only for sign on).
As Password set the password of the synchronization user.

The Test connection button can be used to test if the connection settings to the primary.

Once the configuration has been done, the Replica instance has to be restarted for the synchronization to start.

Adding new tenants

In a multi-tenant environment, new tenants created on the Primary instance are not automatically created on the Replica instances. The new tenant has to be created on the Replica instances that should synchronize the resources for the new tenant.

Make sure to create the new tenant on all required Replica instances before adding any resources to the tenant on the Primary instance. If resources are added to the tenant before the replicas are ready to receive them, the replicas will skip past these updates for tenants they do not know about.

Monitoring replication status

On both the Primary instance and the Replica instances, the current synchronization index number can be seen on the admin system monitor page as below:


If a Replica has an index number that is lower than the index number on the Primary, it means that some changes have not synchronized yet.
    • Related Articles

    • High Availability setup

      For multi-server installations with high availability, we support an active-active setup with multiple active application server instances, supported by a shared relational database. A load balancer should be in front, to distribute HTTPS requests ...
    • Delete resource

      This workflow component deletes a resource from the Library. Related workflow components are: List resources, Resource to payload, Resource to attachment and Save in resources. An example of how this can be used with the List resources can be found ...
    • From resource to file

      With the advanced, utilities workflow component, From resource to file you can save a resource in the file system. The parameters are: Resource type The type of resource. You can select that via a drop down list or via an Xpath expression, if you ...
    • From file to resource

      You can load a file into a resource within a workflow with the advanced, utilities component: From file to a resource. The component has these parameters: File path The path to the file, that you want to load as a resource. Resource type This is a ...
    • From variable to resource

      You can load data from a workflow variable into a Library resource within a workflow with the advanced, utilities component: From variable to a resource. The component has these parameters: Variable name The name of the variable to copy the data ...