Docker Deployment

Software Requirement

• Docker CE 19+ or equivalent Docker EE

Installation

• Pull docker image online or load docker image offline
  
  • Online: docker pull [docker url]/tag:version
    
  • Offline: docker load -i < 'docker image file' >
• Get docker image ID: docker images
• Start docker container

docker run -itd --net=host 
-v <absolute_path_to_data_folder>:/opt/ultipa-server/data 
-v <absolute_path_to_config_folder>:/opt/ultipa-server/config 
-v <absolute_path_to_log_folder>:/opt/ultipa-server/log 
-v <absolute_path_to_mlog_folder>:/opt/ultipa-server/mlog 
-v <absolute_path_to_algo_folder>:/opt/ultipa-server/algo 
-e TZ=Europe/Paris 
--name ultipa-server --ulimit nofile=90000:90000 --privileged `image ID`
docker exec -it ultipa-server bash 
start

Docker will report 'No config files found' error during the first start, while at the same time, it creates an example file named example.config automatically. Copy and rename example.config to server.config, and modify this file by following Server Configuration (See below) before restarting.

• Docker run options
  
  • Set timezone: -e TZ=Europe/Paris
    
  • Increase number of open files limit: -ulimit nofile=90000:90000 --privileged
    
  • Enable GDB debugging: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined
• Common reasons that docker fails to start

Cluster Deployment

• Prepare cluster machines, ascertain the IP and port for inner communication and external services, ensure that every machine can access the private ports of other machines.
• It is recommended to use ultipa_doctor_linux_v4.0 gen_config to generate cluster configuration file, then copy the file to the corresponding machine, as detailed in documentation Ultipa Doctor 4.0. You may also configure the cluster manually.
• Install ultipa-server on every machine using Docker and start service for all machines.
• Check the following items after the deployment is finished:
  1. Process check. Every machine should have ultipa-server process. If not, locate the problem by checking the server log and htap log.
  2. Cluster status check. It is recommended to use ultipa_doctor_linux_v4.0 health_check as detailed in documentation Ultipa Doctor 4.0. You may also use curl http://private_addr:private_ip/raft_admin, the cluster is successfully deployed if the Leader is elected normally.
  3. Use Ultipa Manager (if has) to check if the cluster is displayed well.
  4. Import testing data. Enter ultipa-server container from any machine to execute the import: cd example && sh import_example.sh.
• Note: When new cluster is being deployed, initial data of each instance must be empty, and do not copy data catalog in production environment!

Server Configuration

Configuration File Path

config/server.config

Global Configuration

section:[Server]

Log Configuration

section:[Log]

Supervisor Configuration

section:[Supervisor]

Persistence Configuration

section:[BaseDB]

Engine Configuration

section:[Engine]

HTAP

section:[HTAP]

Detailed Cluster Configuration

Raft

Raft normally uses odd number of nodes, such as 3, 5, 7, etc. That is because Raft can only serve when more than half of the nodes are online, and 'more than half' means N/2+1 rather than N/2. For example, it requires >=2 nodes online for 3-node cluster, and >=3 nodes online for 5-node cluster.

For clusters with an even number of nodes, Raft requires 2-node cluster's 2 nodes both online, and 4-node cluster has >=3 nodes online.

Although even number of nodes consume more hardware resources than odd number of nodes but bring worse availability, in some strong consistency scenarios, even number of nodes have more advantages in data disaster tolerance to guarantee data safety.

In Ultipa system, each GraphSet has a Raft group, and each Raft group is completely independent with different Leader and Follower.

Note: Newly created GraphSet cannot be used immediately until Raft cluster elects the Leader. The delay is determined by the configured election time.

Role

• Role 0: Node cannot provide reads when it is Follower, it is used for consistent reads
• Role 1: Node can provide reads when it is Follower, it is used for load balancing reads
• Role 2: Algorithm node which can run algorithms, but cannot participate in load balancing reads
• Role 3: Algorithm node which can participate in load balancing reads
• Role 4: Backup node which only synchronizes data, but does not participate in elections

Notes:

• Node of role 2, 3 or 4 cannot be elected as the Leader.
• The number of algorithm nodes (role 2 or 3) in a cluster cannot exceed 50%.

Election Timeout

To avoid simultaneous elections initiated by several nodes, Raft sets random election timeout:

Actual Election Timeout = random(election_timeout, 2*election_timeout)
Heartbeat Time = max(election_timeout/election_heartbeat_factor, 10)

Cluster Operation

Log

Cluster log: htap-yyyymmdd-HHMMSS.log
Server log: server-yyyymmdd-HHMMSS.log

Logs begin with [HTAP] in server log are related to cluster synchronization, to view them: cat server-yyyymmdd-HHMMSS.log |grep "[HTAP]"

Operation Tool

ultipa_cluster_cli

Please use it with extra caution!

Add Node

The initial data of the node to be added must be the same with other nodes. Start the newly joined node while ensuring the data/htapp directory is empty, otherwise new data may not be synchronized.

ultipa_cluster_cli add --peer="$peer_being_added" --conf="$current_conf"

./ultipa_cluster_cli add \ 
--peer="127.0.0.1:60163|127.0.0.1:60063:1" \ 
--conf="127.0.0.1:60161|127.0.0.1:60061:1,\ 
            127.0.0.1:60162|127.0.0.1:60062:1" 

Remove Node

ultipa_cluster_cli remove --peer="$peer_being_removed" --conf="$current_conf"

./ultipa_cluster_cli remove \ 
--peer="127.0.0.1:60163|127.0.0.1:60063:1" \ 
--conf="127.0.0.1:60161|127.0.0.1:60061:1,\ 
					  127.0.0.1:60162|127.0.0.1:60062:1,\ 
            127.0.0.1:60163|127.0.0.1:60063:1" 

Modify Cluster Node

ultipa_cluster_cli change_conf --conf="$current_conf" --new_conf="$new_conf"

Create Snapshot, Compress Log

ultipa_cluster_cli snapshot_leader --peer="$target_peer"

Change Leader

ultipa_cluster_cli change_leader --peer="$target_leader" --conf="$current_conf"

Resynchronization

When a node has data inconsistency, it has to be reset. Strictly follow the steps below to resynchronize data. Note: The following operations must be performed on the node that needs to be resynchronized!

1. Use ultipa_cluster_cli to remove the node to be resynchronized
2. Stop the node to be resynchronized: ./ultipa-server stop
3. Delete the entire data directory of the node to be resynchronized
4. Restart the node to be resynchronized
5. Use ultipa_cluster_cli to add the node to be resynchronized

Cluster Monitoring

Monitoring page: http://private_host:private_port/raft_admin

Cluster FAQs

1. Common Errors

• This follower is not readable: Read request was sent to an unreadable node.
  
• This follower is loading snapshot: Node is loading snapshot, services temporarily unavailable.
  
• This peer is not leader, please redirect to the leader peer: Node is not leader.
  
• This raft cluster has no leader: No Leader was elected in cluster.

2. If the following error occurs when restarting ultipa-server, it indicates that server may have data inconsistency due to abnormal exit during the log apply process. For data consistency, use ./ultipa-server -d -safe startup to repair data automatically.

[HTAP] IMPORTANT! xxx pre_applied_index xxx != applied_index xxx, there might be data inconsistencies. 
[HTAP] Run ./ultipa-server -d -safe to safe-start the server. 

If ./ultipa-server -d -safe fails to start, please strictly follow the corresponding steps to resynchronize.

[HTAP] safe-start failed! Please Resync this peer according to the document!

3. Error 'reject term_unmatched AppendEntries' occurs when restarting

It indicates that the cluster status of the node is incorrect, which may be caused by the deletion of the htap directory. Since the last prev_log_term of the node is recorded in the cluster, and local_prev_log_term becomes 0 after the data is deleted locally, data synchronization is rejected as local_prev_log_term < prev_log_term.

When there is problem with the cluster status of the node, use ultipa_cluster_cli to resynchronize.

4. Modify the configuration file and restart in order to change the listening IP or port of the node does not take effect. It is because cluster has already recorded the node's information, so you need to use ultipa_cluster_cli change_conf to update cluster information after the restart.

./ultipa_cluster_cli change_conf \ 
--conf="127.0.0.1:60161|127.0.0.1:60061:1,\ 
127.0.0.1:60162|127.0.0.1:60062:1,\
127.0.0.1:60163|127.0.0.1:60063:1" \ 
--new_conf="127.0.0.1:60261|127.0.0.1:60061:1,\ 
            127.0.0.1:60262|127.0.0.1:60062:1,\ 
            127.0.0.1:60263|127.0.0.1:60063:1" 

5. Modify the configuration file and restart in order to change role does not take effect. It is because cluster has already cached this configuration, so you need to use ultipa_cluster_cli change_conf to update cluster information after the restart.

./ultipa_cluster_cli change_conf \ 
--conf="127.0.0.1:60161|127.0.0.1:60061:2,\ 
        127.0.0.1:60162|127.0.0.1:60062:1,\ 
        127.0.0.1:60163|127.0.0.1:60063:1" \ 
--new_conf="127.0.0.1:60161|127.0.0.1:60061:1,\ 
            127.0.0.1:60162|127.0.0.1:60062:1,\ 
            127.0.0.1:60163|127.0.0.1:60063:1" 

6. If deletes htap directory without deleting GraphSet data directory, there will be problem when you use ultipa_cluster_cli tool to remove node and re-add node after the service is restarted. It is because data is synchronized from scratch due to the deletion of htap directory, and while data directory is not deleted, there will be data inconsistency due to duplicate execution logs! Thus, remember to delete all GraphSet data directories (including global) when you delete htap directory!

7. When neither htap directory nor data directory is deleted, data will be synchronized from the last location when node is removed and re-added by using ultipa_cluster_cli tool. It is because the cluster still remembers the location when synchronizing log last time.