This manual covers the usage of Ultipa Transporter (Go).

Function Preview

Ultipa Transporter is a command-line-based lightweight tool for fast import/export meta-data to/from the Ultipa Graph database. Batch import/export is supported in remote mode.

A command-line can import or export multiple node files and edge files.

Local operation (to be declared in local in the yml file) is executed against graphsets in a local Ultipa database. The directory of the local database server needs to be declared via parameter path (by default is './data').

Local import operation is normally used for initializing the Ultipa server, in which case the Ultipa server should be stopped and there is no previous import executed.

We strongly recommend the local import operation to be implemented under the supervision of a certified Ultipa Graph Database engineer.

• Remote Operation (to be declared in server in the yml file)

Remote operation is executed against graphsets in a remote Ultipa database. The IP and port of the remote database server should be declared via parameter host，as well as username and password if required.

Node/Edge File

Files carrying node/edge information (either exported or to be imported) are stored in the local directory. A file contains either nodes of a specific schema or edges of a specific schema; each row (except the header) in the file represents a node or an edge, and each column represents a property of the node/edge. Files to be imported can be encoded in the format of csv, tsv, txt, etc., and delimiters supported include ,, \t, |, ;. Files to be exported are in the format of csv with , as delimiter.

Data contained in the files (either node or edge) should be declared via nodeConfig and edgeConfig. Format (and delimiter if contained) of all files in one operation should be consistent.

Import Declaration

Information that are consistent for all files (to be declared in settings in the yml file): delimiter separator, number of threads threads and so on, see settings introduced later for details. A csv or tsv file to import can have header or not, and the column name should be <property_name> or <property_name>:<property_type>.

Information to be declared for each file (to be declared in nodeConfig, edgeConfig in the yml file): file directory file, schema of data - schema, start position of import skip, number of rows to import limit, and property list properties or types:

(for csv or tsv file)

• when the file is headerless, for each column in turn, use properties to:
  • declare property name and type using - name and type;
  • declare property name and type to 'string' using - name only;
  • when there are less - name than expected, columns will be omitted from the left;
• when the file contains header, for some columns, use types to:
  • declare or modify property type using - name and type;
  • declare or modify property type to 'string' using - name only;
  • for columns whose - name does not appear, the property type will either be 'string' if the header is <property_name>, or be the <property_type> from the header if the header is <property_name>:<property_type>.

Valid type are listed below:

• _id, _uuid, _from, _to, _from_uuid, _to_uuid (for declaration of Ultipa system properties);
• string, float, double, int32, uint32, int64, uint64, datetime, timestamp;
• _ignore (for ignoring a column)

The data type of properties declared should satisfy the data in the columns. A misparsing of data, reading a column of integers as strings for example, will be prompted as an error.

Export Declaration

Information that is consistent for all files (to be declared in settings in the yml file): the csv file directory outPath, whether to include header writeHeder (with <property_name>:<property_type> as the column name if includes header). Batch size is auto-defined and need not be set.

Information to be declared for each file (to be declared in nodeConfig, edgeConfig in the yml file): schema of data - schema, property list (not including Ultipa system properties) properties. Ultipa system properties will always be exported and need no declaration.

Metadata Knowledge

Unique Identifier

Node has two types of unique identifiers which are _id (32bit-string) and_uuid (uint64-integer); edge has therefore starting node as either _from or _from_uuid, and ending node as either _to or _to_uuid. Edge, on the other hand, has only one identifier _uuid. These six properties are Ultipa system properties.

Import Mode

When the import mode importMode is set to upsert or overwrite, a node/edge that already exists (meaning that its identifier is provided in the file and the identifier is found in the graphset) will update or overwrite its corresponding record in the graphset; otherwise (which is, either there is no identifier in the file, or the identifier is not found in the graphset) the node/edge will be inserted into the graphset as a new record.

When the import mode importMode is set to insert, only new node/edge (either there is no identifier in the file, or the identifier is not found in the graphset) will be inserted; any identifier that already exists in the graphset will trigger error.

The absence of _id or _uuid from the file will be automatically generated by the system up insertion.

Special Requirements on Edges

An edge file must contain starting node and ending node of edges.

To successfully import edges, both ending nodes of the edges should already exist, otherwise the import will fail. In this case, set createNodeIfNotExist to 'true' and let the system create the non-existing ending nodes so as to import the edge.

How does Transporter judge whether an ending node exists when importing edge files? During remote operation, Transporter will search the node file being imported in the same command line as well as the graphset for the ending nodes.

What will happen if only system properties _from and _to are provided as unique identifiers of ending nodes, or only _from_uuid and _to_uuid are provided? During remote operation, Transporter will automatically project the other two system properties.

Error

Error Before Importing

Error berfore importing is triggered when checking configuration in the yml file, creating graphset or creating schema. Possible causes:

1. unconformity of the yml file content with yml format;
2. parameter config error, such as property name or data type mismatching UQL specification;
3. failure when creating graphset and/or schema;

Error During Importing

Error berfore importing is triggered when imporing data records into the server. Error types:

1. server returned error;
2. network error;
3. data format error (inconsistency between declared file header and data columns);
4. duplicated identifier (node or edge already exists under insert mode).

By default when an error occurs during importing a data batch, the whole batch will be skipped and the operation will continue from the next batch; one can use parameter stopWhenError to make the import operation fully stop once an error occurs, without importing any later batches. The error type, skipped data rows (represented by the start row position and the total number of rows) will both be recorded in the log file, which is for easy re-import using parameters skip and limit.

The 3rd type of error which is data format related occurs when:

• in the case of a headerless file, the number of - name declared under properties is different than the number of columns in the file;
• in the case of a file with header, some data columns have no column name, or some column names in the tail have no data.

To avoid the 3rd type of error, set fitToHeader to 'true', and let Transporter import data columns according to the properties declared, which means columns without names will be ignored and names with no data will be taken as properties and filled with empty value or 0.

Command and Parameters

Ultipa Transporter has two tools --- ultipa-importer and ultipa-exporter --- and both have below parameters:

Example of Import:

/opt/ultipa-transporter/ultipa-importer --config ./in.yml

Example of Export:

/opt/ultipa-transporter/ultipa-exporter --config ./out.yml

Example of Help:

/opt/ultipa-transporter/ultipa-exporter --help

Configuration File

Both import and export operations need a configuration file in yml format. There are four parts in a yml config file:

• server
• nodeConfig
• edgeConfig
• settings

server

nodeConfig | edgeConfig

Note: Parameters with a dash '-' ('- schema' and '- name') and their sub-parameters carried can appear multiple times.

settings

Note: exported files (node and edge) are automatically named, eg., node file will be named with format: <schema>.node.<file_type>, which can be 'default.node.csv'

YML Samples

Remote Import

Example: Given the following 3 files, import them into a remote server.

Node file-A (txt): @student, in which the proterty type of 'age' is mistakenly written as 'string'; please revise it into 'uint32' during import:

stuNo:_id,name:string,age:string,gender:string
20215865,Alice,24,f
20215925,Jack,25,m
20215973,John,28,m
20215990,Grace,25,f

Node file-B (txt): @course; please create an empty column for property 'professor':

crsNo,title,credit
CS202104,Computer Principle and Application,4
SH202127,Art of File and Television,2.5
MS202104,Calculus,3

Edge file (csv): @enroll, in which the 1st column is stuNo number and the 2nd column is crsNo; please ignore the 3rd column when importing:

20215865,SH202127,84.5
20215925,MS202104,77.5
20215973,CS202104,86
20215990,SH202127,64.5

Command line:

./ultipa-importer --config ./in.yml

in.yml：

server:
  host: "192.168.35.151:60024"
  username: employee533
  password: joaedSSGsdf
  crt: ""
  graphset: test_graph
  
nodeConfig:
  - schema: student
    file: ./student.txt
    types:
      - name: age
        type: uint32
  - schema: course
    file: ./course.txt
    types:
      - name: title
      - name: professor
      - name: crsNo
        type: _id
      - name: credit
        type: float

edgeConfig:
  - schema: enroll
    file: ./enroll.csv
    properties:
      - name: _from
        type: _from
      - name: _to
        type: _to

settings:
  separator: ","
  yes: true
  fitToHeader: true

Remote Export

Example: export from a remote database all node properties of @student, node properties _id and 'title' of @course, and all edge properties of all schemas.

Command line:

./ultipa-exporter --config ./out.yml

out.yml：

server:
  host: "192.168.35.151:60024"
  username: employee533
  password: joaedSSGsdf
  crt: ""
  graphset: test_graph
  
nodeConfig:
  - schema: student
  - schema: course
    properties:
      - name: _id
      - name: title

edgeConfig:
  - schema: *

settings:
  writeHeader: false
  outPath: ./export/temp

FAQ

Q: What system environment supports running Transporter (GO Version)?

A: Transporter (GO Version) is supported by MacOS, Windows and Linux. Make sure to place the node/edge file to be imported, the yml file and the importer/exporter under the same directory, and run the importer/exporter with command line tool.