General Format

• Configure schemas under nodeConfig and edgeConfig:

nodeConfig:

  # Configure the first node schema
    # Schema name
  - schema:
    # (mysql/postgresSQL/sqlserver/bigQuery) Sql that extracts data fields
    sql:
    # (csv/json/jsonl) Path of data file
    file:
    # (csv) Whether the CSV file has header, 'true' by default
    head:
    # (kafka) Kafka topic
    topic:
    # (kafka) Kafka offset, supports 'newest', 'oldest', '5', '2006-01-02 15:04:05 -0700' formats
    offset:
    # (neo4j) label name in Neo4J
    label:
    # (neo4j) Use variable 'n' to filter the data of this label, eg, n.released > 1999
    where:
    # (Optional) Manually configure the name and/or type of data fields
    properties:
    # (Optional) The number of data rows to skip from the beginning
    skip: 
    # (Optional) The maximum number of data rows to inject
    limit: 

  # Configure the second node schema
  - schema:
    sql/file&head/topic&offset:
    properties:
    skip: 
    limit: 

  # Configure more node schemas
  ...
  
  # (csv) Configure the first node schema folder, in which the CSV files should follow the naming conventions same as those generated by an export operation, which is <schema>.node.csv with complete headers
  - dir:
  
  # (csv) Configure more node schema folders
  ...

edgeConfig:  

  # Configure the first edge schema
  - schema:
    sql/file&head/topic&offset:
    properties:
    skip: 
    limit: 

  # Configure more edge schemas
  ...
  
  # (csv) Configure the first edge schema folder, in which the CSV files should follow the naming conventions same as those generated by an export operation, which is <schema>.edge.csv with complete headers
  - dir:
  
  # (csv) Configure more edge schema folders
  ...

• Congifure data fields under properties:

nodeConfig:
  - schema:
    sql/file&head/topic&offset:
    properties:
    
      # Configures the first data field
        # Name of data field
      - name:
        # (Optional) Property name
        new_name:
        # (Optional) Property type
        type:
        # (Optional) Prefix to be appended before the data of _id, _from or _to
        prefix:
        
      # Configures the second data field
      - name:
        new_name:
        type:
        prefix:
        
      # Configures more data fields
      ...  
      
    skip: 
    limit: 
  ...

• When and how should data fields be configured:
  • when field name is not existent, illegal or inconsistent with the name of target property, configure field name with new_name
  • when field type is not existent, illegal or inconsistent with the name of target property, configure field name with type

Legality of field name: refer to Property Naming Convention.

Legality of field type：
For system properties: _id, _uuid, _from, _to, _from_uuid, _to_uuid
For custom properties: refer to Property Types
For fields to be ignored: _ignore

Example: MySQL/PostgreSQL/SQLServer/BigData

• Use sql to acquire data fields
• Configure data fields under properties (for system properties, type must be configured)

Example: Import field _id as _id of node, import fields payer and payee as _from and _to of edge

nodeConfig:
  - schema: "card"
    sql: "select id from account"
    properties:
      - name: _id
        type: _id

edgeConfig:
  - schema: "transfer"
    sql: "select payer, payee from transaction"
    properties:
      - name: payer
        type: _from
      - name: payee
        type: _to

Example: Import field a as node property age (with same type); import field level (type text) as node property level (type int32); import field username (type text) as node property name (type string)

nodeConfig:
  - schema: "human"
    sql: "select a, level, username from entity where type = 1"
    properties:
      - name: a
        new_name: age
      - name: level
        type: int32
      - name: username
        new_name: name
        type: string

Field type that is ultimately illegal will trigger error.

Example: Kafka

Example: Ignore the first field and import the rest of fields as node _id, name and age, prefixing 'order_' to _id

nodeConfig:
  - schema: "student"
    topic: "studentTest"
    offset: newest
    properties:
      - name: ignore
        type: _ignore
      - name: _id
        type: _id
        prefix: order_
      - name: name
        type: string
      - name: age
        type: int32

Example: Neo4J

Example: Import all properties of label 'client' from Neo4J to Ultipa schema 'customer', where property level is greater than 3, retain the property names and types

nodeConfig:
  - schema: "customer"
    label: "client"
    where: n.level > 3
    properties:
      - name: CustomerID
        type: _id

Example: CSV (With Header)

• Header format: <name> or <name>:<type>
• Use file to acquire data file
• Configure data fields under properties

Example: Revise the configuration of fields of node file reviewer.csv based on its header

username:string,age:string,birthday:datetime
Dean J Copely,10,1987-11-02
Teresa Halbert,5,2001-08-14
Mike C,9,1998-02-19

nodeConfig:
  - schema: reviewer
    file: ./reviewer.csv
    head: true
    properties:
      - name: age
        type: uint32
      - name: username
        new_name: name

Example: Supplement the configuration of fields of node file city.csv based on its header

_id,name,address
CITY001,Beijing,POINT(39.9 116.3)
CITY002,New York,POINT(40.7 -74)
CITY003,Paris,POINT(48.5 2.2)

nodeConfig:
  - schema: city
    file: ./city.csv
    head: true
    properties:
      - name: _id
        type: _id
      - name: address
        type: point

Field type not existent and not configured will be considered as string; field type that is ultimately illegal will trigger error.

Example: CSV (No Header)

• Use file to acquire data file
• Configure data fields under properties (sequence sensitive)

Example: Configure fields of edge file review.csv, knowing that the fields represent start node, end node, rating, comment adnd tags

A2CMX45JPSCTUJ,B0002CZSJO,5,The Best Cable,"[possitive,rebuy]"
A3EIML9QZO5NZZ,B0000AQRSU,5,awesome,"[possitive]"
A3C9F3SZWLWDZF,B000165DSM,2,worse than the one i bought last time,"[negative,rebuy]"
A1C60KQ8VJZBS5,B0002CZV82,4,Makes changing strings a breeze,"[possitive]"

edgeConfig:
  - schema: review
    file: ./review.csv
    head: false
    properties:
      - name: _from		
        type: _from
      - name: _to
        type: _to
      - name: rating
        type: int32
      - name: comment
      - name: tags
        type: string[]

Field type not configured will be considered as string; field type that is ultimately illegal will trigger error.

Example: CSV Folder

• Use dir to acquire data files in a folder
• Data files should be named as <schema>.nodes.csv or <schema>.edges.csv, with complete headers as <name>:<type>

Example: Import CSV files card.node.csv and transfer.edge.csv from local folder './import/'

_id:_id,level:int32,type:string
CARD001,3,credit
CARD002,4,credit
CARD003,1,debit
CARD004,4,debit

_from:_from,_to:_to,amount:double
CARD001,CARD002,2672.97
CARD002,CARD004,100
CARD003,CARD001,6.5
CARD004,CARD003,235.34

nodeConfig:
  - dir: "./import/"
  
edgeConfig:
  - dir: "./import/"

Example: JSON

• File content: array of json
• Use file to acquire data file
• Configure data fields under properties
• Keys not configured will not be imported

Example: Configure node file reviewer.json and import

[
  {"username":"Dean J Copely", "age":10, "birthday":"1987-11-02"},
  {"username":"Teresa Halbert", "age":5 "birthday":"2001-08-14"},
  {"username":"Mike C", "age":9 "birthday":"1998-02-19"}
]

nodeConfig:
  - schema: reviewer
    file: ./reviewer.json
    properties:
      - name: age
        type: uint32
      - name: username
        new_name: name
      - name: birthday
        type: datetime

Example: JSONL

• File content: lines of json
• Use file to acquire data file
• Configure data fields under properties
• Keys not configured will not be imported

Example: Configure node file city.jsonl and import

{"_id":"CITY001", "name":"Beijing", "address":"POINT(39.9 116.3)"}
{"_id":"CITY002", "name":"New York", "address":"POINT(40.7 -74)"}
{"_id":"CITY003", "name":"Paris", "address":"POINT(48.5 2.2)"}

nodeConfig:
  - schema: city
    file: ./city.jsonl
    properties:
      - name: _id
        type: _id
      - name: name
      - name: address
        type: point

Field type not configured will be considered as string; field type that is ultimately illegal will trigger error.