General Format
- Configure schemas under
nodeConfig
andedgeConfig
:
nodeConfig:
# Configure the first node schema
# Schema name
- schema:
# (mysql/postgresSQL/sqlserver/bigQuery) Sql that extracts data fields
sql:
# (csv/json/jsonl) Path of data file
file:
# (csv) Whether the CSV file has header, 'true' by default
head:
# (kafka) Kafka topic
topic:
# (kafka) Kafka offset, supports 'newest', 'oldest', '5', '2006-01-02 15:04:05 -0700' formats
offset:
# (neo4j) label name in Neo4J
label:
# (neo4j) Use variable 'n' to filter the data of this label, eg, n.released > 1999
where:
# (Optional) Manually configure the name and/or type of data fields
properties:
# (Optional) The number of data rows to skip from the beginning
skip:
# (Optional) The maximum number of data rows to inject
limit:
# Configure the second node schema
- schema:
sql/file&head/topic&offset:
properties:
skip:
limit:
# Configure more node schemas
...
# (csv) Configure the first node schema folder, in which the CSV files should follow the naming conventions same as those generated by an export operation, which is <schema>.node.csv with complete headers
- dir:
# (csv) Configure more node schema folders
...
edgeConfig:
# Configure the first edge schema
- schema:
sql/file&head/topic&offset:
properties:
skip:
limit:
# Configure more edge schemas
...
# (csv) Configure the first edge schema folder, in which the CSV files should follow the naming conventions same as those generated by an export operation, which is <schema>.edge.csv with complete headers
- dir:
# (csv) Configure more edge schema folders
...
- Congifure data fields under
properties
:
nodeConfig:
- schema:
sql/file&head/topic&offset:
properties:
# Configures the first data field
# Name of data field
- name:
# (Optional) Property name
new_name:
# (Optional) Property type
type:
# (Optional) Prefix to be appended before the data of _id, _from or _to
prefix:
# Configures the second data field
- name:
new_name:
type:
prefix:
# Configures more data fields
...
skip:
limit:
...
- When and how should data fields be configured:
- when field name is not existent, illegal or inconsistent with the name of target property, configure field name with
new_name
- when field type is not existent, illegal or inconsistent with the name of target property, configure field name with
type
- when field name is not existent, illegal or inconsistent with the name of target property, configure field name with
Legality of field name: refer to Property Naming Convention.
Legality of field type:
For system properties:_id
,_uuid
,_from
,_to
,_from_uuid
,_to_uuid
For custom properties: refer to Property Types
For fields to be ignored:_ignore
Example: MySQL/PostgreSQL/SQLServer/BigData
- Use
sql
to acquire data fields - Configure data fields under
properties
(for system properties,type
must be configured)
Example: Import field _id as _id
of node, import fields payer and payee as _from
and _to
of edge
nodeConfig:
- schema: "card"
sql: "select id from account"
properties:
- name: _id
type: _id
edgeConfig:
- schema: "transfer"
sql: "select payer, payee from transaction"
properties:
- name: payer
type: _from
- name: payee
type: _to
Example: Import field a as node property age (with same type); import field level (type text) as node property level (type int32); import field username (type text) as node property name (type string)
nodeConfig:
- schema: "human"
sql: "select a, level, username from entity where type = 1"
properties:
- name: a
new_name: age
- name: level
type: int32
- name: username
new_name: name
type: string
Field type that is ultimately illegal will trigger error.
Example: Kafka
Example: Ignore the first field and import the rest of fields as node _id
, name
and age
, prefixing 'order_' to _id
nodeConfig:
- schema: "student"
topic: "studentTest"
offset: newest
properties:
- name: ignore
type: _ignore
- name: _id
type: _id
prefix: order_
- name: name
type: string
- name: age
type: int32
Example: Neo4J
Example: Import all properties of label 'client' from Neo4J to Ultipa schema 'customer', where property level is greater than 3, retain the property names and types
nodeConfig:
- schema: "customer"
label: "client"
where: n.level > 3
properties:
- name: CustomerID
type: _id
Example: CSV (With Header)
- Header format:
<name>
or<name>:<type>
- Use
file
to acquire data file - Configure data fields under
properties
Example: Revise the configuration of fields of node file reviewer.csv based on its header
username:string,age:string,birthday:datetime
Dean J Copely,10,1987-11-02
Teresa Halbert,5,2001-08-14
Mike C,9,1998-02-19
nodeConfig:
- schema: reviewer
file: ./reviewer.csv
head: true
properties:
- name: age
type: uint32
- name: username
new_name: name
Example: Supplement the configuration of fields of node file city.csv based on its header
_id,name,address
CITY001,Beijing,POINT(39.9 116.3)
CITY002,New York,POINT(40.7 -74)
CITY003,Paris,POINT(48.5 2.2)
nodeConfig:
- schema: city
file: ./city.csv
head: true
properties:
- name: _id
type: _id
- name: address
type: point
Field type not existent and not configured will be considered as string; field type that is ultimately illegal will trigger error.
Example: CSV (No Header)
- Use
file
to acquire data file - Configure data fields under
properties
(sequence sensitive)
Example: Configure fields of edge file review.csv, knowing that the fields represent start node, end node, rating, comment adnd tags
A2CMX45JPSCTUJ,B0002CZSJO,5,The Best Cable,"[possitive,rebuy]"
A3EIML9QZO5NZZ,B0000AQRSU,5,awesome,"[possitive]"
A3C9F3SZWLWDZF,B000165DSM,2,worse than the one i bought last time,"[negative,rebuy]"
A1C60KQ8VJZBS5,B0002CZV82,4,Makes changing strings a breeze,"[possitive]"
edgeConfig:
- schema: review
file: ./review.csv
head: false
properties:
- name: _from
type: _from
- name: _to
type: _to
- name: rating
type: int32
- name: comment
- name: tags
type: string[]
Field type not configured will be considered as string; field type that is ultimately illegal will trigger error.
Example: CSV Folder
- Use
dir
to acquire data files in a folder - Data files should be named as
<schema>.nodes.csv
or<schema>.edges.csv
, with complete headers as<name>:<type>
Example: Import CSV files card.node.csv and transfer.edge.csv from local folder './import/'
_id:_id,level:int32,type:string
CARD001,3,credit
CARD002,4,credit
CARD003,1,debit
CARD004,4,debit
_from:_from,_to:_to,amount:double
CARD001,CARD002,2672.97
CARD002,CARD004,100
CARD003,CARD001,6.5
CARD004,CARD003,235.34
nodeConfig:
- dir: "./import/"
edgeConfig:
- dir: "./import/"
Example: JSON
- File content: array of json
- Use
file
to acquire data file - Configure data fields under
properties
- Keys not configured will not be imported
Example: Configure node file reviewer.json and import
[
{"username":"Dean J Copely", "age":10, "birthday":"1987-11-02"},
{"username":"Teresa Halbert", "age":5 "birthday":"2001-08-14"},
{"username":"Mike C", "age":9 "birthday":"1998-02-19"}
]
nodeConfig:
- schema: reviewer
file: ./reviewer.json
properties:
- name: age
type: uint32
- name: username
new_name: name
- name: birthday
type: datetime
Example: JSONL
- File content: lines of json
- Use
file
to acquire data file - Configure data fields under
properties
- Keys not configured will not be imported
Example: Configure node file city.jsonl and import
{"_id":"CITY001", "name":"Beijing", "address":"POINT(39.9 116.3)"}
{"_id":"CITY002", "name":"New York", "address":"POINT(40.7 -74)"}
{"_id":"CITY003", "name":"Paris", "address":"POINT(48.5 2.2)"}
nodeConfig:
- schema: city
file: ./city.jsonl
properties:
- name: _id
type: _id
- name: name
- name: address
type: point
Field type not configured will be considered as string; field type that is ultimately illegal will trigger error.