Loader

You can import data from various sources into Ultipa Graph in the Loader module. Supported data sources include CSV, JSON, JSONL, PostgreSQL, SQL Server, MySQL, BigQuery, Neo4j and Kafka.

Loader and Task

Within this module, you can create loaders, each containing multiple import tasks. Tasks within a loader can be executed individually or simultaneously (either serially or in parallel).

Create a Loader

Clicking the New Loader button on the Loader main page will create a loader with the default name My Loader.

On the loader configuration page, you can rename the loader and modify its description by clicking the icon located next to the loader name.

Create a Task

To add a new task on the loader configuration page, click the New Task button and choose the type of data source for the task. You can add multiple tasks in a loader.

You can set the task name by double-clicking the corresponding area on the left side task card. On the right side is the task configurations. Please refer to the Appendix: Task Configurations for details.

Import

Import Single Task

Click the Import button in certain task will execute the task immediately.

Run Loader

Click the Run button above the task list will trigger the import for all tasks within the loader, adhering to the designated order and processing mode:

• Tasks are executed from top to bottom. You can reorder tasks by dragging and dropping the task cards.
• Use the S/P tag on each task card to toggle between the Serial (one after the other; default) and Parallel (simultaneous) processing modes.
• Adjacent P tasks along with the S task immediately preceding the top P task form a parallel group, imported concurrently.

NOTE

Errors may arise when importing nodes and edges concurrently, given that the creation of edges depends on the existence of their terminal nodes. Alternatively, consider importing nodes before edges, or ensure proper configuration of the Create Node If Not Exist option and the Insert/Upsert/Overwrite import mode.

Import Log

Typically, the SPEED indicator and process bar become visible on the task card once the import begins. A 100% successful import is denoted by a full green progress bar. However, if any errors occur during the task, a portion of the process bar turns yellow, reflecting the percentage of failed data rows.

At the top of the right-side configuration panel, the latest import log is displayed. Here, you can download the Failed Rows and Logs files.

Import/Export Loader

You can export any loader as a ZIP file from the loader card. The option to import a loader is available at the top right of the Loader main page.

Appendix: Task Configurations

Settings

Task configurations vary for different data sources. However, the Settings part is common to all.

NOTE

Any new graph, schema, or property is flagged with a warning mark, indicating that they will be created during the import process.

CSV

File

Here is an example CSV file for @account nodes, including a header with property names and types:

account.csv
_id:_id,username:string,brithYear:int32
U001,risotto4cato,1978
U002,jibber-jabber,1989
U003,LondonEYE,1982

Here is an example headless CSV file for @follow edges:

follow.csv
103,risotto4cato,jibber-jabber,1634962465
102,LondonEYE,jibber-jabber,1634962587

Mapping

After uploading or selecting the file, the Mapping section becomes visible:

JSON, JSONL

File

Here is an example JSON file for @user nodes:

user.json
[{
	"_uuid": 1,
	"_id": "U001",
	"level": 2,
	"registeredOn": "2018-12-1 10:20:23",
	"tag": null
}, {
	"_uuid": 2,
	"_id": "U002",
	"level": 3,
	"registeredOn": "2018-12-1 12:45:12",
	"tag": "cloud"
}]

Here is an example JSONL file for @user nodes:

user.jsonl
{"_uuid": 1, "_id": "U001", "level": 2, "registeredOn": "2018-12-1 10:20:23", "tag": null}
{"_uuid": 2, "_id": "U002", "level": 3, "registeredOn": "2018-12-1 12:45:12", "tag": "cloud"
}

Mapping

After uploading or selecting the file, the Mapping section becomes visible:

PostgreSQL, SQL Server, MySQL

Source Database

SQL

Write the SQL query to retrieve data from the database, then click Preview to map the results to the properties of the nodes or edges.

Here is an example SQL to return the name and registeredOn columns from the users table:

SQL
SELECT name, registeredOn FROM users;

Mapping

Once the SQL returns results, the Mapping section becomes visible:

BigQuery

Source Database

SQL

Write the SQL query to retrieve data from BigQuery, then click Preview to map the results to the properties of the nodes or edges.

Here is an example SQL to return all columns from the users table:

SQL
SELECT * FROM users;

Mapping

Once the SQL returns results, the Mapping section becomes visible:

Neo4j

Neo4j

Cypher

Write the Cypher query to retrieve nodes or edges from Neo4j, then click Preview to map the results to the properties of the nodes or edges.

Here is an example Cypher to return all nodes with the user label:

Cypher
MATCH (n:user)

Mapping

Once the Cypher returns results, the Mapping section becomes visible:

Kafka

Kafka

Mapping

Once there are messages returned after clicking Preview, the Mapping section becomes visible:

NOTE

The Kafka task (consumer), once started, will continue running to import newly published messages. You need to manually stop the task as needed.