This section introduces how to manage, configure, and start projects.
View Project
User can view multiple projects' information in Project page. Viewable items are listed below:
Items |
Description |
|---|---|
| Project Name | The name of a project. |
| ETL | Total ETLs in a project. |
| Created at | The time when a project is started at. |
| Started at | The last time when a project is started at. |
| Status | The status of a project. "Started" represents that a project has been started as scheduled and is put into action. "Stopped" represents that the operation of a project has been paused. |
| Description | Remarks about a project. |
Create Project
To create a project, click "+ Create" in Project page, give a name and a description to the new project as shown below:
Delete Project
To delete a project, find target project and click "Delete" in its accordant row.
Configure Project
To enter project configurations, find target project in Project Page and click "View" under "Action".
ETL (Extract, Transfer, Load) is the smallest unit of data migration in a project. It represents a single and complete data migration process composed of extracting, transfering, and loading of data. Each project must contain at least one ETL.
Hence to create, configure, and sequence ETLs are the major steps for a project's data migration.
Create ETL
Click "Create ETL" under the project and enter an ETL name as shown below:
Complete ETL Setting
Complete the settings below after clicking "Configure" for each ETL.
| Parameters | Description |
|---|---|
| Load Number (per batch) | The maximum amount of data allowed to load, must be equal to or less than 1,000,000. |
| Inserting Method | When there is no data in current ETL, select "Normal" as the mode of imports; when current ETL has history data, select "Overwrite" to update or clear all properties other than unique identifiers UUID and ID, and properties carried by data inserting will be updated with uncarried data; "Upsert" means to update values of properties other than UUID and ID; only properties carried in the data will be updated, while those are not will remain unchanged. User can decide based on projects. |
| Concurrency Number | The maximum concurrent threads allowed, must be equal to or less than 100. |
| Create nodes if not exist | Switching it on means to automatically create nodes when there are no projection existing for import data in the graph; switching it off represents ignoring them in the graph. |
| Stop when error | This switch is used to automatically stop ETL process; switching it off represents continuing ETL progress in the occasion of error. |
When "Stop when error" is switched on, system will stop data importing in the occurence of errors, and keep data that has been imported. User needs to select "Overwrite" as the inserting method, and operate this ETL again to complete this data migration process; if options other than "Overwrite" are selected, there might be chances that this ETL fails again.
For instance:
Set Extraction Source
Click "Configure" icon of Extraction Source module in current ETL and complete the extraction source settings below.
| Parameters | Description |
|---|---|
| Source | All of the available extraction sources for the user to select from. |
| Database | The database under selected data source to extract data from. |
| Query Scripts | ETL query scripts come from query statements from different data platforms and are syntactically different. Ultipa Maker has embedded query templates of each data platform for the user to match with selected extraction sources. Not applicable for CSV and DAT files. |
An example of Neo4j query scripts:
MATCH (n:Person) RETURN n
Meaning: find all Person nodes and return.
An example of Hive query scripts:
select * from mini_circle_edge_agree2 limit 10
Meaning: find 10 nodes from graphset named mini_circle_edge_agree2.
As shown below:
Set Loading Source
Click "Configure" icon of Loading Source module in current ETL and complete the loading source settings below.
| Parameters | Description |
|---|---|
| Instance | All of the available Ultipa Graph instances for the user to load data to. |
| Graphset | The available graphset under the selected Ultipa Graph instance for the user to load data to. |
| Schema | The graph schema for the user to create projection between extracted data and the graph to load data to. |
As shown below:
Set Graph Schema
After configurations for both extraction source and loading source are completed, a projection setting area will show up below. User can select schema properties in Ultipa Graph for import data to form mapping relations:
Set ETL Order
When more than one ETLs are created for one project, the operations of ETLs should be sequenced based on projects' demands, for instance:
Schedule Project
User can schedule the occurrences of a data migration project by project scheduler and project trigger.
Set Scheduler
Project scheduler sets the frequency and the time when the project should start. User can complete parts of or all of the settings below.
Set Trigger
Project trigger sets the rules where the project should start. User can complete parts of or all of the settings below.
Start Project
After all ETLs in a project are configured and scheduled successfully, user can click "Start" to start operating the project as scheduled:
Note: projects in starting status cannot be modified, the user needs to stop projects before any re-configurations.
View Logs
After a project is started, user can click the "Log" button in each ETL to view the data migration details for each ETL.
