Vector Index

Overview

A vector index is designed to efficiently store and manage high-dimensional vectors (or embeddings), enabling fast retrieval of similar vectors based on a chosen similarity metric. Instead of performing an exhaustive search across all stored vectors, the vector index significantly reduces the search space, making nearest-neighbor retrieval more efficient.

Vectors and Embeddings

Vectors can be viewed as an ordered list of numbers. For example, the vector [1, 2] represents a direction from the origin to the point (1, 2) in a two-dimensional space, and the distance (or magnitude) to that point.

In the context of machine learning and natural language processing (NLP), the term embedding is commonly used when referring vectors. An embedding is a high-dimensional vector that represents data, often with hundreds of dimensions.

Creating Embeddings

You can create embeddings for both structured data (e.g., text) and unstructured data (e.g., images, graphs) using various models. Some popular models include:

• OpenAI: A collection of models to turn text into embeddings.
• Node2Vec: A model that generates embeddings for graph nodes based on their relationships.
• ResNet (Residual Networks): A series of models for generating image embeddings.

Why Embeddings

Embeddings are often generated by considering the context of the data entity rather than just the data itself. This is especially true for tasks like NLP, where the meaning of a word is influenced by its surrounding context.

Consider the word "bank", which has multiple meanings depending on the context:

1. Bank as a financial institution: "I went to the bank to deposit some money."
2. Bank as the side of a river: "We sat on the bank of the river to watch the sunset."

This contextual information makes embeddings much more powerful for downstream tasks like semantic search, recommendation and classification.

Loading Embeddings into Ultipa

After creating embeddings for entities, they can be imported or stored into Ultipa as properties of type list<float> or list<double>. By creating vector indexes for these properties, you can perform vector searches.

Vector Search

Vector search refers to the process of finding vectors that are most similar to a given query vector, using a similarity measure. Ultipa supports the following similarity measures:

• L2 Distance (Euclidean Distance): Calculates the straight-line distance between two points in a high-dimensional space. This measure is sensitive to the magnitude of the vectors.

• Cosine Similarity: Measures the cosine of the angle between two vectors, indicating their orientation in space. It is independent of vector magnitude.

• Inner Product: Computes the dot product between two vectors, often used when the magnitude of the vectors is important.

Example Graph

The example graph consists of 10 Book nodes, each containing the properties name, author, summary, and summaryEmbedding. The summaryEmbedding holds the 384-dimensional text embeddings of the summary, generated using the all-MiniLM-L6-v2 model from Hugging Face.

Create the graphInsert data to the graph
CREATE GRAPH myGraph { 
  NODE Book ({title string, author string, summary text, summaryEmbedding list<double>})
} PARTITION BY HASH(Crc32) SHARDS [1]

Showing Vector Indexes

To retrieve node vector indexes in the current graph:

GQL
SHOW NODE VECTOR INDEX

The information about vector indexes is organized into a _nodeVectorIndex table with the following fields:

Creating a Vector Index

You can create a vector index using the CREATE VECTOR INDEX statement for a node property of the list<float> or list<double> type. The vector index creation runs as a job, you may run SHOW JOB <id?> afterward to verify the success of the creation.

To create a vector index named summary_embedding for the property summaryEmbedding of Book nodes:

GQL
CREATE VECTOR INDEX "summary_embedding" ON NODE Book (summaryEmbedding) OPTIONS {
  similarity_function: "COSINE",
  index_type: "FLAT",
  dimensions: 384,
  vector_server: "vector_server_1"
}

To create a vector index for all node labels:

GQL
CREATE VECTOR INDEX "all_embeddings" ON NODE * (embedding) OPTIONS {
  similarity_function: "COSINE",
  index_type: "HNSW",
  dimensions: 1536
}

Details

• The vector index name must be unique. Naming conventions are:
  • 2 to 64 characters.
  • Begins with a letter.
  • Allowed characters: letters (A-Z, a-z), numbers (0-9) and underscores (_).
• A vector index is applied to a single label and a single property. Use * to apply to all labels.
• Configurations for a vector index:

Dropping a Vector Index

You can drop a vector index using the DROP VECTOR INDEX statement. Dropping a vector index does not affect the actual property values stored in shards.

NOTE

A property with a vector index cannot be dropped until the vector index is deleted.

To drop the node vector index summary_embedding:

GQL
DROP NODE VECTOR INDEX summary_embedding

Use IF EXISTS to avoid errors when dropping a non-existent index:

GQL
DROP NODE VECTOR INDEX IF EXISTS summary_embedding

Using Vector Indexes

You can use a vector index for vector search by calling the vector.queryNodes() procedure.

Syntax

Syntax
CALL vector.queryNodes("<vectorIndexName>", <numMostSimNodes>, <targetVector>)

Parameters

Returns

• _uuid: _uuid of the retrieved node.
• score: The similarity score between the vector of retrieved node and the target vector.

Example

Finds the top two books most similar to Pride and Prejudice by the vector index summary_embedding, return the book names along with the similarity scores between them:

GQL
MATCH (target:Book {title: "Pride and Prejudice"})
CALL vector.queryNodes('summary_embedding', 3, target.summaryEmbedding)
YIELD result
MATCH (book WHERE book._uuid = result._uuid)
RETURN table(book.title, result.score)

Result:

Annex: Index Types