index¶

S3 Vector Index Management

This module provides functionality for managing S3 vector indexes, including index creation, vector storage, and vector querying operations. It integrates with AWS S3 Vectors service to provide a Pythonic interface for vector search operations.

The module includes classes for:

Index management and operations (Index)
Query result processing (QueryVectorsOutput)

These classes handle the complexities of AWS S3 Vectors API interactions, data format conversions, and error handling.

class s3vectorm.index.VectorsOutputMixin(data_type: Literal['float32'])[source]¶

Mixin providing common functionality for vector operation outputs.

This mixin provides functionality for converting raw AWS S3 Vectors responses into Vector objects. It is used by both QueryVectorsOutput and ListVectorsOutput to provide consistent vector conversion capabilities.

Parameters:

data_type – The data type of the vectors in the response
boto3_raw_data – The raw response data from AWS (inherited)

Example:

>>> output = QueryVectorsOutput(
...     boto3_raw_data=s3vectors_client_query_vectors_response,
...     data_type="float32"
... )
>>> vectors = output.as_vector_objects(Vector)

as_vector_objects(vector_class: Type[VectorT]) → list[VectorT][source]¶

Convert query results into Vector objects.

This method transforms the raw AWS query response into a list of Vector objects, handling data type conversion, metadata extraction, and validation errors. It provides helpful error messages for common issues like missing metadata fields.

Parameters:: vector_class – The Vector class to use for creating vector objects
Returns:: A list of Vector objects created from the query results. Returns an empty list if no vectors were found.

Raises:

ValidationError: If vector creation fails due to missing required fields ValueError: If metadata fields are missing from the response, with

guidance on enabling metadata in the query

Example:

>>> output = QueryVectorsOutput(
...     boto3_raw_data={
...         "vectors": [
...             {
...                 "key": "doc1",
...                 "data": {"float32": [0.1, 0.2, 0.3]},
...                 "distance": 0.95,
...                 "metadata": {"category": "documents"}
...             }
...         ]
...     },
...     data_type="float32"
... )
>>> vectors = output.to_vectors(Vector)
>>> print(vectors[0].key)  # "doc1"

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/query_vectors.html

class s3vectorm.index.QueryVectorsOutput(data_type: DataTypeType, boto3_raw_data: type_defs.QueryVectorsOutputTypeDef)[source]¶: Query vectors operation output.

See also

VectorsOutputMixin

class s3vectorm.index.ListVectorsOutput(data_type: DataTypeType, boto3_raw_data: type_defs.ListVectorsOutputTypeDef)[source]¶: List vectors operation output.

See also

VectorsOutputMixin

class s3vectorm.index.Index(*, bucket_name: str, index_name: str, data_type: Literal['float32'], dimension: int, distance_metric: Literal['cosine', 'euclidean'])[source]¶

Represents a vector index in AWS S3 Vectors service.

This class provides a Pythonic interface for managing vector indexes, including creation, vector storage, and similarity search operations. It handles the complexities of AWS API interactions and data format conversions.

Parameters:

bucket_name – Name of the S3 vector bucket containing the index
index_name – Unique name for the vector index
data_type – Data type for vector embeddings (e.g., “float32”)
dimension – Dimensionality of the vectors (e.g., 768 for many LLM embeddings)
distance_metric – Distance metric for similarity calculations (e.g., “cosine”, “euclidean”)

Example:

>>> index = Index(
...     bucket_name="my-vectors",
...     index_name="documents",
...     data_type="float32",
...     dimension=768,
...     distance_metric="cosine"
... )
>>> # Create the index
>>> result = index.create(s3_vectors_client)
>>> # Store vectors
>>> vectors = [Vector(key="doc1", data=[0.1, 0.2, 0.3])]
>>> index.put_vectors(s3_vectors_client, vectors)
>>> # Query similar vectors
>>> results = index.query_vectors(s3_vectors_client, [0.1, 0.2, 0.3])

create(s3_vectors_client: S3VectorsClient, vector_bucket_arn: str = OPT, metadata_configuration: MetadataConfigurationTypeDef = OPT) → dict[str, Any] | None[source]¶

Create the vector index in AWS S3 Vectors service.

This method creates a new vector index with the specified configuration. It handles the common case where an index with the same name already exists by returning None instead of raising an exception.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
vector_bucket_arn – Optional ARN of the vector bucket. If provided, takes precedence over bucket_name
metadata_configuration – Optional configuration for metadata fields that can be used for filtering

Returns:

A dictionary containing the AWS response if the index was created successfully, or None if the index already exists.

Raises:

botocore.exceptions.ClientError: For AWS errors other than index: name conflicts (e.g., permission errors, invalid parameters).

Example:

>>> index = Index(
...     bucket_name="my-bucket",
...     index_name="documents",
...     data_type="float32",
...     dimension=768,
...     distance_metric="cosine"
... )
>>> result = index.create(s3_vectors_client)
>>> if result:
...     print("Index created successfully")
... else:
...     print("Index already exists")

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/create_index.html

delete(s3_vectors_client: S3VectorsClient, index_arn: str = OPT)[source]¶

Delete the vector index.

Note

Delete index will also delete all vectors in the index. You don’t need to delete vectors one by one before deleting the index.

Reference:: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/delete_index.html

classmethod get(s3_vectors_client: S3VectorsClient, vector_bucket_name: str, index_name: str = OPT, index_arn: str = OPT)[source]¶

Retrieve an existing vector index from AWS S3 Vectors service.

This method fetches the index configuration and metadata from AWS, allowing you to reconstruct an Index object for further operations. You can specify either the index name or the index ARN to identify the index.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
vector_bucket_name – Name of the S3 vector bucket containing the index
index_name – Optional name of the vector index to retrieve
index_arn – Optional ARN of the vector index to retrieve

Returns:

An Index object representing the retrieved index

Raises:

botocore.exceptions.ClientError – If the index does not exist or AWS returns an error

Example:

>>> index = Index.get(
...     s3_vectors_client,
...     vector_bucket_name="my-vectors",
...     index_name="documents"
... )
>>> print(index.dimension)  # Prints the index dimension

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/get_index.html

classmethod new_for_delete(bucket_name: str, index_name: str)[source]¶

Create an Index object for deletion operations only, we don’t need data_type, dimension, distance_metric.

Parameters:

bucket_name – Name of the S3 vector bucket containing the index
index_name – Unique name for the vector index

Returns:

An Index object configured for deletion operations.

classmethod new_for_delete_from_list_index_response(response: ListIndexesOutput)[source]¶

Create Index objects for deletion from s3vectorm.bucket.Bucket.list_index() response.

Parameters:: response – The response from the list_index operation
Returns:: A list of Index objects configured for deletion operations.

put_vectors(s3_vectors_client: S3VectorsClient, vectors: list[Vector])[source]¶

Store vectors in the index.

This method uploads a list of vectors to the S3 Vectors index, automatically converting each vector to the appropriate format required by the AWS API.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
vectors – List of Vector objects to store in the index

Example:

>>> vectors = [
...     Vector(key="doc1", data=[0.1, 0.2, 0.3], category="documents"),
...     Vector(key="doc2", data=[0.4, 0.5, 0.6], category="documents")
... ]
>>> index.put_vectors(s3_vectors_client, vectors)

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/put_vectors.html

query_vectors(s3_vectors_client: S3VectorsClient, data: list[float], top_k: int = 10, filter: Expr | CompoundExpr | None = None, return_metadata: bool = False, return_distance: bool = False) → QueryVectorsOutput[source]¶

Query the index for vectors similar to the provided query vector.

This method performs a similarity search in the vector index, returning the most similar vectors based on the configured distance metric. It supports filtering by metadata and optional return of metadata and distance values.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
data – Query vector as a list of float values
top_k – Maximum number of similar vectors to return (default: 10)
filter – Optional filter expression for metadata-based filtering
return_metadata – Whether to include metadata in the results (default: False)
return_distance – Whether to include distance values in the results (default: False)

Returns:

A QueryVectorsOutput object containing the search results

Example:

>>> # Basic similarity search
>>> results = index.query_vectors(
...     s3_vectors_client,
...     data=[0.1, 0.2, 0.3],
...     top_k=5
... )
>>> vectors = results.to_vectors(Vector)

>>> # Search with metadata filtering
>>> from s3vectorm.metadata import Expr
>>> filter_expr = Expr(field="category", operator="$eq", value="documents")
>>> results = index.query_vectors(
...     s3_vectors_client,
...     data=[0.1, 0.2, 0.3],
...     filter=filter_expr,
...     return_metadata=True,
...     return_distance=True
... )

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/query_vectors.html

list_vectors(s3_vectors_client: S3VectorsClient, index_arn: str = OPT, segment_count: int = OPT, segment_index: int = OPT, return_data: bool = OPT, return_metadata: bool = OPT, page_size: int = 100, max_items: int = 9999) → Generator[ListVectorsOutput, None, None][source]¶

List all vectors in the index with pagination support.

This method retrieves vectors from the S3 Vectors index using pagination to handle large result sets efficiently. It supports segmentation for parallel processing and optional return of vector data and metadata.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
index_arn – Optional ARN of the vector index. If provided, takes precedence over index_name
segment_count – Total number of segments for parallel processing
segment_index – Index of the segment to retrieve (0-based)
return_data – Whether to include vector data in the results
return_metadata – Whether to include metadata in the results
page_size – Number of vectors per page (default: 100)
max_items – Maximum total number of vectors to retrieve (default: 9999)

Yields:

ListVectorsOutput objects containing paginated vector results

Example:

>>> for page in index.list_vectors(
...     s3_vectors_client,
...     return_data=True,
...     return_metadata=True,
...     page_size=50
... ):
...     vectors = page.as_vector_objects(Vector)
...     print(f"Retrieved {len(vectors)} vectors")

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/paginator/ListVectors.html

delete_vectors(s3_vectors_client: S3VectorsClient, keys: list[str], index_arn: str = OPT)[source]¶

Delete specific vectors from the index by their keys.

This method removes vectors from the S3 Vectors index based on their unique keys. This is useful for selective removal of vectors that are no longer needed or need to be updated.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
keys – List of vector keys (unique identifiers) to delete
index_arn – Optional ARN of the vector index. If provided, takes precedence over index_name

Example:

>>> # Delete specific vectors by their keys
>>> keys_to_delete = ["doc1", "doc2", "doc3"]
>>> index.delete_vectors(s3_vectors_client, keys_to_delete)

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3vectors/client/delete_vectors.html

delete_all_vectors(s3_vectors_client: S3VectorsClient, page_size: int = 100, max_items: int = 9999) → int[source]¶

Delete all vectors in the index.

This method provides a convenient way to delete all vectors from the index by first listing them with pagination and then deleting them in batches. This is useful for clearing an index without deleting the index structure itself.

Parameters:

s3_vectors_client – The AWS S3 Vectors client to use for the operation
page_size – Number of vectors to process per page (default: 100)
max_items – Maximum total number of vectors to delete (default: 9999)

Returns:

The total number of vectors that were deleted

Example:

>>> deleted_count = index.delete_all_vectors(s3_vectors_client)
>>> print(f"Deleted {deleted_count} vectors from the index")

Note

This operation may take some time for large indexes as it processes vectors in batches. For very large indexes, consider using segmentation parameters in list_vectors for parallel processing.

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].