s3vectorm Comprehensive Tutorial

This tutorial provides a complete guide to using the s3vectorm library for managing vector data in AWS S3 Vectors service. We’ll walk through creating buckets and indexes, storing and querying vectors, and performing cleanup operations.

Setup and Imports

First, let’s import all the necessary components and set up our AWS client:

import boto3
from s3vectorm.api import (
    Bucket,
    Index,
    Vector,
    BaseMetadata,
    MetaKey,
)

# Configure your AWS credentials and region
bucket_name = "s3vectorm-tutorial-bucket"
index_name = "document-embeddings"

# Create AWS S3 Vectors client
boto_ses = boto3.Session()
s3_vectors_client = boto_ses.client("s3vectors")

Bucket Management

Creating a Vector Bucket

Before working with vector indexes, we need to create an S3 vector bucket. The bucket serves as a container for all your vector indexes:

# Create a bucket instance
bucket = Bucket(name=bucket_name)

# Create the bucket in AWS (returns None if already exists)
create_result = bucket.create(s3_vectors_client)
if create_result:
    print("✅ Bucket created successfully")
else:
    print("ℹ️ Bucket already exists")

✅ Bucket created successfully

Listing Indexes in a Bucket

You can list all indexes within a bucket using pagination. This is useful for discovering existing indexes:

# List all indexes in the bucket
print("📋 Indexes in bucket:")
for page in bucket.list_index(
    s3_vectors_client,
    prefix="document-",  # Optional: filter by prefix
    page_size=50
):
    indexes = page.indexes or []
    for index_summary in indexes:
        print(f"  - {index_summary.indexName} (dim: {index_summary.dimension})")

📋 Indexes in bucket:

Index Management

Creating a Vector Index

An index defines the structure and properties of your vector data, including dimension and distance metric:

# Create an index with specific configuration
index = Index(
    bucket_name=bucket_name,
    index_name=index_name,
    data_type="float32",
    dimension=768,  # Common dimension for many LLM embeddings
    distance_metric="cosine"
)

# Create the index in AWS
create_result = index.create(s3_vectors_client)
if create_result:
    print("✅ Index created successfully")
else:
    print("ℹ️ Index already exists")

✅ Index created successfully

Retrieving an Existing Index

You can retrieve an existing index configuration from AWS to work with it:

# Retrieve an existing index by name
existing_index = Index.get(
    s3_vectors_client,
    vector_bucket_name=bucket_name,
    index_name=index_name
)

if existing_index:
    print(f"📊 Retrieved index: {existing_index.index_name}")
    print(f"   Dimension: {existing_index.dimension}")
    print(f"   Distance metric: {existing_index.distance_metric}")
else:
    print("❌ Index not found")

📊 Retrieved index: document-embeddings
   Dimension: 768
   Distance metric: cosine

Creating Index Objects for Deletion

When you need to delete indexes without knowing their full configuration, you can create lightweight index objects:

# Create index object specifically for deletion operations
deletion_index = Index.new_for_delete(
    bucket_name=bucket_name,
    index_name=index_name
)

Vector Data Models

Defining Custom Vector Classes

Define your vector data structure by extending the base Vector class with custom metadata fields (Vector is just a subclass of pydantic.BaseModel):

from pydantic import Field

class DocumentChunk(Vector):
    """Custom vector class for document chunks with metadata"""
    document_id: str = Field(description="ID of the source document")
    chunk_seq: int = Field(description="Sequence number of the chunk")
    title: str = Field(description="Document title")
    category: str = Field(description="Document category")
    owner_id: str = Field(description="ID of the document owner")
    created_at: str = Field(description="Creation timestamp")

Understanding Vector Conversion

Vectors can be converted to different formats for AWS operations:

# Create a sample vector
sample_vector = DocumentChunk(
    key="doc-1#chunk-1",
    data=[0.1, 0.2, 0.3] * 256,  # 768-dimensional vector
    document_id="doc-1",
    chunk_seq=1,
    title="Introduction to AI",
    category="technology",
    owner_id="user-123",
    created_at="2025-01-01T10:00:00Z"
)

# Extract just the metadata
metadata = sample_vector.to_metadata_dict()
print("📋 Metadata only:", metadata)

📋 Metadata only: {'document_id': 'doc-1', 'chunk_seq': 1, 'title': 'Introduction to AI', 'category': 'technology', 'owner_id': 'user-123', 'created_at': '2025-01-01T10:00:00Z'}

Storing Vector Data

Inserting Vectors

Store multiple vectors in your index efficiently with batch operations:

# Create a collection of document vectors
vectors = [
    DocumentChunk(
        key="doc-1#chunk-1",
        data=[0.1] * 768,
        document_id="doc-1",
        chunk_seq=1,
        title="Introduction to Machine Learning",
        category="technology",
        owner_id="user-alice",
        created_at="2025-01-01T10:00:00Z"
    ),
    DocumentChunk(
        key="doc-1#chunk-2",
        data=[0.2] * 768,
        document_id="doc-1",
        chunk_seq=2,
        title="Introduction to Machine Learning",
        category="technology",
        owner_id="user-alice",
        created_at="2025-01-01T10:05:00Z"
    ),
    DocumentChunk(
        key="doc-2#chunk-1",
        data=[0.3] * 768,
        document_id="doc-2",
        chunk_seq=1,
        title="Business Strategy Guide",
        category="business",
        owner_id="user-bob",
        created_at="2025-01-01T11:00:00Z"
    ),
    DocumentChunk(
        key="doc-2#chunk-2",
        data=[0.4] * 768,
        document_id="doc-2",
        chunk_seq=2,
        title="Business Strategy Guide",
        category="business",
        owner_id="user-bob",
        created_at="2025-01-01T11:05:00Z"
    )
]

# Store all vectors in the index
index.put_vectors(s3_vectors_client, vectors)
print(f"✅ Successfully stored {len(vectors)} vectors")

✅ Successfully stored 4 vectors

Metadata Query System

Defining Metadata Models

Create queryable metadata models using inheritance for better organization:

# Base metadata class with common fields
class BaseDocumentMeta(BaseMetadata):
    document_id = MetaKey()
    chunk_seq = MetaKey()

# Extended metadata class with additional fields
class DocumentMeta(BaseDocumentMeta):
    title = MetaKey()
    category = MetaKey()
    owner_id = MetaKey()
    created_at = MetaKey()

Understanding Query Operators

The metadata system supports various comparison operators for building complex queries:

# Demonstrate all available operators
meta = DocumentMeta()

# Equality operators
equality_filter = meta.category.eq("technology")
not_equal_filter = meta.owner_id.ne("user-deleted")

# Comparison operators
sequence_filter = meta.chunk_seq.gt(1)        # Greater than
recent_filter = meta.chunk_seq.gte(2)         # Greater than or equal
early_filter = meta.chunk_seq.lt(5)           # Less than
boundary_filter = meta.chunk_seq.lte(3)       # Less than or equal

# List operators
multi_user_filter = meta.owner_id.in_(["user-alice", "user-bob"])
not_category_filter = meta.category.nin(["draft", "archived"])

# Existence operators
has_title_filter = meta.title.exists(True)
no_title_filter = meta.title.exists(False)

print("🔍 All operators available for metadata filtering")

🔍 All operators available for metadata filtering

Vector Querying

Basic Similarity Search

Perform similarity search to find vectors similar to your query vector:

# Query vector (representing a search query embedding)
query_data = [0.15] * 768

# Basic similarity search
results = index.query_vectors(
    s3_vectors_client,
    data=query_data,
    top_k=5,
    return_metadata=True,
    return_distance=True
)

# Process results
print("🔍 Basic similarity search results:")
for i, vector in enumerate(results.as_vector_objects(DocumentChunk), 1):
    print(f"  {i}. {vector.title} (distance: {vector.distance:.6f})")
    print(f"     Key: {vector.key}, Owner: {vector.owner_id}")

🔍 Basic similarity search results:
  1. Introduction to Machine Learning (distance: -0.000001)
     Key: doc-1#chunk-1, Owner: user-alice
  2. Business Strategy Guide (distance: -0.000001)
     Key: doc-2#chunk-2, Owner: user-bob
  3. Introduction to Machine Learning (distance: -0.000001)
     Key: doc-1#chunk-2, Owner: user-alice
  4. Business Strategy Guide (distance: -0.000001)
     Key: doc-2#chunk-1, Owner: user-bob

Filtered Similarity Search

Combine similarity search with metadata filtering for more precise results:

# Search within specific category
category_filter = DocumentMeta.category.eq("technology")
tech_results = index.query_vectors(
    s3_vectors_client,
    data=query_data,
    top_k=3,
    filter=category_filter,
    return_metadata=True,
    return_distance=True
)

print("🎯 Technology documents:")
for vector in tech_results.as_vector_objects(DocumentChunk):
    print(f"  - {vector.title} (chunk {vector.chunk_seq})")

🎯 Technology documents:
  - Introduction to Machine Learning (chunk 1)
  - Introduction to Machine Learning (chunk 2)

Complex Query Combinations

Build sophisticated queries using logical operators (AND, OR):

# Complex query: Technology documents owned by Alice, chunk sequence > 1
complex_filter = (
    DocumentMeta.category.eq("technology") &
    DocumentMeta.owner_id.eq("user-alice") &
    DocumentMeta.chunk_seq.gt(1)
)

complex_results = index.query_vectors(
    s3_vectors_client,
    data=query_data,
    filter=complex_filter,
    return_metadata=True,
    return_distance=True
)

print("🧠 Complex filtered results:")
for vector in complex_results.as_vector_objects(DocumentChunk):
    print(f"  - {vector.title} (chunk {vector.chunk_seq}, owner: {vector.owner_id})")

🧠 Complex filtered results:
  - Introduction to Machine Learning (chunk 2, owner: user-alice)

Multi-User Query Example

Search for content from multiple users using the IN operator:

# Find documents from specific users
multi_user_filter = DocumentMeta.owner_id.in_(["user-alice", "user-bob"])
multi_user_results = index.query_vectors(
    s3_vectors_client,
    data=query_data,
    filter=multi_user_filter,
    return_metadata=True
)

print("👥 Multi-user search results:")
for vector in multi_user_results.as_vector_objects(DocumentChunk):
    print(f"  - {vector.title} by {vector.owner_id}")

👥 Multi-user search results:
  - Business Strategy Guide by user-bob
  - Introduction to Machine Learning by user-alice
  - Business Strategy Guide by user-bob
  - Introduction to Machine Learning by user-alice

Vector Listing and Management

Listing All Vectors

Retrieve all vectors in the index using pagination. This is useful for data auditing and bulk operations:

# List all vectors with metadata
print("📃 All vectors in index:")
all_keys = []

for page in index.list_vectors(
    s3_vectors_client,
    return_metadata=True,
    return_data=False,  # Don't return vector data for performance
    page_size=100
):
    for vector in page.as_vector_objects(DocumentChunk):
        all_keys.append(vector.key)
        print(f"  - {vector.key}: {vector.title} ({vector.category})")

print(f"📊 Total vectors found: {len(all_keys)}")

📃 All vectors in index:
  - doc-2#chunk-1: Business Strategy Guide (business)
  - doc-2#chunk-2: Business Strategy Guide (business)
  - doc-1#chunk-2: Introduction to Machine Learning (technology)
  - doc-1#chunk-1: Introduction to Machine Learning (technology)
📊 Total vectors found: 4

Listing Vectors with Data

When you need the actual vector embeddings, enable data return:

# List vectors with their embedding data
print("🔢 Vectors with embedding data:")
for page in index.list_vectors(
    s3_vectors_client,
    return_data=True,
    return_metadata=True,
    page_size=2  # Small page size for demo
):
    for vector in page.as_vector_objects(DocumentChunk):
        data_preview = vector.data[:3] if vector.data else None
        print(f"  - {vector.key}: data preview {data_preview}...")

🔢 Vectors with embedding data:
  - doc-2#chunk-1: data preview [0.30000001192092896, 0.30000001192092896, 0.30000001192092896]...
  - doc-2#chunk-2: data preview [0.4000000059604645, 0.4000000059604645, 0.4000000059604645]...
  - doc-1#chunk-2: data preview [0.20000000298023224, 0.20000000298023224, 0.20000000298023224]...
  - doc-1#chunk-1: data preview [0.10000000149011612, 0.10000000149011612, 0.10000000149011612]...

Segmented Vector Listing

For large indexes, use segmentation to process vectors in parallel:

# Process vectors in segments (useful for parallel processing)
segment_count = 2
for segment_index in range(segment_count):
    print(f"📦 Processing segment {segment_index + 1}/{segment_count}:")

    for page in index.list_vectors(
        s3_vectors_client,
        segment_count=segment_count,
        segment_index=segment_index,
        return_metadata=True,
        page_size=50
    ):
        vectors_in_segment = list(page.as_vector_objects(DocumentChunk))
        print(f"   Found {len(vectors_in_segment)} vectors in this segment")

📦 Processing segment 1/2:
   Found 2 vectors in this segment
📦 Processing segment 2/2:
   Found 2 vectors in this segment

Vector Deletion Operations

Deleting Specific Vectors

Remove individual vectors by their keys when you need selective deletion:

# Delete specific vectors by key
keys_to_delete = ["doc-1#chunk-1", "doc-2#chunk-1"]
index.delete_vectors(s3_vectors_client, keys=keys_to_delete)
print(f"🗑️ Deleted {len(keys_to_delete)} specific vectors")

# Verify deletion
remaining_count = 0
for page in index.list_vectors(s3_vectors_client, return_metadata=True):
    for vector in page.as_vector_objects(DocumentChunk):
        remaining_count += 1

print(f"📊 Remaining vectors: {remaining_count}")

🗑️ Deleted 2 specific vectors
📊 Remaining vectors: 2

Deleting All Vectors

Clear all vectors from an index while preserving the index structure:

# Delete all vectors in the index
deleted_count = index.delete_all_vectors(
    s3_vectors_client,
    page_size=100,
    max_items=1000
)
print(f"🧹 Deleted {deleted_count} vectors from index")

# Verify the index is empty
verification_count = 0
for page in index.list_vectors(s3_vectors_client):
    for vector in page.as_vector_objects(DocumentChunk):
        verification_count += 1

print(f"✅ Index now contains {verification_count} vectors")

🧹 Deleted 2 vectors from index
✅ Index now contains 0 vectors

Advanced Index Operations

Deleting an Index

Remove an entire index and all its vectors permanently:

# Delete the index (this also deletes all vectors)
index.delete(s3_vectors_client)
print("🗑️ Index deleted successfully")

🗑️ Index deleted successfully

Bulk Index Management

Manage multiple indexes efficiently using list operations:

# List and delete all indexes in a bucket
print("🔍 Finding all indexes in bucket:")
for page in bucket.list_index(s3_vectors_client):
    # Create index objects for deletion
    index_list = Index.new_for_delete_from_list_index_response(page)

    for idx in index_list:
        print(f"  Deleting index: {idx.index_name}")
        idx.delete(s3_vectors_client)

print("✅ All indexes deleted")

🔍 Finding all indexes in bucket:
✅ All indexes deleted

Cleanup Operations

Deleting the Bucket

Finally, remove the entire bucket. Note that all indexes must be deleted first:

# Delete the bucket (must be empty of indexes)
try:
    delete_result = bucket.delete(s3_vectors_client)
    print("🗑️ Bucket deleted successfully")
    print(f"Response: {delete_result}")
except Exception as e:
    print(f"❌ Failed to delete bucket: {e}")
    print("💡 Make sure all indexes are deleted first")

🗑️ Bucket deleted successfully
Response: {'ResponseMetadata': {'RequestId': '65a9389e-7a52-4ac2-940c-b7b7bbb4a37c', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Sat, 27 Sep 2025 23:19:58 GMT', 'content-type': 'application/json', 'content-length': '2', 'connection': 'keep-alive', 'x-amz-request-id': '65a9389e-7a52-4ac2-940c-b7b7bbb4a37c', 'access-control-allow-origin': '*', 'vary': 'origin, access-control-request-method, access-control-request-headers', 'access-control-expose-headers': '*'}, 'RetryAttempts': 0}}

Best Practices and Tips

Performance Optimization

Tips for optimal performance:

# 1. Use appropriate page sizes for your use case
# Small pages for interactive applications
for page in index.list_vectors(s3_vectors_client, page_size=10):
    pass

# Large pages for batch processing
for page in index.list_vectors(s3_vectors_client, page_size=1000):
    pass

# 2. Only return data when needed
# Metadata only (faster)
results = index.query_vectors(
    s3_vectors_client,
    data=query_data,
    return_metadata=True,
    return_distance=False
)

# 3. Use segmentation for parallel processing of large datasets
segment_count = 4  # Adjust based on your processing capacity

Type Safety

Leverage the type system for better development experience:

# The library preserves vector subclass types
results = index.query_vectors(s3_vectors_client, data=query_data)
typed_vectors = results.as_vector_objects(DocumentChunk)

# Now your IDE knows these are DocumentChunk instances
for vector in typed_vectors:
    # Auto-completion works for custom fields
    print(vector.document_id)  # ✅ Type-safe access
    print(vector.title)        # ✅ IDE knows about this field

This concludes the comprehensive S3VectorM tutorial. You now have all the tools needed to build sophisticated vector search applications with AWS S3 Vectors service!