Distributed Training with LanceDB and Tigris | Tigris Object Storage

A cartoon tiger meditating in an open office on top of a runic magic circle with the LanceDB logo projecting out of his crown chakra.

Training datasets, especially multi modal ones, tend to grow rapidly, quickly outgrowing any single disk that any cloud provider can provide. A common solution has been to use a networked filesystem, accessible to all the nodes in your distributed training job. You figure out how to chunk your datasets to be smaller than ram, and swallow the cost of managing this overhead.

This does work, but what if there was a better way to do this? What if you could just store however much data you wanted and then let the infrastructure layer figure it all out for you? This is possible with the combination of LanceDB, a Multimodal Lakehouse layer for your data, and Tigris, a globally performant object storage backend that puts your files close to where they’re being used. LanceDB uses the Lance columnar format (built on Apache Arrow) to store data ranging from images, videos, text, or embedding vectors, all with features like versioning, fast random access, and eager look-ahead caching. LanceDB can also store raw bytes as columns, so you don’t need to manage your data separately from your database.

In this guide, we’ll show how you can use LanceDB on top of Tigris to manage massive training datasets and stream them into model training, all without having to manage any infrastructure yourself. We’ll walk through practical steps and code examples for:

By the end, you’ll be able to scale your training workflows with a LanceDB + Tigris stack, this lets you keep your datasets close to where they’re being used in a multimodal lakehouse on the cloud without having to worry about local storage, data placement, or egress fees. LanceDB and Tigris handle those hard parts for you so you can get back to training and using your models.

Prerequisites

To get started, you need the following:

Loading LanceDB data for Training with PyTorch

Once your dataset is in Lance format on Tigris, you can train your models without ever fully downloading the dataset to your training machine. LanceDataset treats the dataset as a stream of data. It reads batches of data directly from the LanceDB store instead of relying on the PyTorch DataLoader class. In our case, LanceDataset will stream from the Lance file we created on Tigris and yield one batch at a time.

One thing to note: LanceDataset yields data in sequence. Any preprocessing should be done within the LanceDataset’s to_tensor_fn callback. In other words, since we aren’t using the native DataLoader collation function to transform raw data (as we would in the in-memory style of dataset), we have to provide a function that takes the raw batch and convert it into PyTorch tensors. This is the same rough idea as a collation function, but it happens when the dataset is being iterated. If your LanceDB table contains multimodal data (like our image bytes), you must decode them in to_tensor_fn because the raw bytes can’t be directly turned into Torch tensors without conversion.

Let’s create a LanceDataset for our training data and include a decoding function:

import os

os.environ['AWS_ACCESS_KEY_ID'] = 'tid_access_key_id'

os.environ['AWS_SECRET_ACCESS_KEY'] = 'tsec_secret_access_key'

os.environ['AWS_REGION'] = 'auto'

os.environ['AWS_ENDPOINT_URL_S3'] = 'https://t3.storage.dev'

import torch

from PIL import Image

from torchvision import transforms

from lance.torch.data import LanceDataset

from torch.utils.data import DataLoader

# Define transformation for images (resize to 224x224 and convert to tensor)
_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

def decode_batch(record_batch):
    images = []
    labels = []

for item in record_batch.to_pylist():
        img = Image.open(io.BytesIO(item["image"])).convert("RGB")
        img_tensor = _transform(img)
        images.append(img_tensor)
        labels.append(item["label"])

return {
        "image": torch.stack(images),
        "label": torch.tensor(labels, dtype=torch.long)
    }

bucket_name = "my-bucket"  # Replace with your Tigris bucket name
train_uri = f"s3://{bucket_name}/food101_train.lance"
test_uri  = f"s3://{bucket_name}/food101_test.lance"

dataset = LanceDataset(train_uri, batch_size=128, to_tensor_fn=decode_batch)
loader = DataLoader(dataset, batch_size=None, num_workers=0)

batch = next(iter(loader))
print("Batch images shape:", batch["image"].shape)
print("Batch labels shape:", batch["label"].shape)

In this snippet, we open the data in the LanceDataset and use our decode_batch function to process them fragment by fragment. We set num_workers=0 here to disable the native DataLoader parallelism because that will stomp over what LanceDB does internally to make things safely parallel. Internally, LanceDB uses a sharded batch sample to read contiguous chunks of rows from each fragment across multiple workers or processes (keep reading for multi-GPU training tips). The result is that each iteration of the DataLoader returns a dictionary that’s ready to feed into the model.

In the code above, there’s a commented out normalization transformation. If you are using a pretrained model, uncomment that transformation.

Choosing Between LanceDataset and SafeLanceDataset

Both dataset types let you stream your training data from LanceDB to Tigris, but picking the right option depends on your use case:

In practice, many teams will find that SafeLanceDataset meets all their needs for training on fixed datasets. Use LanceDataset when you have a compelling reason like data streaming or when experimenting with LanceDB’s advanced sampling capabilities.

What About Distributed Training?

If you use Distributed Data Parallel (DDP) for multi-GPU training with SafeLanceDataset, you should use DistributedSampler or the shuffle=True option to ensure each process gets different data. In practice, you might initialize the DataLoader with something like:

sampler = DistributedSampler(train_dataset_map) if using_ddp else None
train_loader = DataLoader(train_dataset_map, batch_size=32, sampler=sampler, ... )

Just make sure to keep this set of tradeoffs in mind: With LanceDataset (iterable), Lance provides its own ShardedBatchSampler and ShardedFragmentSampler that are already aware of DDP. You can specify these when creating the LanceDataset (via the sampler= argument).

Here’s what it looks like to use LanceDataset with DDP:

from lance.torch.data import ShardedBatchSampler

sampler = ShardedBatchSampler(rank=rank, world_size=world_size)
dataset = LanceDataset(train_uri, batch_size=128, to_tensor_fn=decode_batch, sampler=sampler)
loader = DataLoader(dataset, batch_size=None, num_workers=0)

This ensures each GPU gets its own portion of each batch, splitting the work up equally. Either way, LanceDB on Tigris enables distributed training directly from cloud storage without manually sharding the dataset or duplicating data per node. Each worker will fetch exactly the data it needs from the shared Lance dataset in Tigris thanks to these samplers.

How can you make training with object storage fast?

Normally training from object storage can be slow compared to training from the disk. Lance supports look-ahead caching, meaning that fetching one group from your dataset actually results in fetching many fragments in advance. This means that making your data load fast is as easy as changing your dataset initialization code:

from lance.torch.data import LanceDataset

# Configure caching and performance tuning options
reader_options = {
    "max_lookahead_batches": 3,
    "fragment_readahead": 2,
    "prefetch_queue_depth": 8,
    "cache_enabled": True,
    "cache_size_limit": 8 * 1024**3
}

dataset = LanceDataset(
    "s3://my-bucket/my-dataset/",
    batch_size=1024,
    reader_options=reader_options
)

Conclusion

When you use LanceDB as a multimodal lakehouse on top of Tigris, this lets ML engineers scale their training pipelines with ease. Today we learned how to take a dataset that lived on the local disk and move it into LanceDB (with all the benefits of versioning, efficient I/O, and raw multimodal data storage) so you can leverage Tigris’ bottomless object storage and global availability. The LanceDB Python integration via LanceDataset makes it simple to load this remote data into your training loop with minimal changes.

When you combine LanceDB and Tigris, you get an invincible power team that lets you treat data as data, no matter where it happens to be located; this gives you fearless scaling from research to development to production. Happy training!