Fearless dataset experimentation with bucket forking | Tigris Object Storage

A blue tiger in several parallel universes demonstrating different kinds of things that can be done with AI.

Our new feature, bucket forking, lets you make an isolated copy of your large dataset instantly with zero copying. No more colliding in shared datasets or waiting hours for bytes to copy, fork your dataset like you fork your code, and experiment away.

Experimentation enabled by bucket forking

Imagine a world where an AI company's researchers can instantly experiment with an entire, massive training dataset in object storage, without the days-long wait for copies or the uncertainty of using live data. This addresses a common pattern where developers create and then abandon numerous copies of central datasets for individual experiments, leading to significant duplication and wasted time.

We’re proposing a new workflow: create per-user (or per-run) forks of your dataset, do your experimentation and development, and then merge your updated data back into main. Just like git, and it’s as fast as forking a git repo.

When you fork a bucket, you get an isolated copy that’s a metadata reference to your original dataset. Writes to the source bucket aren’t replicated to the fork: their timelines have diverged at the moment of the fork. Tigris only stores the changes, so there’s no duplication.

Bucket forking and the scientific method

Let’s follow an experiment using the scientific method: you want your model to match the painterly aesthetic of a video game, but you aren’t sure which subset of screenshots will finetune your model best. Should you train on screenshots of the entire game? Should you make individual “experts” for deserts vs ocean scenes? Should you remove the borders and menus? Do you really need to downscale the images to 512x512 like you did in the early days of Stable Diffusion v1.5? How about the aspect ratio or greyscaling... the list goes on.

Each of these variations is an experiment, a parallel timeline for your data. Without forking, you’d need a frozen copy for each experiment for control, so you’d cull the list to minimize the number of parallel datasets. Or you’d share data across experiments and track the changes. But with forking, you can instantly make a copy so you can try all of them at once. Trivially.

We're going to make changes across the entire dataset to optimize the data for the models we want to train. Instead of making multiple copies of the dataset, we're going to use bucket forking for these experiments. But, in order to talk about that, first we need to talk about parallel universes.

Dataset experimentation with parallel universes

In the real world, datasets arrive as messy unlabeled piles of bytes that we have to make sense of in order to do useful things. As an example, let’s take our example, a dataset of Nintendo Switch game screenshots. With all this data, you could do any number of things, such as:

Today I’m going to show you how you would do this kind of experimental massaging from a bucket-forking native mindset. In my case, I want to take that dataset of Nintendo Switch screenshots and isolate things out so that I can train a Stable Diffusion LoRA on screenshots from The Legend of Zelda: Breath of the Wild. This will require the following steps:

We'll end up with four parallel timelines for our data, each a controlled lab for our experiments. Here's a sketch.

Fork 1: Data cleaning and labeling

Right now my data is a giant pile of thousands of flat files I copied off of my Switch’s SD card. It’s got a bunch of filenames that look like this:

screenshots/2022/03/03/2022030300000900-1E1800B8D04F999C436DDFE2B8CD0B81.jpg

The filenames are broken down like this:

${date}-${titleID}.jpg

So that example would be a screenshot of Dark Souls Remastered that I took in early March 2023.

I had Claude write a little shell script that broke down this input folder and renamed the files like this:

./var/switch-screenshots/train/${titleID}/${date}.jpg

Then I imported it to a Tigris bucket with a little bit of Python code:

from datasets import load_dataset

import os

BUCKET_NAME = "xe-screenshots-multiworld"

storage_options = {

"key": os.getenv("AWS_ACCESS_KEY_ID"),

"secret": os.getenv("AWS_SECRET_ACCESS_KEY"),

"endpoint_url": "https://fly.storage.tigris.dev"

}

ds = load_dataset("imagefolder", data_dir="./var/switch-screenshots", split="train")

ds.save_to_disk(f"s3://{BUCKET_NAME}/images", storage_options=storage_options)

Then let’s freeze this state in time by creating a snapshot:

import boto3

from botocore.client import Config

def create_bucket_snapshot(bucket_name, desc):

tigris = boto3.client(

"s3",

endpoint_url="https://t3.storage.dev",

config=Config(s3={'addressing_style': 'virtual'}),

)

tigris.meta.events.register(

"before-sign.s3.CreateBucket",

lambda request, **kwargs: request.headers.add_header(

"X-Tigris-Snapshot", f"true; desc={desc}"

)
    )

return tigris.create_bucket(Bucket=bucket_name)["ResponseMetadata"]["HTTPHeaders"]["x-tigris-snapshot-version"]

create_bucket_snapshot(BUCKET_NAME, "imported dataset from the disk")

And then we can make sure it’s there by listing all the snapshots:

def list_snapshots_for_bucket(bucket_name):

tigris = boto3.client(

"s3",

endpoint_url="https://t3.storage.dev",

config=Config(s3={'addressing_style': 'virtual'}),

)

tigris.meta.events.register(

"before-sign.s3.ListBuckets",

lambda request, **kwargs: request.headers.add_header("X-Tigris-Snapshot", bucket_name)
    )

return tigris.list_buckets()

for snapshot in list_snapshots_for_bucket(BUCKET_NAME)['Buckets']:

name, desc = snapshot["Name"].split("; desc=")

snaptime = snapshot["CreationDate"].strftime("%s")

print(f"name={name} time={snaptime} desc=\"{desc}\"")

That returns something like this:

{'name': '1760036788104497556', 'time': '1760054788', 'desc': 'imported dataset from the disk'}

So we can use this snapshot to create a fork of the bucket:

def create_bucket_fork(bucket_name, from_bucket, snapshot_id=None):

tigris = boto3.client(

"s3",

endpoint_url="https://t3.storage.dev",

config=Config(s3={'addressing_style': 'virtual'}),

)

tigris.meta.events.register(

"before-sign.s3.CreateBucket",

lambda request, **kwargs: (

request.headers.add_header("X-Tigris-Fork-Source-Bucket", from_bucket),

)
    )

if snapshot_id is not None:

tigris.meta.events.register(

"before-sign.s3.CreateBucket",

lambda request, **kwargs: (

request.headers.add_header("X-Tigris-Fork-Source-Bucket-Snapshot", snapshot_id),

)
        )

tigris.create_bucket(Bucket=bucket_name)

botw_only_bucket = f"{BUCKET_NAME}-botw"

create_bucket_fork(botw_only_bucket, BUCKET_NAME, "1760036788104497556")

And then make some helpers to load the dataset from that fork:

from datasets import load_from_disk

def load_timeline(bucket_name):

return load_from_disk(f"s3://{bucket_name}/images", storage_options=storage_options)

def save_timeline(ds, bucket_name):

ds.save_to_disk(f"s3://{bucket_name}/images", storage_options=storage_options)

Filtering

From here we can filter everything that isn’t from Breath of the Wild out of the dataset. According to the Switchbrew wiki, the title ID for Breath of the Wild is F1C11A22FAEE3B82F21B330E1B786A39. Let’s set this as a global variable and then filter everything else out:

BOTW_TITLE_ID = "F1C11A22FAEE3B82F21B330E1B786A39"

ds = load_timeline(botw_only_bucket)

ds = ds.filter(lambda x: ds.features['label'].names[x['label']] == BOTW_TITLE_ID)

print(f"filtered dataset size: {len(ds)}")

save_timeline(ds, botw_only_bucket)

Then we can make a snapshot of the bucket in this state:

botw_only_snapshot_id = create_bucket_snapshot(botw_only_bucket, "Dataset filtered down to only images of Breath of the Wild")

Fork 2: Caption Synthesis

I use this dataset for other training projects so I don't want to apply the captions to the common dataset. I want to leave the underlying / central data the same and add my captions in its own little fork. Let’s start by diverging the timeline for captioning and make a fork:

caption_bucket = f"{BUCKET_NAME}-captions"

create_bucket_fork(caption_bucket, botw_only_bucket, botw_only_snapshot_id)

ds = load_timeline(caption_bucket)

From here we add an empty column for the text caption (our training workflow will require this to be called text):

text_data = [""] * len(ds)

ds = ds.add_column("text", text_data)

Then we can generate high-quality captions using a few-shot process. For this I went into Breath of the Wild and captured some screenshots I’ll use to make my own high quality captions as examples for the language model. I’m including a few images in my dataset, capturing the following scenarios/scenes:

These base captions will help “ground” the model so it creates more captions like my examples. For the captioning I’m going to be using gemma3:4b on a local device, but you can use whatever model you want.

Fork 3: Better captioning and different models

When I was looking through the dataset I noticed that some of the captions weren’t ideal, so I thought that I should redo them by changing the prompting theory to be closer to what Stable Diffusion XL natively prefers. However, I don't know if this new method will be any better. I want to preserve the old captions so I can compare them. Let’s see if a different captioning method will work better. I want to preserve the first experiment so I can compare; thus I forked the bucket.

Fork 4: Resizing to train Stable Diffusion

Now that I have the images and captions, I want to start optimizing the image size for the model I want to train. This requires a destructive action across every image in the dataset.