Snapshot and Fork put performance

🍴Bucket Forking

Hi, I was testing snapshots and forking and noticed for each additional layer there was a pretty significant increase in latency for different ops. For example, a 1kb put could go from 7-8 ms to 15 ms on a fork and each additional fork caused an additional increase in latency.

My assumptions with how forks work is it's basically like a CoW filesystem. Out of curiosity is there some compaction step that brings down this latency over time or is this additional latency always present with an increased number of layers?

ot•3/16/26, 11:43 AM

ot•3/16/26, 11:43 AM

This is unexpected

ot•3/16/26, 11:44 AM

Which op you are seeing the latency increase?

vmg OP•3/16/26, 12:35 PM

I noticed it across all ops

vmg OP•3/16/26, 12:36 PM

I was testing against a single region in iad from a hetzner vm

vmg OP•3/16/26, 12:42 PM

I'm going to rerun the benchmark and I'll also share the repo with you

ot•3/16/26, 12:42 PM

That would be great

vmg OP•3/16/26, 1:41 PM

https://github.com/vmg-dev/tigris-storage-bench/tree/main

I ran the benchmark with the ts and go sdks just to confirm it wasn't an sdk issue

vmg OP•3/16/26, 1:42 PM

https://github.com/vmg-dev/tigris-storage-bench/blob/main/GO_BENCHMARK_RESULTS.md

vmg OP•3/16/26, 1:42 PM

Notice how latency climbs as fork depth increases

ot•3/16/26, 6:01 PM

Thanks for sharing this

ot•3/16/26, 6:01 PM

Let me run it as well

ot•3/16/26, 6:53 PM

So I just tested by running this tool

ot•3/16/26, 6:53 PM

I don't see variance in write operations:

ot•3/16/26, 6:54 PM

I do see variance in get results

ot•3/16/26, 7:11 PM

I will get back to you on this

ot•3/16/26, 7:14 PM

And I have added support for --location``--location https://github.com/vmg-dev/tigris-storage-bench/pull/1

GitHub

feat: make bucket location configurable via --location flag by ovai...

vmg OP•3/16/26, 8:10 PM

Weird, I definitely didnt make up that result but I realize now for any test over a network I need to start testing from multiple datacenters.

vmg OP•3/16/26, 8:10 PM

Ill merge your pr once im back to my laptop

vmg OP•3/16/26, 8:11 PM

I dont see why the dc would influence this though. I got this result consistently.

ot•3/17/26, 1:34 AM

I am not saying you made up the result. I don't expect overhead in write requests which my tests show

ot•3/17/26, 1:34 AM

While read overhead is clear in your results and mine

ot I am not saying you made up the result. I don't expect overhead in write request...

vmg OP•3/17/26, 8:52 AM

I know, just wanted to emphasize it, thanks for looking into this

ot•4/8/26, 3:11 PM

Hi @vmg we have made improvements to the performance of snapshots. Do you mind running your benchmarks again?

vmg OP•4/8/26, 8:45 PM

sure ill try running them

ot•4/8/26, 8:50 PM

Cool

vmg OP•4/9/26, 12:44 AM

It seems like the perf hit is still there for deep forks

vmg OP•4/9/26, 12:44 AM

Writes are unaffected by fork depth but reads are

ot•4/9/26, 11:26 AM

I see

himank•4/13/26, 3:26 PM

@vmg we are looking into it actively, should've an update for you this week.

himank•4/22/26, 12:48 PM

@vmg do you want to give it a try? We rolled out improvements for deep forks.