Snapshot and Fork put performance

🍴Bucket Forking

Hi, I was testing snapshots and forking and noticed for each additional layer there was a pretty significant increase in latency for different ops. For example, a 1kb put could go from 7-8 ms to 15 ms on a fork and each additional fork caused an additional increase in latency.

My assumptions with how forks work is it's basically like a CoW filesystem. Out of curiosity is there some compaction step that brings down this latency over time or is this additional latency always present with an increased number of layers?

This is unexpected

Which op you are seeing the latency increase?

I noticed it across all ops

I was testing against a single region in iad from a hetzner vm

I'm going to rerun the benchmark and I'll also share the repo with you

That would be great

https://github.com/vmg-dev/tigris-storage-bench/tree/main

I ran the benchmark with the ts and go sdks just to confirm it wasn't an sdk issue

https://github.com/vmg-dev/tigris-storage-bench/blob/main/GO_BENCHMARK_RESULTS.md

Notice how latency climbs as fork depth increases

Thanks for sharing this

Let me run it as well

So I just tested by running this tool

I don't see variance in write operations:

I do see variance in get results

I will get back to you on this

And I have added support for --location``--location https://github.com/vmg-dev/tigris-storage-bench/pull/1

Weird, I definitely didnt make up that result but I realize now for any test over a network I need to start testing from multiple datacenters.

Ill merge your pr once im back to my laptop

I dont see why the dc would influence this though. I got this result consistently.

I am not saying you made up the result. I don't expect overhead in write requests which my tests show

While read overhead is clear in your results and mine

I know, just wanted to emphasize it, thanks for looking into this

Hi @vmg we have made improvements to the performance of snapshots. Do you mind running your benchmarks again?

sure ill try running them

Cool

It seems like the perf hit is still there for deep forks

Writes are unaffected by fork depth but reads are.