[Golang] PutObject Failing with StatusCode: 501 (NotImplemented)

Hello everyone :wave:

I hope you are having a good end to the week. Me, on the other hand, I went on an S3 scavenger hunt :sweat_smile:

For context, my production code works fine but I haven't run any deployments or dependency upgrades in a few days. Today, I was working on a feature on my local machine and after upgrading my deps (go get -u ./… and go mod tidy) all my requests to Tigris started failing with the following error:

operation error S3: PutObject, https response error StatusCode: 501, RequestID: 1737729871240111208, HostID: , api error NotImplemented: A header you provided implies functionality that is not implemented”

I tried connecting to a different bucket, changing permissions etc. but with no luck – and the error message isn't very clear as to what is not implemented. However, I came across the AWS S3 Go SDK changelog for v1.73.0:

Feature: S3 client behavior is updated to always calculate a checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). The checksum algorithm used by default now becomes CRC32.

So it seems that the checksum calculation is causing errors on Tigris' side. I fixed the error by requesting the checksum be calculated only when required:

svc := s3.NewFromConfig(config, func(o *s3.Options) {
    o.BaseEndpoint = aws.String(u)
    o.Region = r
    o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired // new line
})

I am not sure if this is a robust solution as I haven't tested DeleteObject yet, but it's something that might affect people upgrading their dependencies.

Hope this helps :love_you_gesture:

Solution

Hi, you are indeed right, there are breaking changes in the newer versions of the AWS S3 SDKs.

We have captured the details in our blog post: If you've upgraded boto3 or the JavaScript S3 client in the last week, uploading files won't work. Here's how to fix it.

If you've recently upgraded boto3 or the JavaScript S3 client, file uploads may fail due to a required Content-Length header. Downgrade boto3 to version 1.35.x and the JavaScript client to v3.728.0 to resolve this issue until Tigris releases an update for compatibility with new versions.

Discussion

Hi there,

I get a similar 501 error when using the PutObject Kotlin API, but only when my file is larger than 1024 * 1024 bytes. I'm using aws-sdk-kotlin 1.4.9, which just came out and supposedly contains a fix for a header signing issue, but I don't know if this is the same issue.

When uploading the same file via the AWS S3 CLI everything works, but running with --debug shows that it's doing a multipart upload.

// S3 client setup with endpoint and credentials
val s3 = S3Client {
    region = "auto"
    endpointUrl = Url.parse("https://fly.storage.tigris.dev")
    credentialsProvider = EnvironmentCredentialsProvider()
}

// Upload test - fails at exactly 1MB boundary
val file = File("1gb_file")
s3.putObject {
    bucket = "<MY_BUCKET>"
    key = "tigris_upload_test"
    // body = file.asByteStream() // Fails with 501 NotImplemented
    // body = file.asByteStream(0, 1024 * 1024) // Same - fails at 1MB
    body = file.asByteStream(0, 1024 * 1024 - 1) // SUCCEEDS - 1MB-1 works
}

Follow-Up

Thanks for sharing the example!

I was able to reproduce the issue you're encountering. It appears to be caused by a breaking change introduced in the recent versions of the AWS SDK.

We are currently working on a fix and will notify you once it's deployed to production.

In the meantime, clients using version 1.4.9 or earlier should continue to work as expected.

Thanks for your patience!

Conclusion

We've rolled out the fix! Please give it a try and let us know if you encounter any further issues.

Thank you. I can confirm this now works.