# seeing continuous HTTP 500 errors on HeadObject calls for ~3 hours with no recovery

▎ Details:

▎ 
- Endpoint: [https://t3.storage.dev](https://t3.storage.dev/)

▎ 
- Bucket: `meshy-learning`

▎ 
- Affected paths: `checkpoints/meshylatv_avocado/VAE_20251019_d0e667e7/` and `checkpoints/meshylatv_avocado/GSHead_GANS_20251109_38541c8e/`

▎ 
- Operation: HeadObject via boto3/s3fs

▎ 
- Error: ClientError: An error occurred (500) when calling the HeadObject operation (reached max retries: 4): Internal Server Error

▎ 
- Started: ~20:03 UTC, April 5, 2026

▎ What we've ruled out:

▎ 
- Same bucket and paths work fine from AWS us-west-2

▎ 
- Fails from multiple independent non-AWS networks (bare metal datacenters and DigitalOcean)

▎ 
- Rotated credentials, same 500 with new key

▎ 
- Other paths in the same bucket accessible from AWS

▎ 
- Status page shows all operational

### Solution

To help others find answers, you can mark your question as solved via `Right click solution message -> Apps -> ✅ Mark Solution`

[Jump to solution](https://community.tigrisdata.com/m/1490502674360176727#solution-1490502674360176727)

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:06 PM

We will take a look to see what's going on

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:12 PM

[@Jie Wang](https://community.tigrisdata.com/u/1457776617290989861) it looks like this bucket has shadow bucket configured

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:12 PM

The access keys for the source bucket seem to be incorrect

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:12 PM

Tigris is internally getting 403 when trying to fetch data from source bucket

[Jie Wang](https://community.tigrisdata.com/u/1457776617290989861) OP•4/5/26, 8:14 PM

interesting...Thanks for looking into it. Let me see what's going on with our key, we just created a new one and still seeing the same issue

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:15 PM

That would be the access key for your S3 bucket that is having issues

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:15 PM

You can update the access key for shadow bucket in the bucket settings on our dashboard

[Jie Wang](https://community.tigrisdata.com/u/1457776617290989861) OP•4/5/26, 8:15 PM

so it's realted to AWS_ACCESS_KEY_ID not the AWS_SECRET_ACCESS_KEY?

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:16 PM

I mean it could be either

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/5/26, 8:16 PM

We are just seeing 403 when trying to read from S3

[Jie Wang](https://community.tigrisdata.com/u/1457776617290989861) OP•4/5/26, 8:17 PM

okay, thanks

[ot](https://community.tigrisdata.com/u/938129329953591377)•4/6/26, 1:21 AM

[@Jie Wang](https://community.tigrisdata.com/u/1457776617290989861) is the issue resolved for you now?

[Jie Wang](https://community.tigrisdata.com/u/1457776617290989861) OP•4/13/26, 10:36 AM

Yes, it's resolved. Thank you!