seeing continuous HTTP 500 errors on HeadObject calls for ~3 hours with no recovery - Tigris
seeing continuous HTTP 500 errors on HeadObject calls for ~3 hours with no recovery
▎ Details:
▎
- Endpoint: https://t3.storage.dev
▎
- Bucket:
meshy-learning
▎
- Affected paths:
checkpoints/meshylatv_avocado/VAE_20251019_d0e667e7/andcheckpoints/meshylatv_avocado/GSHead_GANS_20251109_38541c8e/
▎
- Operation: HeadObject via boto3/s3fs
▎
- Error: ClientError: An error occurred (500) when calling the HeadObject operation (reached max retries: 4): Internal Server Error
▎
- Started: ~20:03 UTC, April 5, 2026
▎ What we've ruled out:
▎
- Same bucket and paths work fine from AWS us-west-2
▎
- Fails from multiple independent non-AWS networks (bare metal datacenters and DigitalOcean)
▎
- Rotated credentials, same 500 with new key
▎
- Other paths in the same bucket accessible from AWS
▎
- Status page shows all operational
Solution
To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution
ot•4/5/26, 8:06 PM
We will take a look to see what's going on
ot•4/5/26, 8:12 PM
@Jie Wang it looks like this bucket has shadow bucket configured
ot•4/5/26, 8:12 PM
The access keys for the source bucket seem to be incorrect
ot•4/5/26, 8:12 PM
Tigris is internally getting 403 when trying to fetch data from source bucket
Jie Wang OP•4/5/26, 8:14 PM
interesting...Thanks for looking into it. Let me see what's going on with our key, we just created a new one and still seeing the same issue
ot•4/5/26, 8:15 PM
That would be the access key for your S3 bucket that is having issues
ot•4/5/26, 8:15 PM
You can update the access key for shadow bucket in the bucket settings on our dashboard
Jie Wang OP•4/5/26, 8:15 PM
so it's realted to AWS_ACCESS_KEY_ID not the AWS_SECRET_ACCESS_KEY?
ot•4/5/26, 8:16 PM
I mean it could be either
ot•4/5/26, 8:16 PM
We are just seeing 403 when trying to read from S3
Jie Wang OP•4/5/26, 8:17 PM
okay, thanks
ot•4/6/26, 1:21 AM
@Jie Wang is the issue resolved for you now?
Jie Wang OP•4/13/26, 10:36 AM
Yes, it's resolved. Thank you!