Getting Started with Bufstream on Tigris | Tigris Object Storage Documentation

On this page

Learn how to configure Bufstream, a Kafka-compatible message queue, to use Tigris Object Storage as its backend.

What is Bufstream?

Bufstream is the Kafka-compatible message queue built for the data lakehouse era. It's a drop-in replacement for Apache Kafka®, but instead of requiring expensive machines with large attached disks, Bufstream builds on top of off-the-shelf technologies like Object Storage and Postgres, providing a Kafka implementation designed for the cloud-native era.

Tigris is a globally distributed, multi-cloud object storage platform with:

• Native S3 API support
• Zero egress fees
• Globally distributed

When combined, Bufstream and Tigris provide:

• Unlimited message retention
• Global scalability
• Cost-efficient and resilient data pipelines

Requirements

• Docker Desktop or Podman Desktop
• A Tigris account

Tools Used

• Bufstream — Kafka-compatible broker
• kafkactl — CLI for Kafka and compatible tools
• Docker — to containerize the setup

Step 1: Clone the Example Repository

Clone the demo project to your local machine:

git clone https://github.com/tigrisdata-community/bufstream-tigris

cd bufstream-tigris

This repository includes configuration files and sample data for testing.

Step 2: Create a Tigris Bucket

1. Go to storage.new and create a new bucket in the Standard access tier.

2. Copy the bucket name for later.

3. Create an Access Key with Editor permissions: storage.new/accesskey

4. In the repository’s .env file, add your credentials:

# .env

TIGRIS_ACCESS_KEY_ID=<your-access-key-id>

TIGRIS_SECRET_ACCESS_KEY=<your-secret-access-key>

Step 3: Configure Bufstream

Open the bufstream.yaml file in your cloned repo and update the bucket field with your Tigris bucket name. Leave region set to auto to let Tigris route to the nearest region.

storage:

provider: S3

region: auto

bucket: <your-tigris-bucket-name>

endpoint: https://t3.storage.dev

access_key_id:

env_var: TIGRIS_ACCESS_KEY_ID

secret_access_key:

env_var: TIGRIS_SECRET_ACCESS_KEY

Step 4: Start Bufstream

Start the Docker environment:

docker compose up -d

You should see output similar to:

 ✔ Network bufstream-on-tigris_bufstream_net  Created

✔ Container cli                              Started

✔ Container postgres                         Healthy

✔ Container bufstream                        Started

Step 5: Create a Topic

Use kafkactl to create a topic:

docker exec cli kafkactl create topic bufstream-on-tigris

Expected output:

topic created: bufstream-on-tigris

Step 6: Produce Messages

A sample message file (messages.txt) is included in the repository. To publish its contents to your topic, run:

docker exec cli kafkactl produce bufstream-on-tigris --file=/messages.txt

Expected output:

7 messages produced

Step 7: Consume Messages

Read the last 100 messages from the topic:

docker exec cli kafkactl consume bufstream-on-tigris --tail=100

Example output:

Hello, world!

This

is

Bufstream

running

on

Tigris!

Verify in the Tigris Console

Open your Tigris Console and navigate to your bucket. You’ll see Bufstream-created keys representing your topic data.

Bufstream now uses Tigris for:

• Durable message storage
• Unlimited retention
• Globally accessible streams

You can continue producing and consuming messages using kafkactl or integrate with your existing Kafka-compatible apps.

🎉 Congratulations, you’ve successfully deployed Bufstream on Tigris.