Warpstream on Tigris | Tigris Object Storage Documentation

On this page

Warpstream Overview

Warpstream lets you store an unlimited amount of data in your message queues, but when you set it up with S3 or other object stores, you end up having to pay egress fees to read messages. Tigris is a globally distributed, multi-cloud object storage service with built-in support for the S3 API and no egress fees. When you combine the two, you get a bottomless durable message queue that lets you store however much you want without having to worry about where your data is.

Before we get started, let’s cover the moving parts:

Today we’re going to deploy a Warpstream Broker backed by Tigris into a Docker container so you can create your own bottomless durable message queue. This example will use Docker compose, but it will help you understand how to create your own broker so you can deploy it anywhere.

Prerequisites

Clone the warpstream-tigris demo repo to your laptop and open it in your favourite editor, such as VS Code.

Make sure you have the following installed on your computer:

You will need the following accounts:

Building a compose file

First, clone tigrisdata-community/warpstream-tigris to your laptop and open it in your favourite text editor. If you use development containers, tell your editor to open this repository in a development container to get up and running in a snap!

Take a look at the docker-compose.yaml file in the root of the repository:

services:

warp:

# Grab the latest copy of the warpstream agent for your computer

image: public.ecr.aws/warpstream-labs/warpstream_agent:latest

# Run warpstream in "playground" mode for testing

command:

- playground

- -advertiseHostnameStrategy

- custom

- -advertiseHostnameCustom

- warp

environment:

# this is a no-op as it will default on the custom advertised hostname defined above, but you can change this if you want to use a different hostname with Kafka

- WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=warp

healthcheck:

# Wait for the Agent to finish setting up the demo before marking it as healthy

# to delay the diagnose-connection command from running for a few seconds.

test: ["CMD", "sh", "-c", "sleep 10"]

interval: 5s

timeout: 15s

retries: 5

Open a new terminal in your development container and make sure Warpstream is up and running:

warpstream kcmd --bootstrap-host warp --type diagnose-connection

This should return output like the following:

running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092

Broker Details

---------------

warp:9092 (NodeID: 1547451680) [playground]

ACCESSIBLE ✅

GroupCoordinator: warp:9092 (NodeID: 1547451680)

ACCESSIBLE ✅

Excellent! Create a new topic with warpstream kcmd:

warpstream kcmd --bootstrap-host warp --type create-topic --topic hello

This should return output like the following:

running create-topic sub-command with bootstrap-host: warp and bootstrap-port: 9092

created topic "hello" successfully, topic ID: MQAAAAAAAAAAAAAAAAAAAA==

Perfect! Now let’s make it work with Tigris. Create a .env file in the root of the repository:

cp .env.example .env

code .env

Create a new bucket at storage.new in the Standard access tier. Copy its name down into your notes. Create a new access key with Editor permissions for that bucket. Copy the environment details into your .env file:

## Tigris credentials

AWS_ACCESS_KEY_ID=tid_access_key_id

AWS_SECRET_ACCESS_KEY=tsec_secret_access_key

AWS_ENDPOINT_URL_S3=https://t3.storage.dev

AWS_ENDPOINT_URL_IAM=https://iam.storage.dev

AWS_REGION=auto

Then fill in your Warpstream secrets from the console, you need the following:

If your bucket is named xe-warpstream-demo, your bucket URL should look like this:

s3://xe-warpstream-demo?region=auto&endpoint=https://t3.storage.dev

Altogether, put these credentials in your .env file:

## Warpstream credentials

WARPSTREAM_AGENT_KEY=aks_agent_key

WARPSTREAM_BUCKET_URL='s3://xe-warpstream-demo?region=auto&endpoint=https://t3.storage.dev'

WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID=vci_cluster_id

WARPSTREAM_REGION=us-east-1

Edit your docker-compose.yaml file to load the .env file and start warpstream in agent mode:

# docker-compose.yaml

services:

warp:

image: public.ecr.aws/warpstream-labs/warpstream_agent:latest

command:

- agent

environment:

WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE: warp

WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE: 9092

WARPSTREAM_REQUIRE_AUTHENTICATION: "false"

env_file:

- .env

Then restart your development container with control/command shift-p "Dev Containers: Rebuild Container". Test the health of your Broker:

warpstream kcmd --bootstrap-host warp --type diagnose-connection

You should get output like this:

running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092

Broker Details

---------------

warp:9092 (NodeID: 1415344910) [warpstream-unset-az]

ACCESSIBLE ✅

GroupCoordinator: warp:9092 (NodeID: 1415344910)

ACCESSIBLE ✅

It’s working! Create a topic and publish some messages:

warpstream kcmd --bootstrap-host warp --type create-topic --topic hello

warpstream kcmd --bootstrap-host warp --type produce --topic hello --records "world,,world"

This should create the topic hello and two messages with world in them. You should get output like this:

result: partition:0 offset:0 value:"world"

result: partition:0 offset:1 value:"world"

Now let’s read them back:

warpstream kcmd --bootstrap-host warp --type fetch --topic hello --offset 0

You should get output like this:

consuming topic:"hello" partition:0 offset:0

result: partition:0 offset:0 key:"hello" value:"world"

result: partition:0 offset:1 key:"hello" value:"world"

It works! You’ve successfully put data into a queue and fetched it back from the queue. From here you can connect to your broker on host warp and port 9092. All your data is securely backed by Tigris and you can access it from anywhere in the world.