[WarpStream](https://www.warpstream.com/) is a [Kafka®](https://kafka.apache.org/)-compatible streaming platform that stores data directly in object storage instead of relying on local disks. This architecture unlocks virtually unlimited retention and simplifies operations. Because WarpStream is zone-aligned, it avoids costly inter-AZ network traffic by ensuring that agents and object storage communicate within the same availability zone. However, in multi-cloud architectures where clients and agents communicate across clouds, egress fees from cross-cloud traffic can quickly dominate your cloud bill as you scale. Tigris is a globally distributed, multi-cloud object storage platform with native S3 API support and zero egress fees. It dynamically places data in the region where it’s being accessed—eliminating cross-cloud data transfer costs without sacrificing on performance. By combining WarpStream with Tigris, you get a bottomless, durable, and globally aware message queue. Your data is stored efficiently, close to where it’s consumed, without incurring hidden transfer fees or needing to plan for regional bucket placement. ## The Moving Parts Apache Kafka is a durable, distributed messaging system. In Kafka, Producers write records into Topics hosted by Brokers that are read by Consumers. Used by 80% of Fortune 500 companies, Kafka connects applications and data stores using a real-time publish/subscribe model with strong durability guarantees. One of the main differences between Kafka and a traditional queueing system is _persistence._ Data written to Kafka is retained, even after it is read by a Consumer, which enables interesting patterns such as multiple consumers reading from the same topic. The main downside is that Kafka relies on local storage, meaning that your Kafka Brokers require fast disks and intra-cluster replication to ensure durability and availability. WarpStream is fully compatible with the Kafka protocol, but it fundamentally reimagines the architecture. Instead of using local disks, it writes all data directly to object storage, completely decoupling compute and storage. While Apache Kafka 3.6.0 introduced Tiered Storage to enable more flexible scaling of compute and storage, it stops short of full decoupling—brokers still require local disk, retain state, manage replication, and serve all client traffic. With WarpStream, the compute layer is completely stateless. The data does not need to be replicated between nodes since durability is handled entirely by the object store. This architecture is much simpler to operate: you don’t need to set up and manage all of Kafka’s dependencies (Zookeeper, the JVM, etc). WarpStream also ships an easy to use command line utility that helps you administrate your message queue and test functionality. In addition, we’ll use Docker, the universal package format for the Internet. Docker lets you put your application and all its dependencies into a container image so that it can’t conflict with anything else on the system. Today we’re going to deploy a WarpStream cluster backed by Tigris into a Docker container so you can create your own bottomless durable message queue. This example will use Docker compose, but it will help you understand how to create your own broker so you can deploy it anywhere. ## Prerequisites Make sure you have the following installed on your computer: - [Docker Desktop](https://www.docker.com/products/docker-desktop/) or another similar app like [Podman Desktop](https://podman-desktop.io/). - The [AWS CLI](https://aws.amazon.com/cli/). - [Warpstream’s CLI](https://docs.warpstream.com/warpstream/reference/cli-reference). You will need the following accounts: - A Tigris account from [storage.new](https://storage.new/). - A Warpstream account from [console.warpstream.com](http://console.warpstream.com/login). ## Building a compose file Clone the [warpstream-tigris](https://github.com/tigrisdata-community/warpstream-tigris) demo repo to your laptop and open it in your favourite editor, such as VS Code. ```yaml services: warp: # Grab the latest copy of the warpstream agent for your computer image: public.ecr.aws/warpstream-labs/warpstream_agent:latest # Run warpstream in "playground" mode for testing command: - playground - -advertiseHostnameStrategy - custom - -advertiseHostnameCustom - warp environment: # this is a no-op as it will default on the custom advertised hostname defined above, but you can change this if you want to use a different hostname with Kafka - WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=warp healthcheck: # Wait for the Agent to finish setting up the demo before marking it as healthy # to delay the diagnose-connection command from running for a few seconds. test: ["CMD", "sh", "-c", "sleep 10"] interval: 5s timeout: 15s retries: 5 ``` Open a new terminal in your development container and make sure WarpStream is up and running: ```text warpstream kcmd --bootstrap-host warp --type diagnose-connection ``` This should return output like the following: ```text running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092 Broker Details --------------- warp:9092 (NodeID: 1547451680) [playground] ACCESSIBLE ✅ GroupCoordinator: warp:9092 (NodeID: 1547451680) ACCESSIBLE ✅ ``` Excellent! Create a new topic with `warpstream kcmd`: ```text warpstream kcmd --bootstrap-host warp --type create-topic --topic hello ``` This should return output like the following: ```text running create-topic sub-command with bootstrap-host: warp and bootstrap-port: 9092 created topic "hello" successfully, topic ID: MQAAAAAAAAAAAAAAAAAAAA== ``` Perfect! Now let’s make it work with Tigris. Create a `.env` file in the root of the repository: ```sh cp .env.example .env code .env ``` Create a new bucket at [storage.new](https://storage.new/) in the Standard access tier. Copy its name down into your notes. Create a new [access key](https://storage.new/accesskey) with Editor permissions for that bucket. Copy the environment details into your `.env` file: ```sh ## Tigris credentials AWS_ACCESS_KEY_ID=tid_access_key_id AWS_SECRET_ACCESS_KEY=tsec_secret_access_key AWS_ENDPOINT_URL_S3=https://t3.storage.dev AWS_ENDPOINT_URL_IAM=https://iam.storage.dev AWS_REGION=auto ``` Then fill in your Warpstream secrets from the console, you need the following: - Cluster ID from the virtual clusters list (begins with `vci_`) - Bucket URL (explained below) - Agent key from the agent keys page for that virtual cluster (begins with `aks_`) - Cluster region from the admin panel (such as `us-east-1`) If your bucket is named `xe-warpstream-demo`, your bucket URL should look like this: ```text s3://xe-warpstream-demo?region=auto&endpoint=https://t3.storage.dev ``` Altogether, put these credentials in your `.env` file: ```sh ## Warpstream credentials WARPSTREAM_AGENT_KEY=aks_agent_key WARPSTREAM_BUCKET_URL='s3://xe-warpstream-demo?region=auto&endpoint=https://t3.storage.dev' WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID=vci_cluster_id WARPSTREAM_REGION=us-east-1 ``` Edit your `docker-compose.yaml` file to load the `.env` file and start warpstream in agent mode: ```yaml # docker-compose.yaml services: warp: image: public.ecr.aws/warpstream-labs/warpstream_agent:latest command: - agent environment: WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE: warp WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE: 9092 WARPSTREAM_REQUIRE_AUTHENTICATION: "false" env_file: - .env ``` Then restart your development container with control/command shift-p “Dev Containers: Rebuild Container”. Test the health of your Broker: ```text warpstream kcmd --bootstrap-host warp --type diagnose-connection ``` You should get output like this: ```text running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092 Broker Details --------------- warp:9092 (NodeID: 1415344910) [warpstream-unset-az] ACCESSIBLE ✅ GroupCoordinator: warp:9092 (NodeID: 1415344910) ACCESSIBLE ✅ ``` It’s working! Create a topic and publish some messages: ```text warpstream kcmd --bootstrap-host warp --type create-topic --topic hello warpstream kcmd --bootstrap-host warp --type produce --topic hello --records "world,,world" ``` This should create the topic `hello` and two messages with `world` in them. You should get output like this: ```text result: partition:0 offset:0 value:"world" result: partition:0 offset:1 value:"world" ``` Now let’s read them back: ```text warpstream kcmd --bootstrap-host warp --type fetch --topic hello --offset 0 ``` You should get output like this: ```text consuming topic:"hello" partition:0 offset:0 result: partition:0 offset:0 key:"hello" value:"world" result: partition:0 offset:1 key:"hello" value:"world" ``` It works! You’ve successfully put data into a queue and fetched it back from the queue. From here you can connect to your broker on host `warp` and port `9092`. All your data is securely backed by Tigris and you can access it from anywhere in the world.