Growing Fast, Paying Faster
A WhatsApp-based B2B messaging platform serving 1,000+ businesses was running on Google Cloud Platform. As their customer base grew, so did their cloud bill — proportionally.
- Peak load: 93–155 messages per second
- 15,000 concurrent MQTT connections (5K agents + 10K customers)
- Multi-terabyte storage needs with 36+ month retention
- Sub-second latency requirement for real-time messaging
- Growing GCP bill with no ceiling in sight
20 Dedicated Servers. 8 Clusters. 12 Weeks.
Full bare-metal infrastructure deployed in Hetzner (Falkenstein, Germany) with every service clustered for high availability.
Databases (12 nodes)
- PostgreSQL — 3-node Patroni cluster, pgBouncer, pgBackRest PITR
- MongoDB — 3-node Percona replica set, encryption at rest, PBM backups
- Cassandra — 3-node cluster, 11.5 TB raw, TWCS compaction, RF=3
- Elasticsearch — 3-node cluster, ELK stack, ILM, S3 snapshots
Messaging & Platform (8 nodes)
- RabbitMQ — 3-node cluster, quorum queues, delayed exchange
- EMQX — 3-node MQTT broker, 300K connection capacity
- K3s — 3-node Kubernetes (EPYC, 256GB RAM each)
- HAProxy — Dual-node with Keepalived VIP failover
Monitoring
Prometheus + Grafana + Loki + Alertmanager. Pre-built dashboards for every service. Multi-channel alerting.
Security
Hardened bastion host (fail2ban, PSAD, auditd). Private VLAN. TLS automation. UFW on every node. WireGuard VPN. Bitwarden Vault for secrets management.
Automation
100% Ansible-automated. Automated S3 backups for every service. Version-controlled infrastructure as code.
What "5x Performance" Actually Looks Like
Side-by-side comparison of every major service — GCP managed vs. bare-metal dedicated hardware.
PostgreSQL
| Spec | GCP Cloud SQL | Bare-Metal (Patroni HA) |
|---|---|---|
| Nodes | Single instance + read replica | 3-node Patroni cluster (auto-failover) |
| CPU | 4 vCPUs (shared, burstable) | 8 cores / 16 threads per node (dedicated) |
| RAM | 16 GB | 64 GB ECC per node (192 GB total) |
| Storage | 500 GB SSD (pd-ssd) | 2x 3.84 TB NVMe RAID-1 per node (3.84 TB usable each) |
| IOPS | 15,000 (30 IOPS/GB, capped) | 1,000,000+ (NVMe Gen4, included) |
| WAL | Same disk as data | Dedicated 512 GB NVMe per node (I/O isolation) |
| Connection Pooling | None built-in | pgBouncer on every node |
| Backups | Automated (7-day retention) | pgBackRest PITR to S3 (30+ day retention) |
IOPS improvement: 67x. RAM: 12x. Storage: 23x.
Redis
| Spec | GCP (before) | Bare-Metal (Redis Sentinel) |
|---|---|---|
| Instance | 2 vCPU, 4 GB RAM VPS | 3 servers, 16 cores / 128 GB ECC each |
| HA | None — single point of failure | Redis Sentinel (auto-failover <30s) |
| Failure Impact | Full outage — sessions lost, cache cold | Automatic failover, zero downtime |
| Available RAM | 4 GB (maxed out) | 128 GB per node (3 GB active, 125 GB headroom) |
| Persistence | RDB only | RDB snapshots + AOF |
| Risk Level | Extremely fragile | Production-grade HA |
RAM: 32x per node. From single-point-of-failure to automatic failover.
MongoDB
| Spec | MongoDB Atlas (GCP M50) | Bare-Metal (Percona) |
|---|---|---|
| CPU | 8 vCPUs (shared) | 8 cores / 16 threads per node (dedicated) |
| RAM | 32 GB | 64 GB ECC per node |
| Storage | 1 TB (max 4 TB, $0.25/GB extra) | 3.84 TB NVMe per node |
| IOPS | 7,500 (tied to storage size) | 1,000,000+ |
| Encryption at Rest | Enterprise tier only ($$$) | AES-256 included free (Percona) |
| Audit Logging | Enterprise tier only ($$$) | Included free (Percona) |
| Cost (3-node RS) | ~$1,200/month | ~$175/month |
IOPS: 133x. Storage: 3.8x. Cost: 85% less. Enterprise features included free.
Kubernetes
| Spec | GKE (Google Kubernetes Engine) | Bare-Metal (K3s) |
|---|---|---|
| Control Plane | $73/month (managed) | $0 (K3s built-in) |
| Worker Nodes | 3x n2-standard-8 (8 vCPU, 32 GB each) | 3x EPYC (16+ cores, 256 GB RAM each) |
| Total RAM | 96 GB | 768 GB |
| Storage per Node | 100 GB pd-ssd ($17/month extra) | 2x 3.84 TB NVMe RAID-1 (included) |
| Load Balancer | $18/month + $0.008/GB | MetalLB (free, unlimited) |
| Cost | ~$808/month (on-demand) | ~$306/month |
RAM: 8x. Storage: 38x per node. No control plane fee. No per-GB load balancer charges.
RabbitMQ
| Spec | Managed (CloudAMQP / Amazon MQ) | Bare-Metal (Self-hosted) |
|---|---|---|
| Cluster | 3-node HA (mq.m5.large) | 3-node cluster with quorum queues |
| RAM per Node | 8 GB | 128 GB |
| Message Throughput | Rate-limited by plan | Unlimited — hardware is the only limit |
| Cost | ~$660/month (Amazon MQ 3-node) | $0 marginal (runs on messaging servers) |
RAM: 16x per node. No message rate limits. No per-message pricing.
Network & Egress
| Spec | GCP | Hetzner Bare-Metal |
|---|---|---|
| Egress Pricing | $0.08–$0.12 per GB | $0 — unlimited included |
| Bandwidth | Metered, pay per GB | 1 Gbps unmetered per server |
| Internal Network | VPC (cross-zone: $0.01/GB) | Private VLAN (vSwitch, free) |
| At 5 TB/month egress | ~$500/month | $0 |
The Numbers Speak
| Metric | GCP (Before) | Bare-Metal (After) | Change |
|---|---|---|---|
| Monthly Infra Cost | $22,000 | $4,200 | -81% |
| Annual Savings | — | $213,600/year | |
| Database IOPS | 15,000 (provisioned, capped) | 1,000,000+ NVMe Gen4 | 67x |
| Total RAM | ~200 GB across all services | 2,560 GB across 20 servers | 12x |
| Storage Capacity | Limited (expensive to scale) | 30+ TB usable | 5–10x |
| MQTT Capacity | Managed limits | 300,000+ connections | 20x headroom |
| Egress Fees | $500+/month (metered) | $0 (unlimited) | -100% |
| Message Retention | Limited by cost | 36+ months hot storage | 3x duration |
Key Architectural Decisions
- Separated NVMe (WAL/commitlog) from SSD (data) — I/O isolation for databases
- RF=3 on Cassandra for maximum durability with QUORUM consistency
- External PostgreSQL datastore for K3s (not embedded etcd) — proven reliability
- MongoSync for zero-downtime migration from GCP to bare-metal
- Patroni health checks in HAProxy for intelligent read/write routing
- EMQX with Docker host networking — native performance for MQTT
Migration Timeline
Assessment & Architecture
Cloud spend analysis, server specification, network topology design.
Server Provisioning & Bootstrap
20 servers provisioned, OS hardened, private VLAN configured, monitoring agents deployed.
Cluster Deployment
All 8 clusters deployed, configured, tested. Ansible automation for every service.
Data Migration & Sync
Cross-cloud replication established. Data integrity verified. MongoSync for live MongoDB migration.
Cutover & Verification
DNS switched. Traffic migrated. Full health check. Monitoring verified. Handover complete.
"We went from a growing GCP bill to a fixed bare-metal cost. The performance is better, we have 10x more storage, and we actually understand our infrastructure now. The migration paid for itself in the first quarter."
— CTO, B2B Messaging Platform (anonymized)