Scaling Slack and GitHub: Lessons from Maude Lemaire

[Source: https://newsletter.pragmaticengineer.com/p/github]

Controlling the thundering herd

Slack’s growth exposed how seemingly harmless features can explode at scale. When a CEO announced “everyone gets Friday off!” in a 100,000-member channel, users flooded the conversation with custom emojis. Each emoji required a fan-out to every member, turning ten emojis into one million network pushes and a hundred emojis into ten million. To prevent this, Slack capped custom emojis at 23, a limit that “is not missed” in practice. A related issue, termed an “event flood,” hits power users whose WebSocket connections receive a sudden deluge of messages without deduplication, overwhelming client resources.

Cost-effective load testing with Koi Pond

A prospective Slack customer with 500,000 users forced the team to validate capacity. The naïve approach—spinning up headless Chrome instances for each simulated user—would have cost $0.37 per five-hour session per instance, amounting to $150,000 per run. Maude’s response was to build Koi Pond, a Go-based tool that creates lightweight Slack “Koi” clients to establish the required WebSocket connections.

Database abstraction and shard limits at GitHub

GitHub’s Rails monolith relied on Active Record, which abstracted database access but obscured the true query load. Many of these queries were scatter queries—lacking a primary-key shard key—and required execution across all shards. The resulting latency prevented GitHub from expanding beyond 16 shards, illustrating how developer-friendly abstractions can create infra-level bottlenecks.

Human-in-the-loop AI and engineer rotation

When using large language models for foundational platform work, Maude insists on personal verification of every output, separating “foundational” from “non-foundational” tasks to safeguard stability. She also advocates rotating platform engineers onto product teams and vice-versa. This practice keeps platform engineers grounded in real user workflows and gives product engineers exposure to systemic tooling, reducing the drift that can occur when teams become siloed.

Our read: Maude Lemaire’s experiences underscore that scaling is as much about disciplined limits and visibility as it is about raw capacity. Simple policy changes—emoji caps, event-flood handling—prevent exponential load growth, while bespoke tooling like Koi Pond demonstrates the ROI of targeted engineering solutions over generic, expensive alternatives. The GitHub case warns that abstraction layers can mask costly patterns, demanding vigilant performance awareness. Embedding human judgment in AI-assisted work and fostering cross-functional rotation builds resilience against the entropy that accompanies rapid product expansion.