SaaStr AI Agent Finds Challenger Email Vendor Doubles Opens in 500K-Message Test

SaaStr ran an email deliverability bake-off across three vendors, sending more than 500,000 messages, with the process fully automated by its AI “VP of Marketing,” 10K Source. The AI agent captured every issue, performed multivariate diagnostics, and produced a one-page recommendation that the team acted on the same day.

In a split test of a 10K-person “AI Day” campaign, two matched groups of about 2,300 recipients each received identical content at the same time. The challenger vendor logged 1,168 unique opens (50.5%) and 281 unique clickers (12.2%), while the incumbent recorded 485 opens (21.0%) and 96 clickers (4.2%). The challenger therefore delivered 2.4× more opens and 2.9× more clicks, despite identical audience and timing.

Scoring across seven criteria—including confirmed delivery, inbox placement, spam complaints, human clicks, sponsor reporting, and operator effort—the AI agent recommended moving core newsletter sends to the challenger, ramping volume from 20K to full scale while syncing unsubscribe data and mapping dependent workflows.

Our read: The test demonstrates that AI can execute production-scale deliverability audits far faster and more rigorously than manual processes, surfacing hidden failures and quantifying operator overhead. By embedding such agents in the procurement loop, B2B firms can maintain continuous vendor competition and avoid long-term lock-in based on limited test data.