/

Company

Judgment vs. Execution: how I over-automated AEO

Speech bubble line drawing

Last Updated

Published On

When I joined Plain in June 2025, I made a big bet to go all in on AI search visibility (aka AEO, or GEO). It's where I was already doing pretty much all of my own research, for everything from fantasy football to the B2B software I was thinking about buying. I figured our buyers were probably a lot like me, and that AI search was only going to grow as the place people do their research. Honestly, I think we're getting to a point where humans won't read most of what B2B companies write. The agents will.

And for most of my first year, it was one of the biggest slam dunks of my career.

How it started

I put together a core set of prompts I thought people would ask when buying B2B customer support tooling. If the models knew us at all, we barely showed up. Not good. So I started launching content around those prompts.

I'd never built a content engine before, never managed SEO, never even really wrote a blog post. So I did what made sense to me. I got all of our sales call transcripts in one place so they could be analyzed. I got all the competitor content that was getting cited in one place, so I could find what those pages had in common.

Then I'd ask: what are we missing, what spin can I add, and how can I tell our story through a similar framing?

Over time I broke it down to a science, at a really specific level. These words go here, the H2 says this, the FAQ schema says this, and the structured data table goes here. JavaScript rendering and JSON-LD were my Roman Empire. It was basically my full-time job for months.

And it worked. The content would get cited almost immediately. After a few weeks, real demos were coming in from LLMs, driving real pipeline, then real revenue. It was off to the races.

The part I didn't realize was the important part

The whole time, I was still in the tools every day, looking at what went up, what went down, and why I thought it moved. I'd find one of our pages that was dropping in citations and a page that was rising, grab both URLs, and stare at them. Then I'd find more pairs like that across the board and send them to an AI agent that pulled them in with Firecrawl and picked apart every technical difference.

From there I started scoring things like how neutral the cited content was, how technical it was, which LLMs thought G2 was still cool. That turned into a skill for our agents, basically a rubric for what was good to write and what wasn't. But it was still based purely on what I was seeing, and on my guesses about what the new models (coming out almost daily) would favor and what they'd start cutting off.

Then I tried to automate it

My thinking was: if I'm just looking at this data every day, and all these tools have APIs, why not funnel it all into Postgres tables? I used Firecrawl to scrape any new URLs we didn't have. Every time URLs moved up or down, the system would look at everything that was moving, and try to figure out why.

So I spent weeks building it. Then I wired the whole loop together with n8n and GitHub Actions:

  1. Review all the data

  2. Read our sales call transcripts

  3. Check live content

  4. Pick the opportunities that will boost visibility

  5. Write it

  6. Launch it

  7. Repeat

That's where it broke down. The agents were good at seeing what was going up and what was going down.

They weren't good at making an educated guess about why. They couldn't tell whether a page was dropping because it was old, or because of some new pattern across all the pages that were rising and falling that the different LLMs were starting to respect differently. They weren’t listening to the same podcast episodes.

Instead, the system kept saying, "Other companies have this, we don't, so you need to do this or we won't get citations." It wanted me to write about Plain’s CRM functionality (we’re not a CRM).

For 3 days straight, all it wanted to launch was in-depth pricing breakdowns, because pricing comes up on every sales call. Of course it does. And we already have a really thorough pricing page. Every time I looked at the topics it picked, I thought, "There's no way. This is not what I would pick."

And here's my mistake. Instead of just picking my own topics, I'd go back to the drawing board: "This is what I would pick. How do I make the system pick it?" I'd spend a few days building, and a few weeks later we'd launch content I felt better about. Something that used to take me less than a day now took 2 weeks.

It kept happening. I'd run the system, get a response, and think, "I don't think so." I'd look at the data myself for a few minutes and know it wasn't what I'd do. I'd spend a day figuring out what I would do, then weeks trying to get the system to say the same thing. Meanwhile, the rules for what was working kept changing, and so did my hypothesis about where the channel was going.

Every few weeks I'd try to get the system to say something new that matched my gut. When the goalposts keep moving, you can't really train agents for that.

Here's what the data was doing while I was building:

  • Visibility on the core prompts we track peaked at 42.8% on July 24.

  • By October 3, it had fallen to 28.1%, about a third of our visibility gone over 11 weeks.

  • Citations fell harder. By early October our pages were cited in just 18.8% of answers, about half our July rate.

Going back into the data

In October, I finally dug back into the data myself, for multiple days. It was incredibly clear what was winning and what wasn't.

The rubric we'd built at the very beginning, based on the best content getting cited back then, was out of date. What was getting cited had completely changed.

That's when I realized what had actually made the whole thing work: the assumptions I was making along the way. Those assumptions led to new experiments the agents couldn't see.

For example, when I saw overly opinionated listicles start to drop, I had a hunch that a list needed a real reason behind its picks, some kind of proprietary data, not just G2 reviews. So we built our picks on a dataset of 1,900 support leaders and technical founders evaluating support tooling, and used what mattered to them to say which tools were best for technical B2B SaaS companies. Making that change was a huge boost for us in early 2026.

That kind of judgment comes from doing this regularly: watching the market, listening to podcasts, and reading the LinkedIn posts where people share their results. You take all of that with the right-sized grain of salt and use it to steer. That ended up being the most important part of the whole thing.

Once I got back to making those calls myself, it turned around. The recovery started the week of October 12, and by November 12 visibility was back to 41.6%, just under the July peak.

Citations are still catching up and Google's AI Overviews haven't budged, but the recovery traces straight to the work we launched starting in October. For example, we refreshed our guide to the best support tools for Slack and MS Teams and citations went from 239 to 431.

What I'd tell anyone doing this

I still use agents for almost everything in this loop: the research, the scraping, the comparisons, most of the writing, the launching. What I don't do anymore is try to train them on the judgment call for one of the most important channels at the company.

I spent over a year learning everything I could about AEO, running tests across industries, and I still can't fully write down how I decide what to work on next. I don't think it's worth spending weeks or months trying to teach that to an agent right now, when that time is better spent making the calls myself, or building the things agents can decide well today.

Get your support running on Plain in minutes.

Get your support running on Plain in minutes.

Get your support running on Plain in minutes.