Episode 17

Kelly Wortham (Forward Digital, Test & Learn Community, Experimentation Island) on Why Agentic Commerce Breaks Experimentation, Measuring What Visitors No Longer Do, and Optimizing for Machines

May 19, 2026 · Knowledge Distillation Podcast
Listen on Spotify Apple Podcasts Download MP3

Key Takeaways

  • Agentic commerce turns experimentation into a measurement problem
  • AI-referred visitors arrive already persuaded
  • Traditional A/B tests break when persuasion happens upstream
  • Segmenting AI-referred traffic matters more than perfect data
  • Optimizing for AI agents increasingly means optimizing for humans

In this episode of Knowledge Distillation, Katrin Ribant speaks with Kelly Wortham - founder of Forward Digital and the Test & Learn Community, a network of several thousand experimentation practitioners, and curator of the Experimentation Island conference. Kelly came into the industry from academia and social sciences, where running experiments meant clipboards in shopping malls and t-tests on hiring processes. That grounding in the messy real world is exactly what makes her take on agentic commerce so sharp: she sees the current shift not as a technology problem, but as a measurement crisis hiding in plain sight.

The core of the episode is a problem most experimentation teams haven’t fully reckoned with yet. As AI-referred traffic grows, visitors arrive at websites already persuaded - they’ve done their comparison shopping in ChatGPT or Perplexity and are now just confirming what they already know. A/B tests designed to persuade don’t work on people who’ve already been persuaded. And because most companies have no clean way to separate AI-referred from non-AI-referred traffic, results get blended into noise. Kelly’s practical advice: start segmenting AI-referred traffic now, even if the data is messy, because building that muscle early is more valuable than waiting for a clean solution that doesn’t exist yet.

The episode closes on a category Kelly calls brand impact tests - experiments that happen entirely off your website, in the third-party content ecosystem that AI models are trained on: reviews, product descriptions, social mentions. These are the inputs that shape what an AI recommends before a customer ever lands on your page. And on a provocation both find genuinely exciting: after years of optimizing for clicks and dark patterns, agentic commerce may be forcing brands back to clarity and human-first design - because optimizing for machines increasingly means optimizing for humans.

All episodes on our website: www.ask-y.ai/knowledge-distillation-podcast

Learn more about Prism by Ask-Y: www.ask-y.ai

Chapters

  1. 00:00 Introduction and Guest Background
  2. 03:00 Why Kelly Got Interested in Agentic Commerce
  3. 06:30 How This Shift Is Different: Customer Behavior, Not Just Technology
  4. 09:00 Why Agentic Commerce Breaks Experimentation
  5. 12:30 The Jarvis Problem: Visitors Who Arrive Already Persuaded
  6. 16:00 How to Start Segmenting AI-Referred Traffic Today
  7. 18:30 What AI-Referred Visitors Actually Do on Your Website
  8. 21:45 The Venn Diagram: SEO, Accessibility, and What AI Agents Need
  9. 23:30 Why Optimizing for Machines Means Optimizing for Humans
  10. 25:00 Brand Impact Tests: Experimenting Off Your Website
  11. 28:00 How to Run a Test on Reddit Reviews and Third-Party Content
  12. 30:00 Where to Find Kelly + Closing Thoughts

Resources & Links

Companies & Organizations
  • Forward Digital – Kelly Wortham’s experimentation and optimization consultancy
  • Test & Learn Community (TLC) – experimentation-focused Slack community founded by Kelly
  • Experimentation Island – conference curated by Kelly Wortham for experimentation practitioners
Analytics & Measurement
  • Google Analytics 4 (GA4) – referenced in the context of segmenting AI-referred traffic
  • A/B Testing – discussed in relation to experimentation challenges in agentic commerce
AI & Commerce
  • ChatGPT – referenced as a source of AI-referred traffic
  • Perplexity – referenced as a source of AI-referred traffic
  • Gemini – referenced as a source of AI-referred traffic
  • Agentic Commerce – discussed in the context of experimentation and optimization challenges
Industry Experts & Research
  • Jono Alderson – technical SEO expert referenced in Kelly’s GEO research
  • Sani Manić / NoHacks – referenced in Kelly’s research on machine-readable websites

About the Guests

Host: Katrin Ribant Guest: Kelly Wortham Podcast: Knowledge Distillation
Episode: 17 Runtime: ~51 minutes Release Date: 05/18/2026

Full Transcript

Katrin (01:39)
Welcome to Knowledge Distillation, where we explore the rise of the AI analyst. I'm your host, Katrin Ribant, CEO and founder of Ask-Y. This is episode 17, and we're continuing our exploration of how agentic commerce is reshaping the world of digital practitioners, this time turning the lens on the experimentation discipline. And who better than Kelly Wortham to help us with this? Kelly, you've spent 25 years (I'm aging you, I'm sorry, but it's true), you did spend 25 years building experimentation programs from the inside. You've trained probably hundreds, thousands of practitioners, because you're a coach as well. And you also founded and run Forward Digital and the Test & Learn Community, where probably about a couple of thousand experimentation practitioners are trading notes every day at this point. And obviously you curate the wonderful Experimentation Island conference, which is the best conference I have been to in so long. It was amazing, and I am so thrilled that you're doing two a year next year.

Kelly Wortham (02:51)
I'm really excited, because I'm, what, like now two months out of Experimentation Island, and I'm like, hmm, I could do another one of these. So two a year is going to be great.

Katrin (03:01)
So, to kick us off, Kelly, could you please tell us why you got interested in agentic commerce?

Kelly Wortham (03:12)
Awesome. Well, first of all, thank you for having me. And yeah, let's talk a little bit about this. So I actually came into the experimentation industry from academia. So at my core, I am a teacher. But honestly, I probably love learning even more. As far as I'm concerned, you don't know a subject until you can teach it. And it's probably why I created the Test & Learn Community in the first place. You want to surround yourself with people that are smarter than you so that you can learn as much as you can from those folks. And so, 12 years ago, I went out and said, hey, all you smart people, let's create this community. And we gather, and we have this amazing Slack community, and we get to learn from each other. The TLC was never about being the expert in the room. It's about creating a space where we could all learn from each other. So when I started seeing all these conversations around AI and GEO and agentic commerce and AI-mediated discovery, my first reaction, honestly, wasn't fear or hype, like all this other stuff I've been hearing and reading. It was curiosity. I really found it fascinating, and I wanted to research as much as I could. I wanted to learn, because that's what I do. That's part of teaching. It's just part of the process. So I knew absolutely nothing in the beginning. This is not my field of expertise. Nowhere near it.

Katrin (04:59)
The good thing is nobody knew anything, because it didn't exist.

Kelly Wortham (05:04)
So cool. Right, right. And so that's why I started, if anybody has seen any of it, this blog series called Mapping GEO. I just started interviewing anybody in the industry who did know something: some SEO experts, AI folks, technical practitioners. I basically went into this full research mode, and the more conversations I had, the more fascinated I became. And the more I realized that, unlike some of these other big shifts that we've had, like mobile, GDPR, GA4, all of those shifts were kind of a little slower. They were dramatic, but this time is different, because it's a customer behavior shift, and it's really fast. And then I started looking at data that different people were sharing, and I started realizing that now people are arriving further down the funnel. They're arriving already having more information. They're already making up their minds. And I realized, oh shit, we maybe aren't measuring the right things. We're maybe not ready, and we're maybe underreacting. And that's when I realized it's needed.

Katrin (06:46)
It's interesting, because when you talk about the behavioral shift, my theory is that this is sort of a mix between the mobile replatforming, because it really is a different customer behavior, right? Somewhat the rise of Amazon, because there is a disintermediation of the brand and the customer in terms of relationship management. There is a sort of breakdown there in the relationship of the brand with the customer. And some of the GA4 replatforming, because it affects measurement in a really, really strong way. And I'm thinking that we are just on the cusp of an enormous work stream on the part of marketers to decide how they're going to restructure their websites. And there are many theories out there of how you should do it. Some of the people I've had on the podcast were like, build for humans and bots will figure it out. Other people are like, no, you'll end up with two websites, a website for humans and a website for bots. So there's really a lot of different ways of looking at it. And that's kind of just step one, because you also then have to figure out, okay, how do we evolve the measurement in the data layer, what we capture, and also in the KPIs, what we measure, right? So we really have to think about how we're doing that, and that's very much an open question. And then there is a specific sort of targeting to be done to this non-human audience that is very, very different, I imagine, from what you would do for humans. So I think that's going to be a major theme of work and projects to be done. And I don't really see it yet. And I'm thinking, am I overreacting? Is the market underreacting? I'm not sure. What do you think?

Kelly Wortham (09:12)
No, no, no, no, no. I think the market is underreacting, personally. Well, I think there's a mix. I think there are people that are panicking and there are people that are underreacting. But I think today we do not have a clean way. Most companies do not have a clean way to effectively measure AI-referred traffic versus non-AI-referred traffic, and it's all being blended together in a really messy way. And on the experimentation side, this is a real problem, because we already had noisy data. This was always a problem. Sample size and power analysis have always been an important part of experimentation. And the data needs to be really clean in experimentation, in a way that it doesn't matter as much with your marketing analytics and your product.

Katrin (10:15)
Can you explain that? Can you explain why, for people who are not experimentation specialists? Why is that?

Kelly Wortham (10:23)
Yeah. So when you're talking about statistical analysis versus just looking at the performance of your website or your marketing campaign: you can sample data when you're talking about the performance of your website or the performance of your marketing campaign. When you're talking about a statistical test, if you're looking at sampled data, then you have issues, because the sampled data has to be the same for A and B. And that's statistically impossible unless your sample is very, very, very large. And you have to have increasingly large sample sizes, which increases your runtime to the point of making your test ridiculous to run. So what we do is we say, okay, we want a complete sample or a full population, and then we don't sample our data. That's why when GA4 came out and they had sampled data, everybody said, oh, well, we can't test with GA4 data, because they were running with sampled data. So you don't want incomplete data. You don't want sampled data. You want accurate data. So the noisier the data is, the greater the risk you introduce that, for example, of your control versus your treatment, maybe one of them is inaccurately sampled and the other one has the full sample, and then they're incomplete, and you can't accurately measure. You can kind of think of scales and weights: it's like putting your fingers on the scales a little bit, if that makes sense.

Katrin (12:23)
Yes, it totally does. So how does the presence of agents play into this?

Kelly Wortham (12:32)
So the best way that I can think of: we used the analogy when we were talking back and forth ahead of time. If we think of Jarvis as our AI agent: before anybody comes to your website, somebody's shopping, and they're using ChatGPT or Perplexity or Gemini or whatever, and they're comparison shopping. Maybe they're trying to pick running shoes or whatever, and they haven't come to your website yet. They're just comparison shopping between you and a competitor. And at the end of the day, they're pretty sure they've made their decision before they get to your website. Then they get to your website, and they're just coming to confirm what they already learned from Gemini, for example. When they get to the website, if they get to the website, number one, only the survivors have made it, which is an important consideration. You're running an A/B test on the product that they happen to have been doing their research on, right? Now you're running this A/B test, and maybe you're running an A/B test that is on persuasion. They've already been persuaded. That persuasion A/B test is going to be less effective for the people who came through from that AI referral than for the people who didn't get there from an AI referral. The people who came through direct or through some other channel, they need the persuasion that your A/B test provided, right? Because they haven't been persuaded yet. But it's going to be watered down, because the percentage of people who got to your website from an AI referral is higher today than it used to be. So you're not going to see as much of a lift from your persuasion test as you used to, because more people are getting there through AI referrals than they used to. But most people don't have a way to split out AI-referred traffic from non-AI-referred traffic. So you've got noise in your data, and you're going to see neutral results, when in fact, if you could say, this is the AI-referred traffic, this is the non-AI-referred traffic, and I only want to see the results of a persuasion test for non-AI-referred traffic, that would make sense.

Katrin (15:25)
Yes, I see. And obviously you talk to a lot of different people who do experimentation, right? And when you say most people don't have a way to separate AI-referred traffic from non-AI-referred traffic, do you know any people who have some workable way, who have figured something out?

Kelly Wortham (15:51)
I know people who are claiming they do, yes.

Katrin (15:54)
And do you have any idea how they do that? Share the secret.

Kelly Wortham (15:59)
Yeah, I would love for people to share that secret, really cleanly. I know people that are claiming they have, but I don't think anybody is doing it cleanly. And I think the problem is that there are people who can do pieces of it, but it's not all clean. And having said that, I think one of the questions you sent over ahead of time is, what should people start doing right away in the next quarter?

Katrin (16:41)
I was just going to ask that.

Kelly Wortham (16:44)
Yeah, yeah. And my answer is: start doing that. I don't care if it's not clean. You still need to do it.

Katrin (16:53)
But do what?

Kelly Wortham (16:55)
Start creating a segment of AI-referred traffic, however unclean it is.

Katrin (17:04)
Okay. Even if it's unclean.

Kelly Wortham (17:07)
Because we all recognize that it's going to be messy in the beginning. It's kind of like when we all just recognized that mobile-first was important, and boy, did it look gross in the beginning.

Katrin (17:25)
Oh, yes, it did.

Kelly Wortham (17:27)
Oh, it was terrible, right? It was funny, actually. It was so bad. But if you don't start, you won't get there. And it's the same with accessibility.

Katrin (17:42)
And so what you're saying basically is: with whatever you have today, start creating a baseline and monitor your baseline. And that will give you some information, which is better than no information. And if you see growth in that, you can assume that you are underestimating the growth, because you're not measuring correctly anyway.

Kelly Wortham (18:13)
Yeah. And to me, it's less about measuring your baseline than it is about recognizing the segment in your experiments, because I'm an experimenter. So I want experimenters, product owners and product managers that are running tests, marketers that are running tests, whoever's running your experiments: I want you to run experiments and look at the segment of people that came from AI referrers. And I want you to run tests targeted to AI-referred traffic, and I want you to run experiments targeted to people who did not come from AI. You should have entire experiments targeted to people who came from AI-referred traffic, because they are behaving differently. We have a lot of data now that shows that people who came from AI referrals come in further down the funnel. So we should be targeting different experiments to those people. And a lot of people are like, oh, so we just change our website...

Katrin (19:24)
How does that manifest, when you say they behave differently? I understand they come persuaded and come further down the funnel. But how does that actually manifest in terms of behavior? They browse quicker through product pages? What do you see?

Kelly Wortham (19:48)
Yeah. So if I come in already convinced, then there are things that I don't do. For example, if I'm coming as a browser, as a researcher, then I'm going to look at more product pages. I'm going to go in through search. I'm going to go into the FAQ. I'm going to do a lot more research than somebody who will often just add an item they've already decided on to their cart and make a purchase. It's a totally different experience from somebody who's already made a decision. The purchase path, or the number of pages seen, of somebody who is AI-referred: the time on site is shorter, the number of pages seen is lower. What I would do as an experimenter is look at the number of pages seen and the number of products added to cart. What we're typically seeing is they land on the site. They often go directly to a product. They might compare two products on the site, and then they're adding it to the cart and they're peacing out. What they might be doing is confirming what the AI told them. Yes, this product is available. Yes, it's on sale. Yes, it's this price. Okay, I will make my purchase, or not, because they're impatient. All they're doing is validating. It's like in a good crime novel: you're looking for what the person didn't do, and that's what gives you the clue.

Katrin (21:38)
Yeah, I see what you mean. That makes total sense. And the other thing I'm curious about is, you have this "how to win" hypothesis, right? You described it as a Venn diagram between SEO, accessibility and what AI agents need to navigate, because that's another thing that people really have to think about, right? How do you attract humans, but how do you get into that result window? Because the AI agent typically does sets of 10 queries at a time until it considers that it has enough in its context window, and then it makes a selection based on that. Nobody really knows how. And if you're not in there, then you're not.

Kelly Wortham (22:35)
Yeah. So this is one of my hypotheses right now. I've been talking to, like I said, people smarter than me in these subjects. One is a specialist in accessibility. You know Jono Alderson; you spoke with him. He's my technical SEO person that I had been interviewing on the Mapping GEO series. And I was talking to lots of other folks, but I started noticing this overlap. And then Sani at NoHacks. And I was like, wait a minute. They're saying the same things. There are these trends that they all have. SEO wants machine-readable structure. Accessibility wants machine-readable structure. AI agents require machine-readable structures. So if they all need machine-readable structures, at the end of the day we may finally be arriving at this moment where, if we're optimizing for machines and we're optimizing for humans, like for the accessibility point, maybe they're pointing in the same direction. And I find this really, really exciting, because for a really, really long time as experimenters and optimizers, we've been talking about human-first, or at least trying to. We've been preaching it. And people treat accessibility as compliance, and they treat SEO as gaming algorithms. But what if all of this is actually converging around clarity and usability and understanding? And what if optimizing for machines increasingly just means optimizing for humans? Because humans built the machines. So, I don't know, maybe after years of optimizing for clicks and engagement hacks and dark patterns and all this complexity, we finally end up back at human-first design. Yay!

Katrin (24:56)
Yeah, well, that's something to celebrate. But you also talk about this other category, which you call brand impact tests. I just want to talk about all the things we cannot control. So how do you actually execute on something like that? Because...

Kelly Wortham (25:16)
I think we can control it. Very carefully. So again, I came to this world from academia. Digital experimentation, to me... I came here from social sciences, not from marketing, not from product, not even from analytics. I used to run experiments in the field. We studied human behavior in the very messy real-world environment. So when I started in experimentation, I was telling people to run t-tests on...

Katrin (25:54)
Sorry, I just have to interrupt you for a second. So for people who have always lived in the digital world, what that means is that you were actually walking in the physical world with clipboards, and you were asking questions.

Kelly Wortham (26:09)
That's right.

Katrin (26:12)
That's what experimentation used to be. You asked people questions. That's how people used to run tests.

Kelly Wortham (26:17)
That's exactly right. So I was working at, you know, like Best Buy but red; it was called Circuit City. And I was trying to convince them that they needed to run t-tests on their hiring and retention processes. That's how I got my start. And email, social, subject lines; I mean, we did it all. But decades ago, that's how I was running tests. So to me, it's come full circle: two and a half decades later, we need to still be doing those types of tests, not just on subject lines and websites. We also need to be running tests on things like reviews and Reddit discussions and social mentions. You need to have your subject matter experts engaging, and the employees that you trust engaging, in social media. You need to find out how customers are looking at these signals. You need to look at third-party content.

Katrin (27:29)
But practically, how do you run a test on, say, Reddit reviews?

Kelly Wortham (27:38)
Okay, let's imagine a company realizes that maybe its products have outdated reviews, weak product descriptions, inconsistent positioning, low review recency, whatever.

Katrin (28:01)
Which is a real problem, right? Because the agents are going to pick up on... they don't really look at recency very well. And if you have a review from 2010 that says your product is outdated, that it looks like Windows-whatever-it-was at the time, that's what the agent is going to come back with.

Kelly Wortham (28:23)
100%. So you develop a hypothesis. You say: if we improve the freshness, the specificity, the consistency, whatever (you come up with a campaign to encourage your top customers, maybe a customer base that you trust, and you give them gift cards or whatever) for product group A, then AI-referred traffic will increase, and downstream conversion efficiency for that product line will improve compared to product line B. You pick a product line that you're not going to focus on and a product line that you are going to focus on. And that's important, because you pick product lines as your units of measure, not people. This is how you run these tests. And then for that product line, you run a coordinated effort to refresh those third-party listings. You encourage specific customer reviews. You update those outside product descriptions and the ones on your website. You improve responses to the reviews. You encourage, again, employees you trust to engage with that content. And then you look at AI-referred traffic, conversion efficiency, sales velocity, assisted conversions, and how often each product line is mentioned and recommended by the AI systems. But you also want to compare against the competitors that were doing better for those product lines. Over time, are AI tools mentioning product line A more often after the intervention? Are they describing it more accurately? Are they comparing it more favorably? Is it showing up in the same consideration set as the competitor? All that stuff. And what's really cool is that you're now not measuring whether people clicked; you're measuring whether the upstream representation of the product changed.

Katrin (30:18)
I see.

Kelly Wortham (30:21)
Does that make sense?

Katrin (30:22)
Yeah, it does. It does. Absolutely.

Kelly Wortham (30:26)
Someone super smart, who you were talking about earlier, pointed out something really brilliant to me after he read my blog post about this. He said experimentation folks are really good at optimizing for action. They always have been. The hardest part has always been this long persuasion cycle, and we nailed that part. So if AI is increasingly helping customers move through the consideration cycle, we just remove the friction, confirm trust, and make it easy for them to act. So this part's easy.

Katrin (31:01)
Yeah, that makes a lot of sense. So the data point that seems to have moved your interest was the sign change on AI referral traffic, right? It used to convert not so great; it now apparently converts a lot better, visibly better. What does that mean for an experimentation person? You said you want people to run experiments specifically for AI referral traffic. But we also have sample sizes to take into account. This is not obvious practically, right? How do you see people do that successfully? How do you recommend people design this?

Kelly Wortham (31:56)
So again, the main thing that they need to take into consideration is the messiness of the sample, because of the issue with AI-referred and non-referred traffic being aggregated together. So if you have begun measuring that at your organization, then you can divide them out, which is ideal. But what you really want is for your experimental design to take into account what you're measuring. So your metrics are the most important piece. And then in your sample size, you want to take into account what your experimental design is. Are you measuring top of funnel? If so, it's going to be really messy. If it's bottom of funnel, it's still messy. And what I mean by that is: are you trying to persuade? If persuading is the goal of your experiment, then you need to divide out by whether they're AI-referred or not AI-referred. If it's something simple, like you're just measuring whether they converted or not, then it doesn't matter as much, and it might be a super clean experiment that isn't as impacted by AI-referred versus non-AI-referred. AI-referred traffic is coming in wanting to validate and confirm what they've already learned ahead of time. And that's what's messing things up and making things noisy.

Katrin (33:47)
Let's imagine that I have some way to identify some users that are AI-referred. We don't know if this way is great or not great; it's the way I have, with the tools I have. That creates a segment of users that are AI-referred. Would I run experiments on these users alone, or would I expose them together with a mixed group where I may have some AI-referred users, but I won't know?

Kelly Wortham (34:25)
It only depends on what the experiment is. So let's say you have a feature that you are testing, and for that feature, AI-referred versus non-AI-referred doesn't matter. In that case, it genuinely doesn't matter. But if we have a prior that says the knowledge the person has before they land on your page is going to have a lot of influence, like if they came to your site already influenced, then that matters. Because maybe the feature is a "help me choose," for example. They probably don't need it if they came through AI search. So you would maybe want to exclude the AI-referred traffic from your test entirely.

Katrin (35:37)
I see. Yeah, that totally makes sense. And I'm sorry, this was not in our prep questions, but I really want to ask it. If you just look into the future: we're going to solve some of these measurement problems, because we've solved them before. Maybe not perfectly, but to the level that we solve measurement problems. And obviously agentic commerce is going to grow, and people are going to do some work on their websites, some better than others, some more than others. We'll have a messy reality, as we always have, but we'll have better tools and better techniques. Let's say six months from now and a year from now, because I don't think anybody can possibly have any idea beyond that. How do you see this evolve for experimentation teams in terms of measurement and tooling? Because there are a number of tools out there to execute experiments, and those are, I suppose, evolving very quickly as well, although I don't really have a very good notion of how. What do you see, six months, a year out?

Kelly Wortham (37:15)
I'm not entirely sure, but in my humble guess, they'll catch up, because they have to. Some of them already claim to. The thing that they have to figure out, again, is the AI-referred traffic. Some of them are claiming that they already have it. I think, just like browsers already have the native ability to capture certain things, the tools will too. But I'm not sure. I don't know.

Katrin (38:07)
And let's just talk about people. You talk to a lot of people who do experimentation. With all the changes that AI brings to work, what do you see affecting experimentation practitioners in their day-to-day, in terms of work processes or tooling that has to do with using large language models in one shape or another? How do you see the craft evolving? Maybe some friction points are being removed, or maybe some types of processes are changing?

Kelly Wortham (38:55)
What do you mean? I don't understand the question.

Katrin (39:04)
So imagine you're an experimentation person. I don't actually know specifically where the friction points are in the workflow of an experimentation person, but I imagine there are things to do with image generation, variations of copy, variations of colors. So developing and creating all of that, making a decision about what we're going to test and how we're going to test it. What are the creatives going to be? What are the variants going to be, with which hypothesis, with which thresholds, et cetera. All of that has a lot of bits and pieces to it. And all of that is ultimately based on some precise parameters and some softer parameters: blue versus red, this button versus that button. I don't really know what that workflow typically looks like for an experimentation team. I don't know where AI helps them, and I don't know how scared they are for their jobs. That's kind of what I'm getting at, because a lot of people are very scared for their jobs.

Kelly Wortham (40:21)
Yeah, there's a lot of vibe coding happening on the engineering side. There's a lot of vibe coding going on where people create an experiment, where they're designing the whole experiment around it. It's clunky, it's bad, it's not good experimentation, in my opinion. They fail to have an experimental design. They fail to have a measurement plan. They fail to have a full decision set, the measurement strategy; all of it's missing. And at the end of the day, what you don't have is the continuous learning that is required. We experiment to learn. That's the whole reason. It's the education that feeds your organization. It's not a one-and-done. You'll have false positives if you just run an A/B test and say, this is what I learned, and you do it with vibe coding. Nobody knows if it's real. And I'm not saying that one time couldn't have been a win. But if it's not...

Katrin (42:08)
Do you think that the profession is, if not immune, at least resistant to being cannibalized by AI?

Kelly Wortham (42:27)
Okay. So back in the beginning, platforms first made certain promises, like a single line of code and you're ready to go. That wasn't quite true, but it took many, many years before we all figured out that it actually took a while to get your platform implemented. And then there were the WYSIWYG editors, which make really messy code, but they allowed a whole bunch of people to test who couldn't test before. And that's a good thing. That's what I think vibe coding does for experimentation. It's good, it's bad, it's somewhere in between. It allows people who otherwise couldn't run experiments to think through and build experiments. But what I would really like is some kind of middleman in there, to make sure that they've thought through the measurement plan and the strategy and the experimental design, to make sure that we're running experiments in the right way. And I think that's really, really important. And I think there are people thinking that through. I think tools and really smart people are building that now, and they will continue doing that. And there are companies doing that. Dylan Lewis at Atlassian: they are doing that. He did a great talk about it. And June Dershewitz talked about how you can be the middleman, where you build almost like MVPs with AI and then use those as a communication tool. And I think that's where we are. I think it's a good thing, using it as a communication tool but not the end product. And I think that's the important thing to communicate here. Vibe coding as an end product is really, really dangerous, for risk, for privacy and for quality. So I urge everyone to not run an A/B test that you vibe coded. Please don't do that.

Katrin (45:11)
That's a good warning. So for the analysts and experimentation leads listening to us, who maybe just ran their Monday stand-up and are wondering whether their test backlog is pointed at the right problems: what are two or three things that you would like them to rethink before Friday?

Kelly Wortham (45:35)
Two or three things. Number one: have you looked off-site for ideas about your brand? Because I guarantee AI is. So what is your brand doing? How is your brand doing? And what hypotheses can you come up with that you've never thought of before, ways that you might improve your brand identity? These are big, bold ideas, and I promise they'll be exciting to think about. So that's one. Two: are you measuring the right thing, and how can you measure it? And then three, and most important: are you tracking yet, and how can you track your AI-referred traffic?

Katrin (46:32)
And so number one, what you described with brand experimentation, is really a whole bucket of things that people can look at, right? And for the AI traffic measurement: don't look for perfection, work with what you have.

Kelly Wortham (46:51)
Yeah. Absolutely.

Katrin (46:53)
Well, this was really, really great. Thank you. I learned a lot. I really did learn a lot. That was really amazing, and thank you so much for the conversation. Shameless plug time: where should people read your writing? How can they work with you? What type of projects do you actually work on, what is the core of what you're focusing on and specializing in these days, and how can they join the TLC community? And we'll put everything in the show notes.

Kelly Wortham (47:24)
Okay. First of all, to find the TLC, come to testandlearn.community and sign the code of conduct. The first rule of the TLC is don't be a dick. It's free to join, and it's wonderful. You can read my writing on the TLC as well; there's a blog there, and you can learn about Experimentation Island. And as Kat said, yes, we're going bi-coastal. There'll be two events starting in 2027. Sneak peek: we're changing the name.

Katrin (48:01)
What? What?

Kelly Wortham (48:03)
I know, I love Experimentation Island, but we felt like it was exclusive, and we want to broaden it. It's not just for experimenters. It's for those who run experiments in marketing, in product, in growth, right? It's data people and UXers. It's for everybody. So the name is going to change. The announcement will come soon; if you have ideas, send them my way. But the name will change before the next event. So that's the TLC. My company is a consulting firm, Forward Digital, forwarddigital.org. We are a for-profit for good; we pay it forward. And I am an experimentation coach. I mostly do training and process, and help companies figure out why they're stagnating, and lots of training related to those things. So if you want to talk, reach out. What else? Can you think of anything else, Kat?

Katrin (49:11)
So basically, if people are stuck in their experimentation program, they're not really understanding why they're not getting results, getting learnings, et cetera, that's when they call you.

Kelly Wortham (49:24)
That's right. We do a whole lot of that. Yep.

Katrin (49:28)
Well, I'll put all of those links in the show notes: TLC, Forward Digital, Experimentation Island with the new name and the old name so that people know what we're talking about, and all of your writing and your blogs. You're also quite active on LinkedIn. Some people don't respond to their DMs or messages on LinkedIn, so... okay, cool. So I'll put that too. And so, thank you very much. This was really great. And that's it for episode 17 of Knowledge Distillation. If today's conversation made you want to experiment with AI for analytics, visit us at ask-y.ai and try Prism. Thank you for listening, and remember: bots won't win, AI analysts will.

← Previous Episode Gal Rapoport (Kahoona, Amazon, AWS Inferentia) on Digital Body Language, Why Agentic Commerce Needs a New Web, and How to Derive Intent from Anonymous Traffic Next Episode → Johan van de Werken (Superform Labs, GA4Dataform, Select Star, Simmer) on Why Trust Starts at the Data Layer, Knowledge Compiled as Code, and the Hidden Cost of AI-Written BigQuery Queries