Katrin (00:01)
Welcome to Knowledge Distillation where we explore the rise of the AI analyst. I'm your host, Katrin Ribant, CEO and founder of Ask-Y. This is episode eight and today we're tackling a shift that's reshaping e-commerce
and the web at large. Our websites must now satisfy two audiences, humans and AI agents. For this, my guest today is Slobodan Manic, though everybody calls you Sani. Sani, you're website optimization consultant with 15 plus years of web performance, CRO, also technical SEO. You work with teams at Instacart.
You also wrote for Search Engine Journal and certainly did a whole bunch of things that my research hasn't yet unearthed. You're also a WordPress call contributor and obviously you host the Nohack podcast, which I think has 200 plus episodes at this point. ⁓ Now subtitled, Optimizing the Web for AI Agents. Is that correct? I think.
Sani (00:46)
You
That is
correct. First of all, what a wonderful intro and thank you for having me. It's an honor to be a guest on the podcast.
Katrin (01:09)
Well, let's thank
Claude for the help. Not that I didn't know some of these things and I can read LinkedIn, but Claude was really helpful. We're dead. We're dead.
Sani (01:12)
No, this is great. Come on, we had multiple conversations already before we even talked about the
recording. So thank you, Claude. Let's say that.
Katrin (01:23)
So, you know, all of these tells us exactly where your focus has landed and ⁓ why you're my host on this episode. would you tell us a little bit about why you and like how and why you decided to focus on this?
Sani (01:26)
Hmm.
So I, like many people, I fell into the trap of needing to know everything about AI as soon as it's out. And we've been living that hamster wheel life for the last two or three years. Like everything is the most important thing in the world. Everything, it's a right around the corner. Some big things are happening. It's going to be amazing. I'm tired of that. Honestly, I spent the second half of last year just being tired of trying to keep up with everything. I don't know.
Katrin (01:47)
yes, we're all there.
Sani (02:06)
maybe it's ADHD and the need to just know and explore everything, whatever it is, most optimizers have something. I just got tired, honestly. And also I noticed that every new update and release is kind of the same as the old one. ChachiPT5, was that even a new model, Frontier model? Was that just a label on the old model? So I think it doesn't matter. Everything that's happening in the AI world, most things that are happening in the AI world, not everything, don't matter.
And I think the ratio is 99 % hype and bullshit, 1 % signal. If you don't know what the 1 % is, it's best to sit it out and just wait to realize to see what's actually worth paying attention to. Also, at the same time, we have 35 years of web infrastructure, all the websites we've built over the 35 years. And now we have AI that needs to look at them and understand them and who's doing that work. Like that feels urgent.
necessary and extremely important, more important than keeping up with the latest models. So I'm going back to what I know how to do best, optimize websites, make them technically healthy.
Katrin (03:14)
So actually, tell us what it is that you've been specializing on before this shift, like when it comes to web optimization, because I know you're very sort of like specialized in one specific aspect and how that sort of leads into this.
Sani (03:18)
Mm hmm. Right.
So when I started
learning about ⁓ web development back in mid-2000s, how do you call them? The zeros, the first decade of the century, I think that's the term, something like that. We're not native English speakers. They'll forgive us.
Katrin (03:36)
Yes, I think we say notes. Not sure.
We're just going to say notes. There you go. That's the term.
Sani (03:45)
Let's say that. ⁓
So when I started learning about semantic web and how the browser can understand a website and then like why h1 is not the same thing as a span and why you should use one over the other, which it feels stupid that we have a need to talk about that still in 2026. When I was getting started as a web developer, I was dead focused on semantics, accessibility, and learning what code means.
Katrin (04:13)
Mm-hmm.
Sani (04:14)
because at that time the browser was still, we had IE6, IE7, all those stupid browsers that needed everything to be perfect for them to be able to understand it.
Katrin (04:20)
⁓ yes.
Sani (04:26)
Then I moved into technical optimization, speed optimization, core web vitals. I even did some analytics briefly and technical SEO. And the thing that still sticks and always will be there is that you need a healthy website, whether that's for humans, whether that's for people who have accessibility needs or AI agents or LLM systems. So getting the basics done, I think, is the most important thing before you go do your CRO or your SEO or...
whatever else. I think getting that website to be as healthy as possible.
everyone benefits. So that's why I'm kind going back to that because now ⁓ it feels like more people are abandoning websites and just building vibe coding and building whatever else and no one is really taking care of. Shaving two seconds off of every page load. Why are people not doing that?
Katrin (05:19)
Right.
Yeah, and that does change things to the experience for sure. And talking about which we collaborated recently on something really fun. It was that Halloween episode of the floofies for those of you in the audience who don't know the floofies are creatures that I created with AI and I do AI videos with them to explain the functioning of LLMs.
Sani (05:33)
Yep.
Katrin (05:43)
things like what is a context window, what is attention mechanism, how do data flow in an LLM, how do tokens get generated, things like that. And the flu fees are the central characters of that. And so the episode is called the Flu Fees and the Instant Checkout Crisis. It's really a parable about brands becoming ghosts where their data isn't optimized for AI discovery.
So let's dig into the real version of that. What does it actually look like in the real world? And do you have any real example of this?
Sani (06:21)
So the real examples are kind of being released right now. We had the open AI agent, shopping agent, whatever it was, Perplexity has some version of that. So basically what this means is that the user doesn't have to go to your website to complete a transaction anymore. The user can or will be able to just search for a product, ask their LLM or AI assistant on their phone ⁓ or on the computer.
Can you find this and this for me or that product for me? And then the agent will be able to identify five candidate websites and then maybe ask you which one do you want to use? And you say, I want to use this one. And then it goes and completes the transaction for you. The user never sees your colors, your logo, your hero image, your warm copy in your product page. So I think that that kind of, ⁓ I just had an episode of My Podcast released yesterday. We're moving the threshold up.
And we need to upstream engineering. We're not optimizing the product pages for humans and for emotion and for ⁓ triggers or urgency or whatever else anymore. We are optimizing for clarity and we need to make sure that the LLMs and all those AI systems understand exactly why they would want to do business with you on behalf of their human.
Katrin (07:15)
Mm-hmm.
And so ⁓ we are optimizing for those AI agents, but humans still do visit websites and will continue to visit websites. And we still, I suppose, have a need for traditional SEO, but simply to be present in those search results. So how do you look at those two things together?
Sani (07:41)
Absolutely.
That's an excellent question. The way you optimize for two different things is to find what they have in common and optimize for that. So any LLM, any AI will need the code to be semantic, loading fast and be accessible and be clear and the information presented on the page not to be confusing and not to be self-contradictory. What do humans need? They need the same thing. They need a fast website. They want a fast website. This is what Google has been telling us, by the way, for the last seven or eight years that
the experience of the user when they land, even though Google is like traditionally or seven, eight years ago was just SEO, it's just for rankings. No, they've been pushing the narrative of your website needs to be fast with Core Web Vitals. You need to be seen as an expert with EEAT and all those other things that they have been quietly serving for the last decade without telling us this is going to be important in the AI age. Well, all those things are now extremely and even more important than were when they were...
only when applied to humans. you find whatever, basically whatever the machine and the human have in common and what their needs are and optimize for that and go heavily into optimizing for that.
Katrin (09:03)
So you're specialized in ⁓ fixing what we could call the technical foundations, right? The accessibility, the semantic structure, the performance, like all of those really highly technical things that are ⁓ very hard on larger websites, obviously, right? And larger complex websites and, you know, complex e-commerce, et cetera. How do those traditional web quality signals translate to AR readability specifically, like
Sani (09:08)
Yes, absolutely.
Katrin (09:33)
Let's go, you know, our audience is technical. Let's go, let's go technical into like, what is it that you really actually need to do?
Sani (09:42)
⁓ You need to make your website as complicated as it needs to be and nothing more than that. So trim all the fat that you don't want to have and you don't need to have on your website because when an AI or an LLM is processing your websites, there's a token cost. This is not free. This is what currently is being presented as it's free or almost free. So use as much of it as you want. But if an AI agent needs 100,000 tokens to complete an operation on your website.
you'll start paying that at some point, or your users will start paying. This is not going to be ⁓ virtually free forever. And we all know that all these companies that have these frontier models, they're bleeding money left and right. So they will start charging more for these tokens at some point. So ⁓ I think it's really important to, ⁓ when it comes to speed, for example, ⁓ the main problem with speed with most websites is that they're loading assets that they should not be loading. They're loading all those JavaScript files that they don't need.
in a lot of pages, CSS files and CSS rules that they don't need. Just try to trim as much of that fat as possible and you'll do better than 95 % of the websites.
Katrin (10:51)
Could we translate that into an example of the real asset? When you say JavaScript or CS, what does that correspond to in terms of an element?
Sani (10:54)
⁓ Yep, yep.
I'll give you an example. if you have an A-B testing ⁓ script from a vendor on your website, front end A-B testing, so in browser, there's a JavaScript snippet that's being loaded, that JavaScript.
Katrin (11:11)
Mm-hmm.
Let's
take the example just to irritate all the A-B testers out there. Kelly, you in particular, love you. So let's take the example of we are testing the color of the button. They all hate it when you reduce it to that. So let's do that. Yes, exactly.
Sani (11:23)
Yes, we love you, Kelly.
Let's, but let's reduce it to that. Let's say you're testing a button caller in your about page.
You're testing a button caller in your about page and beyond that about page, there's a, or maybe just in that about page, there's a form where you sign up for something. Nothing beyond that. It's one page test. There's no checkout. There's no card. Now, most people will load that snippet in every single page of their website. And in A-B testing, the front end snippet is,
Katrin (11:54)
Mm-hmm.
Sani (11:58)
to prevent the flicker of content shifting and moving left and right between the, when it's loading a variation, it will block showing the page for a second or two until it loads the changes, changes it while it's invisible and then shows you the page. That adds two seconds to every single page load. If you're loading that snippet in all of the pages on your website, you're adding two seconds of page load to every single page of your website. Furthermore, that JavaScript,
Katrin (12:07)
Mm-hmm.
I see.
Sani (12:27)
⁓ file is it's a third party assets is loaded from a CDN that you don't control. It's it's whatever the platform is. Let's say Google optimise because it's dead. So we don't throw anyone under the bus. But let's say you're loading that from a third party CDN. That's a that's a cardinal standing performance. You're loading a third party render blocking asset, which means if their CDN goes down, your website goes down. If that doesn't make you panic and rethink everything you're doing about how you're
Katrin (12:32)
Mm-hmm.
Sani (12:55)
optimizing your website. This is more important than anything CRO can do, in my opinion. So just load the assets you need only when you need them and only at the time when you need them, not soon.
Katrin (13:07)
So what you're just describing is a best practice that is valid AI agent or not AI agent, right?
Sani (13:13)
Those
two seconds will be added for every single user. That's AI agent, that's a human. So you're making your website slower because you just added it to the head section of every page of your website because that's what the instructions said. That's not what you're supposed to do.
Katrin (13:17)
Mm-hmm.
And is there something specific like, so in the, few stories we talked about brands with like sparse descriptions and missing details becoming ghosts, basically present but unseen. When you look at a site for AI agent compatibility specifically, what are the common failures that you see specifically for AI agents of, you know, of that, you know, of that sort of level. And if you could describe.
Sani (13:34)
Right.
Katrin (13:54)
the technical aspect of it, also sort of like what it translates to into a real use case so we can link it to things we do.
Sani (14:01)
So I don't want to make this strictly an accessibility story, but this is
about accessibility. If you have buttons and you're using something like a Tailwind CSS framework ⁓ to style those buttons, you can use a span tag and just give it 20 classes and it's going to look like a button. It looks like a button to you. It doesn't look like a button to a user who uses assistive technologies or to an AI agent. use the correct...
Katrin (14:18)
Mm-hmm.
Sani (14:27)
HTML tag, use all of the correct HTML attributes, make it painfully obvious for the agent. We're talking about agent humans as well, but for the agent, make it painfully obvious what each element is and what it does. Because if you don't, that's when they hallucinate. Not always, they will not always be wrong, but an AI agent, as you know, they guess. If you don't feed them 100 % accurate information, they will guess and they'll guess wrong sometimes.
Katrin (14:50)
Yeah, of course.
Yeah.
Sani (14:55)
So make it painfully obvious on every single level. And that's the visual of your page. That is the code of your page. That is the content you have in the page. Painfully obvious what it's about. That is the number one thing.
Katrin (15:07)
So
I see what you mean when you're saying that it goes back to the fundamentals of what you were doing in the alts. And I'm just going to say that across the episode because I like that we're saying that. When you had to optimize for those crappy browsers that we had at the time that were requiring perfection for everything, basically,
Sani (15:17)
This is...
Katrin (15:31)
Paradoxically, we are working with a guessing engine that tolerates ambiguity, but if we want that engine, so the LLM, to be able to guess correctly about our website, we have to be painfully accurate.
Sani (15:45)
It needs help. It needs as much help as we can give it. Yes. And even then, who knows, but it's more likely to get the right answer. I mean, if you go to any LLM and ask it something that's kind of ambiguous and you expect it not to know the answer, it's going to give you a different answer as many times as you ask it. When you have a messy page with messy and confusing HTML rendered in the browser or messy, which leads to a messy and confusing accessibility tree, which is what these agents are doing when they control the browser.
Katrin (15:46)
Mm.
Sani (16:14)
they're using. It's more likely, it's infinitely more likely to make a mistake if it doesn't know what the subscribe button is or whatever the action is in the page.
Katrin (16:25)
So if we think about this in this way, where it means that basically we have to really fix the ⁓ accuracy of the technical structure of the web page, we are ultimately...
also creating a new data layer for the analyst, right? Because all of these elements are what gets tagged and what then feeds into ⁓ the base data that we use in digital analytics that we all complain about being so messy because, I mean, you you hack one thing on top of one thing on top of one thing. And when you have an all the website, you end up with all of these subsequent layers of hacks where you end up with a data layer that is just like extremely messy.
So good news is this may force a lot of brands to sort of clean up some of that aspect. That would be nice.
Sani (17:17)
sure hope so. I'm not 100
%... That would be amazing because we know the answer. We know the answer to this problem. Just fix... Like if you have a bad road, the answer is not to build a bridge on top of it. It's fix the road. Like we have the foundational layer and that is the HTML of our websites, which is the language of the browser. And these AI agents are using the browser to access and understand and read the website. We have it already.
We know what it needs to be. All the specs, all of the HTML specs, they barely have changed at all in the last 10, 15 years. We know exactly what we need to do. But then again, for, I don't know, developer experience, we have React to make it easy for developers to write components and just do single page apps. We have Tailwind that gives you 20 classes to make the button red and bold. What happened with external CSS files and HTML for just for the meaning and CSS for the decoration?
Like when I was getting started, again, this is old man yelling at the cloud a little bit, but HTML was the meaning, CSS was the decoration, and JavaScript was the interactivity. And there was nothing wrong with that. And there still is nothing wrong with that. But now we have everything in one big JavaScript bundle, and we call it a single page app. And cool. I just don't get it.
Katrin (18:22)
Mm-hmm.
That's how I learned it too. Yeah, it was great.
So note to everyone out there, if you're in an organization where you hear chatter about redoing the website for AI agents, I suppose this is in every organization that has e-commerce at this point, right? Quite frankly, probably every organization period, because everybody needs to be discoverable by LLMs. Try to get into the conversation because this is your chance to maybe fix some, at least some of the data there.
Sani (18:55)
has to be, has to be.
Mm-hmm.
Katrin (19:10)
⁓ It's a chance that's not going to come back because I mean whoever re-does these fundamentals if they're not forced to it.
Sani (19:17)
Yeah.
This is so painfully simple. This is like going to a doctor and they tell you, need to walk 10,000 steps. Yes, but is there a pill? Forget about the pill. Just do the 10,000 steps and you'll be healthy. If they tell you that, just do the basic thing and things are going to be better for you. Doing the fundamentals and the basics. I don't know. Let's make boring sexy again if we're going to have a slogan. Let's try to do that. So yes.
Katrin (19:41)
That's a great slogan, I like it. I just want
to come back to your HTML, CSS, JavaScript ⁓ separation and bundling, et cetera. Because I definitely know your opinion about by coding, you're not shy. You're not shy. ⁓
Sani (19:50)
Mm-hmm.
I'm not shy. I'm not shy about live recording. I'm not. I'm not. Not at all.
Katrin (20:10)
Do you see brands trying to fix these issues with some vibe coding of webpages? And if so, feel free to rant about it.
Sani (20:19)
So that's an interesting question because it's fixing a problem by not understanding what the problem is, essentially, because Vibe coding. Let me just define what I mean when I say Vibe coding and why I'm so, I'm not fired up. I just, I don't think it works. Okay, ⁓ basically AI assisted coding when you understand what you're doing is not Vibe coding.
Katrin (20:37)
Come on, you're fired up.
Hmm?
Sani (20:46)
That's not what I mean at all. If you have a developer who understands what they're trying to build, what good output is versus bad output. If you have that, that's not vibe coding. Vibe coding is I have a dream. I want to build this. I don't know how, but I know how to type a sentence or two and I build it and I call it a product. That's what I call vibe coding. And I don't think long-term back end works. So basically if you have a website that's essentially broken, works, but it's kind of not.
doesn't work really well, and you don't know why, and then you ask AI to fix something you don't know what it is. It just doesn't make any sense on any level to fix it like that. One thing that I saw, I think it applies here, I saw a screenshot of a vibe coding prompt that says, can you add class red 400 to this element? That was a prompt to...
Just type red 400 where the classes are. Why waste anything? So rather than using AI to fix something in a way that we have no idea what it is, maybe just look at the code and understand what's wrong and try to understand, or hire someone who does, because long-term it's going to be cheaper. I guarantee you that.
Katrin (22:02)
Yeah, that's true. ⁓ I suppose we're in that weird limbo where ⁓ a lot of new functionalities needed to handle this efficiently aren't really covered in a mature way, right? And we are stuck between building our own tools or trying to understand the different aspects of what we're confronted with. ⁓ On that note, actually, shameless plug. ⁓
Episode six with Jim Janolio is talking about episode six of knowledge distillation, where Jim Janolio is talking about just that subject, like the AI analyst as a builder of tools and a consumer of tools.
tools. when I was asking the previous question, I was just thinking I can totally see somebody going like, I understand that this page isn't really discoverable by an agent. Let me just take the code, throw it into Clode or ⁓ Gemini, and ask it to fix it for LLMs. Yeah.
Sani (22:58)
I love how you're leaving chat GPT out. That's the way in
2026. Let's just leave it out of the conversation, please. Sorry.
Katrin (23:04)
Yeah, I
tend to. tend to. ⁓ And like my question is, do you see a lot of that? And that was kind of what I was sort of, you know, thinking about it in terms of vibe coding, because it's basically vibe coding the code of the page, right? It's like sort of in a way.
Sani (23:22)
I think there's an assumption in that process. There's an assumption that
the output is going to be correct.
Why are we assuming that?
Do we have a reason to... Why are we assuming that LLMs are correct? They're correct a percentage of the time. We don't know when they're wrong. If you don't know... We don't even know what the percent... Exactly. So if you don't know what the correct answer is, so you can verify what the LLM told you, don't use an LLM. If you know what the correct answer is, you can do it yourself faster. So why?
Katrin (23:35)
It's a very good question.
We don't even know what that percentage is, really.
Mm.
Yeah, well, you know, first of all, where there's a vacuum, there's going to be people trying to fill the gap, right? And one of the things you've written about this space is that it's poisoned by AI, that most AI wrapper sort of SaaS products will be worthless in 12, 24 months. Thinking about, you know, people in the audience, either responsible for websites, building data layers, analysts, that are evaluating some of these new
Sani (24:09)
Yes, yes, yes, absolutely.
Hmm.
Mm-hmm.
Katrin (24:32)
Offerings and I have to confess I know nothing about these offerings I just assume they exist because again when there's a vacuum there's something filling the vacuum, right? ⁓ So with for people evaluating like AI powered e-commerce of analytics tools, etc What are like can you point some red flags that people should look at not to not to fall in that trap?
Sani (24:40)
when there's a vacuum, I love that.
Hmm.
⁓ when analyzing a tool before using it, right? That's question. Okay.
Katrin (25:02)
Yeah, like an
AI rapper, something that would promise to help you with this particular problem.
Sani (25:10)
Well, if you can see who built the tool, just see if they've done it without AI ever.
Katrin (25:13)
Mm-hmm.
Mm-hmm.
Sani (25:18)
If the answer is no, you spare yourself a long thinking session because if that person doesn't know how to do without AI, if it's something new that they figure, if I just write a prompt and have a wrapper, it'll do it for me. Don't just walk away slowly, carefully. But if that person is someone who knows the process and now is using AI to automate part of that process, read on and pay attention to that.
Katrin (25:43)
Mm.
Sani (25:45)
I think that that's the first red flag. If it's someone who never has any track record of doing what they're using AI to do for them now, because they don't know if it's correct or if it's giving them bullshit answers.
Katrin (25:47)
Yeah.
Mm-hmm.
It comes back
to the same thing, right? If you don't know the answer, then it can't help you.
Sani (26:02)
Exactly,
Exactly. mean, there's just no way. So I think that would be number one. Number two, ⁓ if you can try it, just try and see what it does. And then if you try it again and tells you something else that's completely different, just walk away. So I guess those are the ways.
Katrin (26:07)
Yeah.
Good point. Because again, if you don't know what
the answer is, and you can't you can't judge how correct it is or not, at least you can correct it. You can judge is it consistent. And if it says, yeah. Yeah.
Sani (26:31)
⁓ Yes, at least you have that. Great point. So
I think those two would be the first two signs for me, the first two things I would look at. ⁓ Also, if you, the user, assessing the tool you need to be using, if you don't know what good looks like, try to figure that out first before you use AI or any AI tool or any AI wrapper. Also, those thin wrappers that just have a prompt and a UI.
No one needs that. It's time for that to go away.
Katrin (27:03)
Yeah,
yeah. Speaking of which, amongst many things that I think we both agree on is the fact that learning about AI technology is not just prompting and context engineering, but really understanding how the LLMs work. And the difference between the models, the applicative ecosystem on top of the model.
and what the labs are building around that is really how you spot when someone is selling your hype. it's really the thing that if I were to tell anybody to invest in is actually understand how models work.
Sani (27:48)
Mm-hmm.
Katrin (27:48)
And what
is the model and what is the applicative layer on top of the model? Because even within the labs, they're building applications, obviously, and agents on top of them. And they're vastly different in qualities amongst them. I know you have opinions about the labs and what they offer and in terms of what those applications are. ⁓ Where would you say people should spend some time digging?
Sani (28:15)
I think, again, make boring, sexy again. Learn the basics of how these things work. If you don't know what an LLM is, what it does, what a system prompt is versus a user prompt, all those things. If you don't know any of that, just do that first. It will take a weekend of reading to get the very, very, very basics. That's what I'm talking about. You don't need a PhD in AI. You can watch the...
Katrin (28:40)
You can watch the floofy videos.
Sani (28:43)
Absolutely. No, that's a great example. You need to know that I've had too many conversations with people that take everything Chachi BD told them for granted. And I just said, can you ask it again? ⁓
Katrin (28:55)
That's fascinating. I mean,
that's literally really fascinating because, mean, how do people not understand what token prediction is? Like literally it is probabilistic prediction.
Sani (29:11)
Because no one told them. Because Samultiple is not going to tell them that when he's typing ⁓ chat GPT. That's why. And most people don't need to dig deeper. And look, I'm not blaming them. They just don't need to do that. I'm talking about an average chat GPT user. So ⁓ I think humanity was so primed for something like this to come in and just do all the damage that it's doing and will keep doing that.
A lot of smart people will be analyzing what happened in 2020 for a very long time. That's the way I see AI.
Katrin (29:45)
I'm sure that's true.
I'm sure that's true. It's definitely a historical moment. Well, and you know, let's talk about security, like, you know, AI security for a moment. It's a thorny subject. You did an episode of No Hacks on Chaggpt exposing conversations from users for at least, you know, whose teams are using AI tools and uploading data, ⁓ even if it seems to be non-critical data.
Sani (29:55)
Mm-hmm.
Katrin (30:15)
would you say, like, how would you characterize the real exposure risk?
Sani (30:21)
huge, vast, in one word. I think ⁓ you cannot trust these cloud-based systems that just with any sensitive data. If you want to have ⁓ secure AI working for you, it has to be something you fully control. So something local, something that's open source installed on your machines or your computers. ⁓ I don't think security and any of these major frontier models go hand in hand and never will because
The least bad thing that can happen is them using your data for a next training run. And do you want that? Like, do you really want something like that?
Katrin (31:00)
I think that even if they tell you they don't and you believe them, let's imagine they tell you they don't and you believe them. Working theory, right? They tell you they don't you believe them. The way those systems are architected means that it's not just that you cannot trust them, they cannot trust themselves. They do not actually have control over where the information...
Sani (31:05)
Let's imagine, yes, yes.
Yep. Yep.
Absolutely.
Katrin (31:26)
is when it goes across the layers of the transformer, it just that that information is not traceable at that point. And so ⁓ would they want to sort of spend all of these resources securing it? They wouldn't necessarily have the ability and the way to sort of bypass that if you're a malicious player,
Sani (31:26)
Absolutely.
Katrin (31:56)
There's a very vast attack surface there.
Sani (31:58)
It's extremely easy. especially when you have middleware between the user and the frontier model, there's an app that's a wrapper, basically. Leaks can happen and exposure can happen in more places than you would like to accept. I think, do know what I would like to ask all those big tech AI people? And Sam Altman talked about, he would not be able to raise a baby without chat GPT. Like he had that, it kind of...
Katrin (32:09)
Mm-hmm.
Sani (32:27)
went viral that quote. I want to ask those that have pre-teen and teen kids, do you allow them to use your LLM? Are they on LLMs? Or is that like, know, wealthy people don't have kids on social media and like responsible people don't have kids on social media. Are LLMs the same? If it's not safe enough for a basic user, it's not safe enough for a business. And I mean, another thing that me and Jason Packer and web analyst and other person, when we discovered those leaks in November that
Chat GPT prompts were leaking into random Google Search Console account because Chat GPT was using Google to web search and they're using their ⁓ operator agent, whatever it was, to control the browser, not via API. They literally open a browser, type in the search and give you the results. So we saw prompts like, I think my boyfriend is cheating on me. What should I do? And stuff like that.
How can anyone feel comfortable around this? now have a ChatGPD has a ChatGPD health campaign. They want to manage your health. I'm seeing YouTube ads all the time for this. We should trust them when they prove to us we should trust them and not before that. Like we cannot trust Big Tech after Cambridge Analytica, after Facebook, after social media. We should never trust Big Tech unless it spends a decade convincing us we should trust them. So please don't.
There's no safety, there's no security in using cloud-based LLMs. There are so many levels this could go wrong that it feels like, ⁓ I don't know, Stranger Things Season 5, topical, went sideways in many ways.
Katrin (34:02)
Yes,
yes it did. Well, let's bring it back to the analyst listening. If AI agents are becoming a real traffic source, how do you actually even measure that today? I mean, like technically, right? Like what should analysts be tracking?
Sani (34:15)
Mm-hmm.
I think it's currently impossible to know. There are no tools that will tell you this is an agent. There are IP ranges that you can look at. There are patterns in behavior. So make boring, sexy again for a third time. Server logs. Just look at the way they behave and how quickly they go from one page to another. If it goes from one page to another in five seconds and gets all the information and then another page, another page, that's unlikely to be a human.
So I don't think we have great tools just yet. I'm just like ⁓ LLM visibility. We are in the pre-Semrush and Ahrefs era. There's going to be some tools that tell us that. We don't have good analytics tools for agents just yet. I'm sure they will happen, especially when they do ads finally in those LLMs. They'll be forced to give us a way to track what's happening. So I think the worst thing for LLMs, ads, also the best things for LLMs.
Katrin (35:10)
Yes.
And so, you know, let's imagine we have some data. You've talked about moving from the attention, an attention economy to like a utility economy, right? How would you say analysts should adapt their KPIs when success shifts from capturing eyeballs to delivering utility to AI agents, basically?
Sani (35:29)
Yep.
Yeah, this is fun because attention, just fighting for attention. know, when 10, 15 years ago, our time on page is low, DEF CON 1. Who cares about time on page? Those are stupid. Those were always stupid metrics. You don't want people reading an intro to your blog post before they get to the call to action. Nobody cares about that. moving to utility economy, capturing that, I think it's tracking failure rate for those agents because if an agent is on your website,
has a goal, it already knows it once you complete the goal. Tracking failure rate is really more important than anything else. So how quickly does it do it? How many steps? How many wrong turns does it take along the way tracking those things? And hopefully we get good analytics tools for that. I feel like they should be obligated to provide the tools, the providers of these agents and models. Once we get that, we will be able to fully track. But I think seven extra seconds on page,
is less meaningful than 85 % completion rate on a task.
Katrin (36:46)
Yes, I can see that. And obviously, we don't really have the tracking tools yet, right? But let's put ourselves into a hypothetical, probably relatively close future, where we have some of those. And let's imagine that some of those tools are built the way you think they should be built, and they're tracking useful. And they're useful, right? So for analysts who've built their career on Google Analytics and traditional attribution, ⁓
Sani (36:52)
Mm-hmm. Mm-hmm. Yeah, absolutely.
Mm-hmm.
Right.
We feel
for you. I'll just say that analysts that have built their careers in Google Analytics, feel, especially J4.
Katrin (37:20)
We do, we do.
And traditional attribution, et cetera. ⁓ If you imagine.
Sani (37:25)
Yes.
Katrin (37:30)
those tools and the future of analytics with those tools and, you know, the traditional aspects. What skills from the old era do you think transfer into the old era? What needs to be learned, unlearned? Like, how do you see that necessity to transition from one to the other, knowing that the other doesn't really exist yet?
Sani (37:51)
That's a super important question. don't think I have the best possible answer. think most of it does translate because you're still looking at website usage metrics. I think people who have done server log analysis, some kind of technical ⁓ log analysis are in a great position because session, user, abandon rate, bounce rate, bounce rate is, there's no such thing as bounce rate for an agent.
it's either completing the goal or not when it's on your website. So I think we're looking at different metrics, but the same kind of people in the same tool set still will apply. And I'm not 100 % confident in that, but I think that's how it's going to be.
Katrin (38:36)
You see a return to an era of ⁓ looking at very granular data and rebuilding KPIs from very granular data, emulating what we think is important in the behavior and in the sort of like hitting the goals, while we're sort of like...
guessing that this might be important and these other things might be important and and and so we know because that's what we were doing in the beginning like in the beginning we thought that time onsite was important in the beginning of the web we thought that yes
Sani (39:15)
number of page views were like the hottest metric in the world. We had those trackers,
you know, on every website. That was the most important thing in the early days. And that was exactly. That's why. Absolutely. But internet and we evolved from that into something that was kind of meaningful, not perfect. It was never perfect, but it was more meaningful than
Katrin (39:22)
yes. Yes. Yes. Well there also was nothing else.
To be fair. ⁓
Mm-hmm.
Sani (39:43)
How do you call them? Stat counters? Pit counters? Something like that. I think ⁓ there are two things here with doing analytics for non-human users. Let's just call them that because LLMs, AI agents, whatever there is. There's the standard website that was built for humans and how they interact with that. there's also going to be, there are new protocols. There are going to be new ways that they get things done on a website. There's going to be an... ⁓
Katrin (39:56)
Mm-hmm.
Sani (40:10)
agent handshake with the website and then you know this is an agent, this is what it's trying to do and then you can do analytics based on that. We're not there yet. We have 35 years of legacy infrastructure. We need to make better before that can work.
Katrin (40:23)
Well, also until
now, I mean mostly, we wanted to filter out bot traffic, not analyze it.
Sani (40:28)
That's
another can of worms. yes, this is something that I honestly, I don't know what the solution for this was because there's so many bots. mean, bot traffic is now, is it 60 % or on the internet? It went above 50 % sometime last year. And a lot of people were blocking all kinds of bots except for Google and Bing and whatnot. Now we will have actual agents that are not bots that are scraping. You don't need CAPTCHA. If you put CAPTCHA on a website,
And it needs to check, I'm not a bot before it can access your website. What are you even doing there? But there has to, there are some protocols emerging. If the agent can shake hands with the website and introduce itself and say, I do this, this is my goal. I want to do this. You can, the website will be able to identify that and you'll be able to track and analyze. We're not even close. We're not even close to this being a reality, which, you know, in, in how fast everything is moving, it could be three months. It could be.
a year or three years, who knows.
Katrin (41:29)
I was just gonna
say we're not even close. That means it could be here in three months, right?
Sani (41:33)
Yes,
yes, yes. But it's nowhere in sight right now.
Katrin (41:38)
The good news though is that all of those GA4 analysts that had to ⁓ learn to use BigQuery and go back to event level data and have built some of that skill set, that's actually probably their best preparation for what's coming.
Sani (41:48)
Yes.
100%.
If we still had universal analytics and that's the only thing people know how to do now, we're screwed. It's not happening. But GA4 was there because it's event driven. This is significantly closer to what we will be doing.
Katrin (42:11)
And if you had to guess, because obviously that's all we can do is guess, right? ⁓ Would you think that GA4 is going to be able to evolve its current structure and capture structure to accommodate for analyzing this type of traffic within its current capabilities or minimal evolution of its capabilities?
Sani (42:36)
J5? Maybe?
I don't know. But if there's a tool that's going to be broadly accepted and used, what's the most likely one? It's GA, whatever version.
Katrin (42:50)
for
sure, but there's also a situation where you can say, well, first of all, Snowplow might be faster or something along those lines might adapt faster and then you need to do an integration or you can just possibly have ⁓ niche tools that only do that. And then you have to sort of overlay somehow.
Sani (42:57)
Mm-hmm.
In the early days, in the early days, that makes a ton of sense. And it's probably going
to be like that. But if this agentic web and agents sort of being our avatars online and doing things for us online fully happens, I think it will. I think there's no way to put that genie back in the bottle. When that happens, Google is going to want to have a piece of that pie. And which company has the most expertise to build something like that, this and to make it broadly accepted? Also,
Katrin (43:24)
Mm-hmm.
for sure.
Sani (43:42)
They still control most of web traffic online. A lot of it goes to Google still. Even if it's AI overviews and whatever else, it's still Google. ChatGPT, as hypey as it is, it doesn't matter compared to Google. It really doesn't matter. whatever, everything that Google has in its ecosystem, not just Google Gemini. So I think if there's a tool, absolutely will be broadly accepted tool. It absolutely will be something from Google.
Katrin (43:47)
Mm-hmm.
Sani (44:11)
And why would they have a new tool for this? They just put it in Google Analytics.
Katrin (44:14)
All
All you know, I feel like after having spoken about AI for about an hour to wrap up, we should just talk about AI fatigue. ⁓ You called it like digital tinnitus, I think, which I love that term. I'm stealing it, if I may. I will attribute it as much as I can, but like, like I really like that.
Sani (44:24)
⁓ Let's do that. Let's... Yes. Yes. Yes. Please, please do. Yes.
Katrin (44:38)
And as a result of two years of AI hype and pivot or die mandates and things like that. for others who feel that, who are burned out by the constant pressure to adopt AI and productivity, et cetera, ⁓ what's your advice to protecting mental clarity while staying current? Because you really have to balance the two, right? You kind of have to stay current.
But you kind of also have to not get caught into the hamster wheel.
Sani (45:09)
And it's really difficult to find a balance. It's really, really, really, especially if you're on LinkedIn or following something like it. It's just, it's nearly impossible. I struggled with this a lot. As I told you earlier, I struggled with, I tried to keep up with everything, the good, the bad. You need to stop caring about it because most of it legitimately doesn't matter and never will matter. If you want to bet on finding the 1 % of things that will matter.
obsess over it and say, I said it first, go ahead, by all means do that. But you will also waste 99 % of your time following trails that lead to nowhere. So I think maybe just catching up, a newsletter that gives you monthly updates or something like that, or once a month, on the first of every month, just ask AI overviews or Gemini or whatever tool what happened last month. Like, is there something worth paying attention to? Because when they release
whatever new frontier model. I mean, when TadGPT releases a new model, Gemini does it next week and then Anthropic does it the week after, which means they're just pretending that they're doing something, to me at least. I don't see any change. I think just ignoring most of it, ignoring all of it and occasionally paying attention and just focusing on doing your job as well as you can and being as good as you possibly can be at what you do is more important than...
using AI to squeeze five extra percent of productivity or whatever bullshit. ⁓
Katrin (46:43)
Well, as you
said, the fundamentals still matter, right? If you don't know the answer, you really shouldn't trust what the AI tells you, especially if it tells you different things with different versions of the question, right? And it's... Yeah.
Sani (46:48)
Exactly.
And it does with the same version, with the same question. It tells you different things
most of the time. And I know it's difficult. know people are afraid job security is the what is it? People forget about it completely. People feel stressed, anxious. I'm just saying what if most of it doesn't matter? What if most of the stories they did, they're trying to hype and give us are just to to make the bubble bigger and do whatever they're trying to do. just maybe it's not worth it.
Katrin (47:23)
And that is
the characteristic of a bubble, right? You have to keep it going. So that's going to continue for a while. ⁓ Well, you know, we've talked about a lot and ⁓ we've covered ⁓ a sort of vast array of subjects from technical foundations to the new data layer to security, vibe coding. ⁓ So for specifically for analysts in ⁓ e-commerce teams, which are
Sani (47:27)
Exactly.
Katrin (47:52)
probably really at the forefront of this, right? ⁓ What's the one thing that you'd like them to think about illustrating, you know, making boring sexy?
Sani (48:04)
⁓ Great question. Another great question. So specifically for analysts in e-commerce teams.
I mean, if you can find the three most important things you should be doing and just spend time doing only that for a week and ignoring everything else, what would that life be like? And this is something anyone can test. Just focus on the most important thing and nothing else, because there's always running in too many directions. Every single person in 2025, 26 is running in all the possible directions. What if you just chose one thing to do for a week and just making it as perfect as you can?
And this applies to analysts and e-commerce to anyone. But I think if you just have, what's the most important KPI for us? What is everything we can do to make this better in a week or for one week? Maybe try that.
Katrin (49:03)
That's great advice. So for people who want to learn more, listen to know hacks, work with you on website optimization or accessibility to AI agent, just follow your thinking, where should they go?
Sani (49:16)
So NoHacks Podcast, obviously, so nohackspod.com. There's also a sub stack. It's kind of a sibling to the podcast. It's nohacks.substack.com or my personal website, my first name, last name, I'm not going to spell it, .com. If you just search for me. Thank you, perfect, excellent. But yeah, the podcast is the ⁓ best way to follow my work, get in touch, because if you care about optimizing,
Katrin (49:35)
We'll put all the links in the show notes.
Sani (49:45)
for AI agents optimizing the web infrastructure and websites for AI agents. Guess what? I'm talking about nothing else this year. This is the only thing I care about.
Katrin (49:55)
Great! So we can learn all we need to learn about it there. We'll put all those links in the show notes, including our Fluffy's episode on GPT and commerce for anyone who wants a lighter take on why brands become invisible to AI. And so that's it for episode eight of Knowledge Distillation. Sanny, thank you very much.
Sani (50:16)
Thank you so much.
Katrin (50:18)
If today's conversation made you think about how AI is changing data analytics, visit us at ask-y.ai and try Prism, our platform helping analysts navigate complexity with context. Thanks for listening and remember bots don't think, AI analysts do.
