00:00 The Impact of ChatGPT on AI Development
05:24 Building AI Models: Pre-ChatGPT Experiences
10:55 Balancing Creativity and Accuracy in AI-Generated Content
16:51 Trust and Data Integrity in AI Analytics
22:18 Architectural Considerations for AI in Enterprises
27:31 Understanding Context in AI Models
30:57 The Role of Analysts in AI Integration
35:30 Navigating the Application and Model Layers
39:31 Use Cases for AI in Marketing and Analytics
42:50 Future of AI Architectures
47:21 Skill Evolution in Marketing and Engineering
Katrin (00:00)
Welcome to Knowledge Distillation, where we explore the rise of the AI analyst. I'm your host, Katrin Ribant, CEO and founder of Ask-Y. This is episode seven. And today we have something special, a conversation between two people who have each spent at least 15 years building AI and ML powered software.
Guys, I think between the two of you, you're probably total close to 40 years of experience in AI engineering without wanting to out you. But that's a lot of experience. That's kind of the dream of every recruiter today, know, X years of AI experience on technologies that mostly people have known for about two years. So here we go, 40 years of AI engineering in this room and me. I'm here too.
today we have Yaniv Makover. Yaniv, you are the CEO and co-founder of Anyword, where you've been working with LLM for about six years before ChatGPT launched, is quite a thing. We'll talk about that. In fact, me look like a genius here for a second. I invested in Anyword in 2021, which is, for the record, a full year and a half before ChatGPT. That's like...
dinosaur times before Chad GPT. And I kind of love that we literally talk about AI history as pre and post Chad GPT. It's like, it's the defining moment of where everything changed. And let's talk about that for a second before we go on with the intros. So I remember you and I talking maybe a month after Chad GPT was released.
And at that point, you had been building and training generative AI model for content generation for about six years, first with Kiwi and then with Anyword. And you were telling me about ⁓ how you were evolving Anyword, the application and the workflow layer of Anyword, because you felt ChatGPT changed everything. I remember you saying exactly that sentence. So tell us about the ChatGPT release moment in your life as an AI engineer and founder.
Yaniv Makover (02:04)
Yeah, was an annoying moment. My mom called me. So I actually went to like to school master's degree in NLP and like the text mining lab. And so I've been working on, you know, understanding language, language generation for a long time before, before even LLNs. It was like, don't get into it. But it was like, there was like a lot of algorithms and a lot of like hard work to do stuff that didn't work that well. But, and then I worked on
Katrin (02:08)
You
Yaniv Makover (02:33)
kind of like my first company in QE, we worked on smaller LLMs like GPT-2, BERT, try to make them work. And I was like really, really excited about the fact that one out of seven generations was like very cool. So two plus two equals, and then you'd get four and four plus four, get eight. You do eight plus eight, you get 25. It's like, and so, but I thought that was the coolest thing. Like a model that understands words, learns math is very, very cool. And I thought it was,
Katrin (03:02)
It is
cool.
Yaniv Makover (03:03)
And back then everybody was talking about blockchain and it couldn't stand blockchain. Like, what's the problem? Like, what's the, it's like, everybody was talking about blockchain and I was like, and I talked to other investors. Anyway, I'm working on this. got models to work. And then Shadgy P launched and I got a call from my mom. You heard about this really cool thing called an LLM. I was like, I'm working on this for like 10 years. It's like so annoying. And, ⁓
Katrin (03:08)
Neither could I.
Yaniv Makover (03:32)
But chat GPT changed everything from an adoption perspective, from a business perspective, how enterprises want to use AI, the understanding of just I think the before. So there was a lot of people working on GPT to GPT three and a half before chat GPT. So there was like people adopting it. within the community, were the breakthroughs already happened, but then it went to mainstream. And then you could go and sell to an enterprise.
something where they were before they were super worried. And so for me, you know, my the phone call for my mom was like the ⁓ my god, this is is amazing.
Katrin (04:11)
I mean your mom asked you about what you were doing for 10 years. have a 4C20 because for most people, we do these conferences that are called measure camp for data people, data analysts. And one of the things that everybody laughs about is when you ask a question in the room, or in a quiz, like what do your parents think you do? The biggest, the most common answer is work with computers.
That's kind of the level of understanding generally. So yeah, I mean, your mom calling you and asking you about other labs, that must have been an interesting moment. So, but before we go on with that, let me introduce our co-host, ⁓ Adhigarh Oren. Adhigarh, I have the great pleasure and honor to work together with you at AskWI, where you are the head of technology.
After completing your master's in machine learning, you led a team at Microsoft Innovation Labs, where you developed ML-based methods for a new search product. In fact, you hold, think, I don't know, I'm just going to say multiple Microsoft patterns in ML-powered ranking and recommendation systems. I had to write that down, not to botch that. And we can talk about those patterns if you know.
if we have time. But you are also a serial entrepreneur. So you founded startups in the marketing area with early ML-based systems for aggregating and analyzing real-time local data for social networks. That was a big thing in the mid-2000s for people in our audience who are not old enough to remember that time. It was really a big thing. I remember that very well.
Then you found a startup in HealthTech using Internet of Things biosensors to predict clinical risks in patients lying in hospital beds. So this is basically predicting when the patients are going to get up, which is a lot of signal to noise sorting.
And then you co-founded Topics in 2019, where you built a conversational AI platform to understand audiences' feelings and needs. And I'd like to talk about all of this, but really Topics is probably the most relevant to our conversation because at Topics you cross the chat GPT chasm while there. So did that change anything for you? And if yes, what did that, what was your chat GPT moment? Did your mom call you?
Avigad Oron (06:34)
I called my father and mostly I remember like spending an evening with Sharon, my wife, and I told her our life has changed forever, you have to try something new and she was very still very skeptical about it but she was she started playing with it.
and we started asking questions about art history and applying some concept from a certain research on another art and give examples. like she took my phone and then like spend the rest of the night playing with this unbelievable new capability.
and still skeptical to this day, but professionally, it really changed everything because where GPT, we had a bunch of tools, really hard to orchestrate and make them solve real chatbot conversationally, accurate every time and.
understanding what you want and able to get relevant data from that. And it's like such a massive, dramatic development for everyone in this space that we were lucky because we already had a lot of infrastructure for the application to support and be able to integrate it. But like immediately starting working off how everything is changing for us.
Katrin (08:11)
And so before we started, before we kind of kicked off the show, Yanniv, you were talking to us about what you were doing pre-Chat GPT at Anyword, you know, like you were fine tuning birds, et cetera. Can you talk just a little bit about like life before chat GPT as an AI engineer?
Yaniv Makover (08:31)
Yeah, so I was kind of lucky where, because I grew up in the NLP lab and I was like very, you know, part of this whole change and like LLMs really changed a lot in the field. ⁓ The first company that we founded called Kiwi, together with my co-founders, basically helped my publishers, big US media companies to ⁓ push around ⁓ ads into platforms like Meta and Google.
And they were creating like articles, writing articles, like normally, and then pushing out social posts or tweets or ads through the platform that my company built for them called Kiwi. And it occurred to me that some publishers did a much better job than others to write the right words to get you to read the article or to subscribe or to buy something. And they're pushing like massive amounts of ⁓
content or posts and ads because a typical publisher writes about 800 articles a day and then they have to like create variations for it. So it's labor intensive and it's very non-scientific. So people have like, yeah, if I use this word, it always works for me or if I use that word, it always worked for me. But some publishers did better than others. So for instance, one of our first customers and investors was the New York Times. New York Times, when they write an article, they'll write
they want to promote it with a tweet on Twitter back then, it was called Twitter. ⁓ They would A-B test that tweet. And the way they do that is before pushing it out, they would basically like write eight ads, push those ads, pay for the ads, see which ad worked the best and use the copy from the ad as the tweet that would push out to their audience. And all of that was pretty clear to me that this is something that AI can solve, right? You cannot now test
Okay, so you can write eight ads, but AI can write a thousand ads, which are similar, saying the same thing, but in different ways. And then, okay, we can solve that problem. Which ones are you going to actually use? ⁓ So we started working on very early-stage ⁓ LLMs like BERT and GPT-2. Now, if you would generate a short enough text, like a tweet, the hallucination...
would not be that bad. if you could generalize, back then, like models would just completely hallucinate about everything because they're very small. They didn't have enough context. But once you saw an article, the tweet was like, oh, this is an amazing article. Look what he did again, like something very generic. And you saw that the model was actually learning about what actually gets engagement. And so we were like super excited about that. And we like pushed it into our product. And we saw that the model was like, even with a very
basic model, fine-tuned on posts from publishers, it was outperforming people. And then we did like a test with copywriters, like a manual test. We gave three copywriters ⁓ lots and lots of like copy for social posts and asked them to kind of decide which ones would work better and which ones wouldn't work better. So we gave them each time, we gave them two copy sets, like a pair of copy sets for the same product or article. And
We asked three kind like copywriters. They kind of like know like 5 % of a random, like not more than random, like 55 % accuracy. And they do not agree with each other about which one will do better. And if you give them the same test three months later, they will choose a different set of copies. So the bias that we have humans have about what actually works is natural, but I...
Katrin (12:01)
Mm-hmm.
Yaniv Makover (12:24)
We were completely sold that AI can solve some of this. And that was what Jumpstart did. we were creating really, really short content, really short text, and that worked for us with small models.
Katrin (12:36)
So you're both experts in building AI, building with AI. mean, like building the actual models, ⁓ building with the actual models ⁓ and having had to do a lot of things with a great degree of pain that are basically solved today. So you find yourself using a lot of the same tech.
both of you, right, using the same underlying foundational models because we all use them today. Whereas, you know, when you were building your previous companies, ⁓ you guys had to sort of fine tune everything very differently and use very different and try very different things, right? But today, using the same tech, your success criteria are completely different. And even new world of AI generated copy,
It's creative, it's engaging, the criteria around accuracy are centered around adherence to guidelines, right? Brand guidelines. You really want to create copy that is going to be on brand, in tone, etc. You're balancing creativity and accuracy. At least that's what I understand. Copy needs to state the right price, use brand guidelines, all of these things, but you need to preserve creativity. How do...
you built to for that balance today.
Yaniv Makover (14:03)
Yeah, so I think there's a trade-off between performance and being on brand. So your tone of voice, let's say you want to sound like some bank or whatever, that's not engaging. Like you want to be always very strict. You always want to be kind of like accurate. You don't want to oversell. So I feel like, and we sell to marketers and I feel like that balance is a balance that has to be struck by ⁓
by the marketers themselves. So sometimes you want to be a bit more aggressive and you want the thing to actually convert. Your email, somebody has to open it or, but sometimes you just want to sound like you, sound like Nike wants to sound like Nike. They don't want to sound like Adidas. And so I think there is some kind of like leeway between those two things. I think this is where humans still matter, right? Because it's a case by case kind of like decision.
And I think when you're building in this world, you're building tools for people or you're building for the agents. You kind of like need to allow them to see how both of these things ⁓ are affected. Creativity is the word you used is much more complicated because there's subjectivity to creativity. Like people see something like this is super creative and something else like that sounds like AI. it's very, very hard when you're building software in the space that is perceived as creative.
you have to convince your user that what the AI generated is good. so like Netflix is trying to convince me to watch that TV show, Jeffrey Dahmer, and it gives me like a 99 % match. Now I don't want to a show about some serial murder, and it's not for me, but they've found out way back when you're doing recommendation engines, like,
Katrin (15:39)
yeah, that's interesting, that's true.
Yaniv Makover (15:59)
trying to predict what someone will choose and trying to ⁓ persuade them to do something, it's two different things, right? So Amazon, when you buy your products, you have to like, they'll show you a history, you're gonna like this because you like that. That's not how the algorithm works. There's lots of different things that are affecting why you might like this or why they're showing you this product, but they're trying to convince you. So I feel like creativity is subjective and that you wanna, when you're building tools with AI, you wanna make sure to give the person ⁓
Like all the dimensions, the dimensions that are helpful for them to decide. Is it on brand? Is it going to perform? Is it different than anything else out there? And then they'll just decide what's creative for them. They won't agree, by the way. Different markers see different content and they kind of don't agree if something is good or bad. Yeah.
Katrin (16:48)
Yeah, people don't agree. That's normal, ⁓
Avigad Oron (16:51)
so in our space in analytics, the starting point is very different. The analyst's job, the core profession is all about creating trust and the correctness of the model and the baseline data. The stakeholders, the boss expecting them to get it right and get it right every time business decisions cascade from those numbers.
So trust is really the currency where we have to start our applications. So there is still a place to ⁓ be creative or insightful, let's say, about how you analyze things, how you model your business decisions. There are still elements of creation as part of analytics, but it has to be founded on a solid foundation of really understanding.
the numbers. So for us, we have to build a system over LLMs to, I think in the same way, give those dimensions, those possibilities, ensure and validate and convince and build a trust with the analyst that these are correct. And then we can start suggesting ways for the analyst to explore the data.
derive different insights and do something with this foundation. So also very similar eventually, you have to build on something and give the person the option to build on something that he can trust.
Katrin (18:30)
Yeah, nobody really loves creative math, right? That's really not a thing. So we're not judged by our degree of creativity and is it good? Although I suppose we judge by how good the queries are to a certain degree, at least by some people.
Avigad Oron (18:44)
Yeah, and also a lot of
creativity in kind of how you model or think of the problem and the solution. But you get the math right, you get the data and the numbers and the dimensions. They have to be trustable.
Katrin (18:54)
Yeah.
And so Yanniv, let's get technical about what you build around LLM for a moment. so you are, I think we're allowed to say that ⁓ amongst your clients, you have Google and LinkedIn, right?
So if Google and LinkedIn are using Anyword to create ads for their marketing, there must be something to using Anyword above Chagypti or Claude, I suppose. And I'm really curious to...
What you actually build around LLMs when someone uses your platform, what's happening behind the scenes that beyond the workflow of the application, which we can talk about as well, but like really what it is that is the engine that makes it so different.
Yaniv Makover (19:49)
Have a good one.
so I think there's some architecture and business considerations when you're building with AI for the enterprise. I think the assumptions that we made at Anyword are every enterprise is going to have some sort of in-house AI, which is going to be horizontal. And it's going to be connected to all of the enterprise's data sources. And that AI will.
you know, new employee might ask it, who already did this project in the past? What is our tagline? When like, just basically a knowledge base connected to NLP. That's kind of our assumption that that is happening, regardless of the vendor that's going to provide it, it's going to be OpenAI or Google or Amazon or whatever, or Microsoft. And enterprises really, and I think there's a security concern that that's going to like charge that. if you, if,
If somebody asks an LLM, you know, give the, you know, uploads their like, ⁓ revenue report from last year and does some analysis on it. And then I like ask about this company X's revenue and LLM tells me the answer because somebody mistakenly uploaded the revenue. That's the nightmare of every enterprise that their data is going to get, you know, is going to join the LLM is going to be part of the LLM data and then it's going to be free for all. So I think enterprises are going to build.
guardrails and security. so the assumption is that it's going to happen. So when we built Anyword, and our mission is to help AI create better content, higher performing, personalized. And we get tested versus a foundation model with Anyword, a foundation model without Anyword, and we're 30 % better on conversion as a engine because we add a lot of A-B tested data and a lot of other things. But the way to build around the architecture is to assume
that they're not going to have 20 different applications with different AIs connecting to their data sources. ⁓ They're going to have one, especially those bigger companies. So how do you affect workflows? How do you inject data into workflow where the model has already been built? You can't fine tune their model. And so the way we decided to do it is we have two places where we interject. One, we use something called RAG, like Retrieve Augment Generate.
introduce data into their prompt. So when they're prompting to their model, a foundational model or their own model, ⁓ we will tell that model, look, if you use this word, that sentence, this image, this thing, this output will be better. So we do that. And the second thing we do is when they generate content with their models, we ask the model to generate lots of variations, not just one. And then we can rank them and say, well, this one's the best one. This is the best output.
effectively for them is we're making the model better. yes, it's based on the metric they care about. So like if they care about some, I don't know, engagement or some conversion or sales or Legion, they will basically set that up. But generally the way we build their architecture is assuming they have their own model. Now, some of our customers would just rather have us house a model that generates for them.
Katrin (22:50)
And how do you rank them?
Yaniv Makover (23:14)
And we do that too. So we have like a private ⁓ fine-tuned model. This is yours. And we'll do that. But I think, ⁓ you know, looking into the future, I think the first architecture is probably the one that will kind of win out, at least in bigger enterprises.
Katrin (23:33)
And so how does that contrast with what we do?
Avigad Oron (23:38)
I think there are many similarities around. We build the extra layers of understanding the data. We build an intelligent data catalog, which is a smart manager of all your data assets, the views, it understands how people reference different data and how to give a business meaning to the data. So that's a component.
over the LLMs. We also have a memory system and a RAD system, which allows you to learn from the process and similar to what you described, inject learnings and generalizations that can improve the LLM analysis. maybe eventually we have a full analytics toolkit.
like to run SQLs, to do Excel operations, to do visualizations, dashboard, so all of these tools. And eventually this all connects to running the LLM with all of the relevant context. And we have our own multi-agent engine that's orchestrating all of this context, taking the user needs, and eventually is running the LLM.
So I think what's maybe different from what Yaniv is doing is kind of assuming he will augment some LLM providers infrastructure with the RUG engine, with the ranking capabilities. We can also use the same LLM infrastructure that the enterprise may provide. But we will orchestrate all of the...
the data needed for those prompts to run the right agents to eventually create the artifacts you need in order to run the analytics.
Katrin (25:48)
What's interesting is this is really something that I hear from most people I talk to who build with AI today. For the most part, building with AI today is about building an application layer and scaffolding around the AI models that has a lot to do with obviously managing security, but really managing context for the effectiveness of the outputs. It's really, really about managing memory and managing context.
project managing the workflow for the user, obviously. And Yanniv, you often told me in the beginning when we were discussing anyone's direction, like we preach at GPT, that creating copy is easy. The hard part is to know what is effective in any given context.
So how do you actually, this ranking, how do you actually manage this context? How do you know what is effective? I know you have some substrate of data that you built about what works and what doesn't, but then for any given sort of prompt and result, how does that work, how this evaluation?
Yaniv Makover (27:04)
Yeah, so I want to kind of get into how I see the context issue for AI. Like, I'll try to give an example and generalize it. So let's say you have you're working in this company and you're prompting your AI, your company, whatever that is. ⁓ I want to generate a visualization of next year's revenue projection. And that's your problem. So now there's two problems, right? One.
Katrin (27:09)
Yeah.
Yaniv Makover (27:31)
there's like 50 files of next year's revenue in the data set. Like, so again, in the database, like somebody in marketing, somebody in finance did something, somebody in sales, which one are you referring to? So this is like the internal company context. Like this is, now you can, maybe if you write a better prompt and you had more information, maybe you could get over it, but that's like the internal context. And the second one is what is visualization for revenue? So the model needs to know a lot of stuff about the world generally, because you're using plain English.
And that context is like, so for the second part is like, what is visualization for revenue? How does that look like? And what is useful? You need a large language model that knows stuff about the world and can basically understand whatever you want to do. And you also, you need that context and you need the context about your company. Now, NLP before LLMs was trying to solve the like small data set, pinpointed problem.
Recommend a movie for me out of all the movies in that place. That's kind of like it was pretty good at that You didn't need all the context for the world So what did LM add? This thing where you just wrote something like a visualization for revenue and already knew how that looks like Without you telling it exactly what to do and I think when you're When you're you're solving for for for with an application ⁓ You you kind of there's a trade-off between both
You can train for short copy a very small model, like I said, what we did early on, that kind of repeats itself and has all the context from the customer itself. So if you look at all the customers' tweets, you will be able to generate similar tweets. But what happens when you get a new piece of content or you get a new problem which necessitates knowledge about the world that the customer hasn't seen? And in creative copy, that happens a lot.
Katrin (29:26)
Mm-hmm.
Yaniv Makover (29:26)
and
content that happens a lot, then you need a larger model. So I think there's always a trade-off between that. And then you have latency. You don't want to wait that long. And so our model, when somebody uses AI and they generate three variations of an email subject line, we'll give them scores from 1 to 100. So this is a 72, this is a 96, and this is a 65. So I always get asked by the customer, so why don't you just generate hundreds? Why am I getting these?
sub-performing ⁓ subject lines. Why don't you just generate the best subject lines ever? Because the search space is so large that you would wait for, we would have to generate lots of variations, check them. And so there's a trade-off between that and also there's a trade-off between your brand voice and stuff like that. So anyway, think context is tricky. For us, it's a split between the enterprise context and a larger understanding of the world.
I'm not sure that's the same for every problem out there.
Katrin (30:27)
Well, think the reason why I'm so interested in this question is because ⁓ we focus on the rise of the AI analyst and sort of like trying to understand what skills do data analysts need to invest in in order to ⁓ sort of have sustainable skill sets through this evolution of disruption basically of the analyst workflows today.
My thesis is really, there's three main things you need to focus on, which are one, you need to really understand how LLMs work. Because if you don't understand your tool, you're not going to be able to master it and to get the best out of it. Two, you need to become really good at prompt and context engineering because you do need to understand what the LLMs have, what they don't have, and how you put your context.
into your prompts and your communication with the LLMs in order to get the best out of it. And three, in our case, you have to become very good at reading code because you might not write as much code, but you're going to read a lot of code if you do analytics with LLMs. And so this management of context, I think it's something that analysts have to sort of...
learn and understand themselves. I think it's really interesting to see how the different tools in the different domains are confronted with context and trying to manage context to make the user's life easier and to get the results to be more accurate. again, we obviously do a lot of context management as well.
Avigad Oron (32:05)
Sure, and same as Yaniv said, it's ⁓ some balance between different properties of how you handle this. So you want to get it accurate, like in Yaniv's example, if you say revenue for the next annual period, the definition of what date exactly does it include, not include.
Do you have permissions to see it? What do you get into the LLM? It's critical to get the right answer. so analysts will have to become better at managing those. The LLMs and the systems will help them find the right data or context for the font, but they will still need to be able to understand how the context the LLM is given.
is affecting the results, the results can get wrong or maybe you need first for the LLM to see more example to search the catalog to get more ideas of what are possible visualizations before you can even make a decision of what's your direction. So this balancing of where are you in the stage, how quickly you wanna go.
how much the LLM already knows, how much you can trust and control what it is the LLM is getting into the prompt and understand the output based on it, are ⁓ critical to get the correct and useful process. And that's our focus. We have the user get the precise context to know that a certain table
or certain data source is the ground truth. If there is any issues with the data, we will store and learn about it and feed this into the LLM. We will help the user select it or control what context goes, which will affect obviously speed and how much the LLM can you more general responses or more accurate responses.
We can use this for validation. And we will also apply the business logic into this context. So when the user is saying the visual analysis, we can maybe understand based on his past experience or the expectations of the stakeholders that we've learned from some document in the RUG how to be more effective into generating the correct visualization for the customer.
Katrin (34:52)
And so all of the applications that I can think of have this sort of,
layer on top of LLMs. In fact, LLMs themselves are building applicative layers on top of themselves. Actually, we could maybe talk about that for a second. Can you explain ⁓ for the non-technical users, today if you talk to Cloud or if you talk to TadGPT and let's say you do an analysis or you ask it to do something a little complicated, what is the model and what is the applicative layer?
Avigad Oron (35:32)
when users go into GPT and write a query, this goes into a multi-agent, distributed, monitored, connected system. It's not a model. It's not like you write something into a machine learning model, get an answer.
It's multiple models running, repossessing the data, applying different filters and caching and mechanisms. Data go distributed across different models in different situations. They will have monitoring systems that will monitor abuse because they have to make sure that you are not trying to extract data or do something that's
is problematic. And since they don't really control the core and how it will respond to each problem, they have to store your data somewhere and to be able to later look at a few of your points and see if you're trying to abuse the system in one way or another. Those tools are connected to different other tools, like they can run code, for example, so they can write the Python code and run it.
in some server and for this they have to put it in another instance of a server that will run it and store it somewhere. So your data really goes into this super complex distributed system with multiple models and engines making it also a concern of first no one I think really knows what will happen with this data.
depending on the problem, depending on the user configuration can go into different systems, be stored in different systems, run different tools. They cannot control it so they build other system to try to control it. And from security perspective, this is also a huge surface attack for any of those system may have some vulnerabilities.
and the LLMs themselves can maybe be tuned to generate some prompt that may take advantage of these vulnerabilities and leak data or write some malicious code doing something that you may have not expected. So I think that kind of for the user putting his query into
Cloud seems innocent. Just be aware that this is a very complex system that may do different things. And if you're planning to actually do something with the result, like run code based on it, or make a decision based on it, or if you're putting your sensitive data, just be very careful that these things have to be controlled, at least for the next few years.
My assumptions, we cannot really trust and know what's happening in those systems.
Katrin (38:55)
And you need that something that you have to face at any word as well, I suppose.
Yaniv Makover (39:01)
Yeah, I think when you're asking a question about how do you split between application and model layer and like the orchestration or the agents, I feel like I think sometimes we skip over the product question about AI. So what are we trying to do with the product? So if we're assuming there's a person in the loop, let's say a marketer or an analyst in these two cases. I think there's some similarities, but I'm not as familiar with analysts as I am from marketers. So marketers are basically two use cases. One, I want the AI to give me a bunch of ideas. So I'm going to prompt it like,
Katrin (39:06)
Mm.
Yaniv Makover (39:31)
give me a bunch of ideas for a social post or whatever and then I get like 20 ideas and I don't want to spend a lot of energy or time and context about what those ideas are so that's one use case and the other use case is I already have something that I've written down so and I want you to try like improve or to do better than that so those are like basically two use cases the way I see this for analysts is here's this e-commerce data set spit out a bunch of like
analysis ideas, people are returning the product, maybe revenue is growing. There's lots of possibilities. And now, by the way, that puts the human in the middle, right? They have to decide which one they care about, right? So like, what's the narrative they care about? Or I already know what I'm looking for. I have a query, make one query much better, so I actually get what I want. Like, I kind like, I know what I'm looking for. So it's kind of like, they are low. I know what I'm looking for is like the marketer that knows what content they kind of like want to write, or they have an idea of what copy they want to generate, just want to improve it. And I found like these like,
I'm not familiar with other use cases outside of marketing, I see this pattern over and over again. I don't want to spend a lot of energy, spit out bunch of initial results, or I do want to spend some more time. I have an idea what I want. Help me get what I want from the AI. And when we look at the separation between our model and application layer, we'll look at it from these two use cases perspective, from these two paradigms.
content going in or just a very small problem going in. So for a small problem going in, we're actually looking at rag. And obviously there's guard rails and there's like a security layer and stuff like that. ⁓ When we're improving content, generally we're just at the end of it. Like the content is going into some model and we're just looking at the outputs and like filtering them a bit more. So I don't know, just how I see the problem.
Katrin (41:24)
So obviously foundational models are creating applications on top of their models to help users get value out of them and to compensate for some of the weaknesses that the models have, including security weaknesses. We as the tool makers, are building scaffolding around ⁓ those models. The analysts and the marketers are wondering what they have to build in their skill sets to be able to be ⁓
employable, valuable, basically like and performant throughout this revolution. But there are alternative architectures being developed, right? Currently we are in this Transformers, ⁓ know, Transformers is the king architecture and everything's a transformer. And this is what we see everywhere. There are alternatives being developed. If, you know, if I were like an analyst in my 20s together to today and I was thinking I have 40, you know, 20,
30, 40 years of career in front of me, thinking what do I, well, I mean, not even thinking about that, literally just thinking what will get me through the next five years if there's something new coming? What are some of these new alternative architectures that you see coming? And I know, I know like this is literally anybody's guess, right? What is going to disrupt the transformers? But what do you see, Enniv?
Yaniv Makover (42:50)
I refer this to a figure. I feel like for me, it's not what we think about it anyway. We're going to stay with this architecture for a long time. Agentic versus static stuff, but generally, kind of the same.
Katrin (43:07)
Mm-hmm
So this is we're not gonna put this on the record but like speculatively you don't actually look at the space of what is being researched and and thinking like literally just speculatively thinking What could possibly be an interesting thing that might win over the transformer in the next, know, two three five years?
Yaniv Makover (43:29)
I think there are better and better feedback loops. I think it's just like small iterations all the time. I don't think there's something out there that's completely disruptive. I don't know, maybe I forgot you know, but it's like,
Avigad Oron (43:40)
Yeah.
Yaniv Makover (43:41)
We always get this model training, this other model, and this other model training. they have this, there's no data, there's a lot of ⁓ artificial data sets and stuff like that. But generally it's the same architecture.
Avigad Oron (43:58)
Yes, I think looking at the core and the existing limitations, so there are a few and we see both the big labs trying to improve it, but based on the same core model.
So they will try to handle hallucinations, for example, ⁓ make the models generate more correct data. So they will try to find new ways to improve the way they train, use reinforcement learning to make the models learn what's correct and to try to internally generate more and more correct outputs. But at the baseline, the baseline architecture has two main...
limitations, it has a limitation of size. Currently, it's limited by the context size and not only that it's practically limiting, but the more context that you have, the results and the ability to make something useful out of it is also decreasing. So there are architectures trying to... ⁓
build a better understanding of the world, a better representation of what are concepts and how things are related, and communicate to be able to represent things like physics or space or time. Those things are not inherently built into the LLM models. So they work on those directions, try to remove the imitation of context size by...
using different space models or hybrid approaches that allows you to understand much bigger data. Like you can look at the video and understand space dimensions or other dimensions that currently LLMs don't have any way to, or transformers are really limited with. Some titles together more deterministic. So currently there is a... ⁓
hint of salt and randomness in every query. You really put in something, it's a very complex system, you don't know what you're getting out. And there are ways both the labs are working on and different architecture that will make the result more possible. If we can know that the LLM is doing something that's more likely good and the correct...
and positive outcome, we may have a chance to control it. Currently, we are not really controlling it. So for the coming future, I think it will evolve, as Yannick said.
see the current models and labs releasing more and more capable and they will internally have some better control, better handling of bigger size, less hallucinations built in on top of the existing architecture. But some breakthrough will come from new models that I believe have a better core training around what's correct and how to model more complex things.
beyond just language.
Katrin (47:18)
⁓
So we'll see what happens. Obviously, we can't know what the breakthrough is going to be. And we'll see how we'll have to adapt to that. But let's talk about how we have to adapt today. Yanniv, I'm sure that you can see across your clients, you can see organizational changes. You can see... ⁓
Avigad Oron (47:26)
Yes.
Katrin (47:42)
how content marketers specifically have to evolve their skill set as a result of LLMs. What do you see from a point of view of skill set of the individual and disruption of the organizations in your clients you work with or have contact with?
Yaniv Makover (48:01)
So I think there's a lot of top-down pressure being applied by boards and management. And I think that is applied on people that don't want to change whatever they're doing, especially bigger enterprises. I think it's an interesting time. there is a camp.
Katrin (48:12)
I can see that too, yes.
Yaniv Makover (48:33)
of actual entrepreneurs and VCs that believe that AI is disrupting SaaS in a way where every organization is going to build their own workflow tools for themselves that are completely personalized. So you're not going to buy Salesforce, you're going to build your own Salesforce. That's tailored to you. And that's like the extreme camp where the extreme camp says,
This AI is going to completely eat up all software. If you talk day to day with marketers, you feel like this is hard. People don't want to do that. And I feel like the hardest thing to do is even codify a workflow. What is the process of my job when I come in the morning to the office and I need to do three things? It's not that easy to define.
Katrin (49:25)
It's not.
Yaniv Makover (49:26)
An agent needs something that's defined, right? So some things are easy. OK, so like, I don't know, maybe a travel agent can do, I don't know, like compare some flights and give you the best flight based on the character.
Katrin (49:39)
That's the example
that everybody uses always, write about agents, book the flights.
Yaniv Makover (49:44)
Yeah, so I'll give you about marketing. I
feel like when we talk to ⁓ our customers, want agents. An agent can assist the person managing social. They can replace the person managing social. But there can also be a lifecycle agent that runs social content. The dream is this person replacing the all-in-one marketer, which is much more complicated, because there's so much context there. And all of it is not defined.
people don't really write down exactly what they do for their job description. They have some sort of metrics, some sort of goals. So I feel like even deciding what you can replace a work process or workflow with an agent is very, complicated. Customer support maybe is easier to replicate with an agent, but a lot of jobs aren't that easy to quantify and define. I feel like that's a barrier. I think we're going to see more and more human-supported agents. So I feel like people should not feel like...
I'm going lose my job. For instance, in marketing, an agent will run your website. So change the copy, change the images, A-B test some stuff, get the website to do better and better. Totally something that agent will do better than a human, much faster. But still, you need a human to approve. ⁓ Do I feel comfortable with what is going to be on my website? And so I feel like that's the way. And this in marketing, I can find a
a few use cases. Usually they're very, very channel specific, like inside social. Every time I post, I want to do a bunch of things and I want to find out who commented and I want to go and kind of react to them. So I feel like the more I can define the workflow, definitely solvable today with AI. I think from a vendor perspective, I think it's really cool. Very interesting. You could now sell an agent that does something out of like your domain and our customers will buy it. Where five years ago,
They had another vendor. You cannot step on Salesforce's toes. Somebody's using Salesforce. They only know how to use Salesforce. Today I can say, well, I have an agent that helps sales, and they'll try it. And I feel like that's complete disruption to every SaaS incumbent out there, somebody coming with AI and saying, well, and most enterprises will try it. So I feel like that's a huge opportunity.
Katrin (52:02)
yeah, that's interesting. And so do you see, ⁓ I see a lot of pressure coming from ⁓ top management for efficiency, this dream of efficiency of replacement, as you said, entire jobs could be replaced, which in practice, from what I see ⁓ of people trying and practicing ⁓ this,
It's not a reality, definitely not yet and definitely not in a near future because jobs are way more complex than, as you said, than your specific tasks. But do you already see, because I have seen some disruption in some companies where basically they sort of like eliminate some positions thinking that they will replace them with AI or, know,
AI sort of like augmented positions and where they kind of pedal back because they realize no actually the jobs are way more complex than that.
Yaniv Makover (53:08)
I think simple roles, simple jobs, like junior programmer, junior copywriter, those are definitely being eroded. I think they're being replaced by I as we speak for lots of companies. Blog tasks that you don't need context and there used to be some person doing them, which was a very junior person. I think you can definitely like...
coding for marketing, like, you know, these like marketing apps, which are like very simple to do and very simple to build like a small website and when you website or a lead magnet or whatever. Those are definitely, you see that in the job market. feel like, but they were always like entry level roles, SDR, that could be automated or some of that could be automated at least. So I do think it's happening. I don't think like people can say it's not happening. It's too soon. It's happening and it's happening at scale.
Katrin (53:43)
Mm-hmm.
Yaniv Makover (53:59)
I just think that... ⁓
the non-entry roles are like, those are gonna stay for a while. ⁓
Katrin (54:10)
And so do you, obviously you have an engineering org, do you see yourself hiring differently today for engineering?
Yaniv Makover (54:20)
So, and it started for me, prospectively, we tend to hire senior people anyway. And so, kind of like, this will help them just augment whatever they're doing. But I think bigger engineering orgs that used to hire lots of juniors, I think they have to think about why. So maybe you need to hire juniors to grow them to be your seniors, because that's how you recruit. But I think they're going to be looking at the efficiency of that.
should they?
Katrin (54:52)
And so, Avigad, you've been writing about this subject quite a bit. How do you see elevating skills with AI, and specifically the difference between engineering and analytics?
Avigad Oron (55:08)
Yes, sure. So starting where Yaniv answered, I think it's pushing everyone's skill to a requirement to skill to a higher, more senior, more capable, integrative level. So what LLMs are doing is they are replacing in practice those more simple routine tasks.
some jobs completed, most of the job is about those routine tasks, but even for a more senior developer, most of the routine tasks of getting some code or some management script or design document done, a lot of the hard work of doing this, collecting the data, filtering data, building the code is done by the LLM. And that's pushing.
your need and your skills to be much better both in first becoming a better product manager like now that you can you should be able to better accurately write those specs and points to better understand of what the stakeholder needs so it's pushing us more towards becoming better at this because the speed of you be able to generate something is increasing so you need to communicate more you need to better
be able to quickly understand what's the right way for the application to work and be able to very accurately specify it. The algorithms can have you build bigger, more complex systems. So you also need to be a better architect for that to understand how systems work together, to have a better sense and taste of what's the right way to build an application so it scales. If you don't have those skills,
um this seniority or experience you'll start building something and pretty quickly you'll see that the application doesn't scale there is some bigger challenge that the LLM um helps you more quickly create but you end up with something that's not useful over time so it's kind of pushing those more senior
system understanding integrative skills higher, but at the same time, and that's what I learned as the progress kind of with the profession, you also need better technical skills. Because now if the LLM helps you write, let's say code now cursor can write a few code branches in parallel.
So you need better Git skills to be able to work with them. These are very technical skills that you may have not needed before because you didn't use such complex functions of Git before and now LLM is pushing you to have those skills. So you need to upgrade also your basic skills to be able to work with the output of the LLM and take use of the most of the power of it. It can write, let's say suddenly as an analyst,
Python code and maybe you don't know Python code, but it's the most effective way to do some analysis. So you will try to run it.
Katrin (58:29)
That's really interesting. Yanniv, do you see that with marketers? Because in the very beginning, when we're talking about this, was saying this, know, marketers have an intuition about what works and what doesn't. And that intuition is like maybe 5 % above, you know, above...
above 50 percent, right? Do you see that with marketers having access to content generation and having to judge the quality of the content, ⁓ obviously they take your word for it to a certain degree, but they also have their intuition. Do you see these kind of contextual skills being something that they have to develop?
Yaniv Makover (59:06)
I think it's a new world. don't know how you get from not doing marketing to being an expert marketer. like, also engineering, how do you skip that stuff? I think, I think we'll have to figure that out. I think I don't know how to do that. I, and I don't know, like, and yes, definitely you need the marketer that has all the context, like how I need to be on brand here. I don't need to be on brand so much here. And this is like,
Katrin (59:17)
Yeah, that's for sure.
Mm-hmm.
Yaniv Makover (59:30)
And I give this example all the time when people are like spending money on ads and they have to like they're running ads on Facebook or Meta or Google. And for like the last 20 years, there's been lots of companies trying to like automatically ⁓ decide what budgets goes to which of these channels and like automatically like switch the budgets. And it's like, it hasn't been done. Like nobody's actually doing that. There's no AI that does that. Like really people at scale, people don't do that. And the reason.
Katrin (59:56)
It's funny you
say that I still have, ⁓ so in 2006, it was the big year where Google was ⁓ going everywhere. I was in an agency at the time, going, the ultimate dream is to get all of your budget and allocate it automatically. And they had created this swag. They had like this, I still have one Google radio, it's this big.
So they have these like, and it works. They had these like mini sort of like, you know, objects that they were distributing to occupy people's minds. And yeah, it has not happened. In fact, I was at NeurIPS, which is the AIML conference ⁓ research oriented in San Diego. I was there two days ago and there was a lady that was explaining something similar in the world of radiology.
So she was telling us, one of the things that was paradigm five years ago was this idea that in five years, AI will be able to read your MRIs, et cetera. We will not need radiologists anymore. So it turns out today there is a shortage of radiologists, because apparently people took that to heart and didn't go into the profession. And so now it's actually very hard to find them, because
⁓ We still need people to actually look at the MRIs, right? The AI cannot do it 100%. So yeah, I agree with you. This dream of all the budget allocation, it hasn't happened. But the intuition and the knowledge that the marketers need to build, that is really something that's never gonna go away. If we think about it... ⁓
The technical skills that you have to build around managing the current unreliability of AI may maybe go away with the new paradigm, I think. Some of them, not all of them. And we can't really know which ones exactly. But really those human skills, this knowledge of your craft and this knowledge of the world and this knowledge of the process, that's something that's not going to go away. And I think that's something that is true. ⁓
across most jobs that are complex. And what's interesting is that I also think that the technical aspects of the job that like Avigad mentioned in engineering, like we can see in analytics, you need to become actually better and better at the basics of analytics. But I'm sure you see that in marketing as well.
Yaniv Makover (1:02:33)
Yes. marketing has a problem. And I think B2B startups, CEOs, you can talk to them and they all share the same thing. Marketing is consistent of like seven different, maybe more ⁓ use cases and people that are very professional on seven different aspects of marketing. So you have your SEO person, you have your ads person, you have your social post person, you have your content person. And
And there's not a of teams in the enterprise that are so heterogenic, like engineers is engineering. Maybe you have backend, frontend, but not like marketing. This makes the team annoying to manage because then you have like a manager and that manager came from one of the skill sets and they have to like, and so I think what AI will do and so AI will basically allow these full stack marketers. I don't need to know exactly how to copyright because that thing does this for me.
I don't know exactly how to manage the website because this thing that's for me. I feel like it's going to help because today, so I'll give you another problem for marketers. Most marketers are like junior. If you do like social posts more than like three years, like there's no senior social posts, social media manager. You either move to a management position or there's something like something you're stuck. There's something wrong with you. So basically a marketing team is made up of junior people managed by one.
or like two senior people. That's not how sales and engineering works. If you have your VP of engineering, they'll have strong senior engineers in the fields that they're not as strong as and they'll help them. And those people have been doing that role for like 15 years. And sales, you have senior salespeople doing sales for 10 years. Marketing doesn't have that. There's no person there that's been doing the same thing for like 10, 15 years. There are some of them, but it's very rare. And I feel like AI,
is going to solve some of that for us. We're going to have these full stack marketers. They're going to start, like I know some SEO, know some social, I know some paid. And I really, I need just to understand the brand and the user kind of like life cycle and the stuff that marketers need to really know. Because when you, when I interview people for like senior marketing roles and they came from SEO and you ask them about like, okay, what's the price point or how do we like do our ROI on ads? They don't know. And they've
don't know, like they don't, they kind of guess and now they've been doing this role and they have to step up to this next role. And I think AI is going to solve that. So if people start as juniors, they're going to start as full stack marketers. I think it's going to be great for them. I think they're going to focus on what they need to be. How do they become senior marketers? What they need to know there. They don't need to know copywriting or SEO or all this stuff that AI is going to solve for them. I think that's kind of.
Katrin (1:05:22)
⁓
Yeah, I mean, I can see it's true that in marketing you have all of these different specialties and you cannot be a specialist in everything. just like, that was possible 20 years ago when you started in digital marketing and all of these specialties kind of came one at a time over the course of time and so you could learn them as they started existing. ⁓
If that's not what you've been doing, then yeah, it's really impossible. But today with AI, I do believe that there is a full stack ⁓ extension of skill sets that is possible with AI that wasn't possible before. You can learn much more about technical aspects. You can learn much more about business aspects because you ultimately have this of helper.
that can give you ad hoc help punctually as you need it in your job. And that does help you extend your skill set. That might get us people who are again in marketing have more breadth across all of the specialties, because I couldn't agree with you more. People are pigeonholed in one sort of silo of marketing. And in fact, organizations are built around those silos and perpetuate that.
Avigad Oron (1:06:42)
I
in engineering and in analysts, even for the junior job, you'll start with a more broad senior, in a sense, job because now we have a junior developer on the team and she's getting tasks that I can give her tasks that are full-stuck.
because she can easily write client-side code even if she doesn't have experience because the LLM and APS is so much faster, she will get a broader skill of full stack. And because LLM allows us to do more complex tasks, she will have to handle more complex system and debugging of those. She'll struggle first, but if the learning abilities and the team is doing a good job.
People will grow with a broader set of skills as they develop. Otherwise, they will just not have a job. You cannot start with a very narrow technical routine job because it will just not exist.
Katrin (1:07:49)
Yeah, I think that's really the key, right? You really do need to focus on that broadening of skill sets. So I would love to do this for another three hours, but we're really coming up on time. So I'm just going to wrap us up here. ⁓ So we've talked about ⁓ differences between content AI analytics, scale evolving, ⁓ you know.
applications, future architecture, security issues, et cetera. Yanniv, if you are sort of thinking about one takeaway that you would like marketers to take from this conversation about their skill sets and how to think about their skill sets and the evolution of their skill sets within a world where content creation is ⁓ more abundant and more efficient, what would that be?
Yaniv Makover (1:08:42)
Yeah, I think they're getting to get comfortable with trying a new AI and agents all the time ⁓ and just be aware of what's going on. don't think it's, ⁓ especially if they're in the industry already, I don't think it's a threat to them. ⁓ It's going to be a threat if they don't know how to use AI because if they can't scale up to being strategic, then that's going to be a challenge for them. I think take a look at all the tools out there, use all of them.
try to incorporate that in the workflows even if it takes more time or a bit more energy. Learning always takes more time. So that's what I would tell them. Just don't shy away. Head first.
Katrin (1:09:22)
Yeah, I couldn't
agree more, embrace it. So Yanniv, for people who want to learn more about Anyword, get in contact with you. I don't know if you're hiring currently, you know, people who want to try Anyword platform. Where should they go? How to talk to you?
Yaniv Makover (1:09:40)
Yeah, we're hiring data scientists. We're hiring go-to-market people. I'm Yanniv, Y-A-N-I-V, at Anyword.com. They can go to Anyword.com, take a look at our product. There's also like a free trial, and there's like an SMB product, so they can use that as well. And if you have general questions around AI and marketing, I'd love to just shoot me an email and get on a
Katrin (1:10:03)
Great, so we'll put all those links in the show notes, no worries. And Yannick, thank you both. This is really sort of the depth of the conversation that I've been wanting to have for a while. It's been amazing. So that's it for episode seven of Knowledge Distillation. If today's conversation made you want to experiment with AI for analytics, visit us at AskWY and Try Prism. Thanks for listening, and remember, won't win.
AI analysts will, but only if they understand the technology they use. Thank you for listening.
