AI Search Attribution, ChatGPT Clicks, Prompt Tracking, Google Search Agents | #MonthlyLandwehr

00:00:00: self-reported attribution is more or less the only tool, probably the best tool marketers have in their toolbox to understand how people came into that website.

00:00:09: So what you do at end of your sign up flow?

00:00:11: You simply ask people hey where did he hear about us?

00:00:15: then?

00:00:15: AI search should be one option and if want it next level.

00:00:19: but as a second question when somebody says yes it was AI search can asked them what prompt they use.

00:00:26: What do you think, what's your take on search agents and their potential relevancy for the future of Search?

00:00:31: I think if Google does them well they can become super relevant because Google already has all my contacts.

00:00:39: They have email, calendar.

00:00:41: that is very powerful!

00:00:43: That could be first thing.

00:00:44: makes me use something else in Claude to be honest a long time

00:00:48: Before we dive-in You're listening to The Masters Of Search podcast with me Your host Niklas Buschner.

00:00:55: Each week, I sit down with some of these modest people around the world in SEO and AI search to bring you their strategies mental models.

00:01:03: And top pieces of actionable advice.

00:01:05: if You enjoy this podcast don't forget to like and subscribe.

00:01:09: then follow it In your favorite podcasting app or YouTube?

00:01:12: It helps us get top-notch guests and create The best possible content for you.

00:01:16: let's dive into today's episode.

00:01:18: It is time again for my favorite format in this podcast, the Master's of Search Podcast.

00:01:24: And it is time for Monthly Landwehr.

00:01:27: Malte has already smiling because I tried to study this intro... ...because someone told me you should start with a same intro so that you have recognition effect.

00:01:38: But no honestly!

00:01:39: I'm always looking forward to this alot Because its MY private hour and as i also posted on LinkedIn I keep Malte's calendar link a well-known secret so that i can just book in and ask all the stuff, which is very interesting.

00:01:59: And get his knowledge on it!

00:02:01: Yeah, Malte thanks for joining again today.

00:02:04: how are you?

00:02:04: How has been your week so far ?

00:02:06: Thanks for having me !

00:02:07: It was great week actually.

00:02:10: yeah busy as always

00:02:13: Busy.

00:02:13: the weather is good, but not too busy.

00:02:17: So I'm I'm good

00:02:19: and The weather's not to good so you're not constantly thinking about going outside But your comfortable staying at your desk And Making if

00:02:30: i have an air-conditioned?

00:02:31: So i will Not leave the office If i can avoid it.

00:02:34: yeah i feel like this Is the ultimate hack To get people back to the office.

00:02:39: if You Have a Good Air Condition in Summer It.

00:02:42: it is much more uncomfortable if you work remotely, but people will come to the office more often.

00:02:51: We have a pet with couple of topics and I want start on an easy warm-up for your topic of attribution!

00:03:01: But i think also seen you starting the video series from PKI where you try to explain some basic concepts and ways of thinking about GEO and AI search.

00:03:19: And I think an important topic is attribution, and that i'm also constantly talking to our clients or potential clients about this topic... ...and I want to start like very easily.

00:03:33: why do YOU think attribution is something people should THINK about when it comes to AI Search?

00:03:41: In the past, if we did attribution on the web it was usually with clicks.

00:03:45: You either look at the HTTP referrer or you have a UTM parameter at end of your URL and especially when you come from performance marketing to use this way of attribution also when you're coming from SEO just used for attribution right?

00:04:00: It's what you know.

00:04:01: but AI search breaks because there are often no links to click on, right?

00:04:07: People get a recommendation about brand what service which company to go with.

00:04:12: And that was recently study from two of our clients and eight NN and graphite.

00:04:18: And together they found out that if you look at.

00:04:22: I think it wasn't leads that they looked at.

00:04:24: If you do it on click based attribution zero point.

00:04:27: nine percent come from AI search for an item.

00:04:31: but if your ask people hey How did you find us?

00:04:35: It's actually nine percent of people.

00:04:37: So in this so-called self reported attribution, it is ten times more than a click based attribution.

00:04:44: And that makes total sense, right?

00:04:45: Because if you get a recommendation from Gemini or Claude and there's no link.

00:04:50: What do you do?

00:04:51: You either type it into your Chrome address bar... ...or go to Google.

00:04:55: So is was be organic SEO or organic SEO brand Or maybe even paid?

00:05:01: because for branded searches often its a paid click How the person lands on your website.

00:05:06: It´s directly typed in which usually also attributed to Brand.

00:05:09: But at end LLM gave this recommendation.

00:05:12: So similar to word of mouth, recommendations from friends, recommendations for coworkers.

00:05:19: Self-reported attribution is more or less the only tool or probably the best tool marketers have in their toolbox to understand how people came to that website.

00:05:29: so what you do at your sign up flow?

00:05:32: You simply ask people hey where did he hear about us?

00:05:36: then AI search should be one option and if want it next level What you can do is as a second question.

00:05:43: when somebody says yes, it was I search.

00:05:46: You can ask them what prompted to use and then you store that in the database?

00:05:51: And then when you wanted one more next level which is what i'm right now doing um...I have a cloud automation That every day checks if people filled this out for pki with a prompt.

00:06:04: If yes, I grab all of the prompts over last couple days.

00:06:06: Check if they are unique and add them to a project via the PKI MCP so that i have a project where there only prompts in from real prompts.

00:06:16: uh... That people have entered who signed up to pki And it's a nice way to see prompt sets that generate.

00:06:25: was this prompt set based on People that found us?

00:06:28: Of course There is bias right because you Only get the prompts From people Who did find You.

00:06:32: But still, it's a very nice data set to have.

00:06:35: To understand how do people actually prompt?

00:06:38: And some of this is super weird stuff that you would not have expected.

00:06:43: and the important thing with self-reported attribution Is if You Have A Big Signup Form You Should Not Add Another Thingy Right That Will Kill Your Conversion Rate Just After Somebody Converted.

00:06:55: Ask Them This And Make It Optional.

00:06:56: So If They Skip It they Still Completed The sign up Flow but It Can Really Really Help With understanding better how people find you.

00:07:05: I personally would always add in something that you don't do like let's say, You Don't Do Snapchat.

00:07:11: just put in Snapchat as an option because if then five percent of people click on Snapchat... ...you can be pretty certain that your Five Percent uncertainty at least has a test from time to Time.

00:07:21: it is important.

00:07:23: also the options should Be In Random Order otherwise The Things That Are On Top will Always Click More Often.

00:07:30: Yeah, and then if you treat this as something that's not perfect but just directionally in like a directional indicator of truth it can be super useful.

00:07:41: And I mean just Google or ask GPD about self-reported attribution.

00:07:46: there are few more things to consider.

00:07:49: for example people typically mention the system that they use the most instead of a system, but I actually used to find you because they don't remember.

00:07:58: it's not that people intentionally lie.

00:08:00: They just don't know any better.

00:08:01: and then also small percentage will also lie intentionally etc.

00:08:05: etc.

00:08:06: But still its very good data.

00:08:09: otherwise we wouldn't get.

00:08:11: It sounds so simple almost too simple.

00:08:15: now i for example am confronted And I don't mean this negatively.

00:08:22: I think it's part of the job with people that have been told and also marketed by for example attribution software attribution providers Software, that provides a media mix modeling multi-touch attributions supposedly perfect technically Perfect solutions to tell you exactly Which touch coins?

00:08:44: A user had during their journey which mostly does not consist only off One channel or one touch point.

00:08:51: Now, how would you maybe also have the same conversations?

00:08:55: How would you try to convince a CMO ahead of marketing whoever in The end does that decision to change an attribution model Or at least allow for another legitimate way To look at attribution.

00:09:10: How Would You Try To Convince Them That This Supposedly Very Simple Fix Is Better Than I don't know, ten K-twenty K attributions of wear license they have been paying for.

00:09:25: I mean i'm not sure this fix is better than that software because the perfect attribution software would include this right?

00:09:32: Otherwise it's a performance marketing attribution softwares and overall attribution.

00:09:38: And if you look at how TV advertising radio out of home digital, out-of-home etc.

00:09:45: Etc.,

00:09:45: I attribute it.

00:09:46: there are just things that you cannot do perfectly with kick based attribution and as soon as we have like marketing mix basically the only thing That you can do is shutting off one channel and seeing The effect on other channels?

00:09:59: Then You always Have to problem that you Cannot really turn Off affiliate Marketing Unless you Really want a mess With your filled with Your publishing partners.

00:10:08: You cannot turn off SEO without causing major long term damage.

00:10:12: Um, you can not really turn of word-of-mouth et cetera, et cetera.

00:10:17: um and you cannot turn on AI search unless your willing to have Major Long Term Negative Effects.

00:10:24: So there you always had some way to deal with it.

00:10:28: And if somebody is convinced that they have the perfect solution I would simply ask them how they account for AI Search?

00:10:34: If They Have No Good Answer Then i am very sure that I should do the self-reported attribution in addition.

00:11:06: Neither is perfect and I need to find a model how-to combine them.

00:11:11: And if i pay A lot of money for a marketing attribution software, I expect that they can somehow model in AI search?

00:11:19: If They don't have a good answer...I still might Have to use Them To allocate my paid CIM Etc budget.

00:11:27: but I also Need to have the second reality That actually a significant part Is driven by AI Search.

00:11:34: it's similar For branded SEO traffic.

00:11:37: A lot of that should not actually be attributed to the SEO team, but rather to TV marketing brand

00:11:43: etc.,

00:11:44: etc.. But yeah it's.

00:11:46: there is no easy answer.

00:11:47: if you want perfect and exact numbers... ...but with self-reported attribution It's very easy to get.

00:11:52: a first indication

00:11:54: Sounds good.

00:11:55: I could relate especially With what you said That you have to accept contradicting realities in a way And its also something Pipp Kleppner once said on a podcast that this is basically part of the definition for someone who's smart and you can accept two contradicting realities living in your head without going crazy.

00:12:17: can you share, obviously adjusted to the public space here?

00:12:24: Although this podcast is a safe space.

00:12:26: Can you share how important self-reported attribution is for example for you and your marketing decisions at PKI?

00:12:32: You gave us already a glimpse into like next level ways of asking people for their prompts.

00:12:43: If you look at your marketing mix, that obviously also already consists of quite a few channels.

00:12:48: How much often importance has self-reported attribution in driving your decisions?

00:12:55: Data wise I would say it's one of the top two or three things we look at but of course We are still an early stage company like not even to years old and developing space so Not every decision can be pure database right.

00:13:11: there is also vision and anticipation And conviction to go in towards certain directions, but data-wise it's one of the top things we look at.

00:13:22: Okay I think that already tells a lot because One can obviously also see from the outside based on all the ad manager transparency centers That you're already do a little bit of LinkedIn advertising.

00:13:34: You do some Google paid?

00:13:42: with a high relevancy.

00:13:43: So thanks for sharing that.

00:13:46: Now the premise or I think so to say, the inception moment Or the renewed inception moment For self-reported attribution as you said has been The decline of importance Of click because AI search interfaces are not so Clickable and have not built To be so clickable A couple of weeks ago, I think.

00:14:12: I guess it was people started reporting it somewhere in June or end of May and they tie back to the beginning of may.

00:14:23: There was a sudden spike in clicks from chat GBT that a lot of people have reported.

00:14:28: i saw post form Adele Cahoe?

00:14:33: I hope its pronounced correctly.

00:14:34: Have no idea.

00:14:35: director product marketing at similar web.

00:14:37: She noted it, other people like Kevin Indyk also confirmed.

00:14:43: What has been happening there?

00:14:45: Has OpenAI, has Sam Oldman suddenly decided that they will be generous with us and give back a little bit of the clicks we lost but still mourn?

00:14:58: So what happened is around May seven chatGPT started adding significantly more links inside their answers And the primary thing that we notice is in, I would say list style answers where set of brands are mentioned.

00:15:14: Every brand mention was suddenly a link to the home page.

00:15:18: That's what you see in the chats that we observe and from clients also talking with various people who have access either log files or web analytics.

00:15:31: data seems to be that clicks at least doubled For most people, for some people up to plus four hundred percent plus five hundred percent clicks.

00:15:46: The effect seems to be stronger in the US than in Europe but I haven't looked into it too much.

00:15:53: if maybe English speaking was non-English speaking.

00:15:56: so there i'm not entirely sure what a differentiator is.

00:15:59: But in general us brands have reported bigger uplift then local European Brands and It was a very, very significant increase in May.

00:16:10: In June it stayed roughly on the same level and now in July I would say its TBD... ...I usually look at this data only on monthly cadence but to me it looks like there might have been a bit of reduction in July.. ..but i haven't looked fully into that so can only answer when month is really over.

00:16:30: What's decision behind us?

00:16:32: I've no idea could be a test just exploring, hey what would happen if we send out more traffic.

00:16:40: From what I'm seeing there's also another potential test running right now with this GPT-Five point six where they are chat GPT seems to be testing to be reliant much less on publishers who are usually the biggest drivers of these like legal pressure and public opinion pleasure that systems need to sent out more clicks.

00:17:01: so it might all be coincidence personally from observing what Google is doing very closely for a long time, with the also lawsuits that I was involved in when I said Idealo.

00:17:14: I have the impression that chat GPT or OpenAI is checking both directions like What happens if we send more traffic to the open web?

00:17:22: In case We are forced to do it but Also what happens If we Are less reliant on Wikipedia and publishers different mix of sources who might be less likely to stir up trouble for us.

00:17:35: And what the final result is, I would say it's still very open.

00:17:40: Okay if you had to make a bet or If this trend continues or Stabilizes Or if It will rather go back To The Level Before May seventh What Would Be Your Prediction?

00:17:57: i'm Pretty Sure.

00:17:58: Unless Something Significant Happens the effect is not going to increase, I'm not sure if it will stay the same or go down.

00:18:08: If i have to make a prediction ,I would say It Will Go Down.

00:18:10: So if we look at the July data there will be less linked mentions than in June and less clicks going out then In June.

00:18:18: but...it's A Prediction Like I've No Idea if Im Right Or Wrong.

00:18:22: Yeah And What Would Be Your Reason To Think That You Make This Prediction?

00:18:27: The Judgeability Wants To Keep People In The App.

00:18:31: I mean, everybody wants to keep people in their app.

00:18:33: That's part of building the super app and building this ecosystem And Google has found ways to do it for at least fifteen years.

00:18:43: All the LLMs are doing it.

00:18:47: Nobody is forcing them to do differently.

00:18:51: They have now licensing deals with a couple publishers Their promise to their users without the big reliance on publishers that they have right now.

00:19:03: So I think if push comes to shove, it will simply accept some publishers block them completely And it's more important to them that people stay in the app, have another chat.

00:19:15: Have a follow-up question share all of their personal information.

00:19:18: make all their purchase in the App whenever you lose someone going out off the app.

00:19:22: That is huge problem for them because they still need to monetize user base much better.

00:19:31: I saw one thing talk about monetization.

00:19:36: if your look at introduction of ads and the introduction of clickable rent mentions.

00:19:42: This is very closely tied together, And one theory also that they are trying to educate their users to click on things.

00:19:50: because why ads in Google work so well?

00:19:52: it's beacause you're used to click.

00:19:54: So if put something on top then click that In chat GBT your code not use to click Why would you click an app?

00:20:00: An ad like training people to click on brand mentions might also be part of making people click-on ads that usually recommend a brand or product.

00:20:14: Again, I can't prove it but its another theory which makes sense for me.

00:20:19: Do you know if ShedGBT has already introduced CPC as a way... Okay yeah they have introduced CosperClick bidding.

00:20:33: I just saw it in their blog post on May fifth.

00:20:38: That would be a close correlation to what you've said, because from my understanding initially, JGBT ads were like closed circle for couple of big advertisers in the US and based on cost per mill, cost per impression billing right?

00:21:00: And now obviously the model that made Google so successful is cost per click and not cost per impression.

00:21:06: You could argue Facebook meta, it's still running mostly on CPM.

00:21:12: but yeah okay... The hypothesis of making people learn how to click sounds almost a little bit stupid But Making them comfortable with this new pattern makes a lot sense.

00:21:29: what your overall Impression of, like?

00:21:33: do you have data or any insights on the adoption of ChatchBT ads?

00:21:39: I mean not global data.

00:21:40: but if you look at the US there are now industries where about one hundred percent of chats haven't had and There's a wide variety of advertisers.

00:21:53: I believe it still only open to advertisers from a small set of countries.

00:21:57: Like you cannot use a German credit card for example.

00:22:00: You need a us credit card probably UK and I don't have the list of other markets that are open right now.

00:22:09: So there is a lot of adoption, i saw message today on x where somebody said they stopped their advertising on chat gpt because the impression count is crazily high higher than what would expect possible for their topic.

00:22:25: so they assume that some irrelevant ads I would say questionable relevance, but i have not run a large-scale data analysis to make any kind of judgment on like how relevant the ads are.

00:22:41: Many are very relevant.

00:22:42: that is what I can... ...say from my observations.

00:22:47: Maybe chgbt doesn't wait as long as Google has waited to introduce stuff like broad match or performance max campaigns But they just start with with potentially showing irrelevant ads right away and not waiting like fifteen years.

00:23:05: so it's not random what they do definitely.

00:23:09: And I have a lot of targeting options where you basically describe but want, and then the ads are shown.

00:23:17: So i think they immediately starting was something like pmax campaigns or something going into that direction.

00:23:23: i think if you want to make the maximum money what should not do is allow people to bid different amounts of money on different kinds of keywords.

00:23:31: but he should convince people to give u. Their product feed or just a rough description like list off their landing pages and maybe little bit of text for that target audience, and then make everything automatically.

00:23:44: that's how you can onboard very inexperienced advertisers very quickly.

00:23:49: But this is also how you can milk the large advertisers, and get their most money out of them.

00:23:56: And to circumvent many advertising agencies if there's really just an option to upload something or describe it which I think a bit of trend right now to circumvents anybody who in between us as platform and customer but could be that they first will more agency friendly to get advertising agencies on board But I think long-term, if they want to monetize as good as possible and believe in their own ability to run ads that are relevant.

00:24:29: There should actually be no targeting where you bid different amounts of money for different keywords like there should been integration.

00:24:34: They showed have conversion events And then run all these ads.

00:24:39: i Have a bit of insights from the couple large companies that are similar to OpenAI, roughly.

00:24:47: And I know many of them consider these things like what kind of advertising options will they actually offer in the future?

00:24:57: Now based on your experience from Idealo especially around shopping placements and the introduction Google Shopping now we obviously also see an increasing relevance.

00:25:12: uh e-commerce and chat gbt, the product carousels being shown etc.

00:25:17: And you have done or your team i don't know if personally your team has done some great research on showing or proving that a lot of products come from google organic shopping results.

00:25:32: but do u see Open AI potentially going the same route as Google that they just sell these slots via their ad engine.

00:25:45: First question and second question, I know you cannot give legal advice but from your experience also with a google lawsuits etc.

00:25:53: do you think this would be legal or would then potentially backfire like legal action?

00:26:04: I don't know, whoever has an interest in that.

00:26:09: So in most countries of the world i would say it's legal just like having Google shopping ads on top of google and The only countries that ever did something against these google shopping ads As far as i know are the european union the uk turkey and russia And with Turkey and Russia i think having their most extreme measures And the UK basically copying the Digital Markets Act and from EU.

00:26:38: The EU introduced a DMA to regulate these large gatekeepers, but right now the DMA is not being enforced.

00:26:46: so... ...and the law that's been enforced isn't a law it just suggestion based on current actions of European Commission or US Administration.

00:27:00: I also don't think that the DNA will be and forced.

00:27:03: And even if it's being enforced, It might take years until OpenAI or chppt is classified as a. What does that cause?

00:27:12: VLOP very large online platform Or what there's like one stage above that in the DMA?

00:27:18: I forgot to name right now basically a gatekeeper because The class of answer engines doesn't even exist In the dma.

00:27:25: Like Is it a search engine?

00:27:27: If yes its not a dominant One Right.

00:27:29: so It will take time to regulate and usually regulation is always too slow, it happens shortly before there's a pivot anyways or disruption.

00:27:40: So I don't think that lawsuits against open AI introducing shopping ads are great tool that i would want to rely on.

00:27:49: I do believe at some point they have super prominent shopping ads but right now the ads are always in the bottom And I think they first need to train people, look at the organic shopping results and buy products before they can monetize it because you usually want to first form a habit.

00:28:11: Then make money out of that.

00:28:15: so we don't immediately see big shopping ads everywhere.

00:28:21: I

00:28:22: remember looking at a study from Cystrix, that is already couple of years old where they tried to assess the impact on Google

00:28:31: Ads.

00:28:32: On amount clicks still left for organic results and i think shopping ads placement or shopping carousel this type of SERP assembly had the strongest negative impact And with other things like featured snippet, etc.

00:28:55: It was a little bit less.

00:28:57: But I remember that roughly so not pointing to specific numbers here you can Google it or check chat GBT for Cystrix click-through rate study That it was roughly half half Like half of the clicks go to ads have off the clicks goto organic which obviously limits the potential for us SEOs organic marketer significantly.

00:29:20: Right now most of the AI search interfaces are still Most so Most strongly organic.

00:29:28: you only have limited at placements.

00:29:30: Do you think that we will in The future, have a similar setup?

00:29:34: That basically half of the clicks Are paid and only a half or not clicks?

00:29:39: It's it's hard even for me to say like what is there?

00:29:42: What does the currency then?

00:29:42: but you know where I'm going Like how well do organic potential over these platforms look like two three years down the road?

00:29:51: So that's the problem with the clicks.

00:29:53: is there are so few organic clicks.

00:29:55: Maybe ad clicks will be fifty percent, maybe they already are because their more paid clicks I can click than organic clicks.

00:30:01: for most prompts or most answers?

00:30:03: Um...I'm not sure what it would look like.

00:30:07: There'll definitely some form of monetization.

00:30:12: There also a question how much we actually prompt on our own and just use an agent.

00:30:20: because like especially in a B to B context, there are already small purchases that have been done without the person involved.

00:30:28: Like if my claw tells me I have to pay for a firecrawl or epi-fi i just do...I don't trust it enough yet to give out credit card.

00:30:36: so that's still what we're doing.

00:30:38: but ummm.. I was not actually involved with deciding on epi and fireclaw or for Cloudflare and GitHub Premium, all of these things that my cloud needs to do.

00:30:51: All the things I wanted to do.

00:30:53: And there is how will ads play a role?

00:30:58: Can there be ads that influence agents Or will agents immediately use different systems if they are any kind of ads where we just ignore them?

00:31:08: There's lot to be figured out because monetization is important While as a consumer on some way, I don't like ads.

00:31:20: Also have enough brain capacity to understand that ads are actually the thing That makes many things accessible to the general public.

00:31:28: Like watching TV would not be free if there were no ads right and There are many people who cannot afford premium TV cable subscription And they're dependent on ads.

00:31:41: same for radio Ads in sports pay for a lot of the salaries off athletes, etc.

00:31:49: So yes ads are good in a way because they make stuff accessible For non-rich people but adds also annoying and There somehow has to be away too.

00:32:00: deal with this indie in an economy where a lot Of decisions I'm made by agents I don't have.

00:32:09: I Don't have the clear answer or perfect vision for it.

00:32:11: I'm also thankful to the advertising industry for allowing me see Kilian Mbappé play and pay for his thirty one point two five million euro salary per year.

00:32:25: No, no just kidding.

00:32:25: i think your point makes a lot of sense.

00:32:32: quick transition from you not trusting your agent enough give it credit card.

00:32:41: You seem to trust your team enough.

00:32:44: To give them your credit card, at least for a Cloudflare account.

00:32:47: because I saw how did you like that bridge?

00:32:52: Great bridge!

00:32:54: Yeah so i saw post from your colleague Mittehan today where he says That you at PKI are now building one billion pages Something.

00:33:08: and you know, I'm just playing stupid here now.

00:33:11: I usually try to be a knowledgeable host But honestly, I don't know what are you doing there?

00:33:16: Why are you creating a billion pages?

00:33:18: What is this all about?

00:33:19: and why did you give me Tehan your Cloudflare account connected to your credit card?

00:33:24: Yeah so that credit card thing was a joke.

00:33:26: of course We some team members have websites that they use for experiments And we noticed that there are certain websites you can build where chat GPT will just crawl it.

00:33:44: Long after Google has given up, um... You still get thousands of ChatGPT crawls every single day and We just wanted to test what is the limitation of this?

00:33:56: So uh It was honestly not a big experiment That we planned for long time or anything.

00:34:05: Then MetaHan started it and the goal is to see how much that GPT will crawl this domain.

00:34:14: And which other LLMs would crawl, how search engines would crawl it?

00:34:18: How much cost did actually creates for a person hosting all of them?

00:34:22: because there's always this debate like hey LLM are crawling your website It's very expensive and doesn't matter for smaller websites.

00:34:30: you just put on Cloudflare and pay basically nothing.

00:34:35: But there are websites that are not on cloudflare.

00:34:37: There are websites That Dynamically have to render every request, they're still websites.

00:34:43: it don't use caching.

00:34:44: so It's just a little test how much chachapity is scrolling at website and how deep it will crawl And so on and so on.

00:34:53: Do you think he would only be able to answer?

00:34:57: Like if Chachapiti crawls the side when for example Google has as you said yourself already abandoned or just like doesn't see it as adding any additional value to its index.

00:35:10: Do you also think that you can answer questions like if chat GBT is building up its own index, et cetera?

00:35:17: How the crawlers are behaving?

00:35:19: so do you think you could generate other interesting insights from this project?

00:35:27: because I mean obviously everybody.

00:35:34: It's quite an interesting project that I have not seen anybody do in that transparency at least.

00:35:43: So from that project, i don't think we can make these conclusions?

00:35:47: Um...I'm very convinced that OpenAI is building their own index.

00:35:51: um..i mean they seem to have four different grounding services that they use and some of them seemed to be Google scraping.

00:36:02: Atleast one seems to be Bing either officially or also in a form of scraping, but scraping Bing is actually super difficult.

00:36:09: And most people who scrape Bing go via Yahoo because they have the same results and scrapping Bing it's difficult because there are strong scraper protection that even send you fake result instead just blocking you.

00:36:26: so like one these seems to be an own index.

00:36:30: There are a lot of people that are convinced this is an own index for high-trust, high priority domains.

00:36:38: I'm actually not sure if it's true like i've seen.

00:36:41: no proofs just theories and we're still exploring the topic.

00:36:48: some things have been published on LinkedIn aren't necessarily true or at least they are not provable.

00:36:53: They are just theories.

00:36:55: So yeah definitely Chatchapiti is building their own index.

00:36:59: How much it is used I cannot answer yet at this time.

00:37:03: Okay, but if It seems to be one of their grounding options

00:37:08: One of the topics i wanted To get your opinion on Is also something that Happened in May?

00:37:15: It was The Google IO Event Where they Also introduced Their.

00:37:23: I mean, they claimed also that Google is now AI search which i felt interesting.

00:37:31: Want to get your opinion on that obviously.

00:37:33: but the thing what was more interested in how you think about this new idea of Search Agents where as Im quoting google's blog post here We're entering the era of search agents where you can easily create, customize and manage multiple AI agents for many tasks right in search.

00:37:54: Starting with information agents operating in the background to twenty four seven.

00:37:58: these agents intelligence reason across information to find exactly what you need at exactly that moment.

00:38:03: so

00:38:04: this obviously seems

00:38:06: as

00:38:07: most off the times with corporate press releases and keynotes

00:38:12: seams

00:38:13: super big, the next big thing.

00:38:15: Big innovation?

00:38:17: What do you think?

00:38:18: what's your take on search agents and their potential relevancy for the future of Search?

00:38:24: I Think if Google does them well they can become super relevant Because google already has all my contacts right.

00:38:32: They have to have an email with a calendar.

00:38:35: So If i'm locked into My google account And after search bar that i Can use for quick information but also to search through my personal context.

00:38:45: And then, for complex chats with AI mode and also to start agents that are scheduled other than Claude who is never running on their own machine or always built to run in the cloud... That's very powerful!

00:39:01: So if it works?

00:39:04: This could be something else that makes me use Claude as an example in a long time.

00:39:10: I also was thinking of something that happened at twenty eighteen when Google announced duplex where they stood on stage and they called a hair salon to make an appointment in the restaurant, to make reservation.

00:39:24: And this would soon come everywhere!

00:39:26: And i think it was rolled out.

00:39:28: two pixel phones?

00:39:30: In the US...I don't know if it ever rolls globally..i dont'know what the adoption feature is but kind-of like They never again talked about it, like it went away.

00:39:40: It wasn't this big thing and if it was worked or adopted that would be a very-very big thing.

00:39:46: but I still believe with what Google is doing there they have the potential since most use search bar in the world to funnel more people into AI mode.

00:39:59: then also these agent experiences.

00:40:02: i mean they want you to wipe code directly from google search interface.

00:40:07: and a behavior that I noticed about myself, i used to use chat gpt app on my computer for like short questions instead of clod.

00:40:19: But now I've gone back to Google because AI overviews have actually become so good?

00:40:25: That is simple question.

00:40:27: it's quicker to open google and write.

00:40:29: then you click in clod on the simpler model.

00:40:35: If I look at some usage data, AI overviews are the number one LLM-based answer engine globally.

00:40:44: I'm very sure.

00:40:45: and then JetGPT is maybe half of their size.

00:40:50: Gemini is probably half the size of JetGPPT.

00:40:54: And there's AI mode, Claude, DeepSeq where especially Claude it super hard to measure because so many power users just using the app.

00:41:04: And after that it's then Grog Rufo's perplexity co-pilot.

00:41:09: I don't know how long Google will keep this AI overview, AI mode, Gemini separation?

00:41:18: It doesn't really make too much sense anymore.

00:41:20: if i now can also vibe code an app from my google search bar and like... at some point they need to merge Gemini and AI mode otherwise its just Not clear anymore.

00:41:33: What is the difference between the Gemini interface and AI mode?

00:41:39: But yeah, I think Google has the potential to make these agents very very relevant.

00:41:46: but many people who would be early adopters of this are already using Chatchi PTO Claude or Already.

00:41:55: beyond that it have their local deep-seek model running or something.

00:41:59: I don't know.

00:42:00: if Google will try to win over these early adopters and power users, which would be difficult.

00:42:07: Or they have to simplify it so much that... It's not even the late adopters but like in the middle of adopters.

00:42:17: So i'm not sure how they'll play that But I believe it can be super powerful.

00:42:23: When I looked at the little visualization that Google has on their side, they have this example where somebody types in keep me updated when any of my favorite athletes announced sneaker collabs or signature drops.

00:42:36: It's a bit like it is mixed with newsletters but then also social feed linked-in feed and maybe Instagram feed because in these examples sneaker drops wouldn't necessarily happen on LinkedIn.

00:42:54: So if you think about LinkedIn, for example or X

00:42:58: obviously

00:42:59: a lot of people say yeah it's becoming more AI slop.

00:43:04: everybody can post.

00:43:05: It is not the highest quality.

00:43:06: but they have also perfected this algorithm Gone above or beyond and try to perfect it.

00:43:17: where you passively give the platform a signal of interest like if You Like If You Stay Have A High Dwell Time.

00:43:23: If You Comment It's The Signal That Is Produced.

00:43:26: So, It Doesn't Need A Lot Of Effort For You To In An Ideal World Create A Curated Feed that You Like.

00:43:34: Now This Search Agents.

00:43:35: They Obviously Need Compared To That A Lot Off Effort You have to Type in.

00:43:41: What Do You Need When You Wanted Um, et cetera.

00:43:45: So do you feel like?

00:43:48: Long context easy question Do you feel that people might potentially be too lazy to use a feature like that?

00:43:55: so I have seen it less of a feat.

00:43:58: i envisioned more off Google alerts Like I set a real current task.

00:44:06: um That's not necessarily just alerting me right.

00:44:08: It can also be an agent that Does something regularly And I don't think that for everything, the feed is the right format.

00:44:19: Because there can also be agents.

00:44:22: just write something to some notebook or update something via an MCP on API of write-something in a Google Sheet and look at it every couple weeks.

00:44:38: so i'm not convinced by the feed as interface with all these agents.

00:44:44: I do believe that Google tried to build social networks multiple times and failed.

00:44:54: Maybe now this will be the one where they can actually win, And also OpenAI had this idea of a social network primarily based on Sora content.

00:45:05: i think That kind went nowhere.

00:45:08: So there probably is an argument LLM providers and social network feeds, and new personalized newsfeeds will merge into one product.

00:45:21: And also Google already ten years ago said it shouldn't be a search engine.

00:45:26: It should anticipate what you want and give it to You!

00:45:31: That would also be very much in line with What you just described.

00:45:34: before.

00:45:35: even thinking about the drop of a sneaker call-up Just by knowing that I by expensive limited edition sneakers.

00:45:44: And I like artists because they went to their concert, there could be something in my Gmail inbox that tells me hey this just dropped but why should it be free?

00:45:55: Why shouldn't it be an ad right?

00:45:57: Because Google also built a system where the people who make these sneakers can advertise to me.

00:46:04: so we will have to see how this looks like.

00:46:08: Hmm, I

00:46:10: just sometimes feel like last note when i look at this.

00:46:14: I Just question myself Our attention span Is somehow a zero sum game?

00:46:24: So we will not gain more Attention or We Will Not Be Able to Consume More and More Content Because The Days Are Not Getting Longer.

00:46:32: so Somehow Probably I mean with AI Search.

00:46:35: Now Somehow the cake has just gotten bigger, but most people also expected it

00:46:41: to be

00:46:42: a shift between usage like from Google to chat GPT From Chatcha BT to Claude and then just with this feature.

00:46:48: I feel Like okay will that beer?

00:46:50: Will People really keep using newsletters And the X feed and The LinkedIn feed and Then the search agents and then their Claude routines and then the chachi PT Whatever It's called there.

00:47:02: That gives you information.

00:47:04: This is Just Something Where i'm cautious if a feature like that has to win over some market shares from other forms of information consumption.

00:47:19: Last topic for today, you guys have

00:47:23: put

00:47:23: out another research paper again on SSRN which is titled Prompt Tracking Works.

00:47:30: It's a good start of the title.

00:47:33: Promptracking works but not as one prompt measurement system, prompt wording and brand visibility in AI answer engines.

00:47:42: now As you know people are lazy And most of them don't have that time Maybe also not willing to read the full paper.

00:47:52: Can You give us quick summary Of what you guys have analyzed there?

00:47:57: Why it is significant?

00:47:59: Yeah, so most of the work has been done by Jan Allen Spiel from my team and What we did is actually it's two separate studies that in the end.

00:48:08: We just decided to publish together.

00:48:10: You maybe remember that about a year ago ran fishkin did this experiment where he asked people Tell me how you would prompt if your looking for new pair of?

00:48:22: I Forgot was it was because we got some but i think It wasn't.

00:48:25: noise can chasing headphones And one other thing like buying software.

00:48:29: And then he had this data set and every single prompt was unique.

00:48:33: Then if you run these prompt once, You also get of course different answers for every prompt.

00:48:38: so everybody made a decision.

00:48:39: Ah!

00:48:40: Prompt tracking is so difficult because Every prompt is unique.

00:48:43: Every answer's unique.

00:48:45: So... He very graciously allowed us to use the status set?

00:48:50: We ran these prompts not once nor twice but many times.

00:48:54: Then we realized that ninety percent of these prompts actually give you the same answer.

00:49:03: So if you run them dozens of times, it's the same set of brands that you get for them and ten percent were like truly unique completely different brands he would get.

00:49:12: so that's a first interesting insight right?

00:49:14: That even though we use different words in the end If the intent into context is the same and DDD like funnel stage it's the same answer.

00:49:28: So that is very good news for people to do.

00:49:31: prompt tracking and a lot of variation just comes from the fact that LLMs are not deterministic, there always some randomness.

00:49:37: you need to run these many times.

00:49:40: And then we did second thing We took a set of prompts around fifty Then modified each of them in the tiniest little bit.

00:49:51: There was this thing called Cosine similarity which can be used to measure difference between prompts.

00:49:57: And we did this again and, until the intent shifted.

00:50:02: Because if you change a prompt too much at some point it is different search intent then you can't compare anymore?

00:50:12: Then also tracked all of these variations We basically could find what are the levels in similarity where suddenly different brands were being recommended.

00:50:25: And also there we found that, hey, prompts can actually be worded pretty differently but still the same brands are being returned in the answer.

00:50:36: Then re-analyzed it a bit more and what I can share with you is this style of a prompt matters.

00:50:43: so if you add a word like list or toplist You get more brands than the answer then shifts over all visibility.

00:50:52: If you write something you are an SEO expert, which SEO tool should I buy?

00:51:02: You will get fewer brands because this role-play export mode seems to trigger more information answers.

00:51:10: Also prompts that a very top of funnel or bottom of funnel is pretty robust for changes in wording.

00:51:20: But prompts in the mid of the funnel, they are the most sensitive to using slightly different wordings.

00:51:26: So there it makes probably the most sense to have more variation off your prompts.

00:51:33: and then The models also act differently if you put a lot of constraints on your prompt like You don't ask for their best CIM but say It has to be open source, it has to have single sign-on.

00:51:49: Has to be SOC II compliant and has to integrate with Microsoft.

00:51:53: JGPD and Perplexity will give you fewer brand suggestions but Gemini and AI Overviews will give more brand suggestions if we add these constraints.

00:52:04: I think that could relate how the systems use fanouts or how they pull in information.

00:52:14: But that was interesting.

00:52:15: and what really doesn't matter is the length.

00:52:18: like if I have a very short prompts, Like noise cancellation headphones.

00:52:24: And you will have...I am looking for Noise Cancellation Headphones or i'm Looking to buy noise cancellation head phones?

00:52:32: blah blah blah lots of filler content.

00:52:34: We get The same results in the answers If we got different Answers.

00:52:38: Of course if You run it our personal accounts because of the context but if You were To Run these two prompts in accounts that have no context or history, you get the same brands.

00:52:49: So all of this is very important for prompt tracking because it means yes, prompt tracking works!

00:52:56: It means you don't need to have the exact phrasing and wording your customers used but still relevant to use their general vocabulary that target audience uses.

00:53:08: Makes a lot sense.

00:53:09: I feel like this obviously great finding But that you have also gained enough trust over the years.

00:53:17: You personally, but also PKI.

00:53:19: now That we feel like this is maybe Also differently to some competitors may be and not some flawed research but as actually proper methodology And proper findings um that just prove your hypothesis Uh that you had in the first place.

00:53:37: Um Now I want to end Today's episode with something that I want to introduce also for a series in the future.

00:53:44: And which is, i wanted you ask beyond what we talked about today What was Something That You Saw In The Last Hours Days Weeks Since We Last Spoke That You Found Interesting Thought Provoking Exciting That Made You Happy Made You Cry Maybe not cry but Something where you want to point people towards and say hey, here's a thought.

00:54:13: Here is the content piece here as opposed that I think maybe some people should look at?

00:54:18: I wish you had prepared me for this spontaneous question because i'm drawing it blank And don't have good answer right now.

00:54:28: Okay no worries Unless i do a self promotional one which would be lame.

00:54:33: No You can do It!

00:54:35: Yeah,

00:54:37: people should follow David from my team.

00:54:41: He has a couple of really interesting insights on LinkedIn just in the last couple days especially about fan out sources if you want to learn something go there.

00:54:55: it's where I learned.

00:54:57: yeah and i mean this is great don't be shy because published something today about, so we're recording on July fifteenth.

00:55:09: About how do LLM secretly change tracking invisible shifts in query fanarts?

00:55:14: And I already congratulated him to this being reposted by Lily Ray and he also got reactions where i think he was pretty happy about from Glenn Gabe!

00:55:25: If you don't know these names it's like... and i think Malta you can confirm Well known people in the SEO space from the US and internationally.

00:55:34: So I think if you follow David, You can learn a lot.

00:55:37: so that's definitely That

00:55:43: is my recommendation for today.

00:55:45: Awesome!

00:55:45: You see we don't need preparation We just need to like share with People what on our mind.

00:55:52: anyways Cool Malte.

00:55:55: thanks so much For making time Today.

00:55:57: It was great as always Fun fact.

00:55:59: I had multiple people reach out to me over the last, i don't know one two weeks that randomly just said hey.

00:56:08: I Just wanted to say That monthly land where format is like these are always The episodes that I liked the most because there's so much value in So little time and they want it too.

00:56:20: pass this on to you.

00:56:21: yeah It's great And I feel Like today we lift up To This Promise again.

00:56:27: Obviously there are sometimes these long questions between your answers that might be a little disturbing, but sorry all of you.

00:56:36: That I have to somehow get Malte to share his thoughts.

00:56:39: But yeah period.

00:56:41: thanks so much Malte and already looking forward to the next one

00:56:45: Thanks for having me and looking forward it

00:56:47: Bye bye.