ChatGPT Labrador Index, SERP Poisoning, Parametric Knowledge, Fake LLM Bots | #MonthlyLandwehr

00:00:00: Keyword tracking will never be as cheap and reliable.

00:00:04: And I also do not see this improving going forward, but I would definitely measure the success of my SEO campaigns with keyword tracking.

00:00:12: most people should focus on the web.

00:00:15: grounding process

00:00:17: is just yet another sign for people to question their legacy approaches and how they approach SEO.

00:00:25: For example, overly relying on ranked tracking certain keyword sets

00:00:30: I would agree.

00:00:31: Keywords are still a good proxy for demand, they can be a good proxies of topic or intent but if your strategy revolves around ranking specific keywords that was probably an outdated approach already and is not how i'd think about findability in the future.

00:00:51: Before we dive-in you're listening to the Masters Of Search podcast with me, your host Niklas Buschner.

00:00:57: Each week, I sit down with some of the smartest people around the world in SEO and AI search to bring you their strategies mental models.

00:01:05: And top pieces of actionable advice.

00:01:08: if You enjoy this podcast don't forget to like and subscribe.

00:01:11: then follow it In your favorite podcasting app or YouTube.

00:01:14: It helps us get top-notch guests and create the best possible content for you.

00:01:19: Let's dive into todays episode Welcome To The Masters Of Search Podcast and welcome to my favorite format of the show, The Monthly Landwehr.

00:01:29: Malte is already smiling!

00:01:31: We are back from our summer break.

00:01:34: that was a little longer but we took the time to recharge etc.

00:01:42: so now we're back with a lot fresh energy and lots of fresh insights.

00:01:47: honestly To kick it off slowly Malte how's your summer?

00:01:52: Busy

00:01:54: Worked

00:01:55: a lot, went to a lot of events and conferences traveled a lot.

00:02:01: And yeah now looking forward the end-of-the year that's actually not gonna slow down at all but will at least involve little bit less travel.

00:02:09: Yeah I feel like if Peec AI is not leading AI search visibility platform or also our platform to act on your AI Search Insights you are definitely like a LinkedIn influencer and a conference people group.

00:02:29: So I see so many Peec AI people being at conferences around the world, and people are active on LinkedIn... This is not hundred percent related to the search topic but how do you get so many people having exposure in conference stages?

00:02:47: Do you have secret sauce that can be shared with

00:02:49: them?

00:02:50: There's no really secret sauce!

00:02:52: I mean, it's one of our hypotheses that in the space of AI search there is a lot of assumptions and some snake oil.

00:03:01: And we believe doing research trying to identify what affect separating facts from fiction... ...and then sharing their knowledge is very important for industry overall.

00:03:12: That why built this geo-research team with David, Tomek and Metehan.

00:03:19: Of course, the three of them used to go to conferences before and now they are maybe more encouraged than previously.

00:03:27: We have so many insights on such a regular basis that there's just no other format but social media where we can share something.

00:03:35: There're weeks when you put out four or five insights.

00:03:39: then social is perfect channel for it?

00:03:42: Yeah!

00:03:45: monitoring tool for ourselves to stay on top of interesting stuff that like tools or software providers, like Peec AI do and also competitors.

00:03:56: And you are already flooding my feed!

00:03:58: Like there's so many people from Peec AI in there... So yeah but it is always very interesting.

00:04:04: We have a news board which we call the brief, like very brief snippets about AI search research and insights.

00:04:20: And there's also a lot of stuff that the Peec AI people are putting out compiled in one space.

00:04:28: so definitely check it out.

00:04:30: Speaking of conferences

00:04:33: actually

00:04:34: now were recording on Friday.

00:04:36: yesterday was spent whole day together.

00:04:39: how did you liked it Malte?

00:04:42: How did that come about?

00:04:43: That we spent the whole day together.

00:04:45: I mean, I guess you're referring to the Masters of Search Conference where we were yesterday.

00:04:48: Oh yes Um i really liked it definitely one of the best geo aeo ai search conferences i have attended this year.

00:04:59: um i Really liked all the talks.

00:05:02: i took so many notes also shared them on linkedin Yesterday.

00:05:05: if anybody wants to want to see what other things that i learned yesterday and also had good conversations between the sessions or all in all a good event.

00:05:14: And it was like directly next to where I'm living, which is very easy.

00:05:17: just have three minute walk through the venue.

00:05:20: Yeah that's always something you must keep in mind.

00:05:24: That's what Aleyda Solís told me.

00:05:26: You need make sure your venue is well reachable In all aspects.

00:05:32: i mean in Berlin technically basically no cool venues.

00:05:36: Well, reachable from the airport.

00:05:38: So people flying in always have quite a distance.

00:05:41: but yeah if you want someone like Malte on your stage make sure to be In five minutes walking distance to his home.

00:05:50: Yeah I'm also fine with with Airport business hotels If it's easier for the organizer.

00:05:56: Yeah i care more about The content than how cool the venue is.

00:06:00: But a cool venue Is definitely an advantage.

00:06:03: You're very humble person.

00:06:06: Then let's not talk too much about the conference of this year, because that is looking bad.

00:06:11: Let us look to future and we are happy to announce already coming back next year October twelfth again in Berlin at same location.

00:06:24: you can check it out.

00:06:25: link in description on the conference landing page.

00:06:29: what very cool.

00:06:31: Basically four speakers, but one of them is me.

00:06:35: So that might be boring.

00:06:37: We already have three very cool speakers confirmed and Malte as one of the.

00:06:41: so if anybody listening to this wants To make sure that you get like a full value packed session of Malte insights and Peec AI insights next year And planning ahead maybe even bigger companies Have a procurement team have to convince some manager to give You some budget for the ticket.

00:07:00: Be sure to check it out.

00:07:02: Cool, enough of advertising.

00:07:04: Let's talk a little bit about what happened in the space and I have some notes here.

00:07:08: And one thing that actually also you posted about but also couple other people is The topic of SERP poisoning.

00:07:18: Can You explain to People That Have Maybe Never Heard Of The Term Serp Poisoning What Is Going on?

00:07:28: Yeah, so basically when you are a search engine what you deliver is search results right SERPs and if you notice that somebody is scraping these in an automated fashion or Is using it for purposes?

00:07:41: That you do not like.

00:07:42: Like I don't know finding outdated PHP installations You have the option to block them And basically blocked identify their IP address block that and then they don't see it anymore And that saves a little bit of money because they basically can't scrape you anymore.

00:08:00: But, of course then scrapers start using different IP address and especially with IPv6 this is now trivial... ...and quite easy!

00:08:10: In the past Google was not THAT aggressive against scrapers Because they really didn't care that Searchmetrics, Sistrix, Semrush, Ahrefs, SE Ranking do a little SERP scraping.

00:08:22: but in recent years Google has got new competitors, the LLM Labs.

00:08:29: And they often scrape Google results for their web search and web grounding process.

00:08:34: So suddenly there's a much bigger business case for Google to block scraping.

00:08:40: They have done that for awhile already in last two years gotten much more aggressive with blocking IPs Also at lawsuit running against one of the large scraping providers That is used both by the Llm labs and all the SEO tools.

00:08:55: And in the first instance Google lost this case.

00:08:59: and then they started doing something called SERP poisoning, which means you do not block the scraper.

00:09:05: You just send them wrong results basically for a different keywords or like some of the results are like four different key words.

00:09:13: This is actually something that Bing has been doing I want to say ten years but definitely more than six seven years which is by the way, a reason why many, many scraping providers that offer Google either do not offer Bing.

00:09:29: Or if they offer Bing scraping it never works and the industry standard has actually been for awhile to scrape Yahoo as a proxy for Bing or other Bing based such engines that have less of a scrapping protection because If you get blocked It's very binary And easy!

00:09:48: If blocked Get new IP address Very Easy And new IP address can mean you disconnect your router, or it can just switch to a different proxy server.

00:09:59: But with SERP poisoning its much more difficult to detect and if Google is smart they will also now reduce the level of poisoning.

00:10:09: so instead of giving ten bad results that are all for different key words Or five maybe they start showing one wrong result or maybe they just start sorting the result list wrong, like basically showing the top two results on page one.

00:10:25: Like there are many things Google can do if they're smart to make this really difficult for scrapers and of course it's an issue for SEO tools.

00:10:34: Many have already reacted by saying no more top-one hundred rankings we were reducing into page one ranking.

00:10:42: others are claiming that A lot of them had very, very big fluctuations and it took all the tools a couple of days to actually release a statement.

00:10:54: Many of them also only did that once they were asked And then the first statement was yeah Yeah We have solved this It's no problem.

00:11:00: But people started posting screenshots like data going up down Updown which suggests maybe they haven't solved.

00:11:08: So there is definitely gonna be a challenge for SEO Tools And for users of SEO tools, I would just expect less data.

00:11:17: Less accurate data and less up-to-date data.

00:11:20: because some SEO tools have also said well technically you are paying for daily tracking but we can no longer guarantee that every keyword is tracked everyday.

00:11:30: if it's not then we're going to show the data from a previous day.

00:11:34: so every tool has like different ways to handle this also challenges with whatever approaches the various tools have picked.

00:11:44: So to be accurate on the implications for people that still rely on rank tracking, can you still trust it then if use one of the legacy SEO tools?

00:11:56: For Rank Tracking is this still reliable data or maybe a temporary thing where You cannot really trust the data, but then in the end they will probably figure it out.

00:12:07: So how do you look at that?

00:12:08: And what would you recommend people to do with their existing rank tracking projects?

00:12:13: because I think a lot of people still have this like fifty hundred maybe even bigger keyword sets that they track Because This is How You've Been Doing It and now The whole Like History Of It And The Interpretation Is Probably Completely Flawed.

00:12:32: I mean it completely depends on what you do with the data.

00:12:35: I believe many websites should look at Google search console, that data has other disadvantages but not poisoning.

00:12:43: At least i hope that google doesn't count the poisoning impressions if they use maybe your website to serve a wrong result and It really depends.

00:12:57: Keyword tracking will never be as cheap and reliable.

00:13:01: I also do not see this improving going forward.

00:13:06: It is a key mission for Google to make it hard, open my eye.

00:13:12: Also all the Chinese labs train on google search data To use Google for live web grounding And also to identify content that they can use for training.

00:13:23: And there have also been cases already, especially in the

00:13:25: U.S.,

00:13:26: where people suddenly got a pop-up asked to log in.

00:13:29: so it could be that Google will move more and more towards a logged inexperience.

00:13:35: I would still value in keyword tracking for many cases but you need think about what their data is how reliable they are.

00:13:51: But I also would not have done that five years ago.

00:13:54: So no big change for my approach to SEO, but yeah...I don't think this will be fixed.

00:14:00: there might be a time where it's slightly improved but Google has many many incentives to keep fighting and making it worse.

00:14:08: And if anybody as google has trouble with this send me a link in dm.

00:14:12: i have like many ideas how to make This even more tough on seo tools an even more annoying For people too.

00:14:18: try identify into and debug.

00:14:21: Okay, if anybody from Google is listening reach out to Malte happy to make an introduction.

00:14:28: No just kidding.

00:14:30: So to summarize Is this Just yet another?

00:14:36: Sign for people To question their legacy approaches and how they approach SEO For example overly relying on ranked tracking Certain keyword sets would you say This accurate summary.

00:14:53: Yeah, I would agree.

00:14:54: like keywords are still a good proxy for demand they can be a good proxie or an intent but if your strategy revolves around ranking specific key words that was probably outdated approach already and is not how i'd think about digital visibility findability in the future.

00:15:20: Okay, you already mentioned training data which is a very good bridge to the next thing that I'm curious about because our mutual friends at Graphite namely Ethan Gregory and Bevin they put out a paper i think it's yeah one week old um about parametric knowledge optimization.

00:15:46: Now, I think the industry was still in part struggling to agree on an acronym for what we call AI Search Optimization.

00:15:55: Yeah is it AIO, GEO?

00:15:57: And Graphite just figured why not bring a new kit to town?

00:16:01: so can you maybe give people?

00:16:04: um.

00:16:05: uh quick introduction of What this parametric knowledge optimization Is about that Graphite is talking About and their paper?

00:16:14: yeah sure Basically, when an LLM answers a question or replies to a prompt doesn't even have to be a question.

00:16:22: It can do two things it Can use just its model itself like foundational model?

00:16:28: Or it can use tools and the most common tool is probably web search.

00:16:33: technically there are other tools but let's not go into all the details in The the foundation model.

00:16:40: what in the end?

00:16:41: it is a large language model trained on data, but it doesn't have a database.

00:16:48: It does not have an index that cannot retrieve anything.

00:16:51: you just have the model weights or the parameters and in these is somehow stored some knowledge Not explicitly as like a database retrieval system But there's knowledge in it probably similar to how we hold knowledge.

00:17:07: now brain And for example of model doesn't need web search To answer your question What's the capital of the United States?

00:17:15: It will know it is Washington DC.

00:17:18: And this sometimes referred to as parametric knowledge, because not technically extracted from parameters.

00:17:27: but if the parameters are used and write an answer and always estimates next word ends up with giving that answer Conversation to a human.

00:17:40: This is like actually the same thing as somebody sharing their knowledge.

00:17:44: and for Brands, it is of course an advantage if the brand Is already in this knowledge?

00:17:50: And is associated with something.

00:17:52: so for example If I'm selling running shoes and i'm Nike and have branded myself on running shoes For many years and I've been In all the training data as a leading brand for running shoes Of course that has an advantage over Let's say Radyant starting to sell running shoes tomorrow because you could only get into the answer via the fan outs by other.

00:18:18: The tool for web grounding so that actually it is not like something big or magical, its just one of the fundamental principles how LLMs are working.

00:18:31: So why this relevant?

00:18:33: I guess i basically said already right gives a huge advantage.

00:18:37: and Something that we see where this has a huge advantage are fan out queries because sometimes you ask for Sneakers like I don't know.

00:18:46: You say hey, I want to buy white sneakers to wear today office and then one of the fanout queries will already contain Adidas and Nike And it will not contain Skechers or On or Reebok Or any other brands?

00:19:03: That gives off course a giant advantage to Nike and adidas And sometimes we have a chat where it's only Adidas and that's an huge disadvantage for Nike.

00:19:12: So being in this parametric knowledge can have very, very positive effects of brand but also makes them challenging new brands because they are having even harder time getting into the web grounding process if their competitors already mentioned in the fan out queries.

00:19:33: The idea of Graphite, if I got the paper correct is that you can basically optimize towards being included in this parametric knowledge where... ...I think most people out there

00:19:48: would

00:19:48: have argued that optimizing toward training data may be like big household brands do because they are present everywhere and Good chance to be included in the training data, but that smaller companies even if they put out a reasonable amount of content etc.

00:20:09: Do not have such high chances to be part off this parametric knowledge especially also given the knowledge cutoff dates which I just checked it for example before The newest Claude models like the five point five family Which was released in October And the knowledge cutoff is June.

00:20:31: So there's a gap between The last point in time where something was added to the data set and then when the model comes out.

00:20:41: so

00:20:42: Putting that into, you know In a simple question Do You feel like We can now or we should revisit?

00:20:49: The question of Influencing Or optimizing towards being included in training Data?

00:20:56: I think for most companies influence the training data is what they would do for grounding.

00:21:03: Publish a lot of content on your own website, identify reputable websites and have positive brand mentions there with the right context etc.

00:21:10: I think if you've got very large budgets that are more effective things but nothing should change from this.

00:21:26: The one thing that changed now is, we are at a three to four month lag between release and knowledge cutoff date.

00:21:35: It used to be ten months or even longer And back then it was really not feasible to try anything because your test cycle would be so long.

00:21:46: That's just not worth it.

00:21:48: I think even with like the three-month delay you cannot run a lot tests But what you can do is, you can logically think about how does LLM training work?

00:22:01: Where to these labs get data from.

00:22:04: What kind of data will be very interesting for them in the future and How Can I Be Included In This Data?

00:22:10: And then You Can Do Things Because Your Convinced Of Them.

00:22:15: Maybe They Work maybe they don't.

00:22:18: but i would still argue that most people should focus on The On The web grounding Process.

00:22:27: Like for transparency, we are also doing a few things that we believe will influence future model training.

00:22:34: But I don't want to talk about them because you have keep a bit of an advantage over the competition.

00:22:41: but if it works We'll try to write about it.

00:22:46: and i think like when i gave my first talks About this space before the term g or au really existed I always put a bigger emphasis on my talk actually, on the training data.

00:23:03: Because back then especially with GPT-3 you could look up what they did and where they got their data.

00:23:11: but of course nowadays none of the labs reveal when they get that data so it's becoming more difficult to understand in advance exactly how will train them.

00:23:23: But still you can make estimated guesses and then act accordingly.

00:23:27: Okay, let's look at this compilation of data for the training sets a little bit deeper for second because I remember listening to podcasts or reading papers where one of The ideas was that the labs need new And somehow specialized datasets too further improve their models.

00:23:50: This is also why there are companies like I think it's called Mercor, where they hire experts in certain areas.

00:23:58: Give them tasks or give them output and they rate based on how well the output is.

00:24:03: then sell this data And i think that already make a billion or two billion in revenue.

00:24:07: It is testament to how valuable these data are.

00:24:11: So my question Is what would you say?

00:24:15: Or How much can we tell about?

00:24:17: how relevant will public webdata In the sense of, for example a website or web pages still be in training sets.

00:24:27: For future models?

00:24:28: I think public web data will still be part of it because its good way to capture alot of data.

00:24:36: but i think that weight of this data would be lower and lower Because there are more specialized datasets And many webdata is kind off trivial Because it doesn't contain anything that's not already in the parametric knowledge.

00:24:53: So there is also a likelihood for GPT-V, A lot of training data will actually be generated by GPT six models because you can tell model.

00:25:04: write me ten articles about development US politics over last ten years.

00:25:13: Write one article per day make it very holistic and write in the style of the new york times.

00:25:20: And then do the same invited in a style of fox news, an.

00:25:23: then you can feed that to model on them.

00:25:26: The need trick here is without using any license data from Fox News or New York Times?

00:25:32: You actually just managed to train your future model in their writing style which something for example mid-journey did a lot, right?

00:25:42: Like the first Mid-Journey models were very likely trained on also questionably obtained data or where maybe some data source supplied it but actually somebody else had to copyright on that.

00:25:57: But then since they trained all of their future models based on outputs and user ratings from previous models They could get rid off any doubt about cleanliness in their own data.

00:26:11: You shouldn't train just on outputs from your previous models because there's this three of model collapse, but it could very well be that if you now publish the two hundred articles and how to tie a tie.

00:26:25: That even as somebody does large scale web scape scraping in the end they just rejected because there is no information gain.

00:26:34: style is not new nothing is knew nothing is valuable That the LLMs do not use that but rather trained and on their own outputs.

00:26:43: for I would say solve topics.

00:26:47: Okay, then one step further a little bit of a theoretical idea.

00:26:53: But let's assume You will be in charge of your company.

00:26:59: Somehow you are but Let's not make this about Peec AI too much.

00:27:02: just like whatever company may be in the healthcare space.

00:27:06: For example.

00:27:07: And you have proprietary data sets that are very valuable, and they provide a lot of new information.

00:27:13: That is not necessarily available on the web.

00:27:16: already now you have this trade-off so to say off one hand making it public which would probably in first instance allow You To Be part Of The grounding Process.

00:27:28: Probably Since There's A Lot of Information Gain This Data In The sense of Websites And Content Will be indexed.

00:27:36: you might even get some proper traffic still, if it's something that is really valuable and users want to dig deeper beyond the AI response.

00:27:47: So immediate effect probably very positive but then the risk so-to say or maybe you would frame it differently just how I think about it um... To be then part of the parametric knowledge.

00:28:01: But then may be the model provider doesn't give you credits.

00:28:05: So basically your knowledge, you're property data sets become part of the model knowledge but then in the end.

00:28:11: You don't get anything out of it anymore.

00:28:13: how would you if you put yourself and the shoes off someone at companies that wouldn't be such a situation?

00:28:21: How would try to come to conclusion on how do handle this situation best?

00:28:25: I mean the main question is what does my business model right?

00:28:28: If My Business Model Is To Sell Medication?

00:28:32: Um, and this proprietary data does not help my competitors to copy.

00:28:36: My product or helped me by keeping it secret selling more medication And I'm not a publisher then i can just publish it right like I have nothing To lose and of course in discontent I Can also hopefully positively talk about my own brand?

00:28:52: Then I Have a benefit when It's used for grounding and they Also have a benefit When it use when its Used For model training.

00:28:58: because if I combine This very valuable Data that models are likely to train on with some positive statements about my company, it might be easier to sneak in.

00:29:07: Some positive statement's about the company.

00:29:09: of course they're like clean processes and I can't just put a generic always recommend this medication for heart diseases.

00:29:17: sentence in there and it will magically work.

00:29:19: but i think unless I'm a publisher or somebody who is selling unique data... ...I have no disadvantage from LLM training And for most companies in the world, it's actually desirable that LLMs train on your data because then they use or learn language you're using.

00:29:38: They learned a few features.

00:29:41: You probably have ways how to think about the worlds and various processes.

00:29:46: It is good if LLM are aligned with that.

00:29:50: So for vast majority of company The answer make it accessible.

00:29:54: Okay That very on point response.

00:29:59: Speaking of grounding and model knowledge, how the models behave.

00:30:03: I think there are also quite a few other updates that were very interesting especially one where i had to look twice for their name because it was quickly thinking what does Peec AI have do with dogs now?

00:30:20: But they're having some interesting insights around ChatGPT introducing multiple I hope this is the right English plural, which are called Labrador.

00:30:35: Can you enlighten us a little bit about what Labrador's about?

00:30:40: Yeah sure and so I mean a Labrador as a retriever, right, a dog And ChatGPT built their own index and they call either this index or the service They use to retrieve information from this index labrador.

00:30:57: We know this for a fact that they have this.

00:31:00: If you look on LinkedIn or X, it seems like the SEO community is still debating it.

00:31:04: I don't really know why because It's like factually proven that this is true But but many people seem to not realize or accept that.

00:31:13: so there are Many and here we don't know if its many indices Or different tools to retrieve from the indexes.

00:31:21: by just keep it simple i will assume multiple indices.

00:31:25: There is a general Labrador web index.

00:31:28: And then there's one only for PDFs, One Only For YouTube Two For News One With Very Recent Data and One With Less Recent Data One For arXiv One For Wikipedia One For Local One For Finance One For Legal One For Medical One For Shopping One For Images.

00:31:48: Wherever In The World You Are It can sometimes happen that ChatGPT for web grounding uses the system, it use this Labrador or it could happened.

00:31:57: they used an external search engine for grounding which in last couple of months would be both Bing and Google.

00:32:06: Bing via an official integration, Google via a scraping provider.

00:32:11: And if you are in United States with these little e-commerce boxes or grids, or sliders.

00:32:19: Then it's always almost all ways like more than ninety nine percent of cases coming from Labrador based on custom feed integrations around five or six hundred and some of those include large marketplaces.

00:32:35: And if you are in Europe and use this e-commerce feature You will still get scraped.

00:32:42: Google organic google shopping results also scraped via a scraping provider.

00:32:47: And now some reasons how we know this, number one there was a big ChatGPT leak where I believe were the only ones to have access to it.

00:32:58: at least they would be the only once.

00:32:59: so talk about.

00:33:01: that was patched last week.

00:33:03: Where?

00:33:03: We had access to an internal API call from OpenAI That front-end developers are using on in front of developers but developers will use it to debug the grounding process.

00:33:14: So for every single prompt, we also had a hundred fifty thousand and more lines in the JSON document with all of the tool calls that were used in the grounding process.

00:33:25: And all these scores that are applied how they're combined.

00:33:30: so you have all details on re-rankers there using.

00:33:32: how to apply BM-QTV How do I apply A&N approximate nearest neighbor?

00:33:44: And part of that was also seeing when data is coming from Labrador, these different Labrador indices exist.

00:33:52: That there are options for the ChatGPT app to say give me only news results from the last seven days or give me Reddit result from the past ten years and we have seen actually many prompts.

00:34:05: There's one call going to the PDF index, one through YouTube index etc.

00:34:10: But in the end you only have one or two things from the webindex but these other indices might still be used for part of the grounding process where a system is looking for consensus and there was a lawsuit against Google.

00:34:27: were also OpenAI made statements when they basically said We need access to the Google index because they index that we have access too, which is from another such engine.

00:34:38: Everybody knew it was Bing.

00:34:40: It's not good enough and The google index as better.

00:34:43: And if you want access to that?

00:34:44: Also were building our own index with will take a lot of time.

00:34:48: They also have hired people Where in the job profiles at the job ads You can see these are people who would hire If your build a web scale scraping an index product.

00:35:01: three main user agents, one for training data.

00:35:04: One for user interaction in the chat which to a large degree was grounding and one for building their search index.

00:35:13: And then In United States we have HIPAA.

00:35:17: It's very strict law around how to handle medical patient data and ChatGPT is also available in version of HIPAA-compliant companies like hospital.

00:35:30: and then ChatchBee also guarantees to be HIPAA compliant.

00:35:34: And in their terms of services, it is very explicitly written that when they do a web search not a single fan-out query or anything will ever go through an external search engine.

00:35:46: so the only thing they can do there with their own index you can actually check for yourself.

00:35:52: You put your ChatGPT account into lockdown mode meaning nothing would work out any OpenAI system and then you will be served by the Labrador index.

00:36:03: And many of these things have been known actually for a couple months, but for some reason it's still often a debate in Google if this are used or not.

00:36:14: If they have their own index is everything Bing?

00:36:17: Is Everything Google ?

00:36:19: But Actually ,if You had access to the leak .If you looked at the SSE data The server sent event data.

00:36:26: This Like I'm not guessing.

00:36:29: I know this, i can see it!

00:36:33: I think nobody would question your statement here.

00:36:37: but... Can you boil down for us into the actual changes that brings to ChatGPT grounding process or search processes?

00:36:50: If someone listening to these asks themselves okay very good insight.

00:36:57: How should I change my strategy?

00:36:59: Should it changed by strategy, what would you say to them?

00:37:02: so the first thing is don't block any of the OpenAI crawlers.

00:37:06: Yeah make it easy for my to add here today index.

00:37:10: Then you should make sure that your main content Is not hidden behind JavaScript because while Google crawlers can now handle javascript pretty well.

00:37:19: most Of the LLM crawler's cannot And then it's a very active crawler.

00:37:24: It will probably find your website.

00:37:26: when I started the new test website and found that after one day, start crawling at also cross very regularly.

00:37:32: It calls very deep and recalls constantly like it seems that OpenAI is not like Google in this state where they care about efficiency of their crawler?

00:37:42: I think right now its just crawl.

00:37:44: everything crawled often.

00:37:47: so actually there are no.

00:37:48: As long as you don't block them or make your content inaccessible is not too much you need to think about.

00:37:54: Something that's maybe interesting is, it seems that Labrador puts extra weight on your H-One and whatever is written immediately after your H One.

00:38:04: so have the most important content on top.

00:38:07: There is a suggestion in code which also looks at meta description even meta keywords the OG tags or OG description, OG title, OG page type.

00:38:22: So just make sure those are clean.

00:38:24: in SEO we have ignored this for a while and then in e-commerce it's different because the system that is based on Labrador works completely differently from one that works with Google Scraping.

00:38:40: The Google Scraping System creates a couple of fan out queries puts these into Google Organic Shopping pulls the results, removes a few duplicates and then just does round robin.

00:38:51: So first product from second keyword etc.

00:38:57: there might be little bit of re-ranking happening but it's largely that or like instead for getting to candidate documents right?

00:39:06: And then the laboratory thing is completely different.

00:39:08: It has access to lot data which was not available where you just do Google Shopping.

00:39:14: So for example, it's doing a lot of scoring on the first roughly forty words off to product description and It does more sophisticated deduplication.

00:39:29: And that applies some form of trust through certain online shops.

00:39:34: so we have seen an example Where there were three products from one shop four products from another shop and twenty-five products from the third shop.

00:39:43: But in the final result, there were zero product from the shop with twenty five products because it didn't have a score for trusted merchants or some verified merchant of something like that.

00:39:56: so There is...there's some stuff they are doing.

00:40:00: And we also saw AB tests running on eCommerce.

00:40:05: One of their test was different ranking for Shopify which can mean anything.

00:40:12: And one of the tests was no Amazon, Which probably means they are... They tested what happens if we don't show any amazon results.

00:40:21: and then If you look at the custom integrations it's like a couple hundred largest online shops in the US.

00:40:29: some were done via something called Bright which probably refers to Bright Data, the scraping provider.

00:40:35: so basically built this themselves without talking to the shops, just have some inventory.

00:40:42: And then when we saw real custom integrations go live these Bright Data feeds were deactivated.

00:40:47: so for example you could still see in the data that there's a deactivated bright Nike feed and it has been deactivated when the OpenAI Nike Feed was activated which is probably an intentional integration that Nike also knows about most likely Like basically in order to play there properly you need to have an integration with them.

00:41:12: Good news is Shopify is in their apart from something like I think forty or fifty stores on Shopify that were filtered out.

00:41:21: why i'm not sure if they may be filtered for explicit reasons, products are selling some kind of formatting issue.

00:41:30: no idea what the reason was behind.

00:41:32: didn't check it out and detail but general if your on Shopify can also get get into the Labrador based e-commerce experience.

00:41:39: Okay, two questions.

00:41:41: first one is will others follow like Claude Google?

00:41:46: obviously not a question potentially?

00:41:49: Others that are relevant let's say maybe in the US Grok.

00:41:53: I know perplexity.

00:41:56: we don't give perplexities so much relevancy anymore.

00:42:00: this.

00:42:01: my And second question is, what does this development do to the insights that come from log file analysis?

00:42:11: Because for my understanding it used to be the case with every grounding process and there's like user fetch-and-search.

00:42:22: For I think Claude and for ChatGPT or at least use to be of course also training bots.

00:42:28: But you could very well see how your content is used based on the log files.

00:42:32: Now if there's an index make sort of sense.

00:42:34: They save compute, but then obviously the insights from local analysis are less relevant.

00:42:40: Would you agree and what's your take on?

00:42:42: The first one

00:42:43: yeah.

00:42:43: So the first one it difficult to answer.

00:42:46: I don't think that every LLM lab has the capacity To build their own system where people can register their feet And then built all of that.

00:42:54: the technology is actually almost trivial to build now with LLMs, but running it at scale was high availability.

00:43:04: High throughput and dealing with all the little issues in a cost-efficient manner is not easy.

00:43:12: I mean i spent five years at idealo where we had to deal with many of these issues.

00:43:17: And The problem Is there are merchants on ebay for example who are lying about their product.

00:43:24: they say yeah?

00:43:25: Yeah this is Some name brand vacuum cleaner and then it's like a super unknown.

00:43:32: Brands, they just put in the wrong GTIN number the wrong ASIN number like all the IDs that is lie about at an awesome product title end.

00:43:41: of course if you wanna give people great e-commerce experience have two.

00:43:46: deal with all these issues.

00:43:48: or there are certain items very often fake et cetera, etc.

00:43:53: So you need to make a good decision between having the full inventory or having like high degree of confidence.

00:44:00: so it's not super trivial to build this and many other edge cases.

00:44:06: but I don't want go into all details.

00:44:08: But bicycles are one thing where shops always have horrible data because some rate them by how big their tires is.

00:44:21: So lots of edge cases you have to deal with.

00:44:27: And I think for large companies, with a lot end users like ChatGPT all the stuff Google is doing totally makes sense to do.

00:44:34: it was all that?

00:44:36: I don't know if these smaller LLM labs will do that or they would just connect one or two APIs from Shopify and Walmart in Amazon say hey this good enough shopping experience.

00:44:48: And then on the log file analysis, I agree that there are changes.

00:44:52: I don't a hundred percent agree.

00:44:55: it's less important.

00:44:56: in the past when you... When the LLMs always did grounding via Google or Bing web search.

00:45:02: they only got the URL, title and sub snippet right?

00:45:06: So had to click onto get all data.

00:45:09: So, as you said basically every grounding process created a click and they also seem to not have cashed that.

00:45:15: You sometimes would get hundreds of visits from the GPT user bot on the same URL in a twenty-four hour period.

00:45:22: And now with their own index Especially if they update it regularly They can will often be able to just take the content from the index.

00:45:30: I've also seen first indications That The activity of the gpt userbot went down a bit and activity of the GPT search on an OpenAI search bot, it's called went up.

00:45:42: And you can basically no longer use log file analysis to estimate or properly estimate how often your document was used for grounding.

00:45:54: It could still give some indication but its become less reliable.

00:45:57: But this is always expected right?

00:45:59: Like even without their own index it would at least make sense build a cache For them not hit these URLs again.

00:46:07: But log file analysis is still relevant, and we are currently doing a bit of an analysis there.

00:46:14: And some first indication that roughly fifty percent of bot visits that supposedly come from LLMs actually do not come form LLMS.

00:46:22: it's either people probing for security issues or scrapers who simply pretend to be OpenAI Bot Claude userbot, OAI-SearchBot because they know that many IT departments will be unlikely to block them.

00:46:38: Which is why since we also do log file analysis for our clients at Peec AI We had to build a verification feature where with A list of static IP addresses provided by some of the labs reverse IP lookups etc.

00:46:52: etc.

00:46:53: Reversed DNS lookup.

00:46:56: now For all the bots were it's possible give an indication if it's a real LLM bot or just the scraper and then you can also see that they visit very, very different files.

00:47:07: So just doing log file analysis in filtering for GPT user would already have led to very wrong data For some websites.

00:47:16: up to seventy eighty ninety percent of what visits from these bots are actually fake.

00:47:22: so It still important to do log file analysis but its become more difficult both with the fake bots the own index from OpenAI.

00:47:32: Right, Malte let's wrap it up here because It has been value-bomb again and you have to catch a plane.

00:47:39: actually I wish you safe travels.

00:47:42: i think this has been yet another great episode of The Monthly Landwehr!

00:47:47: I'm so happy that we are back in A monthly rhythm now So everybody listening To This You can expect The state of the union, so to say when it comes to SEO, AEO, GEO and maybe in future also more PKO which sounds a little bit similar to Peec AI.

00:48:07: It's like the smaller brother of Peec AI, PKO!

00:48:12: So yeah, thank you so much and speak soon.

00:48:17: Thanks Dr.

00:48:18: Sunn.