AI Agents and Your Website: The Secret Your Competitors Know

North London storefront at dusk showing invisible AI agents accessing local business website data through digital overlays

Amazon just sued Perplexity, claiming the AI company scraped its website without permission. But here’s the secret most business owners miss: this lawsuit isn’t really about whether scraping is legal. It’s about who controls access to your digital storefront when machines, not humans, are the visitors.

For your North London business, this matters more than you think.

What the Amazon vs Perplexity Case Actually Reveals

The lawsuit hinges on the Computer Fraud and Abuse Act (CFAA), a US law written in 1986, before anyone owned a website. Amazon claims Perplexity’s AI agents repeatedly accessed Amazon’s website in ways that violated the site’s terms of service. Perplexity says they have a right to access publicly available information.

The real question buried in the legal papers: Who decides what machines can do on your website?

Your robots.txt file already answers this. It’s a simple text file you can place on any website that tells search engine bots (and other crawlers) where they can and cannot go. Google respects it. Bing respects it. But here’s the secret: most AI agents do not.

Perplexity, for example, does not fully honour robots.txt instructions the way traditional search engines do. Neither do dozens of emerging AI companies scraping the web to train their models.

The Amazon case is really asking: Should the law force them to?

Why Your Website Is Already Being Visited by Machines You Don’t Know

Every day, hundreds of different bots visit websites. Some are legitimate: Google, Bing, ChatGPT’s training crawlers. Others are less obvious: data scrapers, price monitors, competitor analysers, and AI training systems.

Your analytics software probably only tracks human visitors. The machines? They arrive silently, copy your content, and leave. You never see them in your traffic reports.

For North London small businesses, this creates an invisible problem. A competitor’s website might be getting cited in AI search results because an AI agent scraped their content more thoroughly than yours. You have no way of knowing which machines visited, when, or what they took.

The Amazon lawsuit is trying to answer a question your business should already be asking: Should you have the right to know? Should you have the right to say no?

The Visibility Risk Hidden in Plain Sight

Here’s where this gets practical for your business. If an AI system cannot easily access your website content (because you’ve blocked it, or because your site is slow, or because it’s behind a login), that AI system cannot cite you in its results.

Your customer asks Perplexity, Claude, or Google’s AI: “Where should I get a website designed in North London?” If the AI scraped your competitor’s site easily but struggled with yours, guess whose name appears in the answer.

This is not a ranking problem in the traditional SEO sense. Google’s search rankings are still determined by links, content quality, and relevance signals. But AI search is different. AI systems cite sources based on what they can access and understand.

Make your content harder to access, and you become harder to cite.

This is why the Amazon case matters. The legal outcome will determine whether businesses have enforceable rights to control machine access. Until that changes, you need a practical strategy: make your site easy for legitimate AI systems to read, but smart about which ones you trust.

Three Practical Steps to Protect Your Visibility

First, audit your robots.txt file. If you have one, check what it allows. If you do not have one, you are currently open to all crawlers. For most North London businesses, this is fine. But if you handle sensitive customer data or proprietary information, you may want one.

Second, check your site speed and accessibility. AI agents have less patience than humans. If your website takes five seconds to load, some AI crawlers will time out and move on. If your critical content is buried in JavaScript or behind navigation menus, AI systems may not see it. A responsive, fast website is cited more often by AI systems than a slow one.

Third, understand that blocking AI access entirely is risky. Yes, you could add a robots.txt rule blocking Perplexity or other AI scrapers. But if you block Perplexity, your business never gets cited in Perplexity results. If you block ChatGPT’s training crawler, your content does not help train the model that millions use. You trade short-term control for long-term invisibility.

The smarter play: stay accessible. Make sure your content is high-quality, trustworthy, and easy for legitimate systems to read. That is how you get cited.

The Real Secret Your Competitors Already Know

Here is what the best-performing North London businesses do that others miss: they focus on being cite-worthy, not just rank-worthy.

A website that ranks #1 in Google but is hard for AI systems to understand will lose visibility as AI search grows. A website that is slower to rank but easier for machines to read and trust will gain citations and traffic from AI sources.

This is especially true for local businesses. When a customer asks an AI system “Which web designer should I use in North London?” the AI is not pulling from a ranking list. It is pulling from its training data and current knowledge. If your website is cited in that knowledge base because it was easy and valuable to access, you win.

The Amazon vs Perplexity lawsuit will eventually settle or go to trial. The court will decide whether AI companies need explicit permission to scrape. But by then, your competitors will have already adapted.

The ones who prepared now have a head start. They have fast, accessible, trustworthy websites that AI systems naturally cite. They did not wait for the legal outcome. They built for the future.

What You Should Do This Week

Check your website’s core web vitals. Use Google PageSpeed Insights to see if your site loads quickly for AI crawlers. Fast websites get scraped and cited. Slow ones get skipped.

Review your content for clarity and accuracy. AI systems cite sources they trust. If your content is vague, outdated, or buried in marketing speak, it is less likely to be cited. Clear, specific, well-sourced content wins.

Finally, if you are unsure whether your website is optimised for AI visibility, ask. A quick audit from your web design team can reveal whether your site is set up to be easily discovered and cited by emerging AI systems.

The secret your competitors know is simple: visibility is shifting from rankings to citations. The machines visiting your website today are deciding whether you get cited tomorrow. Make sure they find something worth citing.

© 2025 Local SEO and Digital Marketing Agency - North London
Privacy policy
Terms of conditions