Davriellelouna@lemmy.world to Technology@lemmy.worldEnglish · edit-211 days agoThe AI company Perplexity is complaining their bots can't bypass Cloudflare's firewallwww.searchenginejournal.comexternal-linkmessage-square229fedilinkarrow-up1645arrow-down15
arrow-up1640arrow-down1external-linkThe AI company Perplexity is complaining their bots can't bypass Cloudflare's firewallwww.searchenginejournal.comDavriellelouna@lemmy.world to Technology@lemmy.worldEnglish · edit-211 days agomessage-square229fedilink
minus-squareubergeek@lemmy.todaylinkfedilinkEnglisharrow-up1·10 days agoAnd I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
minus-squareKissaki@feddit.orglinkfedilinkEnglisharrow-up2·10 days agoNo, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.
And I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.