크롤링 하다가 본건데..
https://cafe.naver.com/robots.txt
# BOT ACCESS FOR THE PURPOSES OF AI TRAINING AND RETRIEVAL-AUGMENTED GENERATION (RAG) IS STRICTLY PROHIBITED.
User-agent: *
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Bingbot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Yandex
User-agent: YandexBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot
Disallow: /
User-agent: facebookcatalog
User-agent: facebookexternalHit
Allow: /
User-agent: OAI-SearchBot
Disallow: /
https://blog.naver.com/robots.txt
User-agent: Yeti
Disallow: /
# BOT ACCESS FOR THE PURPOSES OF AI TRAINING AND RETRIEVAL-AUGMENTED GENERATION (RAG) IS STRICTLY PROHIBITED.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
AI 에이전트들 다 막아 놓은거 같은데... 이러면 GEO?는 티스토리나 다른데서 해야되겠네...
네이버 블로그는 왜 네이버 검색엔진을 막아 놓은거지?
그럼 네이버 검색에 나오는 블로그들은 뭐야..
