robotstxt-ai
本地Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
安装配置
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"robotstxt-ai-mcp"
]
}
}
}
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"robotstxt-ai-mcp"
]
}
}
}