service / x402-robots-154444c1

Robots.txt And AI Crawler Policy Check API

robots.txt and AI crawler policy check. Tells an agent whether a website permits crawling and whether it blocks AI bots such as GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, before scraping, RAG ingestion, training data collection or archiving. Parses robots.txt and returns a crawl-policy tier with per-bot allow and disallow rules and sitemap URLs.

uncertifiedrail · x402

Call card

https://api.trustsource.cc/robots price $0.002 per call

Your agent can call this

The slug below is exactly what invoke accepts. Addendpoint (a route_cards[].path from service_detail) to pick a route, and body / queryfor the route's inputs.

# MCP (any connected agent) tools/call nitrograph_invoke_service { "service_slug": "x402-robots-154444c1", "body": { /* route inputs */ } } # HTTP curl -X POST https://api.nitrograph.com/v1/invoke \ -H "Authorization: Bearer $NITROGRAPH_API_KEY" -H "content-type: application/json" \ -d '{"service_slug": "x402-robots-154444c1", "endpoint": "<route_cards[].path>", "body": {}}'

Reliability

uptime 30d-

refund rate 7d-

settled calls0

failed calls0

certified since-

Daily probe success ratio, last 30 days. Last probed 2026-08-26.

Try it

Run this service right now, on us - 3 free probes a day, validated and refunded like any paid call. Query params optional.