Web Content Toolkit

Endpoints (9 live)

• POST /v1/sitemap — Discover a site's XML sitemaps (via robots.txt Sitemap: directives, an explicit sitemap URL, or the conventional /sitemap.xml) and parse them into a bounded list of URLs with their lastmod, changefreq and priority. Follows sitemap-index files to their child sitemaps. Fetches and URL counts are capped (with a `truncated` flag) to stay fast and memory-bounded. The reliable way for an agent to enumerate a site's pages. (0.006 USDC on Base)
• POST /v1/structured — Fetch a URL and pull out its machine-readable structured data: all JSON-LD blocks (schema.org), Open Graph and Twitter Card tags, standard named meta tags, and inline microdata items. Also returns the page title, canonical URL, language and the distinct schema.org types found. Ideal for turning a product, article, recipe, event or organization page into structured fields without bespoke scraping. (0.006 USDC on Base)
• POST /v1/robots — Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards. (0.006 USDC on Base)
• POST /v1/unfurl — Fetch a URL and return link-preview metadata from Open Graph, Twitter Card and standard meta tags. (0.006 USDC on Base)
• POST /v1/redirects — Follow a URL's redirect chain (301/302/303/307/308) to its final destination and return every hop with its status and Location. Unwraps link shorteners and cloaked/tracking links so an agent can see where a link really goes before following it. Each hop is SSRF-validated and the chain is capped. (0.003 USDC on Base)
• POST /v1/headers — Fetch a URL (following redirects) and return the final response's HTTP headers, the server banner and content-type, plus a security-header audit reporting HSTS, Content-Security-Policy, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy and the cross-origin policies (with the missing ones listed). No body is downloaded or parsed. (0.003 USDC on Base)
• POST /v1/seo-audit — Fetch a public page and grade its on-page SEO: title and meta-description length, H1 and heading outline, canonical, meta-robots indexability, viewport/lang/charset, hreflang, Open Graph and Twitter Card completeness, image alt coverage and word count — plus optional focus-keyword placement and density. Returns a pass/warn/fail verdict per check and an overall 0–100 score. (0.006 USDC on Base)
• POST /v1/links — Fetch a URL and extract every hyperlink on the page, each resolved to an absolute URL and classified as internal vs external (by registrable host), with its anchor text, rel attributes (nofollow, sponsored, ugc) and whether it opens in a new tab. Returns internal/external/nofollow counts. For SEO link audits, internal-linking analysis and outbound-link/nofollow compliance checks. (0.006 USDC on Base)
• POST /v1/extract — Fetch a URL and return its main article as clean HTML, Markdown and plain text. (0.006 USDC on Base)