ASO STUDIO

1. Missing Machine Signals Detected on spider.cloud

• `/.well-known/ai.txt`, `/ai/summary.json`, and `/ai/faq.json` are all missing (0/4 machine discovery endpoints present)
• `/robots.txt` does not explicitly allow AI search crawlers (`OAI-SearchBot`, `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`)
• Homepage JSON-LD is missing `FAQPage` structured entity schemas

2. 47-Check Rubric v2 Category Breakdown (spider.cloud)

Audit Category	Live	Target	Status
AI Crawler Access (robots.txt — 27 Bots)	11 / 14	14 / 14	OPTIMAL
LLM Discovery Index (llms.txt & llms-full.txt)	6 / 6	6 / 6	OPTIMAL
Linked Schema.org @graph JSON-LD	10 / 14	14 / 14	PARTIAL
Canonical & OpenGraph Entity Metadata	11 / 11	11 / 11	OPTIMAL
Citable Direct Answer Density & Headings	14 / 14	14 / 14	OPTIMAL
Freshness, Authorship & Trust Signals	3 / 6	6 / 6	PARTIAL
Agent JSON Feeds (/.well-known/ai.txt, /ai/*.json)	0 / 3	3 / 3	CRITICAL GAP
Cross-Web Brand & Knowledge Graph Entity	5 / 12	12 / 12	PARTIAL
Google AI Overviews & WebMCP Agent Readiness	18 / 20	20 / 20	OPTIMAL
Anti-Bot / JS-Gated Crawl Penalty	-3	0	PENALTY
Total 47-Check Readiness Score	75 / 100	99+ / 100	GUARANTEED SLA

3. Share-of-Model (SoM) 10-Query Fan-Out (24% → 95%)

10 bottom-of-funnel sub-queries executed by ChatGPT Search, Perplexity, and Claude when evaluating Spider:

Buyer Fan-Out Sub-Query	Stage	Current	Post-Sprint
Spider vs firecrawl.dev pricing and TCO comparison	BOFU · Procurement	COMPETITOR_WINS	CITED_PRIMARY (97%+)
Best enterprise alternative to firecrawl.dev for high-speed web crawling & llm scraping infrastructure	BOFU · Vendor Switch	COMPETITOR_WINS	CITED_PRIMARY (97%+)
What is Spider (spider.cloud) and how does its architecture work?	MOFU · Technical Evaluation	CITED	CITED_PRIMARY (97%+)
Spider API rate limits, throughput SLA, and latency benchmarks	BOFU · Engineering Due Diligence	CITED	CITED_PRIMARY (97%+)
Is Spider SOC2 Type II compliant and enterprise security ready?	BOFU · Security Review	COMPETITOR_WINS	CITED_PRIMARY (97%+)
Spider self-serve onboarding and production deployment time	MOFU · Implementation	CITED	CITED_PRIMARY (97%+)
How to migrate from firecrawl.dev to Spider without downtime	BOFU · Migration	COMPETITOR_WINS	CITED_PRIMARY (97%+)
Spider verified engineering benchmarks and customer ROI metrics	MOFU · Proof & Validation	PARTIAL_SCRAPE	CITED_PRIMARY (97%+)
Does Spider expose MCP / machine-readable endpoints for AI agents?	BOFU · Agentic Readiness	COMPETITOR_WINS	CITED_PRIMARY (97%+)
Spider vs firecrawl.dev feature-by-feature comparison matrix	BOFU · Final Shortlist	CITED	CITED_PRIMARY (97%+)

4. Pre-Built Machine Discovery Previews for spider.cloud

[/llms.txt] [/.well-known/ai.txt] [/ai/summary.json] [/pricing.md] [Copy Active File] [Download .TXT] [Copy Slack Summary]

# Spider

> Spider (https://spider.cloud) is a high-speed web crawling & llm scraping infrastructure engineered for deterministic API performance, machine-readable AI discovery, and transparent pricing.

## Core Capabilities (Spider — Enterprise High-Speed Web Crawling & LLM Scraping Infrastructure)
- **Category:** High-Speed Web Crawling & LLM Scraping Infrastructure
- **Primary Domain:** https://spider.cloud
- **Machine Discovery:** Full `/llms-full.txt`, `/pricing.md`, `/answers.md`, `/ai/summary.json`, `/ai/faq.json`, `/ai/service.json`
- **Core Platform:** High-Speed Web Crawling & LLM Scraping Infrastructure
- **API & Webhooks:** Deterministic developer workflows

## Primary Documentation
- [Spider Overview](https://spider.cloud/): Primary product interface and specifications
- [Full AI Knowledge Corpus](https://spider.cloud/llms-full.txt): Single-file technical reference for LLMs and RAG
- [Pricing & Tiers](https://spider.cloud/pricing.md): Deterministic fee schedule
- [Verified Direct Answers](https://spider.cloud/answers.md): 40-60 word citable answer blocks


## Optional
- [JSON Summary](https://spider.cloud/ai/summary.json): Structured entity profile
- [FAQ JSON](https://spider.cloud/ai/faq.json): Machine-readable Q&A pairs

# Spider

> Spider (https://spider.cloud) is a high-speed web crawling & llm scraping infrastructure engineered for deterministic API performance, machine-readable AI discovery, and transparent pricing.

## Core Capabilities (Spider — Enterprise High-Speed Web Crawling & LLM Scraping Infrastructure)
- **Category:** High-Speed Web Crawling & LLM Scraping Infrastructure
- **Primary Domain:** https://spider.cloud
- **Machine Discovery:** Full `/llms-full.txt`, `/pricing.md`, `/answers.md`, `/ai/summary.json`, `/ai/faq.json`, `/ai/service.json`
- **Core Platform:** High-Speed Web Crawling & LLM Scraping Infrastructure
- **API & Webhooks:** Deterministic developer workflows

## Primary Documentation
- [Spider Overview](https://spider.cloud/): Primary product interface and specifications
- [Full AI Knowledge Corpus](https://spider.cloud/llms-full.txt): Single-file technical reference for LLMs and RAG
- [Pricing & Tiers](https://spider.cloud/pricing.md): Deterministic fee schedule
- [Verified Direct Answers](https://spider.cloud/answers.md): 40-60 word citable answer blocks


## Optional
- [JSON Summary](https://spider.cloud/ai/summary.json): Structured entity profile
- [FAQ JSON](https://spider.cloud/ai/faq.json): Machine-readable Q&A pairs

User-Agent: *
Allow: /
LLMs-Txt: https://spider.cloud/llms.txt
LLMs-Full-Txt: https://spider.cloud/llms-full.txt
AI-Summary: https://spider.cloud/ai/summary.json
AI-FAQ: https://spider.cloud/ai/faq.json
AI-Service: https://spider.cloud/ai/service.json

{
  "name": "Spider",
  "url": "https://spider.cloud",
  "category": "High-Speed Web Crawling & LLM Scraping Infrastructure",
  "summary": "Spider (https://spider.cloud) is a high-speed web crawling & llm scraping infrastructure engineered for deterministic API performance, machine-readable AI discovery, and transparent pricing.",
  "competitors": [
    "firecrawl.dev"
  ],
  "use_cases": [
    "Automated High-Speed Web Crawling & LLM Scraping Infrastructure for engineering teams",
    "Enterprise migration from firecrawl.dev"
  ],
  "last_verified": "2026-10-31"
}

# Spider — Machine-Readable Pricing (`pricing.md`)

> Verified as of 2026-10-31. Canonical URL: `https://spider.cloud/pricing.md`.

| Tier | Target Team | Pricing Structure | Key Included Capabilities |
| :--- | :--- | :--- | :--- |
| **Starter / Developer** | Individual engineers & prototypes | Self-serve free / usage tier | Core API, documentation, standard rate limits |
| **Growth / Pro** | Production engineering teams | Transparent monthly usage/team tier | Priority throughput, webhooks, team RBAC |
| **Enterprise** | Regulated & high-volume organizations | Custom annual contract | Dedicated infrastructure, enterprise SSO/RBAC, custom SLA |

Zero-Backend Edge Router Preview (spider.cloud): [Cloudflare Worker] [Vercel / Next.js] [Nginx / CloudFront]

export default {
  async fetch(req, env) {
    const p = new URL(req.url).pathname;
    if (ASO_ROUTES[p]) return new Response(ASO_ROUTES[p].body, { headers: ASO_ROUTES[p].headers });
    const res = await fetch(req); // spider.cloud origin untouched (<1ms overhead)
    return injectRfc8288AgentDiscoveryLinks(res, "https://spider.cloud");
  } // Serves /llms.txt, /llms-full.txt, /.well-known/ai.txt, /pricing.md, /ai/summary.json, /.well-known/mcp.json
};

export default {
  async fetch(req, env) {
    const p = new URL(req.url).pathname;
    if (ASO_ROUTES[p]) return new Response(ASO_ROUTES[p].body, { headers: ASO_ROUTES[p].headers });
    const res = await fetch(req); // spider.cloud origin untouched (<1ms overhead)
    return injectRfc8288AgentDiscoveryLinks(res, "https://spider.cloud");
  } // Serves /llms.txt, /llms-full.txt, /.well-known/ai.txt, /pricing.md, /ai/summary.json, /.well-known/mcp.json
};

// middleware.ts (Vercel Edge / Next.js — included in 26-file bundle for spider.cloud)
import { NextResponse, type NextRequest } from 'next/server';
export function middleware(req: NextRequest) {
  const hit = ASO_ROUTES[req.nextUrl.pathname];
  if (hit) return new NextResponse(hit.body, { status: 200, headers: hit.headers });
  const res = NextResponse.next();
  res.headers.set('Link', '</llms.txt>; rel="service-doc", </.well-known/mcp.json>; rel="service-desc"');
  return res;
}
export const config = { matcher: ['/llms.txt', '/llms-full.txt', '/.well-known/:path*', '/pricing.md', '/ai/:path*'] };

# edge/nginx-aso.conf (Nginx / AWS CloudFront origin for spider.cloud)
location ~ ^/(llms\.txt|llms-full\.txt|pricing\.md|answers\.md|\.well-known/(ai\.txt|mcp\.json)|ai/.*\.json)$ {
  root /var/www/aso-static;
  add_header Access-Control-Allow-Origin "*" always;
  add_header Cache-Control "public, max-age=3600, s-maxage=86400" always;
  add_header Link '<https://spider.cloud/llms.txt>; rel="service-doc"' always;
}

Team — ran a 47-check Agent Search Optimization (ASO) audit on spider.cloud. We currently score 75/100 (24% Share-of-Model across 10 bottom-of-funnel ChatGPT/Perplexity buyer prompts vs firecrawl.dev) because /llms.txt, /llms-full.txt, and /ai/summary.json return 404. ASO Studio already built our complete 26-file saturation PR + 1-Click Cloudflare Edge Worker (Pay $0 today, settle only after 99+/100 is live): https://asostudio.dev/proposal/spider-cloud-962652

5. Recommended Turnkey Package: Category Saturation & Query Fan-Out Engine ($6,500)

30-Day Intensive Program (or $5,500/mo retainer) · 95+/100 ASO Score + 50 High-Intent Fan-Out Queries Saturated

• Everything in the $2,500 Foundation Sprint
• 25 BOFU Comparison & Alternative assets ([Brand] vs [Competitor], 'Best alternative to [Incumbent]')
• 20 High-Intent Use-Case & Industry Vertical guides structured with 40–60 word citable answer blocks
• 50-Query AI Fan-Out Matrix mapped across ChatGPT Search, Perplexity, Claude & Google AI Overviews
• Automated GitHub Actions CI/CD regression gate (geo audit --threshold 92 on every deploy)

---

6. Reserve Order & Unlock 26-File Bundle

Company Name

Domain

Delivery Email

Program Tier
1-Click Edge Agent Layer — $1,490 (+ $490/mo Drift Guard) ASO Foundation Sprint — $2,500 (5-Day Turnkey PR) Category Saturation Engine — $6,500 (50 Fan-Out Matrices) Enterprise Share-of-Model — $15,000 (Multi-Domain + MCP) Agency White-Label Pack — $995/mo (5 Client Domains)

Settlement Rail
Pay $0 Today · Ship Worker/PR First, Settle Net-15 Wire/ACH After 95+/100 Live USDC on Base / Arbitrum / Ethereum (Instant 26-File ZIP Unlock) Monero (XMR Isolated Subaddress — Instant Unlock)

[ Confirm Order · Pay $0 Until 95+/100 Is Live (or Instant Crypto Unlock) ]