Machine-First Flywheel Dashboard
Full pipeline: Universe → Captured → Enriched → Pages → Indexed → Cited. Period: all time
200: 16.4M404: 633.9K301: 564.8K410: 499.9K429: 142.4K
Total AI Usage
2.8M
+new vs prev
Citations: 2.7M · Preflight: 37.1K
AI Citations
2.7M
+new vs prev
Claude: 2.4M
Perplexity: 173.7K
ByteDance: 108.0K
ChatGPT: 89.5K
AI Indexing
1.4M
+new vs prev
GPTBot (building index)
Preflight API
37.1K
+new vs prev
Claude: 11.9K
ChatGPT: 11.1K
ByteDance: 8.5K
Non-AI preflight: 92.5K
Total Crawling
17.6M
+new vs prev
AI Cite: 2.7M · AI Index: 1.4M · Search: 6.1M
Meta: 3.4M · Amazon: 703.6K · Other: 1.4M
Answer Depth
100.0%
2.5M enriched vs prev
of 2.5M entities
0 pipelines
Coverage
6.6M
30 registries · 6 verticals vs prev
18.2M requests · 433.9K IPs
Total requests: 18.2M
Bots: 17.6M (96%)
Humans: 582.0K (3%)
Unique IPs: 433.9K
AI Usage Trend
Last 24h: 2.8M
Previous 24h: 0
0%
ChatGPT: 90.8K new
·
Claude: 2.4M new
·
ByteDance: 116.5K new
·
Perplexity: 179.3K new
·
Apple: 2.9M new
·
AI Humans: 34.3K new
Conversion Funnel
Log-scale bars. 1.6M indexable of 2.5M total entities across 30 registries.
Potential Pages
504.6M
504.6M
1.6M indexable entities × patterns × 23 languages
Live (in sitemaps)
93.8M (18.6% conv.)
93.8M
EN: 22.0M core + Lang: 71.8M (22 langs × 2 patterns/entity)
Submitted (IndexNow)
926.7K (1.0% conv.)
926.7K
Accepted by IndexNow — 1.0% of live pages
Crawled (unique pages, all-time)
12.5M (1344.2% conv.)
12.5M
12.5M new in current period
Cited by AI (all time)
2.5M (20.0% conv.)
2.5M
2.7M total AI hits · 22.0% yield
Biggest gap: Potential → Live — 410.8M pages not yet converted (18.6% conversion rate)
Per-Segment Yield
| Segment | Pages | Hits | AI | Yield |
|---|---|---|---|---|
| other | 9.3M | 11.0M | 3.1M | 27.9% |
| localized | 1.9M | 2.1M | 628.6K | 29.5% |
| pattern_en | 455.8K | 877.5K | 226.0K | 25.8% |
| safe_en | 394.7K | 770.1K | 135.4K | 17.6% |
| compare | 148.5K | 410.3K | 66.8K | 16.3% |
| alternatives | 45.1K | 123.7K | 43.6K | 35.3% |
| crypto | 15.1K | 283.1K | 36.5K | 12.9% |
| huggingface | 157.3K | 269.7K | 17.7K | 6.5% |
| best | 486 | 5.2K | 951 | 18.4% |
| npm | 415 | 553 | 56 | 10.1% |
Per-Language Yield
| Lang | Crawled | AI | Yield |
|---|---|---|---|
| en | 10.2M | 3.3M | 32.5% |
| de | 777.0K | 405.1K | 52.1% |
| pt | 336.7K | 142.6K | 42.3% |
| es | 327.0K | 127.7K | 39.0% |
| ja | 294.7K | 112.6K | 38.2% |
| fr | 503.3K | 104.2K | 20.7% |
Master Pipeline — Universe to Citation
| Category | Universe | In DB | Capture% | Enriched | Enrich% | Pages* | Avg Score | Bottleneck |
|---|---|---|---|---|---|---|---|---|
| nuget | 400.0K | 641.6K | 160.4% | 641.6K | 100.0% | 10.2M | 46.4 | Indexing |
| npm | 3.0M | 528.3K | 17.6% | 528.3K | 100.0% | 112.5M | 50.7 | Indexing |
| website | 200.0M | 501.0K | 0.3% | 500.9K | 100.0% | 172.8M | 59.0 | Capture |
| crates | 150.0K | 204.1K | 136.1% | 204.1K | 100.0% | 55.7M | 52.2 | Indexing |
| packagist | 400.0K | 113.8K | 28.5% | 113.8K | 100.0% | 28.1M | 52.1 | Indexing |
| pypi | 500.0K | 93.8K | 18.8% | 93.8K | 100.0% | 24.1M | 53.8 | Indexing |
| android | 3.5M | 57.6K | 1.6% | 57.5K | 100.0% | 23.7M | 48.5 | Capture |
| wordpress | 60.0K | 57.1K | 95.1% | 57.1K | 100.0% | 15.7M | 54.2 | Indexing |
| vscode | 50.0K | 48.9K | 97.9% | 48.9K | 100.0% | 8.0M | 54.2 | Indexing |
| ios | 2.0M | 48.1K | 2.4% | 48.1K | 100.0% | 19.9M | 51.6 | Capture |
| steam | 70.0K | 45.4K | 64.8% | 45.4K | 100.0% | 8.3M | 53.7 | Indexing |
| chrome | 200.0K | 44.2K | 22.1% | 44.2K | 100.0% | 8.1M | 47.7 | Indexing |
| firefox | 35.0K | 29.1K | 83.2% | 29.1K | 100.0% | 5.3M | 50.5 | Indexing |
| go | 150.0K | 22.1K | 14.7% | 22.1K | 100.0% | 2.0M | 46.3 | Indexing |
| gems | 170.0K | 10.1K | 5.9% | 10.1K | 100.0% | 2.9M | 64.9 | Capture |
| homebrew | 10.0K | 8.3K | 82.9% | 8.3K | 100.0% | 1.8M | 55.1 | Indexing |
| saas | 10.0K | 5.0K | 49.6% | 4.9K | 99.3% | 1.7M | 48.2 | Indexing |
| city | 10.0K | 3.0K | 29.8% | 3.0K | 100.0% | 2.5M | 47.2 | Indexing |
| ai_tool | 5.0K | 2.4K | 47.0% | 2.3K | 99.2% | 346.4K | 46.9 | Indexing |
| ingredient | 5.0K | 669 | 13.4% | 669 | 100.0% | 230.8K | 48.2 | Indexing |
| supplement | 90.0K | 584 | 0.6% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| cosmetic_ingredient | 30.0K | 584 | 1.9% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| charity | 1.5M | 504 | 0.0% | 503 | 99.8% | 173.5K | 48.2 | Capture |
| crypto | 20.0K | 226 | 1.1% | 224 | 99.1% | 51.5K | 49.2 | Capture |
| country | 195 | 158 | 81.0% | 158 | 100.0% | 130.8K | 48.2 | Indexing |
| vpn | 100 | 79 | 79.0% | 79 | 100.0% | 24.2K | 62.4 | Indexing |
| password_manager | 0 | 55 | 0.0% | 55 | 100.0% | 12.7K | 59.4 | Capture |
| website_builder | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 61.7 | Capture |
| hosting | 0 | 51 | 0.0% | 49 | 96.1% | 11.7K | 61.7 | Capture |
| antivirus | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 63.6 | Capture |
* Pages = indexable (score≥30 + desc) × patterns × 23 languages
Citation Quality — by Endpoint
| Endpoint | Claude | ChatGPT | Perplexity | Total |
|---|---|---|---|---|
| other | 2.2M | 50.6K | 83.8K | 3.7M |
| /safe/* | 72.5K | 8.8K | 20.3K | 135.3K |
| /is-*-safe | 27.0K | 6.5K | 15.1K | 113.0K |
| /compare/* | 12.9K | 4.6K | 25.0K | 66.8K |
| /alternatives/* | 20.0K | 3.2K | 6.6K | 43.6K |
| /v1/preflight | 11.6K | 1.3K | 5.6K | 36.8K |
| /review/* | 18.9K | 647 | 4.0K | 36.7K |
| /token/* | 10.6K | 2.1K | 12.4K | 36.5K |
| /privacy/* | 20.1K | 595 | 4.2K | 35.4K |
| /model/* | 187 | 5.3K | 1.1K | 7.1K |
| homepage | 5 | 4.9K | 43 | 5.0K |
| ZARQ | 608 | 2.2K | 1.3K | 4.9K |
Crawl Budget Efficiency
| Bot | Total | 200 | 404 | 429 | Efficiency |
|---|---|---|---|---|---|
| Meta | 3.8M | 3.4M | 207.2K | 86.6K | 91.6% |
| Claude | 2.5M | 2.4M | 60.4K | 0 | 96.4% |
| Yandex | 2.0M | 2.0M | 21.4K | 544 | 98.0% |
| ChatGPT | 1.9M | 1.5M | 62.9K | 1 | 80.1% |
| Bing | 1.6M | 869.0K | 77.9K | 0 | 54.8% |
| 458.2K | 419.5K | 8.0K | 29.7K | 91.5% | |
| Perplexity | 181.1K | 179.3K | 1.6K | 0 | 99.0% |
| ByteDance | 129.4K | 116.5K | 12.1K | 0 | 90.1% |
Demand Without Supply (AI 404s)
| Path | Bot | Requests |
|---|---|---|
| /homebrew | Perplexity | 20 |
| /model/com-github-asus4-onnxruntime-unity | ChatGPT | 8 |
| /model/zk-kit-poseidon-cipher | ChatGPT | 8 |
| /container/tensorrt-llm-benchmarks | ChatGPT | 6 |
| /v1/crypto/compare/gmt-token/bitcoin | ByteDance | 6 |
| /compare/ats-god-optimizer-vs-openscad-mcp | ByteDance | 5 |
| /compare/nanobanana-mcp-server-vs-nvidia-nemo-agent-toolkit | ByteDance | 5 |
| /model/jovotech-platform-googleassistant | ChatGPT | 5 |
| /token/croswap | ByteDance | 5 |
| /best/ai-automation | ChatGPT | 4 |
| /best/other | ChatGPT | 4 |
| /best/research | ChatGPT | 4 |
| /best/translation | ChatGPT | 4 |
| /compare/aashari-mcp-server-atlassian-jira-vs-proffesor-for- | ByteDance | 4 |
Enrichment Pipeline Status
| Registry | Total | Enriched | % | Status |
|---|---|---|---|---|
| nuget | 641.6K | 641.6K | 100.0% | ✓ Complete |
| npm | 528.3K | 528.3K | 100.0% | ✓ Complete |
| website | 501.0K | 500.9K | 100.0% | ✓ Complete |
| crates | 204.1K | 204.1K | 100.0% | ✓ Complete |
| packagist | 113.8K | 113.8K | 100.0% | ✓ Complete |
| pypi | 93.8K | 93.8K | 100.0% | ✓ Complete |
| android | 57.6K | 57.5K | 100.0% | ✓ Complete |
| wordpress | 57.1K | 57.1K | 100.0% | ✓ Complete |
| vscode | 48.9K | 48.9K | 100.0% | ✓ Complete |
| ios | 48.1K | 48.1K | 100.0% | ✓ Complete |
| steam | 45.4K | 45.4K | 100.0% | ✓ Complete |
| chrome | 44.2K | 44.2K | 100.0% | ✓ Complete |
| firefox | 29.1K | 29.1K | 100.0% | ✓ Complete |
| go | 22.1K | 22.1K | 100.0% | ✓ Complete |
| gems | 10.1K | 10.1K | 100.0% | ✓ Complete |
| homebrew | 8.3K | 8.3K | 100.0% | ✓ Complete |
| saas | 5.0K | 4.9K | 99.3% | ✓ Complete |
| city | 3.0K | 3.0K | 100.0% | ✓ Complete |
| ai_tool | 2.4K | 2.3K | 99.2% | ✓ Complete |
| ingredient | 669 | 669 | 100.0% | ✓ Complete |
| supplement | 584 | 584 | 100.0% | ✓ Complete |
| cosmetic_ingredient | 584 | 584 | 100.0% | ✓ Complete |
| charity | 504 | 503 | 99.8% | ✓ Complete |
| crypto | 226 | 224 | 99.1% | ✓ Complete |
| country | 158 | 158 | 100.0% | ✓ Complete |
| vpn | 79 | 79 | 100.0% | ✓ Complete |
| password_manager | 55 | 55 | 100.0% | ✓ Complete |
| website_builder | 51 | 51 | 100.0% | ✓ Complete |
| hosting | 51 | 49 | 96.1% | ▶ In progress |
| antivirus | 51 | 51 | 100.0% | ✓ Complete |
0 enrichment processes running. Master watchdog active.
Kings (High-Yield Entities)
Total Kings
37.7K
Kings Enriched
37.6K
King Pages
3.0M
Kings % of Indexable
1.5%
| Registry | Kings | Enriched | Indexable |
|---|---|---|---|
| android | 13050 | 13040 | 13050 |
| website | 10879 | 10868 | 10875 |
| ios | 5427 | 5427 | 5427 |
| saas | 2806 | 2772 | 2783 |
| ai_tool | 787 | 781 | 787 |
| npm | 501 | 501 | 495 |
| wordpress | 500 | 500 | 500 |
| steam | 500 | 500 | 500 |
| charity | 493 | 492 | 492 |
| chrome | 472 | 472 | 472 |
| pypi | 300 | 300 | 294 |
| vscode | 200 | 200 | 200 |
| nuget | 200 | 200 | 5 |
| firefox | 200 | 200 | 200 |
| crypto | 196 | 196 | 196 |
| city | 185 | 185 | 185 |
| country | 158 | 158 | 158 |
| supplement | 137 | 137 | 137 |
| crates | 100 | 100 | 100 |
| packagist | 100 | 100 | 100 |
| go | 100 | 100 | 100 |
| gems | 100 | 100 | 100 |
| homebrew | 80 | 80 | 80 |
| vpn | 79 | 79 | 62 |
| cosmetic_ingredient | 73 | 73 | 73 |
| ingredient | 65 | 65 | 65 |
Kings = high-yield entities enriched with priority. Yield measurement available after 3-5 days of AI re-crawling.
Growth Engine
Enriched
2.5M
+0 today
Indexable Pages
720.9M
Citations/day
89.2K
Yield (per 1K pages)
0.12
falling
Enriched → Indexable Pages → AI Citations. Yield = citations per 1K indexable pages per day.
Crawl Trend — Daily Bot Requests (200 OK)
AI Citations: 624.3K/7d
AI Indexing: 569.5K/7d
Search: 2.8M/7d
Meta: 1.1M/7d
Amazon: 165.5K/7d
All bots: 6.0M/7d
Language Rollout — Submitted → Crawled → Cited
Languages Live: 22
Submitted URLs: 54.3M
AI Crawls (all langs): 4.2M/day
Citations (all langs): 2.7M/day
| Language | Sitemaps | URLs | AI Crawls | Citations | Human | Status |
|---|---|---|---|---|---|---|
| en English | 246 | 2.5M | 1.1M | 790.2K | 567.6K | ✓ Live |
| de German | 246 | 2.5M | 405.1K | 128.9K | 461 | ✓ Live |
| it Italian | 246 | 2.5M | 218.4K | 176.8K | 567 | ✓ Live |
| cs Czech | 246 | 2.5M | 183.9K | 137.9K | 796 | ✓ Live |
| ru Russian | 246 | 2.5M | 168.7K | 67.4K | 595 | ✓ Live |
| da Danish | 246 | 2.5M | 162.4K | 125.0K | 349 | ✓ Live |
| th Thai | 246 | 2.5M | 154.4K | 108.8K | 524 | ✓ Live |
| ar Arabic | 246 | 2.5M | 151.6K | 111.3K | 1.0K | ✓ Live |
| id Indonesian | 246 | 2.5M | 147.7K | 74.8K | 698 | ✓ Live |
| ro Romanian | 246 | 2.5M | 145.7K | 99.6K | 535 | ✓ Live |
| pt Portuguese | 246 | 2.5M | 142.6K | 102.1K | 550 | ✓ Live |
| tr Turkish | 246 | 2.5M | 138.3K | 94.5K | 543 | ✓ Live |
| pl Polish | 246 | 2.5M | 133.3K | 93.6K | 402 | ✓ Live |
| ko Korean | 246 | 2.5M | 128.8K | 90.2K | 524 | ✓ Live |
| es Spanish | 246 | 2.5M | 127.7K | 82.7K | 2.9K | ✓ Live |
| nl Dutch | 246 | 2.5M | 117.3K | 76.2K | 513 | ✓ Live |
| ja Japanese | 246 | 2.5M | 112.7K | 75.1K | 655 | ✓ Live |
| sv Swedish | 246 | 2.5M | 110.5K | 70.3K | 661 | ✓ Live |
| zh Chinese | 246 | 2.5M | 104.3K | 66.2K | 338 | ✓ Live |
| fr French | 246 | 2.5M | 104.2K | 65.9K | 606 | ✓ Live |
| vi Vietnamese | 246 | 2.5M | 100.2K | 62.4K | 575 | ✓ Live |
| hi Hindi | 246 | 2.5M | 82.9K | 40.1K | 599 | ✓ Live |
Infrastructure
Requests (all time)
18220043
0.3x headroom to 5M limit (CF edge)
System Load
6.34
8 cores available
PostgreSQL Connections
38
Max 100 (default)
Disk Usage
1%
Page Cache
0.8% hit rate
0.8% hit rate
77.7K pages · 2.00G memory · 91.3K hits / 11.3M misses
Scaling triggers:
>500K req/day → page cache (DONE) ·
Cloudflare edge cache (DONE) ·
>2M req/day → migrate to VPS (€18/mo) ·
>5M req/day → dedicated server ·
>15M req/day → Cloudflare Workers
Cached 120min. Updated: 08:14.