Machine-First Flywheel Dashboard
Full pipeline: Universe → Captured → Enriched → Pages → Indexed → Cited. Period: 7 days
200: 6.7M301: 116.5K404: 103.5K429: 7.5K304: 679
Total AI Usage
738.4K
-54% vs prev
Citations: 734.7K · Preflight: 3.8K
AI Citations
734.7K
-54% vs prev
Claude: 646.5K
Perplexity: 36.3K
ByteDance: 31.2K
ChatGPT: 20.3K
AI Indexing
666.0K
+41% vs prev
GPTBot (building index)
Preflight API
3.8K
-70% vs prev
ByteDance: 2.2K
Perplexity: 578
Claude: 526
Non-AI preflight: 32.2K
Total Crawling
6.8M
+29% vs prev
AI Cite: 734.6K · AI Index: 666.1K · Search: 3.1M
Meta: 1.1M · Amazon: 184.7K · Other: 735.6K
Answer Depth
100.0%
2.5M enriched vs prev
of 2.5M entities
0 pipelines
Coverage
6.6M
30 registries · 6 verticals vs prev
7.0M requests · 151.7K IPs
Total requests: 7.0M
Bots: 6.8M (97%)
Humans: 164.0K (2%)
Unique IPs: 151.7K
AI Usage Trend
Last 24h: 738.4K
Previous 24h: 1.6M
-54%
ChatGPT: 20.6K -9%
·
Claude: 647.0K -56%
·
ByteDance: 33.4K -12%
·
Perplexity: 36.9K -38%
·
Apple: 2.0M +135%
·
AI Humans: 8.3K -1%
Conversion Funnel
Log-scale bars. 1.6M indexable of 2.5M total entities across 30 registries.
Potential Pages
504.6M
504.6M
1.6M indexable entities × patterns × 23 languages
Live (in sitemaps)
93.8M (18.6% conv.)
93.8M
EN: 22.0M core + Lang: 71.8M (22 langs × 2 patterns/entity)
Submitted (IndexNow)
926.7K (1.0% conv.)
926.7K
Accepted by IndexNow — 1.0% of live pages
Crawled (unique pages, all-time)
12.4M (1343.4% conv.)
12.4M
5.9M new in current period
Cited by AI (7 days)
712.4K (5.7% conv.)
712.4K
734.5K total AI hits · 5.9% yield
Biggest gap: Potential → Live — 410.8M pages not yet converted (18.6% conversion rate)
Per-Segment Yield
| Segment | Pages | Hits | AI | Yield |
|---|---|---|---|---|
| other | 4.3M | 4.7M | 1.1M | 23.8% |
| localized | 1.0M | 1.1M | 205.0K | 18.9% |
| safe_en | 214.7K | 283.2K | 24.6K | 8.7% |
| pattern_en | 137.5K | 205.5K | 24.5K | 11.9% |
| compare | 90.9K | 139.7K | 21.0K | 15.1% |
| alternatives | 25.1K | 43.0K | 6.3K | 14.7% |
| huggingface | 64.5K | 78.7K | 3.0K | 3.8% |
| crypto | 15.1K | 67.8K | 2.6K | 3.9% |
| best | 135 | 1.1K | 215 | 20.3% |
| npm | 178 | 208 | 18 | 8.7% |
Per-Language Yield
| Lang | Crawled | AI | Yield |
|---|---|---|---|
| en | 4.7M | 1.1M | 24.2% |
| de | 302.0K | 69.7K | 23.1% |
| pt | 206.1K | 60.0K | 29.1% |
| es | 202.2K | 54.3K | 26.8% |
| ja | 195.8K | 47.7K | 24.4% |
| fr | 293.2K | 46.1K | 15.7% |
Master Pipeline — Universe to Citation
| Category | Universe | In DB | Capture% | Enriched | Enrich% | Pages* | Avg Score | Bottleneck |
|---|---|---|---|---|---|---|---|---|
| nuget | 400.0K | 641.6K | 160.4% | 641.6K | 100.0% | 10.2M | 46.4 | Indexing |
| npm | 3.0M | 528.3K | 17.6% | 528.3K | 100.0% | 112.5M | 50.7 | Indexing |
| website | 200.0M | 501.0K | 0.3% | 500.9K | 100.0% | 172.8M | 59.0 | Capture |
| crates | 150.0K | 204.1K | 136.1% | 204.1K | 100.0% | 55.7M | 52.2 | Indexing |
| packagist | 400.0K | 113.8K | 28.5% | 113.8K | 100.0% | 28.1M | 52.1 | Indexing |
| pypi | 500.0K | 93.8K | 18.8% | 93.8K | 100.0% | 24.1M | 53.8 | Indexing |
| android | 3.5M | 57.6K | 1.6% | 57.5K | 100.0% | 23.7M | 48.5 | Capture |
| wordpress | 60.0K | 57.1K | 95.1% | 57.1K | 100.0% | 15.7M | 54.2 | Indexing |
| vscode | 50.0K | 48.9K | 97.9% | 48.9K | 100.0% | 8.0M | 54.2 | Indexing |
| ios | 2.0M | 48.1K | 2.4% | 48.1K | 100.0% | 19.9M | 51.6 | Capture |
| steam | 70.0K | 45.4K | 64.8% | 45.4K | 100.0% | 8.3M | 53.7 | Indexing |
| chrome | 200.0K | 44.2K | 22.1% | 44.2K | 100.0% | 8.1M | 47.7 | Indexing |
| firefox | 35.0K | 29.1K | 83.2% | 29.1K | 100.0% | 5.3M | 50.5 | Indexing |
| go | 150.0K | 22.1K | 14.7% | 22.1K | 100.0% | 2.0M | 46.3 | Indexing |
| gems | 170.0K | 10.1K | 5.9% | 10.1K | 100.0% | 2.9M | 64.9 | Capture |
| homebrew | 10.0K | 8.3K | 82.9% | 8.3K | 100.0% | 1.8M | 55.1 | Indexing |
| saas | 10.0K | 5.0K | 49.6% | 4.9K | 99.3% | 1.7M | 48.2 | Indexing |
| city | 10.0K | 3.0K | 29.8% | 3.0K | 100.0% | 2.5M | 47.2 | Indexing |
| ai_tool | 5.0K | 2.4K | 47.0% | 2.3K | 99.2% | 346.4K | 46.9 | Indexing |
| ingredient | 5.0K | 669 | 13.4% | 669 | 100.0% | 230.8K | 48.2 | Indexing |
| supplement | 90.0K | 584 | 0.6% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| cosmetic_ingredient | 30.0K | 584 | 1.9% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| charity | 1.5M | 504 | 0.0% | 503 | 99.8% | 173.5K | 48.2 | Capture |
| crypto | 20.0K | 226 | 1.1% | 224 | 99.1% | 51.5K | 49.2 | Capture |
| country | 195 | 158 | 81.0% | 158 | 100.0% | 130.8K | 48.2 | Indexing |
| vpn | 100 | 79 | 79.0% | 79 | 100.0% | 24.2K | 62.4 | Indexing |
| password_manager | 0 | 55 | 0.0% | 55 | 100.0% | 12.7K | 59.4 | Capture |
| website_builder | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 61.7 | Capture |
| hosting | 0 | 51 | 0.0% | 49 | 96.1% | 11.7K | 61.7 | Capture |
| antivirus | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 63.6 | Capture |
* Pages = indexable (score≥30 + desc) × patterns × 23 languages
Citation Quality — by Endpoint
| Endpoint | Claude | ChatGPT | Perplexity | Total |
|---|---|---|---|---|
| other | 624.7K | 11.5K | 23.0K | 1.3M |
| /safe/* | 9.7K | 2.1K | 3.8K | 24.5K |
| /compare/* | 3.4K | 1.1K | 5.3K | 21.0K |
| /is-*-safe | 2.5K | 1.2K | 1.5K | 11.0K |
| /alternatives/* | 2.1K | 586 | 817 | 6.3K |
| /review/* | 1.9K | 156 | 354 | 6.0K |
| /privacy/* | 2.2K | 143 | 368 | 4.8K |
| /v1/preflight | 506 | 319 | 578 | 3.7K |
| /token/* | 0 | 388 | 639 | 2.6K |
| homepage | 0 | 1.6K | 7 | 1.6K |
| ZARQ | 70 | 609 | 276 | 1.1K |
| /model/* | 4 | 844 | 140 | 1.0K |
Crawl Budget Efficiency
| Bot | Total | 200 | 404 | 429 | Efficiency |
|---|---|---|---|---|---|
| Meta | 1.1M | 1.1M | 4.2K | 5.7K | 99.0% |
| Yandex | 751.9K | 749.4K | 2.2K | 0 | 99.7% |
| ChatGPT | 691.5K | 686.8K | 2.5K | 0 | 99.3% |
| Claude | 655.2K | 647.0K | 7.3K | 0 | 98.7% |
| Bing | 439.3K | 337.5K | 1.7K | 0 | 76.8% |
| 98.9K | 98.3K | 476 | 0 | 99.4% | |
| Perplexity | 37.1K | 36.9K | 205 | 0 | 99.4% |
| ByteDance | 36.2K | 33.4K | 2.6K | 0 | 92.3% |
Demand Without Supply (AI 404s)
| Path | Bot | Requests |
|---|---|---|
| /homebrew | Perplexity | 8 |
| /model/zk-kit-poseidon-cipher | ChatGPT | 7 |
| /model/jovotech-platform-googleassistant | ChatGPT | 5 |
| /model/zilliz-milvus-sdk-node | ChatGPT | 4 |
| /compare/ats-resume-generation-agent-vs-historybuff | ByteDance | 3 |
| /compare/draft-mcp-server-vs-ruslanmv-news-and-trends | ByteDance | 3 |
| /compare/ibm-mcp-context-forge-vs-mcp-server-tool-vision | ByteDance | 3 |
| /compare/md-exporter-vs-transcrab | ByteDance | 3 |
| /compare/symfony-polyfill-mbstring-vs-geocoder-php-common-ht | ByteDance | 3 |
| /model/react-native-executorch-recognizer-crnn-en | Perplexity | 3 |
| /package/aiphad | Perplexity | 3 |
| /safe/deepseek-r1-distill-llama-8b/privacy | ByteDance | 3 |
| /safe/gsm8k/privacy | ByteDance | 3 |
| /safe/ltx-2/security | ByteDance | 3 |
Enrichment Pipeline Status
| Registry | Total | Enriched | % | Status |
|---|---|---|---|---|
| nuget | 641.6K | 641.6K | 100.0% | ✓ Complete |
| npm | 528.3K | 528.3K | 100.0% | ✓ Complete |
| website | 501.0K | 500.9K | 100.0% | ✓ Complete |
| crates | 204.1K | 204.1K | 100.0% | ✓ Complete |
| packagist | 113.8K | 113.8K | 100.0% | ✓ Complete |
| pypi | 93.8K | 93.8K | 100.0% | ✓ Complete |
| android | 57.6K | 57.5K | 100.0% | ✓ Complete |
| wordpress | 57.1K | 57.1K | 100.0% | ✓ Complete |
| vscode | 48.9K | 48.9K | 100.0% | ✓ Complete |
| ios | 48.1K | 48.1K | 100.0% | ✓ Complete |
| steam | 45.4K | 45.4K | 100.0% | ✓ Complete |
| chrome | 44.2K | 44.2K | 100.0% | ✓ Complete |
| firefox | 29.1K | 29.1K | 100.0% | ✓ Complete |
| go | 22.1K | 22.1K | 100.0% | ✓ Complete |
| gems | 10.1K | 10.1K | 100.0% | ✓ Complete |
| homebrew | 8.3K | 8.3K | 100.0% | ✓ Complete |
| saas | 5.0K | 4.9K | 99.3% | ✓ Complete |
| city | 3.0K | 3.0K | 100.0% | ✓ Complete |
| ai_tool | 2.4K | 2.3K | 99.2% | ✓ Complete |
| ingredient | 669 | 669 | 100.0% | ✓ Complete |
| supplement | 584 | 584 | 100.0% | ✓ Complete |
| cosmetic_ingredient | 584 | 584 | 100.0% | ✓ Complete |
| charity | 504 | 503 | 99.8% | ✓ Complete |
| crypto | 226 | 224 | 99.1% | ✓ Complete |
| country | 158 | 158 | 100.0% | ✓ Complete |
| vpn | 79 | 79 | 100.0% | ✓ Complete |
| password_manager | 55 | 55 | 100.0% | ✓ Complete |
| website_builder | 51 | 51 | 100.0% | ✓ Complete |
| hosting | 51 | 49 | 96.1% | ▶ In progress |
| antivirus | 51 | 51 | 100.0% | ✓ Complete |
0 enrichment processes running. Master watchdog active.
Kings (High-Yield Entities)
Total Kings
37.7K
Kings Enriched
37.6K
King Pages
3.0M
Kings % of Indexable
1.5%
| Registry | Kings | Enriched | Indexable |
|---|---|---|---|
| android | 13050 | 13040 | 13050 |
| website | 10879 | 10868 | 10875 |
| ios | 5427 | 5427 | 5427 |
| saas | 2806 | 2772 | 2783 |
| ai_tool | 787 | 781 | 787 |
| npm | 501 | 501 | 495 |
| wordpress | 500 | 500 | 500 |
| steam | 500 | 500 | 500 |
| charity | 493 | 492 | 492 |
| chrome | 472 | 472 | 472 |
| pypi | 300 | 300 | 294 |
| vscode | 200 | 200 | 200 |
| nuget | 200 | 200 | 5 |
| firefox | 200 | 200 | 200 |
| crypto | 196 | 196 | 196 |
| city | 185 | 185 | 185 |
| country | 158 | 158 | 158 |
| supplement | 137 | 137 | 137 |
| crates | 100 | 100 | 100 |
| packagist | 100 | 100 | 100 |
| go | 100 | 100 | 100 |
| gems | 100 | 100 | 100 |
| homebrew | 80 | 80 | 80 |
| vpn | 79 | 79 | 62 |
| cosmetic_ingredient | 73 | 73 | 73 |
| ingredient | 65 | 65 | 65 |
Kings = high-yield entities enriched with priority. Yield measurement available after 3-5 days of AI re-crawling.
Growth Engine
Enriched
2.5M
+0 today
Indexable Pages
720.9M
Citations/day
89.2K
Yield (per 1K pages)
0.12
falling
Enriched → Indexable Pages → AI Citations. Yield = citations per 1K indexable pages per day.
Crawl Trend — Daily Bot Requests (200 OK)
AI Citations: 623.3K/7d
AI Indexing: 565.2K/7d
Search: 2.8M/7d
Meta: 1.1M/7d
Amazon: 165.2K/7d
All bots: 6.0M/7d
Language Rollout — Submitted → Crawled → Cited
Languages Live: 22
Submitted URLs: 54.3M
AI Crawls (all langs): 1.4M/day
Citations (all langs): 734.4K/day
| Language | Sitemaps | URLs | AI Crawls | Citations | Human | Status |
|---|---|---|---|---|---|---|
| en English | 246 | 2.5M | 228.2K | 159.0K | 158.1K | ✓ Live |
| it Italian | 246 | 2.5M | 72.3K | 43.4K | 184 | ✓ Live |
| de German | 246 | 2.5M | 69.7K | 42.9K | 215 | ✓ Live |
| cs Czech | 246 | 2.5M | 67.7K | 39.2K | 335 | ✓ Live |
| da Danish | 246 | 2.5M | 66.1K | 37.6K | 185 | ✓ Live |
| ar Arabic | 246 | 2.5M | 65.7K | 37.9K | 513 | ✓ Live |
| id Indonesian | 246 | 2.5M | 60.5K | 21.1K | 289 | ✓ Live |
| th Thai | 246 | 2.5M | 60.3K | 31.9K | 144 | ✓ Live |
| pt Portuguese | 246 | 2.5M | 60.0K | 31.8K | 282 | ✓ Live |
| ro Romanian | 246 | 2.5M | 59.9K | 30.5K | 227 | ✓ Live |
| tr Turkish | 246 | 2.5M | 56.4K | 28.0K | 231 | ✓ Live |
| pl Polish | 246 | 2.5M | 54.6K | 27.0K | 187 | ✓ Live |
| es Spanish | 246 | 2.5M | 54.3K | 23.0K | 995 | ✓ Live |
| ko Korean | 246 | 2.5M | 52.7K | 26.0K | 263 | ✓ Live |
| nl Dutch | 246 | 2.5M | 50.5K | 21.7K | 188 | ✓ Live |
| sv Swedish | 246 | 2.5M | 49.9K | 22.2K | 223 | ✓ Live |
| zh Chinese | 246 | 2.5M | 49.3K | 21.1K | 179 | ✓ Live |
| ru Russian | 246 | 2.5M | 48.9K | 19.2K | 291 | ✓ Live |
| ja Japanese | 246 | 2.5M | 47.7K | 21.4K | 209 | ✓ Live |
| fr French | 246 | 2.5M | 46.1K | 19.6K | 201 | ✓ Live |
| vi Vietnamese | 246 | 2.5M | 44.2K | 17.7K | 311 | ✓ Live |
| hi Hindi | 246 | 2.5M | 39.7K | 12.2K | 232 | ✓ Live |
Infrastructure
Requests (7 days)
6978458
0.7x headroom to 5M limit (CF edge)
System Load
8.71
8 cores available
PostgreSQL Connections
37
Max 100 (default)
Disk Usage
1%
Page Cache
0.8% hit rate
0.8% hit rate
79.3K pages · 2.00G memory · 91.0K hits / 11.3M misses
Scaling triggers:
>500K req/day → page cache (DONE) ·
Cloudflare edge cache (DONE) ·
>2M req/day → migrate to VPS (€18/mo) ·
>5M req/day → dedicated server ·
>15M req/day → Cloudflare Workers
Cached 30min. Updated: 07:50.