Machine-First Flywheel Dashboard
Full pipeline: Universe → Captured → Enriched → Pages → Indexed → Cited. Period: 24h
200: 262.0K404: 13.1K301: 3.5K304: 59401: 21
Total AI Usage
22.3K
+34% vs prev
Citations: 21.5K · Preflight: 750
AI Citations
21.5K
+32% vs prev
Claude: 20.0K
ByteDance: 1.3K
ChatGPT: 141
Perplexity: 20
AI Indexing
0
-100% vs prev
GPTBot (building index)
Preflight API
750
+143% vs prev
Claude: 650
ByteDance: 100
Non-AI preflight: 3.2K
Total Crawling
266.9K
-7% vs prev
AI Cite: 21.5K · AI Index: 0 · Search: 180.2K
Meta: 2.2K · Amazon: 12.1K · Other: 35.0K
Answer Depth
100.0%
2.5M enriched vs prev
of 2.5M entities
0 pipelines
Coverage
6.6M
30 registries · 6 verticals vs prev
278.7K requests · 13.0K IPs
Total requests: 278.7K
Bots: 266.9K (95%)
Humans: 11.8K (4%)
Unique IPs: 13.0K
AI Usage Trend
Last 24h: 22.3K
Previous 24h: 16.6K
+34%
ChatGPT: 141 +1%
·
Claude: 20.7K +38%
·
ByteDance: 1.4K -8%
·
Perplexity: 20 -26%
·
Apple: 121.9K +40%
·
AI Humans: 146 +8%
Conversion Funnel
Log-scale bars. 1.6M indexable of 2.5M total entities across 30 registries.
Potential Pages
504.6M
504.6M
1.6M indexable entities × patterns × 23 languages
Live (in sitemaps)
93.8M (18.6% conv.)
93.8M
EN: 22.0M core + Lang: 71.8M (22 langs × 2 patterns/entity)
Submitted (IndexNow)
428.4K (0.5% conv.)
428.4K
Accepted by IndexNow — 0.5% of live pages
Crawled (unique pages, all-time)
17.9M (4167.9% conv.)
17.9M
242.2K new in current period
Cited by AI (24h)
20.4K (0.1% conv.)
20.4K
21.8K total AI hits · 0.1% yield
Biggest gap: Potential → Live — 410.8M pages not yet converted (18.6% conversion rate)
Per-Segment Yield
| Segment | Pages | Hits | AI | Yield |
|---|---|---|---|---|
| other | 87.4K | 94.8K | 10.3K | 10.9% |
| pattern_en | 14.9K | 17.1K | 9.1K | 52.9% |
| safe_en | 9.5K | 9.8K | 1.6K | 16.1% |
| compare | 1.4K | 1.4K | 696 | 49.8% |
| alternatives | 2.1K | 2.2K | 689 | 30.9% |
| crypto | 1.6K | 1.7K | 110 | 6.5% |
| huggingface | 1.1K | 1.1K | 106 | 9.6% |
| localized | 126.8K | 128.4K | 77 | 0.1% |
| best | 36 | 43 | 16 | 37.2% |
| npm | 10 | 10 | 0 | 0.0% |
Per-Language Yield
| Lang | Crawled | AI | Yield |
|---|---|---|---|
| en | 181.0K | 22.5K | 12.4% |
| es | 22.4K | 125 | 0.6% |
| pt | 9.2K | 0 | 0.0% |
| ja | 8.7K | 0 | 0.0% |
| fr | 12.9K | 0 | 0.0% |
| de | 10.5K | 0 | 0.0% |
Master Pipeline — Universe to Citation
| Category | Universe | In DB | Capture% | Enriched | Enrich% | Pages* | Avg Score | Bottleneck |
|---|---|---|---|---|---|---|---|---|
| nuget | 400.0K | 641.6K | 160.4% | 641.6K | 100.0% | 10.2M | 46.4 | Indexing |
| npm | 3.0M | 528.3K | 17.6% | 528.3K | 100.0% | 112.5M | 50.7 | Indexing |
| website | 200.0M | 501.0K | 0.3% | 500.9K | 100.0% | 172.8M | 59.0 | Capture |
| crates | 150.0K | 204.1K | 136.1% | 204.1K | 100.0% | 55.7M | 52.2 | Indexing |
| packagist | 400.0K | 113.8K | 28.5% | 113.8K | 100.0% | 28.1M | 52.1 | Indexing |
| pypi | 500.0K | 93.8K | 18.8% | 93.8K | 100.0% | 24.1M | 53.8 | Indexing |
| android | 3.5M | 57.6K | 1.6% | 57.5K | 100.0% | 23.7M | 48.5 | Capture |
| wordpress | 60.0K | 57.1K | 95.1% | 57.1K | 100.0% | 15.7M | 54.2 | Indexing |
| vscode | 50.0K | 48.9K | 97.9% | 48.9K | 100.0% | 8.0M | 54.2 | Indexing |
| ios | 2.0M | 48.1K | 2.4% | 48.1K | 100.0% | 19.9M | 51.6 | Capture |
| steam | 70.0K | 45.4K | 64.8% | 45.4K | 100.0% | 8.3M | 53.7 | Indexing |
| chrome | 200.0K | 44.2K | 22.1% | 44.2K | 100.0% | 8.1M | 47.7 | Indexing |
| firefox | 35.0K | 29.1K | 83.2% | 29.1K | 100.0% | 5.3M | 50.5 | Indexing |
| go | 150.0K | 22.1K | 14.7% | 22.1K | 100.0% | 2.0M | 46.3 | Indexing |
| gems | 170.0K | 10.1K | 5.9% | 10.1K | 100.0% | 2.9M | 64.9 | Capture |
| homebrew | 10.0K | 8.3K | 82.9% | 8.3K | 100.0% | 1.8M | 55.1 | Indexing |
| saas | 10.0K | 5.0K | 49.6% | 5.0K | 100.0% | 1.7M | 48.2 | Indexing |
| ai_tool | 5.0K | 3.1K | 61.4% | 2.6K | 83.9% | 346.4K | 46.9 | Indexing |
| city | 10.0K | 3.0K | 29.8% | 3.0K | 100.0% | 2.5M | 47.2 | Indexing |
| ingredient | 5.0K | 669 | 13.4% | 669 | 100.0% | 230.8K | 48.2 | Indexing |
| supplement | 90.0K | 584 | 0.6% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| cosmetic_ingredient | 30.0K | 584 | 1.9% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| charity | 1.5M | 504 | 0.0% | 503 | 99.8% | 173.5K | 48.2 | Capture |
| crypto | 20.0K | 228 | 1.1% | 224 | 98.2% | 51.5K | 49.2 | Capture |
| country | 195 | 158 | 81.0% | 158 | 100.0% | 130.8K | 48.2 | Indexing |
| vpn | 100 | 79 | 79.0% | 79 | 100.0% | 24.2K | 62.4 | Indexing |
| password_manager | 0 | 55 | 0.0% | 55 | 100.0% | 12.7K | 59.4 | Capture |
| hosting | 0 | 51 | 0.0% | 49 | 96.1% | 11.7K | 61.7 | Capture |
| antivirus | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 63.6 | Capture |
| website_builder | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 61.7 | Capture |
* Pages = indexable (score≥30 + desc) × patterns × 23 languages
Citation Quality — by Endpoint
| Endpoint | Claude | ChatGPT | Perplexity | Total |
|---|---|---|---|---|
| other | 14.4K | 22 | 12 | 15.1K |
| /is-*-safe | 1.9K | 1 | 0 | 2.1K |
| /safe/* | 1.4K | 2 | 0 | 1.5K |
| /v1/preflight | 650 | 0 | 0 | 751 |
| /compare/* | 632 | 24 | 1 | 683 |
| /alternatives/* | 571 | 0 | 0 | 670 |
| /review/* | 512 | 0 | 0 | 604 |
| /privacy/* | 443 | 0 | 0 | 484 |
| /token/* | 57 | 9 | 0 | 109 |
| /model/* | 96 | 1 | 0 | 97 |
| homepage | 0 | 66 | 1 | 67 |
| ZARQ | 32 | 16 | 6 | 59 |
Crawl Budget Efficiency
| Bot | Total | 200 | 404 | 429 | Efficiency |
|---|---|---|---|---|---|
| Yandex | 36.4K | 34.0K | 2.3K | 0 | 93.4% |
| Bing | 32.1K | 29.9K | 2.1K | 0 | 93.2% |
| Claude | 23.1K | 20.7K | 1.2K | 0 | 89.6% |
| Meta | 2.3K | 2.2K | 42 | 0 | 98.1% |
| ByteDance | 1.6K | 1.4K | 115 | 0 | 88.3% |
| 420 | 389 | 26 | 0 | 92.6% | |
| ChatGPT | 159 | 141 | 10 | 0 | 88.7% |
| Perplexity | 24 | 20 | 4 | 0 | 83.3% |
Demand Without Supply (AI 404s)
| Path | Bot | Requests |
|---|---|---|
| /is-llama-3-1-nemotron-70b-instruct-hf-safe | Claude | 2 |
| /is-locators-safe | Claude | 2 |
| /is-microsoft-phi-4-multimodal-instruct-safe | Claude | 2 |
| /safe/com.alibaba.aliexpresshd | ChatGPT | 2 |
| /safe/com.imo.android.imoim | ChatGPT | 2 |
| /safe/dadf90db-eb1c-495f-bb3f-a5aa6d05036d | ByteDance | 2 |
| /safe/devtools-api | ByteDance | 2 |
| /safe/king-james-bible-verse-audio | ByteDance | 2 |
| /safe/krea-ai-realtime | ByteDance | 2 |
| /safe/projects-m-membership | ByteDance | 2 |
| /compare/2noise-chattts-vs-corentinj-real-time-voice-cloning | Claude | 1 |
| /compare/2noise-chattts-vs-enricollen-rag-conversational-age | Claude | 1 |
| /compare/2noise-chattts-vs-victorcarre6-localmind | Claude | 1 |
| /compare/RetailMeNot-vs-honey | Claude | 1 |
| /compare/a-ghorbani-pocketpal-ai-vs-pro-design-ai | ByteDance | 1 |
Enrichment Pipeline Status
| Registry | Total | Enriched | % | Status |
|---|---|---|---|---|
| nuget | 641.6K | 641.6K | 100.0% | ✓ Complete |
| npm | 528.3K | 528.3K | 100.0% | ✓ Complete |
| website | 501.0K | 500.9K | 100.0% | ✓ Complete |
| crates | 204.1K | 204.1K | 100.0% | ✓ Complete |
| packagist | 113.8K | 113.8K | 100.0% | ✓ Complete |
| pypi | 93.8K | 93.8K | 100.0% | ✓ Complete |
| android | 57.6K | 57.5K | 100.0% | ✓ Complete |
| wordpress | 57.1K | 57.1K | 100.0% | ✓ Complete |
| vscode | 48.9K | 48.9K | 100.0% | ✓ Complete |
| ios | 48.1K | 48.1K | 100.0% | ✓ Complete |
| steam | 45.4K | 45.4K | 100.0% | ✓ Complete |
| chrome | 44.2K | 44.2K | 100.0% | ✓ Complete |
| firefox | 29.1K | 29.1K | 100.0% | ✓ Complete |
| go | 22.1K | 22.1K | 100.0% | ✓ Complete |
| gems | 10.1K | 10.1K | 100.0% | ✓ Complete |
| homebrew | 8.3K | 8.3K | 100.0% | ✓ Complete |
| saas | 5.0K | 5.0K | 100.0% | ✓ Complete |
| ai_tool | 3.1K | 2.6K | 83.9% | ▶ In progress |
| city | 3.0K | 3.0K | 100.0% | ✓ Complete |
| ingredient | 669 | 669 | 100.0% | ✓ Complete |
| supplement | 584 | 584 | 100.0% | ✓ Complete |
| cosmetic_ingredient | 584 | 584 | 100.0% | ✓ Complete |
| charity | 504 | 503 | 99.8% | ✓ Complete |
| crypto | 228 | 224 | 98.2% | ▶ In progress |
| country | 158 | 158 | 100.0% | ✓ Complete |
| vpn | 79 | 79 | 100.0% | ✓ Complete |
| password_manager | 55 | 55 | 100.0% | ✓ Complete |
| hosting | 51 | 49 | 96.1% | ▶ In progress |
| antivirus | 51 | 51 | 100.0% | ✓ Complete |
| website_builder | 51 | 51 | 100.0% | ✓ Complete |
0 enrichment processes running. Master watchdog active.
Kings (High-Yield Entities)
Total Kings
37.7K
Kings Enriched
37.7K
King Pages
3.0M
Kings % of Indexable
1.5%
| Registry | Kings | Enriched | Indexable |
|---|---|---|---|
| android | 13050 | 13040 | 13050 |
| website | 10879 | 10868 | 10875 |
| ios | 5427 | 5427 | 5427 |
| saas | 2806 | 2805 | 2806 |
| ai_tool | 787 | 784 | 787 |
| npm | 501 | 501 | 495 |
| wordpress | 500 | 500 | 500 |
| steam | 500 | 500 | 500 |
| charity | 493 | 492 | 492 |
| chrome | 472 | 472 | 472 |
| pypi | 300 | 300 | 294 |
| vscode | 200 | 200 | 200 |
| nuget | 200 | 200 | 5 |
| firefox | 200 | 200 | 200 |
| crypto | 196 | 196 | 196 |
| city | 185 | 185 | 185 |
| country | 158 | 158 | 158 |
| supplement | 137 | 137 | 137 |
| crates | 100 | 100 | 100 |
| packagist | 100 | 100 | 100 |
| gems | 100 | 100 | 100 |
| go | 100 | 100 | 100 |
| homebrew | 80 | 80 | 80 |
| vpn | 79 | 79 | 62 |
| cosmetic_ingredient | 73 | 73 | 73 |
| ingredient | 65 | 65 | 65 |
Kings = high-yield entities enriched with priority. Yield measurement available after 3-5 days of AI re-crawling.
Growth Engine
Enriched
2.5M
+0 today
Indexable Pages
720.9M
Citations/day
29.9K
Yield (per 1K pages)
0.04
stable
Enriched → Indexable Pages → AI Citations. Yield = citations per 1K indexable pages per day.
Crawl Trend — Daily Bot Requests (200 OK)
AI Citations: 97.6K/7d
AI Indexing: 18.9K/7d
Search: 1.1M/7d
Meta: 76.5K/7d
Amazon: 88.7K/7d
All bots: 1.9M/7d
Language Rollout — Submitted → Crawled → Cited
Languages Live: 2
Submitted URLs: 54.3M
AI Crawls (all langs): 22.6K/day
Citations (all langs): 21.9K/day
| Language | Sitemaps | URLs | AI Crawls | Citations | Human | Status |
|---|---|---|---|---|---|---|
| en English | 246 | 2.5M | 22.5K | 21.8K | 10.9K | ✓ Live |
| es Spanish | 246 | 2.5M | 125 | 125 | 46 | ✓ Live |
| th Thai | 246 | 2.5M | 2 | 2 | 31 | ✓ Submitted |
| zh Chinese | 246 | 2.5M | 1 | 1 | 93 | ✓ Submitted |
| vi Vietnamese | 246 | 2.5M | 1 | 1 | 113 | ✓ Submitted |
| pl Polish | 246 | 2.5M | 1 | 1 | 24 | ✓ Submitted |
| nl Dutch | 246 | 2.5M | 1 | 1 | 24 | ✓ Submitted |
| da Danish | 246 | 2.5M | 1 | 1 | 103 | ✓ Submitted |
| ar Arabic | 246 | 2.5M | 1 | 1 | 123 | ✓ Submitted |
| tr Turkish | 246 | 2.5M | 0 | 0 | 57 | ✓ Submitted |
| sv Swedish | 246 | 2.5M | 0 | 0 | 7 | ✓ Submitted |
| ru Russian | 246 | 2.5M | 0 | 0 | 22 | ✓ Submitted |
| ro Romanian | 246 | 2.5M | 0 | 0 | 10 | ✓ Submitted |
| pt Portuguese | 246 | 2.5M | 0 | 0 | 25 | ✓ Submitted |
| ko Korean | 246 | 2.5M | 0 | 0 | 35 | ✓ Submitted |
| ja Japanese | 246 | 2.5M | 0 | 0 | 43 | ✓ Submitted |
| it Italian | 246 | 2.5M | 0 | 0 | 8 | ✓ Submitted |
| id Indonesian | 246 | 2.5M | 0 | 0 | 44 | ✓ Submitted |
| hi Hindi | 246 | 2.5M | 0 | 0 | 4 | ✓ Submitted |
| fr French | 246 | 2.5M | 0 | 0 | 25 | ✓ Submitted |
| de German | 246 | 2.5M | 0 | 0 | 32 | ✓ Submitted |
| cs Czech | 246 | 2.5M | 0 | 0 | 123 | ✓ Submitted |
Infrastructure
Requests (24h)
278738
17.9x headroom to 5M limit (CF edge)
System Load
7.13
8 cores available
PostgreSQL Connections
11
Max 100 (default)
Disk Usage
1%
Page Cache
1.9% hit rate
1.9% hit rate
47.1K pages · 2.00G memory · 561.5K hits / 28.5M misses
Scaling triggers:
>500K req/day → page cache (DONE) ·
Cloudflare edge cache (DONE) ·
>2M req/day → migrate to VPS (€18/mo) ·
>5M req/day → dedicated server ·
>15M req/day → Cloudflare Workers
Cached 5min. Updated: 05:21.