Machine-First Flywheel Dashboard
Full pipeline: Universe → Captured → Enriched → Pages → Indexed → Cited. Period: 24h
200: 245.0K404: 12.0K301: 3.0K304: 81401: 19
Total AI Usage
21.6K
+52% vs prev
Citations: 21.1K · Preflight: 562
AI Citations
21.1K
+51% vs prev
Claude: 19.6K
ByteDance: 1.3K
ChatGPT: 159
Perplexity: 23
AI Indexing
0
-100% vs prev
GPTBot (building index)
Preflight API
562
+85% vs prev
Claude: 477
ByteDance: 85
Non-AI preflight: 3.4K
Total Crawling
248.4K
-24% vs prev
AI Cite: 21.1K · AI Index: 0 · Search: 158.5K
Meta: 2.1K · Amazon: 12.0K · Other: 40.5K
Answer Depth
100.0%
2.5M enriched vs prev
of 2.5M entities
0 pipelines
Coverage
6.6M
30 registries · 6 verticals vs prev
260.1K requests · 12.9K IPs
Total requests: 260.1K
Bots: 248.4K (95%)
Humans: 11.7K (4%)
Unique IPs: 12.9K
AI Usage Trend
Last 24h: 21.6K
Previous 24h: 14.3K
+51%
ChatGPT: 159 +7%
·
Claude: 20.1K +62%
·
ByteDance: 1.3K -22%
·
Perplexity: 23 -23%
·
Apple: 101.2K -5%
·
AI Humans: 154 +20%
Conversion Funnel
Log-scale bars. 1.6M indexable of 2.5M total entities across 30 registries.
Potential Pages
504.6M
504.6M
1.6M indexable entities × patterns × 23 languages
Live (in sitemaps)
93.8M (18.6% conv.)
93.8M
EN: 22.0M core + Lang: 71.8M (22 langs × 2 patterns/entity)
Submitted (IndexNow)
428.4K (0.5% conv.)
428.4K
Accepted by IndexNow — 0.5% of live pages
Crawled (unique pages, all-time)
18.2M (4242.5% conv.)
18.2M
222.8K new in current period
Cited by AI (24h)
19.7K (0.1% conv.)
19.7K
20.9K total AI hits · 0.1% yield
Biggest gap: Potential → Live — 410.8M pages not yet converted (18.6% conversion rate)
Per-Segment Yield
| Segment | Pages | Hits | AI | Yield |
|---|---|---|---|---|
| other | 81.1K | 88.1K | 9.6K | 10.9% |
| pattern_en | 14.5K | 16.2K | 9.0K | 55.3% |
| safe_en | 8.6K | 8.9K | 1.4K | 16.1% |
| compare | 1.2K | 1.3K | 617 | 47.6% |
| alternatives | 1.9K | 2.0K | 567 | 28.8% |
| crypto | 1.6K | 1.7K | 125 | 7.4% |
| huggingface | 1.1K | 1.1K | 81 | 7.5% |
| localized | 112.7K | 113.4K | 67 | 0.1% |
| best | 36 | 40 | 11 | 27.5% |
| npm | 6 | 6 | 0 | 0.0% |
Per-Language Yield
| Lang | Crawled | AI | Yield |
|---|---|---|---|
| en | 162.5K | 21.4K | 13.1% |
| es | 23.4K | 105 | 0.4% |
| pt | 8.4K | 0 | 0.0% |
| ja | 8.2K | 0 | 0.0% |
| fr | 11.3K | 0 | 0.0% |
| de | 9.0K | 0 | 0.0% |
Master Pipeline — Universe to Citation
| Category | Universe | In DB | Capture% | Enriched | Enrich% | Pages* | Avg Score | Bottleneck |
|---|---|---|---|---|---|---|---|---|
| nuget | 400.0K | 641.6K | 160.4% | 641.6K | 100.0% | 10.2M | 46.4 | Indexing |
| npm | 3.0M | 528.3K | 17.6% | 528.3K | 100.0% | 112.5M | 50.7 | Indexing |
| website | 200.0M | 501.0K | 0.3% | 500.9K | 100.0% | 172.8M | 59.0 | Capture |
| crates | 150.0K | 204.1K | 136.1% | 204.1K | 100.0% | 55.7M | 52.2 | Indexing |
| packagist | 400.0K | 113.8K | 28.5% | 113.8K | 100.0% | 28.1M | 52.1 | Indexing |
| pypi | 500.0K | 93.8K | 18.8% | 93.8K | 100.0% | 24.1M | 53.8 | Indexing |
| android | 3.5M | 57.6K | 1.6% | 57.5K | 100.0% | 23.7M | 48.5 | Capture |
| wordpress | 60.0K | 57.1K | 95.1% | 57.1K | 100.0% | 15.7M | 54.2 | Indexing |
| vscode | 50.0K | 48.9K | 97.9% | 48.9K | 100.0% | 8.0M | 54.2 | Indexing |
| ios | 2.0M | 48.1K | 2.4% | 48.1K | 100.0% | 19.9M | 51.6 | Capture |
| steam | 70.0K | 45.4K | 64.8% | 45.4K | 100.0% | 8.3M | 53.7 | Indexing |
| chrome | 200.0K | 44.2K | 22.1% | 44.2K | 100.0% | 8.1M | 47.7 | Indexing |
| firefox | 35.0K | 29.1K | 83.2% | 29.1K | 100.0% | 5.3M | 50.5 | Indexing |
| go | 150.0K | 22.1K | 14.7% | 22.1K | 100.0% | 2.0M | 46.3 | Indexing |
| gems | 170.0K | 10.1K | 5.9% | 10.1K | 100.0% | 2.9M | 64.9 | Capture |
| homebrew | 10.0K | 8.3K | 82.9% | 8.3K | 100.0% | 1.8M | 55.1 | Indexing |
| saas | 10.0K | 5.0K | 49.6% | 5.0K | 100.0% | 1.7M | 48.2 | Indexing |
| ai_tool | 5.0K | 3.1K | 61.4% | 2.6K | 83.9% | 346.4K | 46.9 | Indexing |
| city | 10.0K | 3.0K | 29.8% | 3.0K | 100.0% | 2.5M | 47.2 | Indexing |
| ingredient | 5.0K | 669 | 13.4% | 669 | 100.0% | 230.8K | 48.2 | Indexing |
| supplement | 90.0K | 584 | 0.6% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| cosmetic_ingredient | 30.0K | 584 | 1.9% | 584 | 100.0% | 201.5K | 48.2 | Capture |
| charity | 1.5M | 504 | 0.0% | 503 | 99.8% | 173.5K | 48.2 | Capture |
| crypto | 20.0K | 228 | 1.1% | 224 | 98.2% | 51.5K | 49.2 | Capture |
| country | 195 | 158 | 81.0% | 158 | 100.0% | 130.8K | 48.2 | Indexing |
| vpn | 100 | 79 | 79.0% | 79 | 100.0% | 24.2K | 62.4 | Indexing |
| password_manager | 0 | 55 | 0.0% | 55 | 100.0% | 12.7K | 59.4 | Capture |
| hosting | 0 | 51 | 0.0% | 49 | 96.1% | 11.7K | 61.7 | Capture |
| antivirus | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 63.6 | Capture |
| website_builder | 0 | 51 | 0.0% | 51 | 100.0% | 11.7K | 61.7 | Capture |
* Pages = indexable (score≥30 + desc) × patterns × 23 languages
Citation Quality — by Endpoint
| Endpoint | Claude | ChatGPT | Perplexity | Total |
|---|---|---|---|---|
| other | 14.9K | 31 | 12 | 15.6K |
| /is-*-safe | 1.5K | 0 | 0 | 1.7K |
| /safe/* | 1.2K | 3 | 0 | 1.4K |
| /compare/* | 566 | 23 | 1 | 615 |
| /v1/preflight | 477 | 0 | 0 | 563 |
| /alternatives/* | 465 | 0 | 0 | 555 |
| /review/* | 449 | 0 | 0 | 538 |
| /privacy/* | 389 | 0 | 0 | 428 |
| /token/* | 74 | 11 | 3 | 124 |
| ZARQ | 31 | 28 | 6 | 69 |
| /model/* | 64 | 3 | 0 | 67 |
| homepage | 0 | 60 | 1 | 61 |
Crawl Budget Efficiency
| Bot | Total | 200 | 404 | 429 | Efficiency |
|---|---|---|---|---|---|
| Yandex | 35.2K | 33.0K | 2.2K | 0 | 93.7% |
| Bing | 30.7K | 28.1K | 2.5K | 0 | 91.6% |
| Claude | 22.0K | 20.1K | 966 | 0 | 91.4% |
| Meta | 2.1K | 2.1K | 37 | 0 | 98.2% |
| ByteDance | 1.5K | 1.3K | 116 | 0 | 87.9% |
| 385 | 360 | 22 | 0 | 93.5% | |
| ChatGPT | 180 | 159 | 11 | 0 | 88.3% |
| Perplexity | 28 | 23 | 5 | 0 | 82.1% |
Demand Without Supply (AI 404s)
| Path | Bot | Requests |
|---|---|---|
| /is-locators-safe | Claude | 2 |
| /is-modelcontextprotocol-sdk-safe | Claude | 2 |
| /is-smart-mcp-proxymcpproxy-go-safe | Claude | 2 |
| /is-tensorflow-models-safe | Claude | 2 |
| /safe/com.alibaba.aliexpresshd | ChatGPT | 2 |
| /safe/com.imo.android.imoim | ChatGPT | 2 |
| /compare/2noise-chattts-vs-corentinj-real-time-voice-cloning | Claude | 1 |
| /compare/2noise-chattts-vs-enricollen-rag-conversational-age | Claude | 1 |
| /compare/2noise-chattts-vs-victorcarre6-localmind | Claude | 1 |
| /compare/a-ghorbani-pocketpal-ai-vs-pro-design-ai | ByteDance | 1 |
| /compare/aardvark-application-winforms-gl-vs-mysql-entityfra | Claude | 1 |
| /compare/abg-vs-cdktf-cdktf-provider-docker | Claude | 1 |
| /compare/abp-vs-certbot-dns-luadns | Claude | 1 |
| /compare/abtem-vs-aliyun-python-sdk-alidns | Claude | 1 |
| /compare/abtem-vs-azure-mgmt-servicelinker | Claude | 1 |
Enrichment Pipeline Status
| Registry | Total | Enriched | % | Status |
|---|---|---|---|---|
| nuget | 641.6K | 641.6K | 100.0% | ✓ Complete |
| npm | 528.3K | 528.3K | 100.0% | ✓ Complete |
| website | 501.0K | 500.9K | 100.0% | ✓ Complete |
| crates | 204.1K | 204.1K | 100.0% | ✓ Complete |
| packagist | 113.8K | 113.8K | 100.0% | ✓ Complete |
| pypi | 93.8K | 93.8K | 100.0% | ✓ Complete |
| android | 57.6K | 57.5K | 100.0% | ✓ Complete |
| wordpress | 57.1K | 57.1K | 100.0% | ✓ Complete |
| vscode | 48.9K | 48.9K | 100.0% | ✓ Complete |
| ios | 48.1K | 48.1K | 100.0% | ✓ Complete |
| steam | 45.4K | 45.4K | 100.0% | ✓ Complete |
| chrome | 44.2K | 44.2K | 100.0% | ✓ Complete |
| firefox | 29.1K | 29.1K | 100.0% | ✓ Complete |
| go | 22.1K | 22.1K | 100.0% | ✓ Complete |
| gems | 10.1K | 10.1K | 100.0% | ✓ Complete |
| homebrew | 8.3K | 8.3K | 100.0% | ✓ Complete |
| saas | 5.0K | 5.0K | 100.0% | ✓ Complete |
| ai_tool | 3.1K | 2.6K | 83.9% | ▶ In progress |
| city | 3.0K | 3.0K | 100.0% | ✓ Complete |
| ingredient | 669 | 669 | 100.0% | ✓ Complete |
| supplement | 584 | 584 | 100.0% | ✓ Complete |
| cosmetic_ingredient | 584 | 584 | 100.0% | ✓ Complete |
| charity | 504 | 503 | 99.8% | ✓ Complete |
| crypto | 228 | 224 | 98.2% | ▶ In progress |
| country | 158 | 158 | 100.0% | ✓ Complete |
| vpn | 79 | 79 | 100.0% | ✓ Complete |
| password_manager | 55 | 55 | 100.0% | ✓ Complete |
| hosting | 51 | 49 | 96.1% | ▶ In progress |
| antivirus | 51 | 51 | 100.0% | ✓ Complete |
| website_builder | 51 | 51 | 100.0% | ✓ Complete |
0 enrichment processes running. Master watchdog active.
Kings (High-Yield Entities)
Total Kings
37.7K
Kings Enriched
37.7K
King Pages
3.0M
Kings % of Indexable
1.5%
| Registry | Kings | Enriched | Indexable |
|---|---|---|---|
| android | 13050 | 13040 | 13050 |
| website | 10879 | 10868 | 10875 |
| ios | 5427 | 5427 | 5427 |
| saas | 2806 | 2805 | 2806 |
| ai_tool | 787 | 784 | 787 |
| npm | 501 | 501 | 495 |
| wordpress | 500 | 500 | 500 |
| steam | 500 | 500 | 500 |
| charity | 493 | 492 | 492 |
| chrome | 472 | 472 | 472 |
| pypi | 300 | 300 | 294 |
| vscode | 200 | 200 | 200 |
| nuget | 200 | 200 | 5 |
| firefox | 200 | 200 | 200 |
| crypto | 196 | 196 | 196 |
| city | 185 | 185 | 185 |
| country | 158 | 158 | 158 |
| supplement | 137 | 137 | 137 |
| crates | 100 | 100 | 100 |
| packagist | 100 | 100 | 100 |
| gems | 100 | 100 | 100 |
| go | 100 | 100 | 100 |
| homebrew | 80 | 80 | 80 |
| vpn | 79 | 79 | 62 |
| cosmetic_ingredient | 73 | 73 | 73 |
| ingredient | 65 | 65 | 65 |
Kings = high-yield entities enriched with priority. Yield measurement available after 3-5 days of AI re-crawling.
Growth Engine
Enriched
2.5M
+0 today
Indexable Pages
720.9M
Citations/day
29.9K
Yield (per 1K pages)
0.04
stable
Enriched → Indexable Pages → AI Citations. Yield = citations per 1K indexable pages per day.
Crawl Trend — Daily Bot Requests (200 OK)
AI Citations: 102.4K/7d
AI Indexing: 19.2K/7d
Search: 1.2M/7d
Meta: 120.9K/7d
Amazon: 100.4K/7d
All bots: 2.2M/7d
Language Rollout — Submitted → Crawled → Cited
Languages Live: 2
Submitted URLs: 54.3M
AI Crawls (all langs): 21.5K/day
Citations (all langs): 20.9K/day
| Language | Sitemaps | URLs | AI Crawls | Citations | Human | Status |
|---|---|---|---|---|---|---|
| en English | 246 | 2.5M | 21.4K | 20.8K | 11.1K | ✓ Live |
| es Spanish | 246 | 2.5M | 105 | 105 | 49 | ✓ Live |
| th Thai | 246 | 2.5M | 3 | 3 | 34 | ✓ Submitted |
| zh Chinese | 246 | 2.5M | 1 | 1 | 102 | ✓ Submitted |
| vi Vietnamese | 246 | 2.5M | 1 | 1 | 122 | ✓ Submitted |
| pl Polish | 246 | 2.5M | 1 | 1 | 24 | ✓ Submitted |
| nl Dutch | 246 | 2.5M | 1 | 1 | 24 | ✓ Submitted |
| hi Hindi | 246 | 2.5M | 1 | 1 | 7 | ✓ Submitted |
| da Danish | 246 | 2.5M | 1 | 1 | 107 | ✓ Submitted |
| ar Arabic | 246 | 2.5M | 1 | 1 | 125 | ✓ Submitted |
| tr Turkish | 246 | 2.5M | 0 | 0 | 71 | ✓ Submitted |
| sv Swedish | 246 | 2.5M | 0 | 0 | 11 | ✓ Submitted |
| ru Russian | 246 | 2.5M | 0 | 0 | 29 | ✓ Submitted |
| ro Romanian | 246 | 2.5M | 0 | 0 | 13 | ✓ Submitted |
| pt Portuguese | 246 | 2.5M | 0 | 0 | 33 | ✓ Submitted |
| ko Korean | 246 | 2.5M | 0 | 0 | 37 | ✓ Submitted |
| ja Japanese | 246 | 2.5M | 0 | 0 | 43 | ✓ Submitted |
| it Italian | 246 | 2.5M | 0 | 0 | 13 | ✓ Submitted |
| id Indonesian | 246 | 2.5M | 0 | 0 | 50 | ✓ Submitted |
| fr French | 246 | 2.5M | 0 | 0 | 31 | ✓ Submitted |
| de German | 246 | 2.5M | 0 | 0 | 37 | ✓ Submitted |
| cs Czech | 246 | 2.5M | 0 | 0 | 123 | ✓ Submitted |
Infrastructure
Requests (24h)
260095
19.2x headroom to 5M limit (CF edge)
System Load
8.17
8 cores available
PostgreSQL Connections
10
Max 100 (default)
Disk Usage
1%
Page Cache
1.9% hit rate
1.9% hit rate
39.8K pages · 1.55G memory · 554.7K hits / 28.4M misses
Scaling triggers:
>500K req/day → page cache (DONE) ·
Cloudflare edge cache (DONE) ·
>2M req/day → migrate to VPS (€18/mo) ·
>5M req/day → dedicated server ·
>15M req/day → Cloudflare Workers
Cached 5min. Updated: 22:31.