Search Console + Bing + IndexNow
Run this after the site is live on its production domain (DNS on Cloudflare,
production branch deployed). It makes Google and Bing aware of the site and keeps
Bing's index fresh automatically.
Only register the live domain. A Worker's Preview links (*.workers.dev) carry
noindex, and so do a Pages site's main and *.pages.dev previews (the kit's
functions/_middleware.ts); on Pages, after go-live, the project alias
<project>.pages.dev also 301s to the live domain (once CANONICAL_URL is set —
new-website/references/CLOUDFLARE_FIRST_DEPLOY.md, "After go-live"). Do not add any
workers.dev or pages.dev host as a property; they can't be indexed.
Human-in-the-loop: creating the Google/Microsoft accounts and adding the DNS
record are actions only the owner can do (like the GitHub/Cloudflare accounts in
SETUP.md). The agent drafts the exact TXT value and the sitemap URL; the owner
pastes/clicks.
The verification record can go in earlier. A Domain property is verified by DNS
alone, so its TXT record (and Bing's CNAME, for the manual route) can be written on
setup day, before the site is live: the new-website setup script writes it, DNS only,
with the owner's yes (--record "TXT @ google-site-verification=…",
new-website/references/CLOUDFLARE_FIRST_DEPLOY.md, "S"). The owner still copies the
value from Google and clicks Verify; the sitemap waits until the site is live.
Console UI labels are localized — the click paths below are quoted from the
English UI; if the owner's account language differs, translate the labels for them
(German: Property hinzufügen, DNS → Einträge → Eintrag hinzufügen, Aus der
Google Search Console importieren, Caching → Konfiguration).
1. Google Search Console (Google)
- Go to https://search.google.com/search-console and sign in.
- Add property → "Domain" (covers http+https, www+apex, all subdomains — the
right default). Google shows a TXT record to add to DNS.
- Add the TXT record in Cloudflare (DNS is on Cloudflare): dashboard → your
domain → DNS → Records → Add record → Type
TXT, Name @, Content = the
google-site-verification=… string Google gave you → Save. Back in GSC click
Verify (propagation is usually seconds on Cloudflare).
- Alternative: a URL-prefix property verified by the HTML
<meta> tag (drop
it in Base.astro's <head>) or an HTML file in public/. Domain+TXT is
preferred — broader coverage, survives scheme/subdomain changes.
- Submit the sitemap: GSC → Sitemaps → enter
sitemap-0.xml → Submit.
The kit's @astrojs/sitemap emits /sitemap-0.xml at the site root; confirm it
loads in a browser first. Google discovers pages from it over the next days. Not
sitemap-index.xml: one file holds every page, and the index is an extra fetch that
Bing once reported as "not accessible" on one site.
- Look at "Search generative AI" in the property's settings, and opt out only if
the owner wants out of Google's AI answers. A property can inherit it from a parent
property, so check what yours shows. Opting out keeps your content out of AI
Overviews, AI Mode and Discover's AI features, usually within a few days. Google
says the control "isn't used as a ranking or inclusion signal affecting other parts
of Search" and "doesn't affect AI training"; that is Google-Extended (Google).
2. Bing Webmaster Tools (Bing + the IndexNow consumers)
Easiest path — import from Google Search Console (auto-verifies and pulls the
sitemap, so no second verification dance):
- Go to https://www.bing.com/webmasters and sign in (Microsoft account).
- Choose "Import from Google Search Console" → Continue, sign in to the Google
account that owns the GSC property, grant access, select the site → Import.
- Within minutes the site is added, auto-verified, and existing sitemaps are
imported. Confirm under Sitemaps; if it didn't carry over, Submit sitemap →
https://<domain>/sitemap-0.xml.
Manual alternative (no Google account): Add a site, verify by the XML file
(place in public/), the <meta> tag (in Base.astro), or a CNAME DNS record, then
submit the sitemap as above.
3. IndexNow — one Cloudflare toggle (Crawler Hints)
IndexNow lets the site push "this URL changed" to search engines instead of waiting
to be crawled. On a Cloudflare-proxied site you get it for free with Crawler Hints —
Cloudflare generates and hosts the IndexNow key and sends the notifications for you, so
there is nothing to add to the repo:
- Cloudflare dashboard → your domain → Caching → Configuration.
- Find Crawler Hints and enable it.
That's it. Cloudflare now notifies the IndexNow network (Bing, Yandex, Naver,
Seznam) whenever your content is created/updated/deleted. Requires the records to be
proxied through Cloudflare (orange cloud) — true for a Cloudflare site (a Worker or
Pages) on a Cloudflare-managed domain.
- Google does NOT use IndexNow — it relies on its own crawl signals, so the GSC
sitemap (step 1) is how Google stays current. IndexNow speeds up Bing & friends.
- No manual key file: don't add an
indexnow.txt/key to public/ — Cloudflare's
managed key would conflict. (Only self-host a key if you're NOT using Crawler Hints.)
Also in the Cloudflare dashboard — keep crawlers allowed
The AI crawler settings are under your domain → Security → Settings → filter by Bot
traffic → AI Crawler and Agent access policies (the place as a dashboard showed it on
2026-10-30; Cloudflare renamed the setting from Configure AI bot policies on 2026-11-04). For the open, indexable posture this kit wants, confirm these (the defaults can
block). docs/CLOUDFLARE-CHECKLIST.md in the Croftweaver folder, section 4, has the details:
- Search, Agent and Training → "Allow (do not block)". Since 15 September 2026,
Cloudflare says, Block and Block on pages with ads on Training also keep out
Googlebot, Bingbot and Applebot, "so either setting impacts search as well as training"
(Cloudflare). Its way to
refuse AI training and stay in Google, Bing and Apple search is Disallow for Training,
with Bot Preference Sync on (next bullet); the kit has not tested how crawlers are treated,
and the file Sync writes also shuts out Baiduspider and PetalBot, the crawlers of Baidu's
and Huawei's Petal search. That is the training choice
ai-seo describes (Step 4).
Bing does not read the no-training preference yet; Microsoft's AI Preferences FAQ says
Microsoft starts honoring its new AI Training preference on 1 February 2027 (ai-seo, Step 4, has
the dates).
- Enable Bot Preference Sync → off, unless you chose Disallow for Training. With it on,
Cloudflare puts its own lines in front of the repo's own
public/robots.txt; with
Disallow, those lines are what publish the no-training preference. Choosing Disallow does
not turn it on by itself.
- If your dashboard still shows the older Block AI bots setting, make sure it is off. A
dashboard on 2026-10-30 no longer showed it.
Assistant: offer to do these Cloudflare dashboard clicks for the user if your
runtime can drive a browser. The Crawler Hints toggle and the AI bot policies
above are dashboard actions, not repo edits — so offering to perform them beats
just listing steps. Pick the path that matches your runtime:
- Claude Code with Claude in Chrome — offer that extension: ask the user to
install it (it's what gives real click/navigate access; plain screen control can
only read a browser).
- Another agent (Codex, Antigravity, …) — offer your own browser-automation tool
if you have one; otherwise walk the user through the clicks.
Whichever path, the shape is the same:
- The user opens the browser and logs in to Cloudflare themselves — never handle
their credentials or 2FA.
- With their explicit go-ahead and the target zone, navigate to Caching →
Configuration (Crawler Hints) and Security → Settings → Bot traffic → AI Crawler
and Agent access policies (Configure AI bot policies on a dashboard not yet renamed),
read the current values back, and only then set any that don't
match the posture above.
If no browser automation is available or the user declines, fall back to walking them
through the clicks. Never attempt these dashboard changes through blind screen control.
4. Confirm it's working
- GSC: the Pages / Indexing report shows URLs moving to "Indexed" over a
few days; Sitemaps shows "Success" with the discovered-URL count.
- Bing: Site Explorer lists your URLs; Sitemaps shows the submitted file.
- IndexNow: Bing Webmaster Tools → IndexNow (or URL submission) shows
notifications arriving once Crawler Hints has fired on a content change.
- AI crawlers (free, on Cloudflare): your domain → AI Crawl Control → Overview shows
how many requests from each AI company's crawlers Cloudflare let through, and offers
Hostname and Path filters. Look in the first days after launch, and again every few days,
and pick a date range that reaches back to your last look. Cloudflare's page says that on
free plans the Metrics tab shows only the past 24 hours; on a free-plan domain we looked
at in November 2026, its Metrics tab offered longer ranges too and showed several days of
data. On the free plan Cloudflare knows a crawler by the name the request gives
(Cloudflare), so a crawler
that doesn't name itself isn't counted as an AI crawler. Croftweaver's Cloudflare check
(
new-website/scripts/cloudflare-check.mjs) is counted too: each time it checks the AI
crawlers (a full run does, and so does --only ai), it sends your home page one request per
AI crawler name, and on that domain every one showed up within a few minutes. Note when you
run it. For each run inside the range you picked, take off the home-page requests it sent
that Cloudflare let through: seven when the check reports no name refused (two each from
Anthropic and OpenAI, one each from Perplexity, Common Crawl and Amazon). A visit is not a
mention: it can't tell you whether any AI answer names you.
- How long AI assistants take: when we last checked, in November 2026, we found no
published measurement, with a stated method, of how long AI assistants take to first cite a
brand-new website. The ranges Google gives, and what companies measured for new pages on
sites that already exist, are in
How long until Google and AI assistants find a new website?
Check that the crawlers come (above), and later Google's and Bing's AI reports
(
search-console-insights).
Scope notes
- This is search-engine registration, not on-page SEO — the head/sitemap/schema
contract is
website-seo-geo; technical-SEO/Core-Web-Vitals is seo-audit; GEO/AI
answers is ai-seo. Run those before this so what gets indexed is already correct.
- GSC is also the free, no-script, no-consent-banner analytics option from the
new-website decision interview (Q5): if the owner only needs "is anyone visiting?",
this skill is the whole analytics setup.
1---2name: search-console-setup3description: >4 Post-launch: register the LIVE site with Google Search Console and Bing Webmaster5 Tools, verify ownership, submit the sitemap, and turn on IndexNow so Bing/Yandex6 get pinged the moment content changes. For a Cloudflare-hosted Astro site: GSC7 Domain property verified by a Cloudflare DNS TXT record + submit `sitemap-0.xml`;8 Bing the easy way via "Import from Google Search Console" (auto-verifies + pulls the9 sitemap); IndexNow by flipping ONE toggle — Cloudflare Crawler Hints — no key file10 to host. Account creation + the DNS record are human actions the agent drafts but11 can't click. Trigger phrases: "set up Google Search Console", "Search Console",12 "GSC", "Bing Webmaster Tools", "submit my sitemap", "register the site with Google",13 "verify site ownership", "enable IndexNow", "Crawler Hints", "get indexed", "submit14 the website to search engines".15metadata:16 source: https://skillmd.com/skills/karero/search-console-setup17---1819# Search Console + Bing + IndexNow2021Run this **after the site is live on its production domain** (DNS on Cloudflare,22`production` branch deployed). It makes Google and Bing aware of the site and keeps23Bing's index fresh automatically.2425> **Only register the live domain.** A Worker's Preview links (`*.workers.dev`) carry26> `noindex`, and so do a Pages site's `main` and `*.pages.dev` previews (the kit's27> `functions/_middleware.ts`); on Pages, after go-live, the project alias28> `<project>.pages.dev` also 301s to the live domain (once `CANONICAL_URL` is set —29> `new-website/references/CLOUDFLARE_FIRST_DEPLOY.md`, "After go-live"). Do not add any30> `workers.dev` or `pages.dev` host as a property; they can't be indexed.31>32> **Human-in-the-loop:** creating the Google/Microsoft accounts and adding the DNS33> record are actions only the owner can do (like the GitHub/Cloudflare accounts in34> SETUP.md). The agent drafts the exact TXT value and the sitemap URL; the owner35> pastes/clicks.36>37> **The verification record can go in earlier.** A Domain property is verified by DNS38> alone, so its TXT record (and Bing's CNAME, for the manual route) can be written on39> setup day, before the site is live: the `new-website` setup script writes it, DNS only,40> with the owner's yes (`--record "TXT @ google-site-verification=…"`,41> `new-website/references/CLOUDFLARE_FIRST_DEPLOY.md`, "S"). The owner still copies the42> value from Google and clicks **Verify**; the sitemap waits until the site is live.4344> **Console UI labels are localized** — the click paths below are quoted from the45> English UI; if the owner's account language differs, translate the labels for them46> (German: *Property hinzufügen*, *DNS → Einträge → Eintrag hinzufügen*, *Aus der47> Google Search Console importieren*, *Caching → Konfiguration*).4849## 1. Google Search Console (Google)50511. Go to https://search.google.com/search-console and sign in.522. **Add property → "Domain"** (covers http+https, www+apex, all subdomains — the53 right default). Google shows a **TXT record** to add to DNS.543. **Add the TXT record in Cloudflare** (DNS is on Cloudflare): dashboard → your55 domain → **DNS → Records → Add record** → Type `TXT`, Name `@`, Content = the56 `google-site-verification=…` string Google gave you → Save. Back in GSC click57 **Verify** (propagation is usually seconds on Cloudflare).58 - *Alternative:* a **URL-prefix** property verified by the HTML `<meta>` tag (drop59 it in `Base.astro`'s `<head>`) or an HTML file in `public/`. Domain+TXT is60 preferred — broader coverage, survives scheme/subdomain changes.614. **Submit the sitemap:** GSC → **Sitemaps** → enter `sitemap-0.xml` → Submit.62 The kit's `@astrojs/sitemap` emits `/sitemap-0.xml` at the site root; confirm it63 loads in a browser first. Google discovers pages from it over the next days. Not64 `sitemap-index.xml`: one file holds every page, and the index is an extra fetch that65 Bing once reported as "not accessible" on one site.665. **Look at "Search generative AI"** in the property's settings, and opt out only if67 the owner wants out of Google's AI answers. A property can inherit it from a parent68 property, so check what yours shows. Opting out keeps your content out of AI69 Overviews, AI Mode and Discover's AI features, usually within a few days. Google70 says the control "isn't used as a ranking or inclusion signal affecting other parts71 of Search" and "doesn't affect AI training"; that is Google-Extended ([Google](https://support.google.com/webmasters/answer/16908024)).7273## 2. Bing Webmaster Tools (Bing + the IndexNow consumers)7475Easiest path — **import from Google Search Console** (auto-verifies and pulls the76sitemap, so no second verification dance):77781. Go to https://www.bing.com/webmasters and sign in (Microsoft account).792. Choose **"Import from Google Search Console" → Continue**, sign in to the Google80 account that owns the GSC property, grant access, **select the site → Import**.813. Within minutes the site is added, **auto-verified**, and existing sitemaps are82 imported. Confirm under **Sitemaps**; if it didn't carry over, **Submit sitemap →**83 `https://<domain>/sitemap-0.xml`.8485*Manual alternative (no Google account):* **Add a site**, verify by the XML file86(place in `public/`), the `<meta>` tag (in `Base.astro`), or a CNAME DNS record, then87submit the sitemap as above.8889## 3. IndexNow — one Cloudflare toggle (Crawler Hints)9091IndexNow lets the site *push* "this URL changed" to search engines instead of waiting92to be crawled. On a Cloudflare-proxied site you get it for free with **Crawler Hints** —93Cloudflare generates and hosts the IndexNow key and sends the notifications for you, so94**there is nothing to add to the repo**:95961. Cloudflare dashboard → your domain → **Caching → Configuration**.972. Find **Crawler Hints** and **enable** it.9899That's it. Cloudflare now notifies the IndexNow network (Bing, Yandex, Naver,100Seznam) whenever your content is created/updated/deleted. Requires the records to be101**proxied through Cloudflare** (orange cloud) — true for a Cloudflare site (a Worker or102Pages) on a Cloudflare-managed domain.103104- **Google does NOT use IndexNow** — it relies on its own crawl signals, so the GSC105 sitemap (step 1) is how Google stays current. IndexNow speeds up *Bing & friends*.106- No manual key file: don't add an `indexnow.txt`/key to `public/` — Cloudflare's107 managed key would conflict. (Only self-host a key if you're NOT using Crawler Hints.)108109### Also in the Cloudflare dashboard — keep crawlers *allowed*110111The AI crawler settings are under your domain → Security → Settings → filter by **Bot112traffic** → **AI Crawler and Agent access policies** (the place as a dashboard showed it on1132026-10-30; Cloudflare renamed the setting from Configure AI bot policies on 2026-11-04). For the open, indexable posture this kit wants, confirm these (the defaults can114block). `docs/CLOUDFLARE-CHECKLIST.md` in the Croftweaver folder, section 4, has the details:115116- **Search, Agent and Training → "Allow (do not block)".** Since 15 September 2026,117 Cloudflare says, **Block** and **Block on pages with ads** on Training also keep out118 Googlebot, Bingbot and Applebot, "so either setting impacts search as well as training"119 ([Cloudflare](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)). Its way to120 refuse AI training and stay in Google, Bing and Apple search is **Disallow** for Training,121 with Bot Preference Sync on (next bullet); the kit has not tested how crawlers are treated,122 and the file Sync writes also shuts out Baiduspider and PetalBot, the crawlers of Baidu's123 and Huawei's Petal search. That is the training choice `ai-seo` describes (Step 4).124 Bing does not read the no-training preference yet; Microsoft's AI Preferences FAQ says125 Microsoft starts honoring its new AI Training preference on 1 February 2027 (`ai-seo`, Step 4, has126 the dates).127- **Enable Bot Preference Sync → off**, unless you chose Disallow for Training. With it on,128 Cloudflare puts its own lines in front of the repo's own `public/robots.txt`; with129 Disallow, those lines are what publish the no-training preference. Choosing Disallow does130 not turn it on by itself.131- If your dashboard still shows the older **Block AI bots** setting, make sure it is off. A132 dashboard on 2026-10-30 no longer showed it.133134> **Assistant: offer to do these Cloudflare dashboard clicks for the user** if your135> runtime can drive a browser. The Crawler Hints toggle and the AI bot policies136> above are dashboard actions, not repo edits — so offering to perform them beats137> just listing steps. Pick the path that matches your runtime:138> - **Claude Code with Claude in Chrome** — offer that extension: ask the user to139> install it (it's what gives real click/navigate access; plain screen control can140> only *read* a browser).141> - **Another agent (Codex, Antigravity, …)** — offer your own browser-automation tool142> if you have one; otherwise walk the user through the clicks.143>144> Whichever path, the shape is the same:145> 1. The user **opens the browser and logs in to Cloudflare themselves** — never handle146> their credentials or 2FA.147> 2. With their explicit go-ahead and the target zone, navigate to **Caching →148> Configuration** (Crawler Hints) and **Security → Settings → Bot traffic → AI Crawler149> and Agent access policies** (**Configure AI bot policies** on a dashboard not yet renamed),150> read the current values back, and only then set any that don't151> match the posture above.152>153> If no browser automation is available or the user declines, fall back to walking them154> through the clicks. Never attempt these dashboard changes through blind screen control.155156## 4. Confirm it's working157158- **GSC:** the **Pages** / **Indexing** report shows URLs moving to "Indexed" over a159 few days; **Sitemaps** shows "Success" with the discovered-URL count.160- **Bing:** **Site Explorer** lists your URLs; **Sitemaps** shows the submitted file.161- **IndexNow:** Bing Webmaster Tools → **IndexNow** (or URL submission) shows162 notifications arriving once Crawler Hints has fired on a content change.163- **AI crawlers (free, on Cloudflare):** your domain → **AI Crawl Control** → **Overview** shows164 how many requests from each AI company's crawlers Cloudflare let through, and offers165 Hostname and Path filters. Look in the first days after launch, and again every few days,166 and pick a date range that reaches back to your last look. Cloudflare's page says that on167 free plans the **Metrics** tab shows only the past 24 hours; on a free-plan domain we looked168 at in November 2026, its Metrics tab offered longer ranges too and showed several days of169 data. On the free plan Cloudflare knows a crawler by the name the request gives170 ([Cloudflare](https://developers.cloudflare.com/ai-crawl-control/get-started/)), so a crawler171 that doesn't name itself isn't counted as an AI crawler. Croftweaver's Cloudflare check172 (`new-website/scripts/cloudflare-check.mjs`) is counted too: each time it checks the AI173 crawlers (a full run does, and so does `--only ai`), it sends your home page one request per174 AI crawler name, and on that domain every one showed up within a few minutes. Note when you175 run it. For each run inside the range you picked, take off the home-page requests it sent176 that Cloudflare let through: seven when the check reports no name refused (two each from177 Anthropic and OpenAI, one each from Perplexity, Common Crawl and Amazon). A visit is not a178 mention: it can't tell you whether any AI answer names you.179- **How long AI assistants take:** when we last checked, in November 2026, we found no180 published measurement, with a stated method, of how long AI assistants take to first cite a181 brand-new website. The ranges Google gives, and what companies measured for new pages on182 sites that already exist, are in183 [How long until Google and AI assistants find a new website?](https://guide.croftweaver.com/how-long)184 Check that the crawlers come (above), and later Google's and Bing's AI reports185 (`search-console-insights`).186187## Scope notes188189- This is **search-engine registration**, not on-page SEO — the head/sitemap/schema190 contract is `website-seo-geo`; technical-SEO/Core-Web-Vitals is `seo-audit`; GEO/AI191 answers is `ai-seo`. Run those before this so what gets indexed is already correct.192- GSC is also the free, no-script, no-consent-banner analytics option from the193 `new-website` decision interview (Q5): if the owner only needs "is anyone visiting?",194 this skill is the whole analytics setup.