Index Now - A new push protocol for Search Engine Discovery

Introduction #

Tired of waiting days—or weeks—for our favourite search engines to notice changes on your website? Imagine publishing a page and having it appear in search results almost instantly. That’s the power of IndexNow—a simple tool from Microsoft and Yandex that’s quietly transforming how the web gets indexed. Here’s why your website can’t afford to ignore it.

What is IndexNow #

IndexNow is an open-source push protocol co-developed by Microsoft Bing and Yandex (launched in October 2021), designed to speed up how quickly search engines discover content changes on your website.

For more information, you can refer to the blog post by Bing explaining

Instead of waiting for search engine crawlers—which can take days or even weeks—to find changed pages the traditional way (“pull indexing”), IndexNow lets website owners instantly notify participating search engines of URLs that needs updating.

Integrate with IndexNow #

Step 1: Getting the API Key #

Visit the IndexNow Page to get a new API key.

Step 2: Verifying your website #

There are 2 options to verify your website, we will go through both of them here.

Option 1: Hosting a file with the api key as the name #

Create a <api_key>.txt file at the /<api_key>.txt path of your website.

# If your API Key is `67fc9881e4a0438d90a4ea8b7bbefbe7`
echo "67fc9881e4a0438d90a4ea8b7bbefbe7" > 67fc9881e4a0438d90a4ea8b7bbefbe7.txt

Your can run the following function below.

Option 2: Hosting a file with a different name #

You can also host the file in other locations. If you do, the API calls will need to add a new field keyLocation. Take note of the path you have chosen.

Step 3: Indexing your pages #

The last step is the index your pages dynamically.

There are also 2 methods that can be used for the indexing here.

Option 1: Update Single URL #

To update single urls, you can make a GET request to this URL, with the page that you want to index.

# Ensure that change_url is URL Encoded
# https://api.indexnow.org/indexnow?url=<change_url>&key=<API_KEY>

curl -X GET https://api.indexnow.org/indexnow?url=https://jh123x.com/blog/2025/index-now/&key=67fc9881e4a0438d90a4ea8b7bbefbe7

If you have made use of custom locations you will have to add an additional parameter. Here is an example below

# add keyLocation query param (Also url encoded)
curl -X GET https://api.indexnow.org/indexnow?url=https://jh123x.com/blog/2025/index-now/&key=67fc9881e4a0438d90a4ea8b7bbefbe7&keyLocation=https://jh123x.com/myIndexNowKey.txt

If you have many URLs, please take a look at option 2 instead

Option 2: Update Multiple URLs #

To update many URLs you can make use of a POST request instead. Here is an example of the same request body as the GET request example above

{
    "host":"https://jh123x.com/",
    "key":"67fc9881e4a0438d90a4ea8b7bbefbe7",
    "urlList":[
        "https://jh123x.com/blog/2025/index-now/"
        // Other URLs can be added here (Up to 10,000 URLs)
    ],

    // Can be omitted if default location is used
    "keyLocation":"https://jh123x.com/myIndexNowKey.txt"
}

An example curl command is given below

curl -X POST https://api.indexnow.org/indexnow \
    -H "Content-Type: application/json" \
    --data "{\"host\":\"https://jh123x.com/\",\"key\":\"67fc9881e4a0438d90a4ea8b7bbefbe7\",\"urlList\":[\"https://jh123x.com/blog/2025/index-now/\"]}"

Use Case: Integrate with Hugo #

Step 1: Build your hugo page #

Build your hugo page with the following

hugo --gc --minify --environment production

This should create a public directory with a sitemap.xml inside

Step 2: Create a script to call Index Now #

Create a script to get your articles from the generated sitemap.xml at public/sitemap.xml. Here is an example script below.

import os

# Sitemap path
SITEMAP_LOC = os.path.join("public","sitemap.xml")

def get_all_articles() -> Set[str]:
    """articles from website"""
    with open(SITEMAP_LOC) as f:
        result = xmltodict.parse(f.read())
    return set(url["loc"] for url in result["urlset"]["url"])

def index_urls(urls: Iterable[str]):
    """Sends the url to index_now to be automatically indexed"""
    with requests.Session() as s:
        resp = s.post(
            "https://api.indexnow.org/indexnow",
            json={
                "host": "jh123x.com",
                "key": "67fc9881e4a0438d90a4ea8b7bbefbe7",
                "urlList": list(urls),
            },
        )

if __name__ == '__main__':
    articles = get_all_articles()
    index_urls(articles)

There are multiple optimizations you can do to improve this script

1. Updating only changed articles.
2. Chunking the POST endpoint to send 10,000 urls at a single time at max.

Step 3: Putting them together #

We can put then together into a github action so that our URLs will be updated whenever we push a new commit. Now all future additional articles or changes will trigger the index to scrap the website again.

name: Check and Push to IndexNow
on:
  push:
    branches:
      - main

jobs:
  resources:
    name: Check links
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true

      - name: Hugo setup
        uses: peaceiris/actions-hugo@v2.6.0
        with:
          hugo-version: "0.136.5"
          extended: true

      - name: Build
        run: hugo --gc --minify --environment production

      - name: Setup Python environment
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Run python script
        run: python3 main.py

Conclusion #

Index Now is beneficial when creating, updating or deleting pages and allows for search indexes to scrape your website as soon as possible when the website is updated.

This reduces the delay between the website update and the update time of the website.

In this article we have taken a look at how it works and what we can do to integrate it with hugo.

Useful Links #

• Bing blog post
• IndexNow
• IndexNow Documentation