Dear Friends, when many people create a blog or website and get it crawled on Google or any other search engine, then some problems come in front of them, one of them is also about robots.txt, people who are about robots.txt. Know that they easily crawl and index their website in any search engine.
But those who know about robots.txt, they get them crawled in any search engine, but their WebPages are not able to be indexed and they keep trying again and again, like if we talk about Google search. Of the engine, those people keep testing the page’s URL in the inspection tool again and again to get the page indexed, but they do not understand why their pages are not getting indexed in search engines like Google.
First of all, tell that Google automatically indexes the page and if there is some issue in any page, then Google gives you an option that you can manually inspect the URL with the inspection tool and remove the issue and test the page again. You can request to index the page again and again, but sometimes there are some issues due to which we have to make changes in robots.txt.
Why robots.txt is used:
To tell the website that our webpage or website is made for any particular search engine or for all search engines. We have to mention this in the robots.txt file.
We also use robots.txt to tell the search engine which of our links and pages should be indexed by the search engine and which URLs should not index, we should not index things like our dashboard in it or any We maintain a robots.txt file for such a post which is your private file that the search engine does not index it.
Sometimes our hosting server is a bit slow and when Google search engine bots come to our website and try to index the changed content or URL, they test all the links by opening them internally instead of doing one-on-one. Due to which the speed of our server slows down a bit, then robots.txt helps in telling the search engine how many milliseconds interval it should work. Due to which your server does not slow down and that googlebots also keeps doing its work and your live visitors also keep doing their work.
Now it comes that it indexes things like any image, file or PDF from your websites, then we can also determine in robots.txt which media file should be indexed by the search engine or not.
This robots.txt belongs to this website of ours, now we will try to understand you step wise how to write this file.
User-agent: *
This means that our website is made for all search engines. If we make our website only for Google search engine and no other search engine can crawl and index us, then we will write it like this:
User-agent: Googlebot
After writing this way, only Google search engine can crawl and index us, other search engines will not even touch our files. Now let’s move on to the next code:
Disallow: /wp-admin/
This code is restricted to any particular director to crawl the search engine. He will not read this director of ours and will not publicly show it in the indexing. Now let’s move on to the next code:
Allow: /wp-admin/admin-ajax.php
This code allows the search engine that it can read only this link from the above mentioned restrict directory, or you can put all the links here one by one which you have to allow special. Now let’s move on to the next code:
Crawl-delay: 15
This code shows the time when the search engine bots come to our website and our server becomes slow and we give a time to the search engine to test our website only at the interval of 15 milliseconds, Crawl-delay Keep the right time between 15 – 20 and do not work more than this. Now we move on to the next code:
In this code we have mentioned all our sitemaps, if your website is customized then you will have only one sitemap and all urls and tags are given in the same sitemap but you can have a lot of sitemaps on WordPress because WordPress will automatically post your posts. Sitemap is created separately from your tags, your categories and the URLs (Permalinks) of your blogs. So you can mention all your sitemap in this code.
For more information, you can also read through the details by clicking on this link given below
If friends, if you want to see your robots .txt file, then first of all you can see the link of your website by writing robots .txt with back slash as we have this robots .txt:
https://viralscripts.co.in/robots.txt
Friends, this post of ours is published in both Hindi and English language, you can see the page for any language by changing it with Google Language Translator.
Friends, if you find our post valuable, then you must share it and if you want any information related to robots .txt files, then you can tell us by commenting, we will definitely give you its solutions. thank you
Spread the love with your friends
1 thought on “What is robots.txt and how to create a robot.txt file | Learn robots.txt file with step by step guide”
1 thought on “What is robots.txt and how to create a robot.txt file | Learn robots.txt file with step by step guide”