What
is Robots.txt files?
Robots.txt
is a text file to instruct web robots (search engine robots) how to crawl pages
on their website.
Format:
User-agent: *
Allow: /
* symbol for
all search engine.
Blocking all web crawlers from all content
User-agent:
*
Disallow:
/
Allowing all web crawlers access to all content.
User-agent: *
Disallow:
Blocking a specific web crawler from a specific folder.
User-agent: Googlebot
Disallow: /example-subfolder/
Blocking a specific web crawler from a specific web page.
User-agent: Bingbot
Disallow: /example-subfolder/blocked-page.html
What is Meta Robots Tags?
You can use a special HTML <META> tag to tell robots not to index the
content of a page, and/or not scan it for links to follow.
<html>
<head>
<title>...</title>
<META
NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
Email ID: - hirededicatedemployee@gmail.com
Website: - www.hirededicatedemployee.com
Skype ID: - Hire Dedicated Employee
Linked
In Page: - https://www.linkedin.com/company/hirededicatedemployee
