.Google.com's John Mueller answered an inquiry concerning why Google indexes pages that are actually disallowed from crawling through robots.txt as well as why the it's safe to ignore the related Browse Console documents regarding those creeps.Bot Visitor Traffic To Question Criterion URLs.The person asking the inquiry documented that crawlers were actually generating hyperlinks to non-existent query specification URLs (? q= xyz) to pages with noindex meta tags that are likewise shut out in robots.txt. What triggered the concern is actually that Google is crawling the web links to those pages, receiving blocked by robots.txt (without envisioning a noindex robotics meta tag) after that acquiring reported in Google.com Explore Console as "Indexed, though shut out by robots.txt.".The individual asked the observing question:." Yet below's the significant concern: why will Google.com mark pages when they can't even see the information? What's the conveniences in that?".Google's John Mueller verified that if they can't creep the page they can not see the noindex meta tag. He likewise produces an exciting acknowledgment of the web site: search operator, encouraging to overlook the outcomes since the "common" customers won't observe those outcomes.He created:." Yes, you're correct: if our team can not creep the page, our company can not observe the noindex. That said, if our team can't creep the webpages, at that point there is actually not a great deal for our company to index. Therefore while you may view a number of those webpages with a targeted website:- query, the common user will not observe all of them, so I definitely would not fuss over it. Noindex is actually also alright (without robots.txt disallow), it just means the Links will find yourself being actually crawled (and also wind up in the Browse Console report for crawled/not listed-- neither of these statuses create problems to the remainder of the site). The vital part is that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response validates the restrictions being used the Web site: hunt progressed hunt operator for diagnostic reasons. Some of those reasons is because it's not connected to the regular hunt mark, it is actually a distinct factor entirely.Google's John Mueller talked about the internet site hunt driver in 2021:." The quick solution is actually that a website: inquiry is actually certainly not implied to become total, nor utilized for diagnostics objectives.A web site question is a specific sort of search that restricts the outcomes to a particular internet site. It's generally simply the word website, a digestive tract, and after that the web site's domain name.This concern restricts the end results to a specific internet site. It's certainly not suggested to be a thorough compilation of all the web pages from that internet site.".2. Noindex tag without using a robots.txt is actually alright for these kinds of conditions where a crawler is actually connecting to non-existent webpages that are actually acquiring found by Googlebot.3. Links with the noindex tag will produce a "crawled/not indexed" item in Browse Console which those will not possess a negative result on the rest of the internet site.Check out the question as well as respond to on LinkedIn:.Why would Google.com mark webpages when they can not even see the material?Featured Picture through Shutterstock/Krakenimages. com.