Recently, some of our users asked us how to keep their WordPress websites private so they weren’t listed in Google search results. One of their websites was not yet ready for the public, while another user had a private website for a specific group of people.
Preventing search engines from crawling your website is a simple process, but it’s important to understand how it works. Search engines use automated programs called ‘crawlers’ to discover and index websites. To hide your site, you need to instruct these crawlers to ignore your content.
This article will show you how to stop search engines from crawling your WordPress website. We will explain the steps involved and offer guidance on the best approach for your specific needs.
Why Stop Search Engines From Crawling a WordPress Site?
For most WordPress websites, search engines are the biggest source of traffic. You may ask, why would anyone want to discourage search engines?
Here are some situations when you won’t want search engines to index your website:
- When starting out, you may not know how to create a local development environment or a staging site, and instead, you develop your website while it is live. You won’t want Google to index your site when it is under construction or in maintenance mode.
- There are also many people who use WordPress to create private blogs. They don’t want them indexed by search engines because they are private.
- You can use WordPress for project management or an intranet. In these cases, you won’t want your internal documents to be publicly accessible.
A common misconception is that if you don’t have links pointing to your domain, then search engines will probably never find your website. This is not completely true.
For example, there may be links pointing to your site because the domain name was previously owned by someone else. Also, there are thousands of pages on the internet that simply list domain names. Your site may appear on one of those.
With that being said, let’s take a look at how to stop search engines from crawling your website. We will cover four methods:
Method 1: Asking Search Engines Not to Crawl Your WordPress Site
This is the simplest method but does not fully protect your website from being crawled.
WordPress comes with a built-in feature that allows you to instruct search engines not to index your site. All you need to do is visit Settings » Reading and check the box next to ‘Search Engine Visibility’.
When this box is checked, WordPress adds this line to your website’s header:
<meta name='robots' content='noindex,follow' />
WordPress also modifies your site’s robots.txt file and adds a disallow rule to it:
User-agent: *
Disallow: /
These lines ask robots (web crawlers or bots) not to index your web pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect the request, there is still a chance that some pages or images from your site may get indexed.
If you want to make it impossible for search engines to index or crawl your website, then you will need to password-protect your WordPress site using Methods 3 or 4.
Method 2: Asking Search Engines Not to Crawl Individual Pages
You might want search engines to crawl and index your website, but not include a certain post or specific page in search results pages.
The easiest way to do that is using the All in One SEO (AIOSEO) plugin. It is the best SEO tool for WordPress and is trusted by over 3 million businesses.
For this tutorial, we will be using the AIOSEO free version as it includes the SEO Analysis tool. There is also a premium version of AIOSEO that offers more features like sitemap tools, redirection manager, schema markup, robots.txt editor, and more.
The first thing you will need to do is install and activate the AIOSEO plugin on your website. You can learn how to install and configure the plugin by following our step-by-step guide on how to set up All in One SEO for WordPress.
Once the plugin is set up, you can use it to ask search engines not to index certain posts and pages. Again, it is totally up to search engines to accept this request or ignore it.
Simply edit the post or page that you don’t want to be indexed. You need to scroll down to AIOSEO Settings at the bottom of the WordPress editor and then click the ‘Advanced’ tab.
Notice that the article is using the default robots settings.
To change this, you need to switch the ‘Use Default Settings’ toggle to the off position.
Now you can click the ‘No Index’ checkbox. Once the post is published or updated, search engines will be asked not to index it.
To learn more, see our detailed AIOSEO review.
Method 3: Password Protecting an Entire Site Using cPanel
If your WordPress hosting provider offers cPanel access to manage your hosting account, then you can protect your entire site using cPanel.
All you have to do is log in to your cPanel dashboard and then click on the ‘Directory Privacy’ icon in the ‘Files’ section.
Next, you need to find the folder where you installed WordPress. Usually, it is the public_html folder.
After that, simply click the ‘Edit’ button next to that folder.
Note: If you have multiple WordPress sites installed under the public_html directory, then you need to click on the public_html link to browse those sites, and then edit the folder for the website you want to password-protect.
This brings you to a screen where you can turn on password protection.
Simply check the box that says ‘Password protect this directory’ and click the ‘Save’ button. If you like, you can also customize the name for the protected directory.
You will see a confirmation message saying that the access permissions for the directory have been changed.
Next, you must click the ‘Go Back’ button.
You will be taken to a screen where you can create a username and password that will need to be used when accessing this directory.
You need to enter a username and password and then confirm the password. Make sure to note your username and password in a safe place, such as a password manager app.
Once you click the ‘Save’ button, you have successfully added password protection to your WordPress site.
Now, whenever a user or search engine visits your website, they will be prompted to enter the username and password you created earlier to view the site.
Method 4: Password Protecting WordPress With a Plugin
If you are using a managed WordPress hosting solution, then you may not have access to cPanel. In that case, you can use a WordPress plugin to password-protect your site.
Here are the two most popular solutions:
- SeedProd is the #1 coming soon and maintenance mode plugin for WordPress, used on over 800,000 websites. It comes with complete access control and permissions features that you can use to hide your website from everyone, including search engines. We have a step-by-step how-to guide for SeedProd.
- Password Protected is a very simple way to password-protect your WordPress site with a single password (no user creation needed). See our step-by-step guide on how to password-protect a WordPress site.
Expert Guides on Search Engine Crawling and Indexing
We hope this article helped you stop search engines from crawling or indexing your WordPress site. You may want to see some other guides related to search engine crawling and indexing:
- How to Hide a WordPress Page From Google
- How Do I Get My WordPress Site Listed on Google? (Beginner’s Guide)
- Why Did My WordPress Site Get De-Indexed From Google?
- How to Ask Google to Recrawl URLs of Your WordPress Site
- How to Optimize Your WordPress Robots.txt for SEO
- The WordPress SEO Crawl Budget Problem and How to Fix It
If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.
Oyatogun Oluwaseun Samuel
Wow!, Thank you wpBeginner, I have never thought about turning on password protection on public_html folder. Thanks for providing this perspective. thank you so much!
Jiří Vaněk
I have read that often these procedures, as well as robots.txt, are only recommended behavior for a robot. How to avoid being searched for real. Is it possible, for example, to block the ranges of Google and Bing, for example, with the help of an .htaccess file?
WPBeginner Support
You would use method 3 or 4 for completely blocking access to your site.
Admin
Jiří Vaněk
I understand that, but I assume that then the website will not be public and only those who know the login and password will be able to access it? My idea was just to try to find the ip ranges of the bots and block them at the htacess file level eg. Does that make sense or is it nonsense?
WPBeginner Support
There is no guarantee that the bots would not find the site at a later date with a different IP. If you want to be certain that your site is not crawled you would want it to be a private site, otherwise the best option is to use the robots.txt to request the site not be crawled.
Jiří Vaněk
I understand, and thank you for the response. I thought I could resolve it by blocking the range, but you’re right that the bot could come from a different IP or a new range. I’ll try to rely on the robots.txt file and hope everything will be fine.
mohadese esmaeeli
Is the prevalent belief correct that having a large number of indexed pages within a site has no effect on indexing the rest of the pages?
WPBeginner Support
Just because you have pages that are indexed on your site does not mean you can’t have other pages hidden from crawlers to prevent indexing.
Admin
Moinuddin Waheed
I have been several times in the thinking while in the settings tab of the wordpress dashboard that why would any one would want to discourage Google to show the page or post from visibility. Today infact after reading this post , I came to know about the reasons why would anyone discourage search engine indexing.
it is very simple to do infact. we need to check the box in settings tab reading section where it asks to discourage search engines for visibility.
Thanks for clearing my long held doubt.
WPBeginner Support
Glad we should share why someone would use the checkbox
Admin
Charlotte
This is was just what I was looking for. After I moved my dev site to the client’s live site, I didn’t want any traffic coming back to the old one. Thank you!
WPBeginner Support
Glad our guide was helpful
Admin
Ryan
I deleted my site permanently from wordpress and when I google my name, the old wordpress site still appears, despite the fact that it’s gone. Can someone from the support team help me to delete it? I don’t want my clients getting confused with my actual, live site.
WPBeginner Support
For your old site in the results on Google if you crawl the site as google it can help update Google’s cache. For more ways to completely remove the site you would want to take a look at our post here: https://www.wpbeginner.com/wp-tutorials/how-to-permanently-delete-a-wordpress-site-from-internet/
Admin
Divyanshu
if we disable indexing the wordpress site to prevent it from hackers, then will it affect google indexing or not, and if then how to allow google to index my pages
Andy
Doesn’t seem to work. Followed the directions with confidence. My subdomain is still publicly viewable.
WPBeginner Support
Hi Andy,
If you have password protected your subdomain directory, then please review its settings again in cPanel. If everything looks alright, then you may want to contact your WordPress hosting provider.
Admin
Jay
Yeah, whats written under “Discourage search engines from indexing site” is very true. Google doesnt seems to honor this
chinju
Hi, I have a doubt that if website have 1 million subpages, search engine block all subpage when we add
User-agent: *
Disallow: /
in robots file?
Shekhar Sardar
Yes. Adding Disallow:/ in robot.txt will tell all search engine bots, do not crawl this site or web page.
anand
i want memebership pages member profile page to be not indes by any search engines how to do ????
Kenneth
I tried this but it messed up my .htccess for my Wordpress install.
So proceed with caution.
Joojo
Hello,
I’m currently having a Wordpress.com Blog (Free Plan) & desire to sell my Digital Stuff online without setting-up a website and relying wholly on the Blog, at first (have been short of funds). Thus, going through with this thought I have made changes to my Blog, under this after making the payment, the customer is redirected to my blog page haivng a link to download the digital content. So, I just want the payment gateway to be able to redirect the traffic to the Download page on my blog and not have it searchable via search engines, etc. & neither be it listed under the Wordpress.com Posts list/Pages, etc. because if the download page is searchable by the Search Engines & listed under my Blog’s post, I wouldn’t earn anything because then the customer would directly download the content without making the payment.
Regards,
Joojo.
Jennifer
I’m learning WP now and designing my site. For now, I don’t want the site crawled, so I’ve checked the Search Engine Visibility Box and I’ve add the password protection. However, in a few months, I’ll want the site to go live and I’ll want visitors to my site. If I uncheck the Search Engine Visibility box and take off the password protection, will the robots nofollow header automatically be updated? Or will I have to manually update this in the header section?
WPBeginner Support
Yes, it will be. However, to be extra sure you need to submit your site’s xml sitemap using Google webmaster tools. This will allow you to see if any crawl errors are found on your site.
Admin
Ryan
Is it possible to block only specific pages on your site, rather than blocking the entire site?
Chad
Wordpress offers a Password Protect Option on any of your pages and posts. Simply change “visibility” in your page or post edit to “password protected”.
WPBeginner Staff
Please take a look at this thread.
Patrick
How can I get Google and bing to crawl my site while stopping SeMalt from crawling my site. SeMalt sends traffic to my site making it look like I have more hits, however thesclicks leave my site straight away and by so doing so increase the bounce rate to my wordpress site.
Any help on this matter would be great.
Patrick
Pawan Reddy
Thanks for this tip. I didn’t know this was possible