+ Post New Thread
Results 1 to 5 of 5

Thread: How to disallow/ban Google bot to view certain webpage

  1. #1
    Administrator Fli's Avatar
    Join Date
    Mar 2013
    Posts
    2,245
    Post Thanks / Like
    Blog Entries
    1

    How to disallow/ban Google bot to view certain webpage



    How to disallow Google bot to view a webpage?

    Here is one idea:
    Code:
    <?
    if(strpos($_SERVER['HTTP_USER_AGENT'],'google') !== false ) { header('HTTP/1.0 404 Not Found'); exit(); }
    if(strpos(gethostbyaddr(getenv("REMOTE_ADDR")),'google') !== false ) { header('HTTP/1.0 404 Not Found'); exit(); }
    ?>
    If user agent contains "google", then header 404 (not found) is sent and script is stopped.
    If IP address host contains "google", then same thing happens.

    ---------
    Similar topic, how to hide part of webpage from bots: https://internetlifeforum.com/html-css-forum/1743-how-hide-link-other-part-webpage-bots-like-googlebot/

  2. #2
    Junior Member ELyon01's Avatar
    Join Date
    Jun 2017
    Posts
    5
    Post Thanks / Like


    Is this useful / helpfull? Yes | No
    Hey there! thanks for that code. Helped a lot.

  3. #3
    Junior Member nesir's Avatar
    Join Date
    Sep 2017
    Location
    South Africa
    Posts
    11
    Post Thanks / Like


    Is this useful / helpfull? Yes | No
    Two way you can use you rbt.txt file , or you can use yout htaccess file on the root of your server ht acces would look like this :

    RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC] RewriteRule .* - [R=403,L] or

    disallow google bor in rbt text file. Htacess is a better option as you can redirect google and also rbt.text makes google think you are hiding pbm networks.

    Also you can use the noffollow , noindex tags for specific pages.

    I should tell you that google visits pages under anonymous user agent strings from time to time , to see what you are up to , u can stop it from being indexed but it will get crawled eventually.

  4. #4
    Senior Member RH-Calvin's Avatar
    Join Date
    Jul 2014
    Location
    Forum
    Posts
    503
    Post Thanks / Like


    Is this useful / helpfull? Yes | No
    You can also use robots.txt to allow and disallow certain webpage on your website as required.
    Cheap VPS Hosting | VPS Starting from $1 per month
    Cheap Dedicated Servers | Free Setup with IPMI

  5. #5
    Junior Member nesir's Avatar
    Join Date
    Sep 2017
    Location
    South Africa
    Posts
    11
    Post Thanks / Like


    Is this useful / helpfull? Yes | No
    Quote Originally Posted by RH-Calvin View Post
    You can also use robots.txt to allow and disallow certain webpage on your website as required.

    Yes but rbt.txt is passable by anyone who wants to , i know backlink bots can bypass it easily and often do. Google also does not obey the rbt txt . Easy way to to test this , set your rules in rbt text and htacess then fetch the page as google and see if google fetch it succesful. if youve disallowed google properly should come up with a 403 error. Also googlebots has a separate crawler for mobile so youd need to block that too. Google also indexes information from bing index so youd need to block bing aswell.

+ Post New Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
 Protected by : ZB BLOCK  &  StopForumSpam