I removed www.example.com/foo/file.html from my domain. But, it is still being crawled by Google/Yahoo and many other bots. How do I return a HTTP-410 gone error i.e. tell these bots that resource permanently does not exist using Apache HTTPD server on a CentOS/RHEL/Fedora/Ubuntu/Debian and Unix-like operating system?
The http error status code 410 indicates that the resource (images,css,js and all other files) requested is no longer available and will not be available again ever. This 410 status code may be used when a resource has been intentionally deleted or part of unique url generated using some sort of token for given session (for example cached pdf file). Upon receiving a 410 status code, the client or bot should not request the resource again in the future. Bots such as search engines should remove the resource from their index. In this quick tutorial you will learn how to serve HTTP status code 410 from Apache rather than a 404 or 403 error code.
Configuration
The simplest configuration is to add the following line in .htaccess file using mod_alias Redirect directive. Cd to your DocumentRoot such as /var/www/html/:
$ cd /var/www/html/
$ vi .htaccess
Append the following line:
Redirect gone /foo/file.html
OR
Redirect 410 /foo/bar/demo.php
You can also use of regular expressions using RedirectMatch directive as follows:
# Syntax RedirectMatch gone regex-here # Match all .png files in /foo/ RedirectMatch gone "/foo/.png$" # Another example for gif files starting with bar name RedirectMatch gone "/foo/bar*.png$" # One more example. We now have resposive site so remove all old mobile friendly html pages RedirectMatch gone "/mobilesite/*.html$" |
Adding friendly message page
Although the Apache Server disparages generic error responses in the event of 4xx or 5xx HTTP status codes, these responses are rather stark, uninformative, and can be intimidating to site users. You may wish to provide custom error responses which are either friendlier, or in some language other than English, or perhaps which are styled more in line with your site layout. So append the following code:
ErrorDocument 410 /errorpages/410-mobile.gone.html |
Save and close the file. Next create 410-mobile.gone.html in your DocumentRoot directory (for e.g. /var/www/html/errorpage/)
$ mkdir /var/www/html/errorpage/ && cd $_
$ vi 410-mobile.gone.html
Append the error message as per your needs:
<html> <head> <title>Page Gone - 410 Error</title> </head> <body> <blockquote> <h1>Error 410 - Page deleted or gone</h1> This might be because: <ul> <li>You have typed the web address incorrectly, or the page you were looking for may have deleted.</li> </ul> Please try the following options instead: <ul> <li>Use <a href="/search.html">search option</a> to see if it's available elsewhere. Or visit our home page for the latest info.</li> </ul> <hr> <small>If you feel like it, mail the url, and where you came from to [email protected]</small> </blockquote> </body> </html> |
Your visitor will see the page as follows along with HTTP/1.1 410 gone status code:
Verify error code
Simply type the following curl command:
$ curl -I www.example.com/foo/page.html
$ curl -I www.example.com/mobilesite/4242.html
Sample outputs:
HTTP/1.1 410 Gone
Server: Apache
Date: Mon, 14 Dec 2015 14:52:28 GMT
Content-Type: text/html
Content-Length: 335
Connection: keep-alive
Please note that you must get ‘HTTP/1.1 410 Gone’ as status code.
How do I generate a HTTP-410 error for complete domain?
Add the following to your VirtualHost or .htaccess. This is done using mod_rewrite:
RewriteEngine On RewriteCond %{HTTP_HOST} ^www.example.com$ [NC] RewriteRule ^(.*)$ - [L,G] |
Save and close the file.