Optimise Google Crawl Efficiency

Crawl and Search Bots Optimisation

HTTP Headers Example

HTTP/1.1 200 OK
Date: Mon, 01 Jan 2018 16:42:19 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=def03863833e8e6322c36b77f0154befc1517503339; expires=Fri, 01-Feb-19 16
:42:19 GMT; path=/; domain=.seomanageruk.com; HttpOnly
Link: <https://seomanageruk.com/wp-json/>; rel="https://api.w.org/", <https://seomanageruk.
com/>; rel=shortlink
X-Powered-By: PleskLin
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/exp
ect-ct"
Server: cloudflare
CF-RAY: 3e665cfd1c923665-MAN

Last-Modified

Last-Modified: Mon, 01 Jan 2018 16:42:19 GMT

If-Modified-Since

# If it's modified since the if-modified-since date
If-Modified-Since: Mon, 01 Jan 2018 16:42:19 GMT
HTTP/1.1 200 OK

# If it's not modified since the if-modified-since date
If-Modified-Since: Mon, 01 Jan 2018 16:42:19 GMT
HTTP/1.1 304 Modified

ETag

ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

If-None-Match

# If Etag does not match
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
HTTP/1.1 200 OK

# If Etag does match
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
HTTP/1.1 304 Modified

Keep-Alive

Setting Keep-alive to keep connection active.

In the “connection” header, set “Keep-alive” to keep the connection action.

Setting a cache Time period

Cache-Control: max-age=2592000, public

Example Cache control for Images

# Cache One Week for image assets
<filesMatch ".(css|jpg|jpeg|png|gif|js|ico)$">
Header set Cache-Control "max-age=604800, public"
</filesMatch>