# robots.txt is PUBLIC by design - crawlers fetch it at the site root before crawling. # It is NOT an access control: Disallow only asks compliant crawlers not to INDEX a # path, it blocks nothing. Sensitive paths (config.php, semlib.php, .env, includes, # admin, etc.) are protected at the server (403) and are intentionally NOT listed # here, so this public file does not advertise the app's internals. # # Below we only mark user-specific / transactional / non-content pages as noindex. User-agent: * Allow: / Disallow: /cart Disallow: /checkout Disallow: /profile Disallow: /wishlist Disallow: /compare Disallow: /search Disallow: /order/completed Disallow: /registration/completed Disallow: /auth/forgot_password Disallow: /reset_password Disallow: /maintenance # Bare category listing with no term/facet (Google/Bing honour the $ end-anchor). Disallow: /category$ Disallow: /category/$