# Marketing Crawlers
User-agent: AhrefsBot
User-agent: AhrefsBot/4.0
User-agent: BacklinkCrawler
User-agent: GSiteCrawler
User-agent: ip-web-crawler.com
User-agent: ips-agent
User-agent: Linkdex
User-agent: linkdex.com
User-agent: linkdex.com/v2.0
User-agent: linkdex.com/v2.1
User-agent: Rogerbot
User-agent: MJ12bot
User-agent: SEMrushBot
User-agent: SpyFu
User-agent: Xenu
User-agent: Xenu Link Sleuth
User-agent: dotbot
Disallow: /
# ALL User-Agents
User-agent: *
Disallow:  ?PageID=*
Disallow:  Lang=*
Disallow:  /verify.aspx?v=*
Disallow:  /translate
Disallow:  /search?search=*
Disallow: /registration.aspx
Disallow: /*/data/products-root/*
 
Sitemap: https://www.xrite.com/sitemap.xml