Robots

Our crawler does respect Disallow rules from robots.txt files. If you feel it needs to support other rules as well please contact our support at tuki@valu.fi.