Here at Helpforce we operate our own purpose build web crawler commonly known as a robot. This is used by our servers to access and retrieve the content of web sites which we believe might contain technical support information. The Helpforce Bot is fully compliant with web crawling best practice and identifies itself as our crawler at all times
How does it work?
The robot will visit your root page and then from there fan out and follow all internal links it can find once to retrieve the content of your site. Whilst the Helpforce Bot is the visible part of our technical support analysis system it really is only a tiny part of a bigger picture. Once we have obtained a snapshot of a website then other mechanisms and analysis techniques are run on the data to identify technical support information which will might be useful in answering questions.
How quickly does it work?
That depends entirely on the size of the site we are accessing. The Helpforce robot is very careful not to overload the site in question and as such will index over a time period rather than just all at once. For most medium sized sites crawling will take around an hour.
How do I hide pages?
The Helpforce Bot will check your robots text file and abide by any rules that you have set up in here, if you want to ensure some content is not indexed then just disallow it
How can I stop it?
Whilst it is a shame that some people are unhappy sharing their technical support information to help others, we do appreciate that sometimes webmasters might want to stop us crawling their site. This is fine and we are happy to honour this request, just contact us
