Why You Should Set Up A robots.txt To Proxy Script

148 days ago

Why you should set up a robots.txt to proxy script? The key factor is prevent hotlinking and bandwidth theft. Hotlinking is an act of embedding images or any media files directly from the third party servers. Here is the risk potential for your proxy script.

1. Bots may crawl the hotlinking and eat up all your bandwidth. It's a huge waste of money for proxy business.
2. You are at risk of being suspended from an ad publisher if the hotlinking is illegal content. You'll lose your money source.
3. Duplicate content penalties may occur on your hotlinking. Also they are at risk of copyright violations.

You can set up a robots.txt as follows. Copy and paste the contents with memopad and save as robots.txt.