A robots.txt file lets search engines (Google, Yahoo, MSN, etc) know which pages on your site you don't want them to index. This is helpful in keeping them from going to pages that will confuse them and/or won't provide any added benefit to you from having indexed.
Since nobody has put one in here before, I've put together a simple version you can simply copy into your web site's root directory (usually called "public_html" or "homepage" or "www").
Once you've copied the file into place and made any desired changes, you're done.
Top 4 Download periodically updates information of robots.txt Sample File script from the developer, but some information may be slightly out-of-date.
Our script download links are directly from our mirrors or publisher's website. robots.txt Sample File torrent files or shared files from free file sharing and free upload services, including Rapidshare, MegaUpload, YouSendIt, MailBigFile, DropSend, HellShare, HotFile, FileServe, MediaMax, zUpload, MyOtherDrive, SendSpace, DepositFiles, Letitbit, LeapFile, DivShare or MediaFire, are not allowed!