Can you tell me if this is good robot txt:
User-agent: * Disallow: /wp-content/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /cgi-bin/ Allow: /wp-content/uploads/ User-agent: Googlebot Disallow: /.pdf$ Disallow: /.php$ Disallow: /.js$ Disallow: /.cgi$ Disallow: /.xhtml$ Disallow: /.php* Disallow: /.inc$ Disallow: /.css$ Disallow: /.txt$ Disallow: /?* Disallow: /wp-* Disallow: /feed/ Disallow: */trackback/ Disallow: /cgi-bin/ Disallow: /go/ Allow: /wp-content/uploads/
User-agent: Googlebot-Image Allow: /
Nobody can help me?
Please tell me what a good Robot txt is.
- Envato Staff
- Has been a member for 4-5 years
- Attended a Community Meetup
- Beta Tester
- Contributed a Blog Post
- Contributed a Tutorial to a Tuts+ Site
- Repeatedly Helped protect Envato Marketplaces against copyright violations
- Sold between 5 000 and 10 000 dollars
A good robot txt will just depend on your needs and your situation. You would probably find more information about best practices, etc. by doing some Google searching. You might also want to look into .htaccess stuff to prevent certain user agents or block certain directories, etc.