Summary
Keywords
Full Transcript
Want to block certain file types or restrict search engine crawlers from crawling certain parts of your website? That’s what a robots.txt file is for & through this video tutorial you will learn how to use robots.txt file like an expert. https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/ Note: If you do not see the "Edit robots.txt" feature in Rank Math's General Settings, make sure to use the "Advanced Mode" of Rank Math, which can be found at the top-right corner of Rank Math's Dashboard. Chapters (Timestamps): 00:00 - Introduction 01:25 - Several Use Cases of Robots.txt 03:03 - Language Of Search Engines 03:16 - User-Agent 03:44 - Disallow Rule 03:57 - Allow Rule 04:25 - Crawl-Delay Rule 04:34 - Sitemap Directory 04:44 - Robots.txt syntax 05:43 - The Location Of Your Robots.txt 06:20 - Managing Robots.txt with Rank Math 07:04 - Default Robots.txt Rules 07:31 - The Structure of a Robots.txt File 08:29 - Block Filters and Sorting Pages 09:44 - Block Internal Search Pages 11:20 - Blocking Certain Paths of Your Website 11:45 - Block Certain File Types On Your Website 12:41 - Block All Web Crawlers Except One 13:06 - How To Debug Robots.txt Errors 14:39 - Outro List Of Search Engine Crawlers And Their Associated Names: ► http://www.robotstxt.org/db.html Robots.txt Testing Tool By Google: ► https://www.google.com/webmasters/tools/robots-testing-tool We hope to have made this Robots.txt tutorial simple and easy for you to understand and follow. It can seem intimidating at first, but it becomes rather easy if you take some time to understand it. All you need is just to understand the language search engines are speaking. Anyway, don’t forget to subscribe to our channel if you want to learn more about how to beat your competitors on search engines: ► https://www.youtube.com/c/RankMath/videos?sub_confirmation=1 Rank Math, the Swiss Army Knife of WordPress SEO, can take your SEO game to the next level. Rank Math already goes above and beyond other top-rated SEO plugins. Do you want to see the magic unfold for your website? ► Visit Rank Math today: https://rankmath.com Here's are 40+ unique features that set Rank Math apart: ► https://rankmath.com/blog/unique-rank-math-features/ Rank Math Links: ► Knowledge Base: https://rankmath.com/kb/wordpress/seo-suite/ ► Support: https://rankmath.com/support/ Connect with Rank Math: ► https://www.facebook.com/groups/rankmathseopluginwordpress ► https://twitter.com/rankmathseo ► https://www.linkedin.com/company/rankmath/
Continue this lesson in the app
Install CourseHive on Android or iOS to keep learning while you move.
