Goal : To properly create or optimize your robots.txt file.
Ideal Outcome : You have an excellent robots.txt on your website that allows search
engines to index your website exactly as you want them to.
Why this is important : Your robots.txt sets the fundamental rules that most search
engines will read and follow, once they start crawling your website. They tell search
engines which parts of your website you don’t want (or they don’t need) to crawl.
