The main job of search engine bots is to crawl and archive the pages of a website. The robots.txt file gives search engine bots information about where they can access and where they cannot, and shows the location of the sitemap.xml file. https://support.google.com/webmasters/answer/6062608?hl=en

« Back to Glossary Index
Emre Ercan on LinkedinEmre Ercan on Twitter
Emre Ercan
Expert at Stradiji
He was born in 1983 in Eskişehir, Turkey.
After graduating from the Hacettepe University School of Mining Engineering, For six years, he has work with various positions at Vodafone Turkey. After that he moved to the Media / Advertising sector and undertook reporting and planning duties within the People Initiative. He has been working as an SEO and Digital Marketing specialist at Stradiji since 2013. He also works as a photography and musician independently.