Robots.txt
Martech Zone articles tagged robots.txt:
-
Search Marketing
Add Your Geographic Data to your Sitemap with KML
If your site focuses on geographic data, a KML sitemap can be a valuable tool for integrating with map services and accurately representing spatial information. A KML (Keyhole Markup Language) Sitemap is a specific sitemap used primarily for websites containing geographical information. While rich snippets and Schema markup can enhance your site’s general SEO, a KML sitemap can specifically aid…
-
Search Marketing
What is a Robots.txt File? Everything You Need To Write, Submit, and Recrawl a Robots File for SEO
We’ve written a comprehensive article on how search engines find, crawl, and index your websites. A foundational step in that process is the robots.txt file, the gateway for a search engine to crawl your site. Understanding how to construct a robots.txt file properly is essential in search engine optimization (SEO). This simple yet powerful tool helps webmasters control how search…
-
Search Marketing
Duplicate Content Penalty: The Myth, The Reality, and My Advice
For over a decade, Google has been fighting the myth of the duplicate content penalty. Since I still continue to field questions on it, I thought it would be worth discussing here. First, let’s discuss the verbiage: What Is Duplicate Content? Duplicate content generally refers to substantive blocks of content within or across domains that either completely matches other content…
-
Content Marketing
How to Block Search Engines from Indexing WordPress
It seems every second client we have has a WordPress site or blog. We do a ton of custom development and design on WordPress – everything from building plugins for companies to developing a video workflow application using Amazon cloud services. WordPress isn’t always the right solution, but it’s quite flexible and we’re pretty good at it. Many times, we…