Chapter 3
If your website isn’t visible on Google, all your online efforts are dead in the water. Google is the largest search engine on the planet. So you must please the Google gods, otherwise, your brand will be invisible to consumers. Nobody will know about your brand or what you do. Online visibility starts with crawling and indexing.
Getting online attention via your blog sounds easy.
Publish mouth-watering content. Share your posts on social media. Sit back and watch droves of visitors flood your site while you sip pina colada.
Perfect, right?
Well, it’s deeper than that.
When people do a Google search, the mighty Google search algorithm kicks into action, crawling, searching, filing, ranking, and finally spitting out the most appropriate results from among billions of web pages.
Crawling is the foundation of all. If Google doesn’t index your web pages, you are pretty much non-existent as far as search engines are concerned.
In this section, we will look at indexing, how it works, and how to get your website indexed faster.
What Is Crawling And Indexing?
Let’s start by explaining a few key terms:
Crawling
Crawling is when Google or another other search engine sends a robot to hyperlinks on the internet to discover new content.
Indexing
Indexing is the process of storing a crawled web page in Google’s extensive database.
Web Spider
A web spider, aka spider bot, is a robot that systematically sweeps or crawls the web at scale to index content across the entire internet.
Googlebot
Googlebot is Google’s web crawler or spider.
Now that you understand the basic terms. Let’s dig deeper into how Google indexing works.
How Google Indexing Works
The information superhighway is like a colossal library with billions of books that cover every conceivable topic on the planet. From sports to nuclear physics, from local news to breaking world news, from politics to animal health, you can find anything and everything on the web.
Google spiders always patrol the interwebs in search of new info. If they deem the content worthy to be indexed, they store it in the gigantic database. When you Google something, the search engine jumps into action. Its automated browsers move from site to site and page to page hunting for relevant content.
They search (and sort out) everything stored in the Google index, fish the pertinent info, and spit it out in the search results. The system grades or ranks the results by putting the most relevant items on the front page and the least relevant ones on the subsequent pages.
Easy, right?
Here’s a useful graphic that shows the entire process.
Source: Crazy Egg
The entire process involves three steps:
- Crawling.
- Indexing.
- Ranking.
While indexing and ranking are related, they are not the same. Indexing is registering for an exam. Ranking is writing the exam and passing it with flying colors. You sit on top of the pile as the best student and have the bragging rights.
But here’s the thing.
You write the exam if you are not officially registered for it. As long as your name isn’t on the list of exam candidates, forget about passing. That’s why indexing is crucial.
How To Check If Your Site Pages Are Indexed On Google
How can you tell if your website is indexed already? There are two simple ways to find out.
1. Check Via Google Search
First up, you can tell if your site has been indexed by doing a simple Google search of your website.
Use this search query site:yourdomain.com Make sure there are no spaces in between. You should get results similar to the one for Freedom To Ascend shown below:
At the top of your search, you will see the approximate number of indexed pages on your site as shown by the arrow. You won’t see any results if your site isn’t indexed yet.
2. Check Via Google Search Console
Alternatively, if you have already set up Google Analytics and Google Search Console, you can use the latter to check if you are indexed.
Both are free tools to help you understand visitors’ behavior and other important metrics on your site.
Go to your Google Search Console dashboard.
Select ‘Coverage’ on the left side menu. You will see results close to this:
Indexed pages show up in the valid or valid with warning boxes. Non indexed pages won’t appear at all or will appear in the red error box, so you have work to do to make sure they eventually show up. We will get to those tactics later.
In Google Search Console, you can also manually check if individual pages are indexed. To do that, grab the URL of the page you want to check and paste it on the search bar on top.
You should see this.
You can see the green ticks informing you ‘URL is on Google’ and under it ‘Coverage, Submitted and indexed’. If the page isn’t indexed, you will see the ‘URL is not on Google’ notification.
Fairly straightforward, right?
Let’s get into the practical steps you can take to get your pages indexed.
Give Google Search Spiders Permission To Crawl Your Site
Google not crawling your site all?
Then your site’s robots.txt. file may be blocking the Googlebot. You need to remove the roadblock so the bot is free to crawl your site.
To find the blockade, go to yourdomain.com/robots.txt.
Look for these bits of code:
Source: Ahrefs
These pieces of code instruct Google spiders not to crawl your site. Remove them and voila! You’ve just given Google the green light to crawl your site.
Now that your site is crawlable, let’s see how to get the Googlebot to index more pages on your site.
How To Get Your Pages Indexed Fast (9 Easy Tactics)
While giving Google permission to crawl your site is good, it’s just that, a start. You must do much more. Here are 9 strategies for faster site indexation.
1. Submit Your URLs To Google Search Console
The first and easiest method to get your site indexed is to request indexing on Google Search Console (GSC).
Submitting a request is simple.
To submit an individual page:
- Hit ‘URL inspection’ on the left of your GSC dashboard.
- Paste the URL of the page you want to submit.
- Click on ‘Request Indexing’ on the right side of the page.
That’s it.
You have queued your page successfully for indexing.
The system also allows you to submit multiple URLs at once. To do that, repeat the above process, but submit a sitemap instead of an individual URL. Please note the following important points about indexing:
- Crawling can take anywhere from a few days to a few weeks. Yes, a few weeks. You read that right. I know it sucks but please be patient.
- No submission method is faster than the other—they both have the same response time.
- There is a limit to the number of URLs you can submit.
- Don’t waste time by requesting crawling for the same URL multiple times because your URL or sitemap won’t get crawled any faster. Once again be patient, pal.
- Asking for a crawl doesn’t mean they will grant it. Ouch. The Google algorithm prioritizes premium, useful content for fast inclusion. So to boost your chances of getting indexed, write exceptional content.
- Crawling isn’t a ranking factor.
Make requesting indexing a part of your workflow. Do it whenever you publish a new post or page on your site. You are giving Google a nudge to come and check out something new you’ve added to your site.
Besides submitting an indexing request, there are other tactics you can use to woo crawlers to your site.
2. Produce Unique, Valuable Content
Google obsesses over top-notch content.
They exist to sift billions of pages in their database and serve up only the creme de la creme content that fits the searcher’s needs. Their purpose is to serve the best content to users, so they are always on the lookout for tremendous content.
So the more you produce fresh content, the more likely spiders will swoop in to check it out.
Research by Demand Metric reveals that companies with active blogs that produce fresh content steadily have an astonishing 434% more indexed pages and 97% more indexed links than those with inactive blogs.
Feed the crawlers with delightful fresh content, they’ll come running.
3. Include The Page In Your XML Sitemap
An XML sitemap is just that—a map of your website.
It’s a roadmap that leads Google by hand to your foremost pages. Your sitemap is the gateway to crawling.
As Google explains:
“The crawling process begins with a list of web addresses from past crawls and sitemaps provided by website owners. As our crawlers visit these websites, they use links on those sites to discover other pages. The software pays special attention to new sites, changes to existing sites, and dead links. Computer programs determine which sites to crawl, how often and how many pages to fetch from each site.”
Notice that the sitemaps guide the whole crawling process.
Without the help of sitemaps, crawlers take too long to crawl your site and squander the crawl budget. No doubt having a sitemap increases the crawl rate because it allows Google to grasp your site structure fast.
How To Create A Sitemap And Submit It To Google
Creating a sitemap from scratch can sound daunting.
But it’s easy with an efficient SEO plugin like Yoast SEO.
Follow these simple steps.
Start by installing the Yoast SEO plugin.
Inside your WordPress dashboard, go to ‘Plugins’ and then ‘Add New’. Search for Yoast SEO and then hit ‘Install Now’ and then ‘Activate’ after you’ve identified the plugin at the top of the results.
Once the plugin is active, click on Y SEO > General > Features. After that, turn the XML Sitemaps toggle on.
Next click on the question mark icon next to XML sitemaps and then hit ‘See the XML Sitemap’.
You will then see your sitemap. It looks like this.
Now that you have a sitemap, it’s time to submit it to Google. To do that, grab your sitemap URL from the top of your WordPress dashboard and go to your Google Search Console dashboard.
Go to ‘Sitemaps’ > ‘Add a new sitemap’
Paste your sitemap URL into the provided space and hit ‘Submit’.
Congratulations. You’ve submitted your site to Google. Search spiders will come calling to crawl your site soon.
4. Build Relevant, High Quality Backlinks To Your Content
A backlink, aka inbound link, is when another website links to a page on your website.
Backlinks play a pivotal role in search engine optimization (SEO).
They are votes of confidence that your content is valuable hence external websites reference it. In indexing, Google views pages with many backlinks as more beneficial than those without. For that reason, they crawl and re-crawl them more often than those with fewer or no links.
That said, Google doesn’t only index pages with links.
There are gazillion indexed pages with zero backlinks. However, it is undeniable that a page with backlinks stands a better chance of getting indexed faster than one with no links.
5. Put A Stop To Duplicate Content
Another blockade that stops your site from being indexed swiftly is duplicate content.
Duplicate content is content that is very similar to other pieces of content on and off your site.
Why is duplicate dangerous?
Because it confuses search engines. Since the page content is almost the same, search robots don’t know whether to index this or that page or the other one.
See the poor guy below.
Source: Content King
The completely flummoxed fellow doesn’t know where to start.
Scour your website and remove all duplicate content. Crawlers will crawl your site more efficiently.
6. Maximize Crawl Budget By Removing Low-grade Content
We’ve said it a bajillion times already, but it bears repeating: Google prioritizes top-drawer content.
It’s the axis on which their complex algorithm revolves. Inferior pages on your site waste your limited crawl budget.
Google shed light on the issue:
“According to our analysis, having many low-value-add URLs can negatively affect a site’s crawling and indexing. Wasting server resources on pages like these will drain crawl activity from pages that do actually have value, which may cause a significant delay in discovering great content on a site.”
Other than that, substandard content doesn’t add value to your brand or readers’ lives. Better remove it altogether so you spread your crawl budget over fewer high-quality posts and get them indexed quicker.
From time to time cull your content to preserve the crawl budget. Target pages with:
- No backlinks
- No traffic
- No shares
- No comments
Cut off these non-performing content whose vital metrics are dismal despite your best efforts to promote and optimize it.
7. Generate Buzz Around Your Content On Social Media
What use is your excellent content if people (and search engines) don’t know about it?
To do that, promote your content aggressively on social media platforms. Get it out there. Share tasty tidbits to lure people to click through to your site to read the rest.
Social media shares and chatter around your content:
- Drives much-needed traffic to your content.
- May get you links from people who enjoy the content.
- Alerts crawlers that your content exists.
Google sees shares, links, and the buzz around your content as flares. They tell spiders there’s new, exciting content on your site, so they rush there to investigate it. Bingo! Your pages get crawled.
8. Maximize Internal Links
Your internal link structure tells Google which pages you deem valuable. Most of your internal links should point to your crucial pages.
Because search spiders follow the links trail on the internet, you can use smart internal links to boost your site’s indexation. Make sure you link to your new content from your strongest pages.
These are pages with a high Page Authority. On reaching the pages that are already ranking, search robots will follow the links and discover your new content. Also, link your new pages to your priority pages.
To discover your strongest pages, use a tool like Ahrefs.
Go to the Best By Links report.
Source: Ahrefs
These are your most authoritative link-juice-dripping pages. Link to and from them to lure search robots to your new, unindexed content. Make sure all the links are do-follow.
9. Increase Your Site Speed
Google loves speed.
They’ve even made site speed a ranking factor.
This need for speed can work against you if your website is slow. Slow sites take too long to respond to requests. As a result, Google may not crawl your site as often as they would if it was fast and efficient. This reduction in crawl budget brings about fewer indexed pages because all your pages aren’t being crawled.
If the Googlebot spends too much time crawling a sluggish page, the crawl budget gets depleted and there’s nothing left for the other pages.
The foundation of a blazingly fast website is getting an excellent host. Look for a web hosting company that hosts your files on super fast servers and has a reputation for both uptime and speed. Compare notes on several hosting review sites and go for the company that’s featured on most lists. For instance, here are recommendations from Bitcatcha based on their first-hand experience:
Source: Bitcatcha
Knowing you have the backing of a lightning-fast hosting company, you then optimize for speed.
- Compress bulky images.
- Avoid multiple page redirects.
- Minify CSS and Java Scripts.
- Remove unused CSS and Java Scripts.
- Serve images in next-gen formats such as JPEG 2000, JPEG XR, and WebP
- Enable text compression.
- Preload key requests.
- Eliminate render-blocking resources.
- Avoid an excessive DOM size.
- Enable efficient caching.
- Avoid long main-thread tasks
If all this sounds like Greek to you, don’t worry. Hire a developer to fine-tune your speed. Or, if your budget is tight, use free tools like GTMetrix to check your site speed and get recommendations on what to do to speed up your site.
Improve site speed to lure crawl robots to visit your site more often.
Having said all this, remember:
Indexing ≠ Ranking
Ranking and indexing aren’t synonyms.
Indexing simply means Google knows your content (and website) exists. But it doesn’t mean they will automatically bump it to the top of the SERPs just because they’ve indexed it.
No ways.
You’ve got to do many things right to rank for the keywords that matter to you.
Target the right keywords.
- Produce unique choice content.
- Do expert-level on-page SEO.
- Build links to your content.
- Promote your content vigorously.
- Have a decent Domain Rating.
All indexing does is help you turn up on the ranking race starting blocks. You still have a long way to go to win the race (get ranked) but at least you know you stand a chance. You’ve got to be in the race to win the race.
Crawling is the entry point into the Google ranking race.
