The Revolution Has Long Been Underway
2021 update by Bob Sakayama, CEO TNG/Earthling, Inc.
It's the kind of seismic change that has largely gone unnoticed even as its influence mushroomed. The introduction of artificial intelligence into the growing data capabilities of every industry, has been the invisible hand disrupting the old ways and providing better outcomes by using machine learning to optimize tasks and automate decision making.
The AI assimilation has long been underway. Individually developed systems can be trained to perform seemingly unrelated tasks, like this pastry shop AI that initially learned to identify pasty by shape and color. Using basically the same kind of visual rules it used on the pastry, it was later trained to identify cancerous cells. AI is everywhere and has for some time been making often significant changes to every aspect of life.
And that includes Google's search results. A long stated goal of Google is to actually understand why the search terms were used so they could provide more useful answers. The intent or motive behind the search terms is the holy grail. Training AI on the search semantics is how they are achieving this goal, enabling high probability interpretations of the meaning and the context using data science, by training the machine to accurately guess the reason for the search.
If you are a student of the search you already know that changes to Google's algorithm happen very frequently, and it's impossible and unncessary to keep track. Most are minor tweaks that don't impact most sites. But some changes create disasters and need to get walked back quickly - loss of ranks is an existential crisis for businesses harmed, and the resulting public anger together with the embarrassing bad press are hugely motivating. That's the major reason their "hyper-local" change happened so slowly. Hyper local showed search results first filtered by closeness to the searcher. It was an improvement for pizza searches but disaster for sites with a national aspirations. Often, Google makes algo changes contrived to correct a problem - eg. the payday loan scams, or seos gaming Google. They recently made huge changes to address the liability in providing search results that could potentially end up harming people. The "Your Money of Your Life" sites sold products or gave information to users where a high ranking site created a liability for Google if it sold harmful or bogus product or gave ruinous financial advice. The algo changes required these YMYL sites to provide more evidence of their authenticity before permitting them to rank well. AI is used to identify YMYL businesses that need a higher level of scrutiny before ranks are awarded.
It only makes sense to apply AI to the semantics of the search query. And that process has been underway for long enough to become noticeable. The result is a deeper understanding of semantics, and natural language process - the association of intention & motive with words. Google has always been a semantic search engine. Once it determined ranks by identifying matching strings of text. The AI attempt is to determine ranks by matching intent.
Applying AI to SemanticsSemantics will always be a core study of the Googlebot and while how it evaluates the words will change, the importance of the motive behind the search has always been the objective. Search relies on languages, which can relate differing ideas and meanings to the same words. With enough supporting information, semantics can become a window into the actual meaning of the words being used. Matching the intent with the search results rather than matching a string of text, is the next generation of search. Where, by having access to huge amounts of data regarding the words being searched, Google may accurately assess that your search for "serendipity" was looking for a movie, and not a definition. This requires AI. Google has been improving its ability to recognize the real motive and meaning of a query, and this has been gradually altering the search results. But the application of artificial intellegence is meaningful because it implies that the rules governing ranks are shifting to decisions and conclusions made by machines trained to recognize alignment with the search engine's rules.
From the published portions of the relevant Google patent it has been obvious for a long time that the organization of the information is critical. Their model for the most semantically relevant document addressing the intent of the searcher was one supported by the most robust knowledge base as well as authoritative inbound links. This has always been the case when described this generally.
What has changed is the sophistication level of the algorithm, which can now reference previous searches, clicks, etc. enabling a better read of the intent of the search. In a former world, the semantic relationships were string based - the same set of words or phrases might match those terms used on another page which would then rank high for those matches.
The amount of information available around any given search term is staggering. This would include some obvious things like historic data about the searcher, language connections between semantics and meaning, data relating idioms and vernacular to meaning, data on synonyms, slang, etc., etc. Machine learning, or artificial intelligence applied to understanding searches can make highly reliable predictions regarding search intent. Because now the search semantics can be compared to other search results that share meaning, but not spelling. It's no longer matching the words searched, but the meaning.
Google has had many years to integrate AI into its systems - especially when you consider that machine learning began in the 1950s. But it's pretty obvious that their success has been spotty. For example, the search results for "books by children" has no books written by children, only children's books when searched in September 2021. But this result may be intentional given the relative Google Ads revenue generated by searches for "childrens books" vs "books by children".
But they are getting better at it. Frequent Google users know the search results have been improving. As the search engines get this right, their results become more valuable.
The other changes underway involve the use of AI to detect efforts to game Google's search results. Some of these impact SEO directly. The most significant very recent change is the transformation of nofollow links from worthless to valuable. This corrects a long overlooked flaw in the handling of text links from other websites. Google encouraged the tagging of certain text links "nofollow" which meant the Googlebot would not crawl the link - it would be ignored and no link equity will be passed. Text ads would need this tag so paid links could not push rank. Paid links that ignored this rule could get the site penalized in Google. This rule has been in place for many years and it is standard practice to tag a sponsor's text link as nofollow.
But very recently, Google made a philosophical change regarding nofollow, adding some qualifiers to the tag, and recognizing the value of a nofollow link in providing information useful for the search results. "Sponsored" and "UGC" can now be added as additional qualifiers to convey an ad and user generated content. But more important than the effort to get more granular information from webmasters is the change that will most impact SEO going forward.
We started running tests on this as soon as we learned about the change, early 2021, starting by actively grooming the nofollow links we control, like all those paid text ads, to use valuable keywords. These experiments have not yet moved the needle - we're seeing no rank improvement when adding nofollows to an existing link profile.
But there's one experiment that is very telling because it was with a relatively new site, and the only links on important keywords were nofollow. 3 months after the links were added the site's organic traffic started hitting new all time highs. This first experiment with important results involved a local consumer service business. Below is the organic traffic chart from SEMrush for their website. The experiment was to use only nofollow links from reasonably authoritive domains. In the chart below, links were added about the time of the first bump in June 2021 when 25 nofollow links on valuable anchors were posted without either of the 2 new qualifiers.
This was an all in experiment, meaning only the anchors of the nofollow links carried important semantics in an attempt to see if Google will use those semantic signals in their search results. It makes sense to use information from ads to determine the nature of the business buying them. If a company has a lot of nofollow ad links with anchors on "widgets" it's pretty clear that's an important semantic for them and their ranks should indicate this. But there is a risk here that Google is definitely aware of as they elevate the status of nofollow. Google has long claimed that money can't buy rank. So Google may not want to be revealed to be enabling paid, nofollow text links to positively influence ranks, but that is what appears to be happening here.
At this point in time we don't know enough to make any kind of claim regarding SEO tactics using nofollow. But this experiment is definitely revealing something important and future experiments are already underway to discover how we can make use of this major change in Google's handling of nofollow.