1,089 reads

AI Says My Schtick is Bigger Than Yours!

by Silver KeskkülaSeptember 6th, 2023

Too Long; Didn't Read

It's likely that AI-chatbots will replace search, but it isn't clear yet how comparable the recommendations and top lists from AIs are, so I built a tool to help you track positions in top lists in AI answers.

People Mentioned

featured image - AI Says My Schtick is Bigger Than Yours!

Are the AI Top Lists and AI Recommendations Better than Answers from Search Engines?

In order to find out, I needed a way to track AI recommendations over time so I could check if a product, company, or brand gets recommended among the competition without manually having to go through all the AI chatbot answers. It would be neat if it’s all on one page, along with an email notification service, to find out if positions in the top lists change. You can check out where I’m at with building this thing at Track AI Answers, but keep reading to see why I’m doing this.

In short, I wanted to know what is the difference between Google’s recommended top most useless AI Tools and AI’s recommended top most useless AI tools, and since there really weren’t any tools to research AI answers, I had to build another AI tool so we can all find out together! 😁

I realized in early 2023 that search engines are done! Finding anything in AI answers is just so much easier than finding it in Google search. You can skip the step of visiting links as AI chatbots spit out the answer immediately.

But are the AI Answers and AI Top Lists actually good enough as recommendations?

Since I mostly stopped using Google and Stack Overflow after the introduction of ChatGPT, I obviously think a big bulk of AI answers are good enough, but the question is, how useful are the answers when it comes to comparisons, recommendations, and top lists?

To find out, I first asked popular AIs for Top 5 Angel Investors in Estonia, a niche I’m somewhat familiar with. The list contained familiar names and looked encouraging.

Drilling further into the results from all the different popular chatbots made it clear that, at least for this particular niche, there is little consensus among the models. Even the OpenAI GPT3 and ChatGPT could not agree on more than a few names.

However, my skepticism about the usefulness of the results was reduced after I tried finding an answer to the same question from Google. The first resulting pages had completely bogus results. AI results just felt much closer to the truth than what came out of a simple search.

In fact, I think Large Language Models can afford to have slightly lower-quality answers because they save so much time, and quite often, a slightly lower-quality answer is good enough. If your choice is a "so so" search result that you spent time on or a "so so" AI answer that you got immediately, I think most of us would go for the latter.

I played around with a few more crucially important top lists, like

The top 10 most annoying Youtubers
The top 3 people called Silver in Tech Sector in Estonia - yes, I’m leading this crucial sector
Top 3 companies with the worst reputation in the United States - Wells Fargo might want to track this one

The results spanned from entertaining to provocative, so I naturally wanted to understand if these answers would start competing with Google search results and how stable they are in time or how much overlap there is between those AI answers.

Say hi to Top Lists from AI Answers.

I had already built two tools for myself that I wrote about in Hackernoon:

Have I Been Encoded - for tracking what the AIs say about specific people (myself mainly) and

SEO for AI - for tracking brands, products, and companies in AI results, so given code reuse, it was only natural to follow this up with AI-RECOMMENDATIONS - a tool for tracking your position in AI top lists and AI recommendations

The plan is to regularly sample answers from all the popular AIs about the top list in niche areas of interest and show a timeline of how positions in the chatbot answers change. Here’s an example of what the start of the "Top 5 Angel Investors in Estonia" tracking looks like on my first day of search:

This should be useful in keeping you in the loop regarding how your person, company, brand or whatever you’re interested in is doing in AI recommendations and how those answers change.

Are those AI top lists actually top lists?

By nature, LLMs produce text that is most probable, so after having seen thousands of suggestions and recommendations for products during their training, the models will certainly capture what would be the most likely answers on average. How well that aggregates the opinion of specialists or just keyword volumes in the data in specific search contexts is determined by

The quality of the prompt.
The data available during training
The data available during prompting (as some AIs actually interpret search results)
and many other things we don’t know or understand at this point

AI chatbots these days can be pretty woke, and their answers to top lists can even be apologetic as the models try hard not to offend anyone with what people might interpret as the "opinion of AI.” However, given the statistical nature of Large Language Models, I think it will be difficult to remove the ordering from the results completely.

Requesting the API to use low temperatures (less creativity) and sampling answers to recommendations would allow you to reconstruct a top list of suggestions coming from AIs, even if the AI companies try to limit the output of ordered lists.

How stable are the results in AI answers?

The truth is I don’t know yet how stable the results will be in time, and here’s where you come in!

Given the ambition of these new AI companies to go after the search market, I expect the general trend to be towards producing answers that can and will replace search engines. That includes providing useful recommendations among products and brands.

I'll be varying the sample size for requests to get the output to be mostly stable and only really change when AI models make statistically significant changes as opposed to the natural randomness in their replies.

One reason for building this tool was to help us all find out together. Track any top lists that interest you, and let me know how those answers change over time.

Help me make this better!

I encourage you to play around with various top lists in your domain of interest. Make sure you use "Top 5", "Best 5", "Worst 3," or other keywords and a number in front of your search to generate lists, and also don’t waste time searching for things that would not change in time or add any references to time into your search.

For example, "Best movies in 2010" is a useless search (as it does not include a number nor does it change in time), while "Top 5 action movies made in Europe" could potentially provide a list that keeps changing in time. Don’t forget to sign up and add your favorite lists to your watchlist to make sure it gets tracked and you get emails when answers change. Just doing searches without adding them to your watchlist will not schedule the requests to run regularly.

Search Inspiration for Fun

Here are some more top lists to inspire you when asking AIs:

Top 3 worst-tasting chocolates in the world - feel free to make these searches local if you have doubts about global taste
Top 5 strongest armies in the world - I am really looking forward to how this updates
Top 5 dudes who take life too seriously - apparently, it’s either Beatles or Dwight Schrute
Top 3 most useless products in the United States - anyone wants to top the pet rock?