Are you Intuitive? Challenge my Machine!

Comparing human intuition against artificial neural nets

Computers Can’t Have Intuition… Right?

The word “intuition” is rarely used outside of the human context so it can seem insulting to apply elsewhere. It may however, be time to expand the application of the term.

Machine learning is now an integral part of modern life and its remarkable capabilities are often compared to those of the human brain. Should intuition be included as a trait that humans and machines share?

Are machines more intuitive than humans?

What is Intuition?

For humans and machines to be compared in a meaningful way, a definition of “intuition” must be settled on. Reflecting on a few living organisms should help inspire a useful definition.

Pattern Recognition in the “Living”

The C. Elegan (a small and well studied worm) is a creature that responds well to patterns in its environment. With just 302 total neurons, it “knows” enough to move and eat. WTG worm!

https://www.k-state.edu/hermanlab/images/elegans%20transparent.jpg

In contrast, humans have about 100 billion neurons (just in the brain). This many neurons helps give rise to emergent behaviors like the ability to use language.

Though vastly different, humans and C. Elegans have a lot in common. They both identify patterns unconsciously. You aren’t consciously decoding the words you are reading and C. Elegans respond to their environment without self-awareness.

A Definition of “Intuition”

There are many definitions of “intuition,” but the following definition works best for this experiment:

Intuition is the application of unconscious pattern recognition

First, this definition allows both humans and C. Elegans to be called intuitive. Most importantly, Since neural networks are not considered conscious (yet) and are great at pattern recognition, the same definition can be applied to the machines.

This experiment now has a basis for comparing intuition.

The Experiment

The goal of this experiment is to compare unconscious pattern recognition between humans and machines. How do you test unconscious pattern recognition in humans? “The Eureka Factor” by John Kounios and Mark Beeman has the answer (I highly recommend this book for anyone interested in their own creativity and intuition).

In “The Eureka Factor”, there is a study sited that was done by Arthur Reber who is “a pioneer of intuition research.” In his study, participants are asked to look at strings of letters like this:

RNIWKQ

Each string of letters is generated by one of two rule sets. The rule sets introduce patterns into the word. One example of a rule in a rule set would be to place the letter “R” before every instance of “N”.

The participants were asked to classify each string into one of two “families”. They were told if they were right or wrong and the process repeated. Reber found that over time, participants could group these strings significantly better than chance.

Kounios and Beeman noted that what is REALLY interesting is that participants could not explain the rules they used during classification. Intuition (aka pattern recognition) had been developed!

If a neural network can be trained to classify the same strings, then measuring the accuracy of classification can be the point of comparison.

Gathering the Data

Humans and machines must use the same sets of strings for classification so building a web application was an efficient way to make this happen.

The following web app was created with an interface for creating rule sets that generate strings:

Configuring the Rule Sets

Once rule sets are created, the app can then present an online variation of Reber’s study to humans:

Human presentation of the modified Arthur Reber Study

Using the exact same rule sets, the application can export labeled data to be used for training an artificial neural network.

Now it’s time to analyze the data.

The Human Results

Results from 73 humans was collected using Mechanical Turk.

Before you look at the results feel free to take the Human Intuition Challenge. It takes about 2.5 minutes on average.

The test stops when you reach a pre-determined upper limit OR you hit 80% accuracy over 20 consecutive strings. For each participant the web application records the highest accuracy achieved over 20 consecutive attempts (MAX INTERVAL SUCCESS PERCENT):

Here is the analysis of the 73 humans:

• The average best scores over 20 consecutive strings was 69%.
• Only 23.3% of participants reached the 80% threshold.
• For the 23% of participants who hit the 80% accuracy threshold, it took an average of 43 strings.

Like Reber’s study, it does seem that on average people did better than chance (50% being chance in this experiment).

Additionally, for the 23% of people who hit the 80% accuracy threshold, they did so in an average of 43 strings! For those that hit the mark, they did so relatively quick!

Results for the Machines

The machines deserve to be challenged so the neural network should be compared to the most intuitive 23% of human participants who achieved 80% accuracy.

To match the 17 human participants that reached 80% accuracy in an average of 43 attempts, 17 training sets of 43 strings were created for the artificial neural networks.

The neural net would need to have 69% accuracy to match the entire human group and it would have to exceed 80% to beat those humans that seem to be particularly intuitive.

Drumroll….

The neural network with a single hidden layer achieved 77% accuracy!

It beat the entire human group by 8%, but lost by 3% to the best humans.

So what was the neural network learning? Take a look at the weights in the trained neural net associated with Scheme “A”:

Notice the “hot” (red) weights in row 5 of the grid at each column where vowels occur. Scheme “A” had a rule that forced the 5th letter in every word to be a random vowel so it’s evident the neural network was learning some of the patterns in the test data.

Shouldn’t the machines be able to do better?

Throwing the Machines a Bone

Machines have every opportunity to notice and respond to all information available in the data, so there was room for improvement.

Debugging a neural net is hard but visualizing the change in accuracy while training is a great place to start. The graph below shows the accuracy of 20 different runs through 150 classification attempts.

Accuracy of neural net (single hidden layer) during each step of training on a data set with 150 entries (x-axis)

At the 43rd attempt, there was still a lot of learning going on. Each trial’s accuracy is bouncing up and down and there is a lot of variation between trials. This large variance indicates that the neural net is still “fuzzy” and isn’t converging as fast as desired.

If it is true that ALL possible information can be seen by the neural net, then the structure of the neural net must not be optimal.

Optimizing the neural network

The first neural network was good at noticing letters in particular positions and detecting complete omissions of letters (see technical analysis).

It was NOT good at reacting to rules that created “relative” relationships between characters. For example, there could be a rule that says every “N” should be proceeded by an “R”. This two letter combination could appear anywhere in the string. The neural net needed to use this patterns better.

The neural network was adjusted to deal with both position dependent and position invariant patterns (see technical analysis). This approach paid dividends.

Drumroll #2…

The optimized neural net achieved 87% accuracy in 43 attempts and had much less variance from the previous neural net.

For comparison, here are graphs for accuracy over 20 different data sets (for 150 training steps) from the original neural net followed by the optimized neural net.

FIRST NEURAL NET (single hidden layer)

OPTIMIZED NEURAL NET (single hidden channel + 2hidden layer channel)

The average accuracy at the 43rd attempt improved by 13% and the variance was lowered substantially by 66%.

Overall, this neural net had 26% more accuracy than the entire human sample and had 7% more accuracy than the high performing group of humans.

The machines have won…. they found a better learning curve.

Conclusion

Are machines more intuitive than humans?

This experiment shows that for the task of classifying “random” strings the answer seems to be “yes”. There are many other recent breakthroughs showing that machines can perform as well as or better than humans. Here are a few examples:

• Google Translate Approaching Human Level Accuracy
• Google AI Beats Humans in ‘Go’
• Machine Learning Diagnoses Diabetic Retinopathy

Will machines prevail in pattern recognition in all possible scenarios?

The brain is a highly efficient parallel processing machine that physically changes as it learns. Machine learning is powerful but right now is mostly static and still relatively slow. A recent Wired article says it best right now: “Artificial Intelligence is More Artificial Than Intelligent”

So rest easy. It will be a few years before you will need to worry about locking your toaster up at night… zzzZZZ

Thanks for reading! Click the 💚 below to recommend this to others!

Related Stories & Links

• “Neural Network Design for the Human Intuition Challenge.”
• “How to Attract “Turkers” and be a Mechanical Turk Hero!”
• Source Code for the Human Intuition Challenge