Shannon: a Hero of the Digital Age

In today's digital world, information has become the lifeblood of progress and empowerment. With unprecedented access to vast amounts of data, individuals and organisations can make informed decisions, innovate rapidly, and connect globally. However, the significance of information also underscores the need for responsible dissemination, critical evaluation, and safeguarding of data to ensure that the benefits of the digital age are harnessed effectively while minimising potential risks. But what exactly is information? Can we measure it? How do we actually communicate information over the internet or any other medium? What do social media apps and all electronic communications have in common? How did we get to today’s digital age?

The answer to the questions above dates back to the 1940s. A story about a visionary endeavour that proved to be not just successful, but transformative, reshaping the course of history itself. An endeavour that seamlessly intertwined engineering, science, and mathematics. From the crucible of this endeavour arose a nascent discipline known as information theory, which would go on to serve as the bedrock for the communication infrastructure that underpins today’s digital world.

While one might anticipate that a realm as expansive as information theory sprang forth from the collaborative synergy of numerous mathematicians, scientists, and engineers, history takes an intriguing twist. In a remarkable turn of events, it appears that this domain was the culmination of a solitary individual's decade-long endeavour. The genius behind this transformative feat was none other than Claude Elwood Shannon.

Transforming Circuit Design From Art To Science

In a convergence of diverse knowledge and unquenchable curiosity, Shannon's journey unfolded. Armed with dual bachelor's degrees in Mathematics and Electrical Engineering, he took an unexpected turn. A seemingly incidental philosophy course, undertaken to fulfill a university requirement, introduced him to the works of George Boole, the autodidact English logician of the 19th century.

Shannon applied Boolean algebra in switching circuits in his master’s degree thesis. According to many people it was the most important master’s thesis of the twentieth century. The starting point of digital circuit design, a transformative work that turned circuit design from an art into a science. Because of this thesis, circuit design is no longer a trial and error process. Designs can be tested mathematically, before they are built. Nowadays, engineers routinely design computer hardware, software and complex systems with the aid of Boolean algebra.

Before Shannon's master thesis, small devices that use magnetism to open and close electrical switches, called electromagnetic relays, were used to build circuits that routed telephone calls. However, there was no consistent theory on how to design or analyse such circuits. People thought about them in terms of the relay coils being energised or not.

Shannon used Boolean algebra to make abstractions. He shifted the focus from the relays themselves to the function of a circuit. He used this algebra of logic to analyze, and then synthesise, switching circuits and to prove that the overall circuit worked as desired. This is when the AND, OR, and NOT logic gates were invented. Logic gates are the building blocks of all digital circuits, upon which computers and computer science is based.

Connections to artificial intelligence

But if there is a way to apply laws of thought in switching circuits, we can apply human thought in machines. Shannon’s master’s thesis also lays the foundation of the idea that machines could be made to think.

The public response

During the 1940’s, applying Boolean algebra in electronic circuits was shocking. It was completely unexpected and many thought that however interesting, it was purely theoretical. Life has proved them wrong, however! When looking under specialised glasses and when our background is not diverse enough, it's often difficult to see the magic that lies underneath.

Science, Engineering and Mathematics: The Marriage

Science seeks the basic laws of nature. Mathematics is the universal language of logic that searches for new theorems to build upon the old. Engineering builds systems to solve human needs. The three disciplines are distinct, yet interdependent. It is knowledge as diverse as it spans all three disciplines, that resulted in groundbreaking social and technological changes.

Turning special cases into a unifying theory

Up to the 1940’s, any attempt for electronic communications was considered as a special case that demanded special theories and implementation tools. The huge advance at that time was the discovery of how to use electricity to communicate. For example, remote communication was done using analogue signals. Sending a message involved turning it into varying pulses of voltage along a wire, which could be measured at the other end and interpreted back into words. This works quite well for short distances but for long distances it becomes unusable. Every meter that an analogue electrical signal travels along a wire, it gets weaker and suffers more from random fluctuations, known as noise. One option was to boost the signal but this would result in boosting noise too. Special cases were unified and problems were solved in light of Shannon’s monumental paper in 1948, titled: “A mathematical theory of communication”.

A mathematical theory of communication

Shannon thought about the problem of communication in a radically new way. The basic problem was how to get a message from one place to another, fast and reliably, even when the two places are very far apart. He asked questions that no one has asked. What exactly is information and is there a universal theory that could allow perfect communication?

Here is a snapshot of his thoughts: “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.”

Information and meaning

To make information measurable, Shannon disregarded semantic content. Information has nothing to do with what we understand or not. Let’s say that we’ve learned the news from the media and most of it did not make much sense. We did get information. What kind of information we’ve got was not considered to be a communications engineering problem. Although it may sound counterintuitive and maybe even restrictive that information has nothing to do with content and meaning, treating information as such was a radical idea that made possible what seemed to be impossible.

Measuring information

According to information theory, information is the amount of uncertainty and it’s measured in bits. Assume for example that we throw a fair dice once. Since a fair dice will give a number from 1 to 6 with equal probability, the information that we get by throwing the dice will be equal to the logarithm of base 2 of 6. This is approximately 2.58 bits of information.

The invention of the “bit” catalysed the birth of information theory and ignited an explosion of innovation that continues to shape industries, economies, and societies. The bit's simplicity, in the form of binary code, belies its profound impact, underscoring the adage that sometimes the smallest units of thought hold the power to reshape the entire world.

Communicating information

If we communicate and say nothing new to each other, we exchange no information. After all, if you knew in advance what I would write in this article, what would be the point of writing it?

Shifting the communication problem

Traditionally, the problem of communications engineering was primarily viewed as a deterministic signal-reconstruction problem: Given that a transmitted signal is distorted by a physical medium, try to transform the received signal in order to reconstruct the original as accurately as possible. The observation that the key to communication is uncertainty, shifted the focus of the communication problem from the physical to the abstract. It was no longer a deterministic problem but a probabilistic one. This was also a huge shock to the engineering community at the time.

Communications engineering

Having defined what information is, Shannon found a way to make the communication problem a general problem. He used a diagram like this:

The first major contribution from this diagram is that one of the transmitter’s jobs is to convert all information to bits. Having successfully applied Boolean logic (true or false, zero or one) to switching circuits, Shannon generalizes the idea. All information can be represented as zeros and ones. Pictures, movies, and sound, for example, can all be treated the same way. This was one of the most important unifications of the twentieth century. Communications engineering is no longer a set of special cases that demand special theories.

The problem of communication now reduces to two questions: How do we convert information to zeros and ones and how can we make sure that the zeros and ones are received accurately.

The second major contribution came from realising the important role of redundancy in information. Is there a minimum number of bits that we can shrink our information without losing anything important? Shannon discovered that minimum and showed how to calculate it. His formula was based on the probabilities in the message and it is almost identical to a fundamental quantity in Physics called entropy (which is related to randomness and disorder).

Shannon calls his quantity entropy too, and he proves that it's the answer to his question. Entropy in information theory is the minimal size that we can compress our messages without any loss. The very idea that we can compress a large amount of data into something very small, is what makes the information age possible. Shannon’s idea that we should first change information to zeros and ones and then compress it is a huge first step towards a universal theory of communication.

The third major contribution is that there is a speed limit in information transmission. Each channel has its own discrepancies and limitations like noise and interference. It could be a wire, air or water for example. Shannon calculated the speed limit beyond which we know that our communication will fall apart. This is known as the Shannon limit. The very idea that there is a speed limit for information communication was another radical idea at that time.

The fourth major contribution is the answer to the question how do we eliminate channel noise. We can use mathematical codes, known as channel codes. Noise is eliminated by carefully introducing redundancy with such codes. Shannon proved that this can work at maximum speed, right at the Shannon limit. If we can find the right codes, we can have perfect communication, with as few errors as we want. He proved that there exist codes that can achieve the Shannon limit but he provided little insight on how to find them. This sparked scientific research for decades.

Shannon showed that redundancy in communications is a two-sided coin. Depending on what we want to do we may remove redundancy or artificially introduce it or both. While compression requires removing redundancy, channel coding involves carefully adding redundancy in order to eliminate the effects of channel noise.

The public response

The publication of the mathematical theory of communication was like dropping a bomb. A new scientific field had been conceived, redefining basic ideas. Information existed before Shannon, just as objects had inertia before Newton. However, information used to be for example a telegram, a photograph, a conversation, a book, or a song. Now, information was bits. And as if that was not shocking enough, another series of shocks hit the public: the sender no longer mattered, the intent no longer mattered, the medium no longer mattered, the meaning no longer mattered.

Electronically communicating information has limits. How much information, how quickly could it be transmitted? It was all calculated in this work. It used to be common engineering sense that noise was a problem in communication that had to be lived with. That common sense was drastically re-evaluated since channel coding was an option. It has taken decades to understand, extend and answer open questions from findings of the scientific bomb that was dropped in 1948.

How Did All This Happen?

So how did all that happen back in the 1940s? What made Claude Elwood Shannon one of the greatest electrical engineering heroes of all time? I think it was a mix of skills.

When Shannon was asked about the success of his idea to apply Boolean algebra in switching circuits he replied: “It just happened that no one else was familiar with both those fields at the same time.” While people who knew him recognise his modesty and humility in this statement, it also explains a wider truth: Diversity of knowledge is key. After all, he was both a genius mathematician and a genius electrical engineer.

The persistent generalist

His genuine curiosity and his need to understand how things work gave him a large number of interests and activities. This made him the perfect generalist. From each and every part of his work he gained the satisfaction of fulfilling his curiosity. He asked questions from new perspectives and would never stop until he found the right answers. Experts say that he was working on the mathematical theory of communication for a decade. This work survived broken marriages and the second world war.

A full cycle specialist

He was a conceptual theorist that would redefine ideas if necessary. Whenever he focused on a problem he would become a specialist. His interest and ability in mathematics, his curiosity to explore, learn and discover, his passion for building things would make him work his way through all aspects of a problem. From understanding concepts and designing, to developing and implementing solutions on gadgets and finally testing them. A theorist and a practitioner that would redefine information and calculate its practical limits.

Curiosity beyond conventional boundaries

Shannon followed his curiosity, asking questions and looking for answers in multiple fields. He didn’t let conventional boundaries define what he could or could not be interested in. “I think the history of science has shown that valuable consequences often proliferate from simple curiosity,” he said.

Conventionally, the founding fathers in a scientific field are usually the people who lead its developments for years to come. It's all about incentive, I think. Viewing a field as their child is usually the incentive to nurture and help their child grow. Shannon's incentive appears to be simply the joy of fulfilling his curiosity.

After his publication of the mathematical theory of communication, people expected him to lead information theory research and practice to new dimensions. MIT hired him for exactly this reason. His curiosity, however, led him to study things like:

1. Gambling games. He built a wearable computer that could improve the odds for Russian roulette.
2. How the brain works. Can we simulate the brain using a machine? How does the brain transform information? This kind of discussions he had with Alan Turing in their quest to understand the brain. He proved that we can make a reliable circuit using faulty components, having enormous applications to brain modeling. The data in the brain is interrelated. There is a huge amount of redundancy. Rules can be developed that depict how neurons connect.
3. The intelligent machine. This idea was also a byproduct of his interest in the brain. He built a chess playing machine. He observed that although there is an astronomical number of possible chess games, we don't have to look all the way to the end of the game to figure out what a good move is. We only have to look ahead a few moves and evaluate the possible moves that we could reach. It is all about finding the path that leads to the best one! This principle is the basis for every chess program that came after.
4. How the stocks work. He did work on the theory of stocks, applying information theory to the stock market. He developed various mathematical models to predict stock performance and has tested them—successfully, he said—on his own portfolio.
5. Biology and genetics. He actually did a PhD in genetics! He found that different genetic possibilities could be mathematically represented in matrices. Using matrix algebra he could predict how traits are passed on through generations.
6. Juggling machines. Juggling relates to patterns according to Shannon. There is a topology and a combinatorial aspect. He wrote a paper on Scientific American and came up with a formula that relates the number of balls we are juggling to how long each is in the air versus in the hand.

Claude Shannon was notoriously curious. At home, he had a basement full of gadgets and machines that he spent hours enjoying making. It was his personal workshop for all kinds of curiosities. His friends called it “a gadgeteer's paradise” and Shannon the “master gadgeteer”!

Fields and disciplines influenced

Shannon revolutionised electronic communications and made the internet, wireless and wireline communications possible, including audio, video and multimedia transmission. Computer programs and apps, gaming, barcodes, QR codes, biometrics, digital clocks, digital photography, streaming services and the like, became a reality. Anytime we store, process or transmit information in digital form, every time we save a file on a hard drive or in the cloud, send an email, load a web page, chat on any social media app, we’re relying on concepts based on Claude Elwood Shannon’s mathematical theory of communication. This publication is considered as the main reason why Shannon is known as the father of information theory.

The fields that information theory influenced are vast, including applied mathematics, theoretical physics, data science, genomics, linguistics, neuroscience, probability and statistics and telecommunications. From classical complexity theory, stochastic thermodynamics, quantum information theory and processing, cryptography, data compression and storage, channel coding and information transmission, cybernetics and control, biochemical signaling, machine learning and neural networks.

According to Eric Schmidt, former CEO of Google: “I often wonder if Shannon understood how much information there would be after his many discoveries. Did he understand that we would have this information explosion? In the last couple of years we’ve generated more information than in the sum of all human history. Literally, he not only won the lottery, he invented the lottery.”