AN INTRODUCTION TO THE SCIENCE OF INFORMATION TRANSFER, UNCERTAINTY, AND A PERSONAL VIEW OF ORGANIZATIONAL INTELLIGENCE

Abstract

An introduction to the concepts of entropy, information transfer, and the uncertainty in transfer of information is given. The specific target is information transfer in voice and speech. The entropies of a square wave, a sinusoid, and a sawtooth are calculated because these waveshapes approximate information carriers in vocalization. This is followed by a less-known concept that the increase of organizational intelligence gleaned from physical systems is proportional to the gradient of entropy. That leads to a multi-dimensional interpretation of diversity. Finally, a personal meta-physical extension is made to a universal reservoir of intelligence from which species can draw to advance civilizations.

Introduction

A major purpose of vocalization is to transmit messages. Based on the carrier-modulation principle in communication theory, the messages are usually modulations of a sound carrier that provides the energy and speed of transmission. In speech, singing, and animal vocalization, the dominant carrier of the message is voicing. Superimposed on this carrier are various frequency and amplitude modulations that constitute the message. Vowels and consonants are modulations of the entire frequency spectrum of the carrier. Prosodic information, like intonation in speech and melody in singing, or tremor and vibrato, are common frequency modulations of the carrier. Similarly, syllable stress and crescendo are common intensity modulations. Some modulations, such as unvoiced consonants, are severe enough that they disrupt and replace the voicing carrier with a noise carrier. All modulations, and the carrier itself, can be corrupted with additive noise and distortion, which results in uncertainty (entropy) of information transfer.

In this article, the relations between probability of occurrence of communicative events and the transfer of these events to a receiver are reviewed. The uncertainty in the transfer, and the intelligence gained from the transfer, will also be highlighted for the benefit of voice scientists and practitioners. In ordinary vernacular, intelligence is the ability to learn, to understand, to deal with new or trying situations, or to organize (after Webster’s Dictionary). Here a limited interpretation of intelligence is entertained, namely organizational intelligence, which may be a universal entity on par with matter and energy. To learn how to organize, one must be exposed to various degrees of organization and disorganization, which scientists call a gradient of entropy. Entropy itself is a degree of disorganization or disorder. If organization is the arrangement of elements into a whole with interdependent parts, then more options for organization will emerge with a higher gradient of entropy. More intelligence is gained by maximizing future options with diversity of organization (the entropy gradient). At the end of this tutorial paper, some personal inferences will be drawn with respect to societal and governmental organization and the overall intelligence of a civilization.

Entropy in a System of Particles

Entropy is defined in physics (thermodynamics) as the measurement of the degree of disorganization in a system of particles with a given amount of mass and energy. The second law of thermodynamics states that, over time, such a system in isolation will always trend toward a lesser degree of organization, or a higher entropy, even though its total mass and energy are conserved. A general formula for thermodynamic entropy $S$, developed by Boltzmann [1], is

$$S=k_b{\log }_2(W)$$ (1)

where $k_b$ is Boltzmann’s universal gas constant (1.380649 × 10⁻²³ J/K, or Joules of energy per degree Kelvin temperature change) and $W$ is the number of microscopic configurations the system can have, or the number of ways particles can be arranged. The more configurations $W$ there are (e.g., number of positions or energy states of particles), the greater the entropy. Note that the relation is logarithmic. It takes large changes in organization to alter the entropy when it is already large. In the extreme limit, entropy is maximized when there are infinite ways the system can arrange itself. In a gas confined to a space (Fig. 1), every molecule can in principle assume any position and carry any amount of kinetic energy. On the left, if blue particles have low energy and red particles have high energy and they are spatially separated, there is considerable organization, or low entropy. To the contrary, if particles are spatially allowed to mix and energy is allowed to be transferred between particles (shown on the right), there are more possible configurations, and hence higher entropy. Thermodynamic entropy will always increase over time and cannot reverse itself in isolation. The trend is always toward thermodynamic equilibrium, which means greater entropy, or less organization.

Fig. 1.

Spatial organization of particles with low energy (blue) and high energy (red).

Probability and Information Transfer

When information is transferred from a source to a receiver, there is organization in the rate and structure of successive events. Information theory quantifies the average amount of information (in bits, or binary digits) that must be provided to avoid uncertainty in the message. Calculated digitally, it reveals a lower bound on the number of binary digits that must, on the average, be used to encode a message [2]. Some noise, distortion, and corruption is always present in the transfer of information. Information entropy is then a measure of the uncertainty in the message. This entropy can rise or fall with the information delivered. Sometimes there are too few bits and sometimes there are too many bits of information. An event that is binary, like switching your voice on or off, requires only one bit of information. Vocal loudness, on the other hand, has a gradual change of acoustic energy. It requires more bits of information to describe the states of loudness. Confusion or uncertainty (entropy) arises when one uses multiple bits for on-off and not enough bits for soft-loud.

If an event $x$ has a probability of occurrence $p(x)$, Shannon [2] reasoned that the information $I(x)$ needed for transfer is

$$I(x)=-{\log }_{2}p(x)\text{in binary digits}\text{(bits)}$$ (2)

It is clear from the negative sign in the equation that high probability requires low information and low probability requires high information.

As an example, consider the rate at which the note C4 is sung in a practice room. The receiver of this message is in another room where the sound of C4 can be detected, by ear or by instrument. Let $I(x)$ be the information needed to describe the event and $p(x)$ the probability of its occurrence. If the singer can sing 24 notes (2 octaves), the information $I(x)$ that the note is a C4 requires 5 bits (the binary number for 24 is 11000). But what is the probability $p(x)$ that C4 is sung by anyone? That depends on many conditions, such as how often the room is used and how often C4 occurs in a piece that is performed. If singers enter the room at highly regular intervals and all sing the same song in the same key, less information needs to be passed to the listener. There is much predictability in the repeated events.

Fig. 2 shows the relation between information needed for a probability of occurrence, a plot of Eq. 2. For example, a coin toss has a 0.5 (50 %) probability of being a “head”. It requires 1 bit of information (0 or 1) to transmit the information. If both sides of the coin are a “head”, the probability is 1.0, meaning that no information transfer is needed. For very low probability (many possible choices), the information trends toward an infinite number of bits, as the figure shows.

Fig. 2.

Information needed to describe the probability of occurrence of an event.

Transmission Entropy

According to Shannon [2], the transmission entropy $S(x)$ for transmission of an event (or a series of events) $x$ is calculated as

$$S(x)=-I(x)[-p(x)]$$ (3)

The first negative sign in the equation expresses the general concept that entropy (uncertainty) increases with decrease of information. If there were only 4 bits of information for the note sung in the above example, C4 may not be detected accurately out of 24 possibilities. The second negative sign [inside the brackets] expresses the above illustrated concept that high probability of occurrence provides greater certainty, and therefore less entropy. Stated differently, there is more order or regularity in the rate of transmission of C4 if there is a higher probability of occurrence. As another example, periodicity in a voice signal guarantees an ordered regularity of occurrence. Therefore, a periodic signal has lower entropy than an aperiodic signal. Fig. 3 shows the relation between entropy and information. Note that entropy is high when only 1 or 2 bits of information are transmitted.

Fig. 3.

Transmission entropy as a function of information transmitted.

How is entropy calculated on a voice signal? So far, the information entropy has been described as a measure of uncertainty of a single repeated event, the transmission and detection of the note C4 to a listener. However, voice signals are more continuous in nature and contain multiple events simultaneously. They contain fundamental frequency, intensity, open quotient, formant ripple, and many other features of the signal. Some may be more regular (periodic) than others. Therefore, a differential entropy

$$d{S}_{\text{n}}=-I\left(x_n\right)\left[-p\left(x_n\right)\right]$$ (4)

is first defined according to Eq. (3) for each of a series of events

$$X=x_1,x_2,x_3,\dots .$$ (5)

Second, the integration of all the differential entropies in the continuous random variable set $X$ over the domain ${\text{Ω}}_x$ becomes

$$S(X)=-\underset{{\text{Ω}}_x}{\int }{p}_X(x){\log }_2{p}_X(x)$$ (6)

Here $p_X(x)$ are the individual probability distributions, described in detail by Nilsson [3, pp. 15–27].

Example of Entropy Calculations

One of the methods to approximate the probability distributions is the use of histograms. Consider the wave shapes shown in Fig. 4. In a speech signal, they could represent fundamental frequency or intensity contours. In each case, the mean value is 0.5. In part (a), there is a square wave modulation around this mean value. This could be an alternating high-low $f_{\text{o}}$ from one pitch level to another. In part (c), the modulation is a sinusoid, as in a vibrato or tremor. In part (e), the modulation is a sawtooth, as in a pitch glide followed by a sudden return to the low $f_{\text{o}}$. All three waveforms are computed with 3000 points over 5 cycles. On the right side of the figure, parts (b), (d), and (f), are the probability distributions shown as histograms. They were computed with the MATLAB function hist with a 10 bin default. The square wave has only two states, and hence only two bins.

Fig. 4.

(Left Panels) Signals with various kinds of modulations around a mean value of 0.5. (Right Panels) Histograms with the entropy calculation.

The entropy $S$, calculated from Eq. 6 with a single state variable $x$, is labeled numerically above the histograms. To protect against singularities (division by 0) in the logarithmic calculation for entropy, any state with zero probability was removed. The probability calculations were then normalized to the number of states (3000 points) in the waveforms to obtain positive entropy values. As an example, the script for the square wave is shown below:

sig1=.5+.4*square(0:.01:31.4);	%square wave, 5 cycles, 3000 points
N=10;	%number of bins
p1=hist(sig1,N);	%histogram
p1(p1==0)=[];	%remove p=0 values
p1n=p1/numel(sig1);	%normalize to number of elements
bar(p1n)	%bar plot
S(1)=-sum(p1n.*log2(p1n))	%entropy

Note from Fig. 4(b) that the square wave has the lowest entropy with a value 1.0. The histogram is binary, having only two states, even though 10 bins were requested. The sine wave has an entropy more than three times greater than the square wave (3.15) and the sawtooth has even a slightly higher entropy (3.32). The entropy difference (gradient) between the sinusoid and the sawtooth is small, however, because both require many states along a continuum in the rise and fall portions. Because the sinusoid has a roundness at the peaks and the troughs, the amplitudes near the peaks have a higher probability, which is reflected in the endpoints of the histogram.

Entropy can also be computed in the frequency domain, but the details are not discussed here. Known as spectral entropy (SE) of a signal, it is a measure of regularity or randomness in its power spectrum. It works on the basis that the voiced portions will have lower entropy due to the presence of clear harmonics. The unvoiced or noisy portions will have higher entropy due to the relatively flatter noisy power spectrum [4]. The concept is similar to cepstral peak prominence (CPP) analysis for determining signal periodicity. CPP increases when entropy decreases.

In summary, the concept of entropy may be more inclusive than other perturbation and irregularity measures currently in practice, like jitter, shimmer, or harmonics-to-noise ratios. To the contrary, the integration of multiple probabilistic events in a speech signal may not allow enough physical interpretations. It remains to be seen how robust and clinically applicable entropy calculations will become in practice. Auditory perception should also be considered in signal transmission. Some sounds become rapidly irrelevant with repeated exposure. While they may have a high entropy as a physical signal, the brain filters out the lack of organization. Survival pressures drive the system to unique sounds in the background of much redundancy.

Disorders in Verbal Communication

There is much mention of inaccurate information in our communication age. Misinformation is incorrect information, often with no intent to deceive or confound. There simply were not enough data to make a strong case for a statement. Disinformation is information intended for deception, which includes bias and lies. Miscommunication is a breakdown of the information carrier; there is much noise or distortion in the signal to prevent accurate decoding of the message.

All of the above are forms of entropy in communication, suggesting that information is not necessarily truth. Relating entropy to our daily communications, fewer precise words convey more certainty than many un-precise words. We sometimes confuse people by being indirect with an attempt to be more accommodating. For example, if a partner says to the other “are you tired?” when the intent was to convey that “I am tired”, a “no” will require further clarity in communication. Words are sometimes invented for kindness and acceptability, but the resulting actions may not be commensurate with the words. In scientific communication, the intent is to use few words with precise definitions, thereby lowering entropy. Unfortunately, words that require a continuum for accurate information are often made binary, such as:

Right/Wrong; High/Low; Black/White; Friend/Enemy; Hot/Cold; Large/Small; Young/Old; Smart/Dumb; Liberal/Conservative

These descriptions do convey information, but sometimes not enough intelligence is gained from them because the states are over-simplified. Recent attempts have been made to relate growth of intelligence to an environment with multiple states of entropy. This is the next topic.

The Physics of Entropy Gradient and Organizational Intelligence

Intuition tells us that some relation should exist between organization and intelligence. Persistent exposure to high degrees of organization, we might reason, will lead to high intelligence. Conversely, persistent exposure to low degrees of organization will lead to little intelligence. Neither is exactly true. Growth of intelligence is a process of exploring many options with many states of organization [5]. Scientists call this a gradient of entropy. Returning to an introductory statement, entropy in the universe is maximized when there is a degradation of organization of matter and energy to an ultimate state of inert uniformity. There is no return from this ultimate uniformity. Nothing more can be learned from it. Wissner-Gross and Freer [5] argue that causal intelligence forces are increased by maximizing one’s future options, or future freedom of action. They give an equation for causal entropic forces $F$ for higher intelligence:

$$F=T\nabla S$$ (7)

where $T$ is a constant of proportionality with units of temperature and $\nabla S$ is a gradient (directional change) of entropy $S$. In non-mathematical language, intelligence grows when there is a large variation in states of entropy, some states being highly organized and others being poorly organized. One can think of it as a ladder, or stepping-stones, of organization. At one extreme, if everything in the universe were perfectly organized, little intelligence would be able to be gleaned from it. In biblical language, this would be the state described as the Garden of Eden.

At the other extreme, if there were an ultimate state of inert uniformity in the universe, again little increase in intelligence would be possible. This might be the state described as “matter unorganized” in biblical terms. Fig. 5 (left) shows a graphical representation of a positive gradient, $\nabla S>0$. The entropy $S$, which may be multidimensional, spreads in all directions from little organization to much organization. This provides many future options. Mathematically, a negative gradient (convergence) is shown in the middle graph. This suggests a decrease in future options. The third graph shows no gradient (uniformity) in one direction, which maintains the options as they are.

Fig. 5.

Left to right, two-dimensional positive gradient, negative gradient, and no gradient.

Figure 6 shows a beautiful drawing entitled Progressive Entropy, painted by Franco DeFrancesca [6]. Note that around the circumference of the circle, color and spatial organizations exists. It is not the same everywhere around the circumference, but there is organization. As one progresses toward the center, color and geometry gets more and more unorganized, or less certain. The circle is a visual representation of $\nabla S$, the gradient of entropy. In the center, there is a complete mixture with no apparent organization. Entropy is maximized. A further widening of the sphere could perhaps result in perfect organization of geometry and color. Entropy would then be minimized. The major point is that growth of intelligence will occur as one moves radially outward, or in mid-radius circles, to observe the organizational patterns. Permanent existence at either end, the center of the circle or possibly an infinitely large radius, provide little growth of intelligence. The path of intelligence growth is away from the endpoints, a path that shows the greatest changes (gradients).

Fig. 6.

Progressive Entropy by Franco DeFrancesca, 2021. Image used with permission.

Stated in very simple language, total uniformity (perfect organization or no organization) leads to no increase in intelligence. This is so evident in raising children. Parents who are over-protective, ultra-organized, always providing the perfect answers, may offer no more potential for growth of intelligence than parents who show no structure, no rules, and allow nothing but clutter in the house.

The same can be said about institutions, governments, and societies as a whole. There is little creative intelligence in highly regulated societies without much freedom to choose different paths, neither is there much creativity in a society without rules and structure. With the right proportion, a society can sustain a long growth of intelligence.

Does a coffee cup have intelligence? Yes, it has molecular and geometric organization from which intelligence can be gleaned. Left in isolation, however, it will eventually disintegrate and lose its organization, and therefore all its intelligence. An intermediate state is a broken cup. With intelligence and a little energy, it can be re-constituted. Left in isolation, it will never reconstitute. Growth of intelligence occurs with multiple states of observation of the cup – fully organized, broken in pieces, and totally pulverized (Fig. 7).

Fig. 7.

Entropy change in breaking a porcelain mug. Photo by Angie Keeton, 2023. Image used with permission.

Limitations of the Wissner-Gross theory [5] about forces of intelligence are acknowledged by current physicists. Causal entropic forces do not explain intelligent behavior from known physical laws. The theory cannot explain how an intelligence agent evolved in the first place, and why this agent seeks to maximize future options. Models of the universe that incorporate causal entropy are more likely to come up with a universe that contains an outside source of intelligence, one that is independent of matter and energy. This is the position taken by this author, expanded further in a Meta-Physical Extension below.

It should also be acknowledged that this physical definition of intelligence is somewhat limited to organizational intelligence. Sentient intelligence, based on feelings and human interaction, is not simply organizational. Autistic individuals are often extremely gifted in organizing physical objects, or solving puzzles. However, they may lack the abilitate to interact and motivate others to share the organization and impose a social meaning onto them. The author does not claim expertise in cognitive psychology and invites contributions beyond the scope of this article.

Transfer of Intelligence by Simulation

Simulation is a mathematical representation of physical phenomena, or their sensory appearance. It is an imitation of the operation of a real-world process of systems over time without the use of matter (material). Simulation requires the use of models that represent the key characteristics or behaviors of sub-systems over time. Generally, computers are used to execute simulations, which require some energy, but mainly a large amount of intelligence (the ability to organize and program physical laws). The hologram shown in Fig. 8 is a representation of the visible appearance of a human body. Light is reflected from a real-world physical body and re-constructed as a three-dimensional image in space. All the information needed to re-construct this image can be stored, transmitted, and duplicated many times in many places. No material is needed.

Fig. 8.

A hologram is created with light and intelligence. (Cropped image of “Ren hologram 1, Rise of the Resistance, Galaxy’s Edge, Disney Hollywood Studios, Walt Disney World, Lake Buena Vista, Florida, USA” photo by Cory Doctrow is licensed under CC BY-SA 2.0. Original image: https://www.flickr.com/photos/37996580417@N01/49280516808)

Figure 9 shows a spectrogram of a simulation of the spoken sentence “I am the light of the world”. The speech was produced by physical laws of air and tissue movement, simulating how a human male speaker moves air from the lungs, vibrates the vocal folds, moves the articulators, and propagates sound waves through the airways. While some data were used from imaging the geometry of a speaker’s airways, the production of sound required no air or tissue material in motion. Mathematics provided the intelligence. When humans claim interactions with supra-natural beings, it is not clear how much material is part of the interaction. Spiritual beings are likely to produce only the information that is needed for human sensory systems. Even the sensation of touch can be simulated with mathematics.

Fig. 9.

A sound spectrogram created with light and intelligence. The sentence “I am the light of the world” was produced with a simulation called TubeTalker by Brad Story. Image used with permission.

Metaphysical Extension

A personal conviction (not physical, but meta-physical) is that a reservoir of intelligence in the universe is as fundamental as the reservoirs of matter and energy. Thus, it is assumed that there are three cosmic reservoirs that have always existed (Fig. 10). Exchanges can occur between them, but in sum they are neither created nor destroyed. Creation is the use of intelligence to organize or re-organize matter and energy. Communication is the transfer of intelligence with energy. The universe may not reach an ultimate state of inert uniformity if intelligence prevails. Cosmic death (also known as the Big Chill or Big Freeze) can be avoided. In other words, the hypothesis that the universe will eventually evolve to a state of no thermodynamic free energy and would therefore be unable to sustain processes that change entropy, may not prove positive.

Fig. 10.

Hypothesis: Three reservoirs in the universe

By analogy, for growth of intelligence on earth, there should be freedom of choice for observation and experimentation. There should be diversity of effort, accomplishment, reward, and outcome. For cognitive growth, there should also be diversity of opinion, ideas, and ideology. This freedom is being threatened by growing populations and limited resources. However, the limitation is primarily on physical resources (matter and energy on our planet). Simulation is a means of accelerating intellectual growth with conservative use of matter and energy.

Opinions on Entropy and Diversity in Society

In sociology and biology, much is said about the importance of diversity. The earth is healthy with diverse species, and most species benefit from genetic diversity. Furthermore, if diversity includes diversity of skills, diversity of effort, diversity of reward (outcome), and diversity of thought and opinion, then intelligence grows in a society. There will be many current and future options with a gradient of entropy. However, often with socio-political intervention, the concept of diversity can lead to its exact opposite, namely uniformity. Truly diverse societies are difficult to govern. It is much easier to promote uniform thinking and behavior. However, if there is no gradient of thought or action, little growth of intelligence can take place.

The remaining statements in this article are personal thoughts derived from this study. A state of uniformity in a society is sometimes promoted as social justice and equality. Well-meaning advocates wish to eliminate Social Darwinism. Freedom to choose from many options creates too much individual disparity. It then requires voluntary acts of sharing with others to promote collective growth. This is relatively easy to accept by people with a belief in a higher intelligence, coupled with a broader concept of existence beyond one life. With a single-existence concept, it is attractive (and indeed noble and virtuous) for people to strive for equality, but this goal often results in compulsive sharing by those who wish to excel. Future options are diminished for them. Unfortunately, highly regulated societies gradually run out of options for constraining individual excellence and growth. A ruling class then creates the utmost disparity with its controlling power, a binary system of “haves” and “have-nots”.

Excessive regulation and group thinking is the enemy of intelligence. Opposition is essential for a society to progress. Social entropy [7] is a theory that attempts to evaluate social behaviors with an analogy to physical entropy. The entropy gradient in a social system is a gradient of wealth, or a gradient of physical and geographical location. While this gradient appears unjust at any moment in time, one might argue that it promotes future options to benefit a society in the long-term. For growth of intelligence, there should be freedom of choice for observation, expression, location, and experimentation. Freedom of expression must include both truth, half-truth, and untruth. Exposure to lies is as important as exposure to truth. Intelligence, not censorship, is the key to discerning veracity.

For this author, a desire to seek virgin truth from a universal intelligence reservoir is paramount. Many of our greatest minds, like Einstein, Beethoven, Gandhi, and Christ spent long periods of time in solitude, presumably seeking cosmic intelligence. If one gains intelligence only from existing facts and data (social media, teachers, friends), there are likely to be few quantal leaps in cognitive ability. Even artificial intelligence, touted as one of the greatest inventions of our time, is based only on existing intelligence.

Finally, the difference between information, knowledge, and wisdom needs to be cultivated. Much information is transmitted to us every second. A modest amount of knowledge is obtained from it, but wisdom is lagging because we cannot apply the knowledge fast enough in daily experiences. Quantification of wisdom is a challenge for the future. Perhaps an equation will emerge.