AI, visual imagery, and a case study on the challenges posed by human intelligence tests

Abstract

Observations abound about the power of visual imagery in human intelligence, from how Nobel prize-winning physicists make their discoveries to how children understand bedtime stories. These observations raise an important question for cognitive science, which is, what are the computations taking place in someone’s mind when they use visual imagery? Answering this question is not easy and will require much continued research across the multiple disciplines of cognitive science. Here, we focus on a related and more circumscribed question from the perspective of artificial intelligence (AI): If you have an intelligent agent that uses visual imagery-based knowledge representations and reasoning operations, then what kinds of problem solving might be possible, and how would such problem solving work? We highlight recent progress in AI toward answering these questions in the domain of visuospatial reasoning, looking at a case study of how imagery-based artificial agents can solve visuospatial intelligence tests. In particular, we first examine several variations of imagery-based knowledge representations and problem-solving strategies that are sufficient for solving problems from the Raven’s Progressive Matrices intelligence test. We then look at how artificial agents, instead of being designed manually by AI researchers, might learn portions of their own knowledge and reasoning procedures from experience, including learning visuospatial domain knowledge, learning and generalizing problem-solving strategies, and learning the actual definition of the task in the first place.

I think in pictures. Words are like a second language to me. I translate both spoken and written words into full-color movies, complete with sound, which run like a VCR tape in my head…. Language-based thinkers often find this phenomenon difficult to understand, but in my job as an equipment designer for the livestock industry, visual thinking is a tremendous advantage.

Temple Grandin, professor of animal science and autism advocate (ref. 1, p. 3)

What I am really trying to do is bring birth to clarity, which is really a . . . thought-out pictorial semivision thing. I would see the jiggle-jiggle-jiggle or the wiggle of the path. Even now when I talk about the influence functional, I see the coupling and I take this turn–like as if there was a big bag of stuff–and try to collect it away and to push it. It’s all visual. It’s hard to explain.

Richard Feynman, Nobel laureate in physics (ref. 2, p. 244)*

Temple Grandin is a well-known animal scientist who is on the autism spectrum. She has had incredible professional success in the livestock industry, and she credits her success to her strong visual imagery skills, that is, abilities to generate, transform, combine, and inspect visual mental representations. (1).

Many physicists such as Richard Feynman (2), Albert Einstein (3), and James Clerk Maxwell (4) used imagery in their creative discovery processes, and similar patterns emerge in accounts by and about mathematicians (5), engineers (6), computer programmers (7), product designers (8), surgeons (9), memory champions (10), and more. People also use visual imagery in everyday activities such as language comprehension (11), story understanding (12), and physical (13) and mathematical reasoning (14).

These observations raise an interesting scientific question: What are the computations taking place in someone’s mind when they use visual imagery? This is a difficult question that continues to receive attention across cognitive science disciplines (15).

Here, we focus on a related, more circumscribed question from the perspective of artificial intelligence (AI): If you have an intelligent agent that uses visual imagery-based knowledge representations and reasoning operations, then what kinds of problem solving might be possible, and how would it all work?

In this paper, we discuss progress in AI toward answering this question in the domain of visuospatial reasoning—reasoning about the geometric and spatial properties of visual objects (16). This discussion necessarily leaves out such intriguing and important complexities as nonvisual forms of spatial reasoning, for example, in people with visual impairments (17); the role of physics and forces in imagery (18); imagery in other sensory modalities (19); etc.

As a case study, we focus on visuospatial reasoning for solving human intelligence tests like Raven’s Progressive Matrices. While many AI techniques have been developed to solve many different tests (20), we are still quite far from having an artificial agent that can “sit down and take” an intelligence test without specialized algorithms having been designed for that purpose. Contributions of this paper include discussions of 1) why intelligence tests are such a good challenge for AI; 2) a framework for artificial problem-solving agents with four components: a problem definition, input processing, domain knowledge, and a problem-solving strategy or procedure; 3) several imagery-based agents that solve Raven’s problems; and 4) how an imagery-based agent could learn its domain knowledge, problem-solving strategies, and problem definition/input processing components, instead of each being manually designed.

Why the Raven’s Test Is (Still!) a Hard AI Challenge

Take a look at the problems in Fig. 1. Can you solve them?

Fig. 1.

Sample problems like those from the Raven’s intelligence test, comparable to ones of easy-to-middling difficulty on the standard version of the test.

While these problems may seem straightforward, consider for a moment the complexity of what you just did. As you were solving each problem, some executive control system in your mind was planning and executing a series of physical and cognitive operations, including shifts of gaze from one element of the problem to another, storing extracted features in working memory, computing and storing the results of intermediate calculations, and so on. And, you did all of this without any explicit instructions as to what cognitive operations to use, or in what order to apply them.

At a deeper level, you may notice that no one actually even told you what these problems were about. Typically, Raven’s test-takers are instructed to solve each problem by selecting the answer from the bottom that best completes the matrix portion on top (21). However, even if you hadn’t seen problems quite like these before, it is likely that you were able to grok the point of the problems just by looking at them, no doubt due to a lifetime of experience with pattern-matching games and multiple choice tests.

From a general AI perspective, intelligence tests like the Raven’s have been “solved” in the sense that we do have computational programs that, given a Raven’s problem as input, can often produce the correct answer as an output. In fact, some of the earliest work in AI was Evans’ classic ANALOGY program from the 1960s—at the time, the largest program written in LISP to date!—that solved geometric analogy problems from college aptitude tests (22).

However, all of these programs have essentially been handcrafted to solve Raven’s problems in one way or another. Humans (at least in theory) are supposed to take intelligence tests without having practiced them beforehand. Thus, intelligence tests like the Raven’s are still an “unsolved” challenge for AI when treated as tests of generalization, that is, generalizing previously learned knowledge and skills to solve new and unfamiliar types of problems.

At an even higher level, the notion of “taking a test” is itself a sophisticated social and cultural construct. In people, for example, crucial research on stereotype threat has observed how stereotypes about race and gender can influence a person’s performance on the exact same test depending on whether they are told it is a “test” or a “puzzle” (23). If we assume that human cognition can be explained in computational terms, then, someday, we ought to be able to have AI agents that model these effects.*

The Raven’s test and similar tests of matrix reasoning and geometric analogy are particularly interesting for AI for several reasons. First, the Raven’s test, originally designed to measure “eductive ability,” or the ability to extract and understand information from a complex situation (21), occupies a unique niche among psychometric instruments as being the best single-format measure of a person’s general intelligence (25). In other words, the Raven’s test seems to tap into fundamental cognitive abilities that are very relevant to many other things a person tries to do.

Second, there are several Raven’s tests that span a very wide range of difficulty levels, from problems that are easy for young children to problems that are difficult for most adults. The developmental trajectories of performance that people show offer a motivating parallel for studying AI agents that meaningfully improve their problem-solving abilities through various learning experiences.

Third, there is evidence that many people use multiple forms of mental representation while solving Raven’s problems, including inner language as well as visual imagery (26, 27). Interestingly, many people on the autism spectrum show patterns of performance on the Raven’s test that do not match patterns seen in neurotypical individuals (28), and neuroimaging findings suggest that many individuals on the spectrum rely more on visual brain regions than neurotypicals do while solving the test (29). Thus, the Raven’s test is a fascinating testbed for AI research on visual imagery in particular and multimodal reasoning more generally.

A Framework for Artificial Agents That Solve Problems

Many approaches in AI can usefully be decomposed according to the framework shown in Fig. 2. The agent is given a problem as input and is expected to produce a correct solution as output.

Fig. 2.

Framework for artificial agents. Pushing the boundaries of what artificial agents can do often involves deriving more and more of the internal structure and knowledge of the agent through learning instead of programming.

The “problem definition” refers to the agent’s understanding of what the problem is actually asking, that is, what constitutes a valid format of inputs and outputs (“problem template”) and what the goal is in terms of desired outputs (“solution criteria”). For example, for a generic Raven’s problem, the problem template might specify a two-dimensional matrix $M$ of images $m_i$, with one entry in the matrix missing, and an unordered set $A$ of answer images $a_i$, and that a valid answer consists of selecting one (and only one) answer $a_i\in A$. The solution criterion is that the selected answer should be the one that “best fits” in the missing slot in $M$.

The “input processing” component refers to how an agent takes raw or unstructured inputs from the “world” and converts them into a usable internal problem representation. For example, what the Raven’s test actually provides is a pattern of ink on paper. At some point, this visual image needs to be decomposed into the matrix $M$ and answer choice $A$ elements in the problem template. For many artificial agents, input processing is performed outside the agent, either manually or by some other system. For example, most chess-playing agents do not operate using a video feed of a chess board, but rather using an explicit specification of where all of the pieces are on the board. While this is a reasonable assumption to make in many AI applications, it does mean that the agent relies on having a simplified and preprocessed set of inputs.

“Domain knowledge” refers to whatever knowledge an agent needs to solve the given type of problems. The Raven’s test can be tackled using visuospatial knowledge about symmetry, sequential geometric patterns, rows and columns, etc.

Finally, the “problem-solving strategy” encompasses what the agent actually does to solve a given problem, that is, the algorithm that churns over the problem definition, domain knowledge, and specific problem inputs in order to generate an answer.

Given this framework, what would it mean for an agent to use visual imagery to solve problems? We offer one formulation: Anywhere beyond the input processing step, the agent needs to use or retain representations of problem information that count as “images” in some way. This includes image-like representations occurring in the problem definition, domain knowledge, problem-solving strategy, and/or the specific problem representations generated by the input processing component.

What counts as an image-like representation? Previous research on computational imagery often distinguishes between spatial representations, that is, those that replicate the spatial structure of what is being represented, versus visual/object representations, that is, those that replicate the visual appearance of what is being represented (30). These categories correspond to findings about spatial versus object imagery in people (31). Thus, we label agents using either type of representation as using visual imagery or being imagery based. The imagery-based Raven’s agents discussed later in this paper primarily use visual/object imagery and not spatial imagery, although, certainly, many other AI research efforts have developed agents that use spatial imagery (32).

Note that imagery here refers to the format in which something is represented, not the contents of what is represented. Many artificial agents reason about visuospatial information using nonimagery-based representations (33); for example, visuospatial domain knowledge can be encoded propositionally, such as the rule left-of (x,y) $\Rightarrow$ right-of (y,x).

Different Types of Raven’s Problem-Solving Agents

Different paradigms of AI agents can now be described according to components in this framework.

Knowledge-based approaches, also associated with terms like cognitive systems (34) or symbolic AI, traditionally rely on manually designed domain knowledge and flexible problem-solving procedures like planning and search to tackle complex problems. The first wave of “propositional Raven’s agents” used manual or automated input processing to convert raw test problem images into amodal, propositional representations, such as lists of attribute–value pairs, and then problem-solving procedures would operate over these propositional representations (33, 35–37). Visuospatial domain knowledge in these agents included predefined types of relationships among elements, like similarity or containment, and methods for extracting and defining relationships.

As foreshadowed in early writings about possible representational and algorithmic strategy differences on the Raven’s test (38), a second wave of “imagery-based Raven’s agents” were also knowledge-based, but their internal representations of problem information remained visual, that is, the problem-solving procedures directly accessed and manipulated problem images, and even often created new images during the course of reasoning (39–43). Visuospatial domain knowledge in these agents included image functions like rotation, image composition, visual similarity, etc.

More recently, a wave of “data-driven Raven’s agents” aims to learn integrated representations of visuospatial domain knowledge and problem-solving strategies by training on input–output pairs from a large number of example problems (44–49).

Which approach is correct? This is a bad question, as different types of agents are used for very different lines of scientific inquiry. Referring again to Fig. 2, most knowledge-based Raven’s agents are used to study problem-solving procedures and assume a relatively fixed set of domain knowledge (although some of these agents certainly include forms of learning as well). Most of the data-driven Raven’s agents are used to study how domain knowledge about visuospatial relationships can be learned from examples, and the problem-solving procedure is often (although not always) fixed.

All of these Raven’s agents have many hand-built components, although the parts that are hand-built differ from one agent to another. Many open AI challenges remain, even within the one task domain of the Raven’s test, to gradually convert the components in Fig. 2 from being manually programmed to being learned or developed by the agents themselves. Next, we discuss how knowledge-based agents can use imagery to solve Raven’s problems in several different ways, and then we examine emerging methods for agents to learn their own 1) domain knowledge, 2) problem-solving strategies, and, finally, 3) problem definitions.

Imagery-Based Strategies for Solving Raven’s Problems

Within the category of imagery-based Raven’s agents, many different formulations are possible, in terms of the problem-solving strategy that is used, the representation and contents of domain knowledge, and even the problem definition.

We describe five imagery-based strategies along with results from research by the author and colleagues. Results are reported for the Raven’s Standard Progressive Matrices test, scored out of 60 problems (21). For comparison, human norm data suggest that average children in the United States would score around 26/60 as 8-y-olds, 40/60 as 12-y-olds, and 49/60 as 16-y-olds.

At a high level, the following strategies are described in terms of two strategy types observed in psychology research (50):In “constructive matching,” the test-taker looks at the problem matrix, generates a guess for the missing element, and then chooses an answer most similar to its generated guess. In “response elimination,” the test-taker looks at each answer in turn, plugging it into the problem matrix, and choosing the one that produces the best overall matrix.

Strategy 1 (Fig. 3A).

Fig. 3.

Raven’s-like problem and four different imagery-based strategies for solving it. A problem consists of matrix $M$ of elements $m_i$ and set $A$ of answer choices $a_i$. (A) First strategy begins with search for transformation $t$ that best transforms $m_1$ into $m_2$, then applies $t$ to $m_3$ to produce an image candidate for $m_4$, and finally searches for answer $a_i$ most similar to $m_4$. (B) Second strategy also begins with search for $t$ that best transforms $m_1$ into $m_2$, then conducts similar searches for transformations $t_{ai}$ that transform $m_3$ into each $a_i$, and finally searches for answer $a_i$ that yields $t_{ai}$ most similar to $t$. (C) Third strategy begins with search for image $m_4$ that maximizes Gestalt metric for matrix $M$, and then searches for answer $a_i$ most similar to $m_4$. (D) Fourth strategy involves search for answer $a_i$ that maximizes Gestalt metric for matrix $M$.

We developed an imagery-based agent that solves Raven’s problems through multistep search, using a constructive matching strategy (39, 43, 51): 1) Using elements from complete rows/columns of the matrix, search among known visual transformations for the one that best explains image variation across parallel rows/columns. 2) Apply this transformation to elements in a partial row or column to predict a new answer image. 3) Search among the answer choices to find the one that is most similar to the predicted answer image.

More formally, problem inputs include a set $M$ of images $m_i$ representing sections of the problem matrix, and a set $A$ of answer choice images $a_i$. Let $C$ be the set of all collinear subsets $c$ of $M$, with $c_x$ referring to the first element(s), and $c_y$ referring to the last element. Each $c$ contains matrix elements along rows, columns, or diagonals. We define an analogy $g$ as a pairing of a single complete collinear subset $c_1$ with an incomplete collinear subset $c_2$ (i.e., $g=\left[c_{1.x}:c_{1.y}::c_{2.x}:c_{2.y}\right]$, where $c_{2.y}$ is the missing element in the matrix). All such analogies that share the same $c_2$ are further aggregated into sets $G_i\in G$.

In addition, let $T$ be the agent’s predefined set of visual transformations. Also, let sim(I₁,I₂) be a function that returns a real-valued measure of similarity between images $I_1$ and $I_2$. First, the agent finds the best-fit transformation,

$$\left(t_{max},g_{max}\right)=\underset{t\in T,G_i\in G}{\text{argmax}}\left(\underset{g\in G_i}{\text{mean}}\left(\text{sim}\left(t\left(g.c_{1.x}\right),g.c_{1.y}\right)\right)\right),$$

Second, the agent computes a predicted answer image as $a_{pred}=t_{max}\left(g_{max}.c_{2.x}\right)$. Third, the agent returns the most similar answer choice: $a_{\mathit{fi}nal}={\text{argmax}}_{a_i\in A}\left(\text{sim}\left(a_{pred},a_i\right)\right)$. Hand-coded domain knowledge is provided in the form of the set $T$ of visual transformations, including eight rectilinear rotations and reflections (including identity) and three to six image composition operations (union, intersection, subtraction, and combinations of these) as well as visual similarity and other image processing utility functions. Steps 1 and 3 above used exhaustive search.

Successive versions of the agent, using more transformations $T$ and more varied ways to optimize over matrix entries in step 1, have achieved scores of 38/60 (39), 50/60 (51), and 57/60 (43) on the Raven’s Standard Progressive Matrices test.

Strategy 2 (Fig. 3B).

In a related line of research, colleagues developed a different imagery-based agent that adopted a response elimination type of strategy (Fig. 3B). In this work (40), a smaller set of visual transformations (rotation and reflection) was used to compute “fractal image transformations,” that is, a representation of one image in terms of another, using techniques from image compression (52).

In particular, to compute a fractal transformation between source image $A$ and target image $B$, $B$ is first partitioned into a set of subimages $b_i$. Then, for each $b_i$, a fragment $a_i\in A$ is found such that $b_i$ can be expressed as an affine transformation $t_i$ of $a_i$. The fragments $a_i$ are twice the size of $b_i$, resulting in a contractive transformations. The set $T$ of all $t_i$ is the fractal transformation of $A$ into $B$.

To solve a Raven’s problem, a fractal transformation $T$ is computed using elements from each complete row/column $j$ in the matrix, and then similar transformations $T_{ij}^{\prime }$ are computed for each of the answer choices plugged into the incomplete rows/columns of the matrix. Finally, the selected answer is the one yielding the fractal transformations most similar to those computed for the original rows/columns of the matrix. Formally, if we let Tsim be a similarity metric across fractal transformations, the final answer is given by

$$a_{\mathit{fi}nal}=\underset{a_i\in A}{\text{argmax}}\sqrt{\sum _j\text{Tsim}{\left(T_j,T_{ij}^{\prime }\right)}^2}.$$

Results using this fractal method were also 50 out of 60 correct on the Raven’s Standard Progressive Matrices test, allowing for some ambiguous detections of the answers, or 38 out of 60 correct with a specific method for resolving these ambiguities (40).

Strategy 3 (Fig. 3C).

The first two strategies consider each matrix element individually. However, people can also use a “Gestalt” strategy to consider the entire matrix as a whole (38, 53). For instance, for the problem in Fig. 3, if one looks at the matrix as a single image, an answer might just “appear” in the blank.

In recent work (42), we attempted to model this kind of strategy using neural networks for image inpainting, trained to fill in the missing portions of real photographs. We used a recently published image inpainting network consisting of a variational autoencoder combined with a generative adversarial network (54), and we tested several versions of the network trained on different types of photographs, such as objects, faces, scenes, and textures. Given an image of the incomplete problem matrix, the network outputs a guess for what image should fill in the missing portion. This guess is then used to select the most similar answer.

Formally, let $F$ be the learned encoder network that converts an image into a representation in a learned feature space, and let $G$ be the learned decoder network that converts a feature-based image back into pixel space, including inpainting to fill in any missing portions. Then, our agent first computes $M^{\prime }=G\left(F\left(M\right)\right)$ to obtain a new, filled-in matrix image, with $m_x$ denoting the new, filled-in portion of $M^{\prime }$. Let $\text{L}2\text{dist}$ represents the L2 norm of a vector in the learned feature space. Then, the final answer is

$$a_{\mathit{fi}nal}=\underset{a_i\in A}{\text{argmin}}\left(\text{L}2\text{dist}\left(F\left(m_x\right)-F\left(a_i\right)\right)\right).$$

Fig. 4 shows examples of inpainting results on several example problems, some of which are filled in more effectively than others. The best version of this agent, trained on photographs of objects, answered 25 out of 60 problems on the Raven’s Standard Progressive Matrices test. While this score may seem low, it is quite astonishing given that there was no Raven’s-specific information fed into or contained in the inpainting network, and, in fact, the network had never before “seen” line drawings, only photographs.

Fig. 4.

Images generated using an inpainting neural network (54) for Raven’s-like problems (42). The network was trained only on real-world photographs of objects.

Strategy 4 (Fig. 3D).

The fourth strategy combines a Gestalt approach with response elimination. We have not yet implemented this strategy, nor do we know of other AI efforts that have, but we present a brief sketch here. Essentially, this strategy works by plugging in answers to the matrix, and choosing the one that creates the “best” overall picture, for some notion of best.

Assume a Gestalt metric $S$ that measures the Gestalt quality of any given image. Images that are highly symmetric, contain coherent objects, etc., would score highly, and images that are chaotic or broken up would score poorly. Then, the agent chooses the answer that scores highest when plugged into the matrix $M$,

$$a_{\mathit{fi}nal}=\underset{a_i\in A}{\text{argmax}}\left(S\left(M\cup a_i\right)\right).$$

Strategy 5 (Not Shown in Figure).

The above four strategies treat Raven’s matrix elements as single images. However, previous computational and human studies have suggested that it can be helpful to decompose Raven’s problems into multiple subproblems, by breaking up a single matrix element into subcomponents (35).

In previous work, we have also explored imagery-based techniques for decomposing a geometric analogy into subproblems, solving each separately, and then reassembling the subsolutions back together to choose the final answer (55), although this method has not yet been tested on the actual Raven’s tests.

Open Questions.

From this small survey, it is clear that there is no single imagery-based Raven’s strategy. Imagery-based agents are like logic-based agents or neural network-based agents; there are a set of generally shared principles of representation and reasoning, but then individual agents are designed to use specific instantiations of these and combine them in different ways to produce very diverse problem-solving behaviors.

Exploring the space of imagery-based agents is valuable, not to find the “best” one but rather to characterize the space itself. Each agent, as a data point in this space of possible agents, is an artifact that can be studied in order to understand something about how that particular set of representations and strategies can produce intelligent task behaviors (56). Future work should continue to add data points to this space and also investigate the extent to which these strategies overlap with human problem solving.

Learning Visuospatial Domain Knowledge

Imagery-based agents use many kinds of visuospatial domain knowledge, including visual transformations like rotation, scaling, and composition; hierarchical representations of concepts in terms of attributes like shape and texture; Gestalt principles like symmetry, continuity, and similarity; etc. These types of knowledge can be leveraged by an agent to solve problems from the Raven’s test as well as many other visuospatial tests (32).

Visuospatial domain knowledge also includes more semantically rich information such as what kinds of objects go where in a scene (57); we do not further discuss this type of semantic knowledge here, although it certainly plays an important role in imagery-based AI, especially for agents that perform language understanding or commonsense reasoning tasks (32).

How is visuospatial domain knowledge learned? One hypothesis suggests that agents learn such knowledge through prior sensorimotor interactions with the world. Under this view, the precise nature of the representations and learning mechanisms involved are important open questions. For brevity, we discuss here AI research on learning two types of visuospatial domain knowledge—visual transformations and Gestalt principles.

Learning Visual Transformations.

In humans, many reasoning operators used during visual imagery (e.g., transformations like mental rotation, scaling, etc.) are hypothesized to be learned from visuomotor experience, for example, perceiving the movement of physical objects in the real world (58). As with the well-known kittens-in-carousel experiments (59), learning visual transformations may rely on the combination of active motor actions coupled with visual perception of the results of those actions. Studies in both children and adults have indeed found that training on a manual rotation task does improve performance on mental rotation (60, 61).

Computational efforts to model the learning of visual transformations have generally represented each transformation as a set of weights in a neural network. In early work, distinct networks were used to learn each transformation individually (62). More recent work combines the visual and motor components of inputs for learning mental rotation (63). While many of these approaches implement visual transformations as distinct operations, a more general approach might represent continuous visual operations as combinations of basis functions that can be combined in arbitrary ways (64). Along these lines, other recent work uses more complex neural networks to represent transformations as combinations of multiple learned factors, although this work still focused on relatively simple transformations like rotation and scaling (65, 66).

People certainly do not learn visual transformations from specialized training on rotation, scaling, etc., taken as separate transformations. More generally, we have access to a very robust and diverse machinery for simulating visual change, and the simple “mental rotation” types of tasks often used in studies of visual imagery tap into only very tiny slices of this knowledge base. In line with evidence of the importance of motor actions and forces on our own imagery abilities (18), we expect that work in AI to model physical transformations—especially work in robotics that combines visual and motor inputs/outputs—will be essential for producing the kinds of capabilities agents need for visual imagery.

There is starting to be a wave of relevant work in AI in the area of “video prediction,” which involves learning representations of the appearance of objects as well as their dynamics (67–69), including for increasingly complex forms of dynamics, as with a robot trying to manipulate a rope (70). Importantly, these efforts focus on learning and making inferences about object dynamics directly in the image space, as opposed to computational approaches that rely on explicit physics simulations and then project predictions into image space. Thus, these new approaches offer intriguing possibilities as potential models for how humans might learn naive physics as a form of imagery-based reasoning.

Learning Gestalt Principles.

Many visuospatial intelligence tests rely on a person’s knowledge of visual relationships like similarity, continuity, symmetry, etc. Simple tests like shape matching require the test-taker to infer first-order relationships among visual elements, while more complex tests like the Raven’s often progress into second-order relationships, that is, relations over relations.

In one sense, a test like the Raven’s ought to be agnostic with respect to the specific choice of first-order relationships, and, indeed, in many propositional AI agents, a relation like CONTAINS(X, Y) can be replaced with any arbitrary label, and the results will stay the same. However, for people, the actual visuospatial relationships at play do deeply influence our problem-solving capabilities. For example, isomorphs of the Tower of Hanoi task are more difficult if task rules are less well aligned with our real-world knowledge about spatial structure and stacking (71). Similarly, the perceptual properties of Raven’s problems have been found to be a strong predictor of item difficulty (72).

A person’s prior knowledge about visuospatial relationships is closely tied to Gestalt perceptual phenomena. In humans, Gestalt phenomena have to do, in part, with how we integrate low-level perceptual elements into coherent, higher-level wholes (73), as shown in Fig. 5. Psychology research has enumerated a list of principles (or laws, perceptual/reasoning processes, etc.) that seem to operate in human perception, like preferences for closure, symmetry, etc. (74). Likewise, work in image processing and computer vision has attempted to define these principles mathematically or computationally, for instance, as a set of rules (75).

Fig. 5.

Images eliciting Gestalt “completion” phenomena. Left contains only scattered line segments, but we inescapably see a circle and rectangle. Right contains one whole key and one broken key, but we see two whole keys with occlusion.

However, in more recent computational models, Gestalt principles are seen as emergent properties that reflect, rather than determine, perceptions of structure in an agent’s visual environment. For example, early approaches to image inpainting—that is, reconstructing a missing/degraded part of an image—used rule-like principles to determine the structure of missing content, while later approaches use machine learning to capture structural regularities from data and apply them to new images (76). This seems reasonable as a model of Gestalt phenomena in human cognition; it is because of our years of experience with the world around us that we see Fig. 5, Left as partially occluded/degraded views of whole objects.

Image inpainting represents a fascinating area of imagery-based abilities for artificial agents (54), which we used in our model of Gestalt-type problem solving on the Raven’s test (42), as described earlier. Other work in computer vision and machine learning studies the extent to which neural networks not explicitly designed to model Gestalt effects might exhibit such effects as emergent phenomena (77–81).

Learning a Problem-Solving Strategy

Relatively little research in AI has proposed methods for automatically generating problem-solving procedures for intelligence tests, despite the extensive research on manually constructed solution methods or methods that rely on a large number of examples (20). How does a person obtain an effective problem-solving strategy for a task they have never seen, on the fly and often without explicit feedback? Some human research suggests that children learn to solve a widening range of problems through two primary processes of 1) “strategy discovery,” that is, discovering new strategies for certain problems or tasks, and 2) “strategy generalization,” that is, adapting strategies they already know for other problems or tasks (82, 83).

Some AI research on strategy discovery can be found in the area of inductive programming or program synthesis; that is, given a number of input–output pairs, constraints, or other partial specifications of a task, together with a set of available operations, the system induces a “program” or series of operations that produces the desired behaviors (84). In other words, “Inductive programming can be seen as a very special subdomain of machine learning where the hypothesis space consists of classes of computer programs” (85). Inductive programming has been applied to some intelligence test-like tasks, such as number series problems (86), and to simple visual tasks like learning visual concepts (87, 88). However, more research is needed to expand these methods to tackle more complex and diverse sets of tasks. For example, given the imagery-based strategies described above, a challenge for imagery-based program induction would be to derive these strategies automatically from a small set of example Raven’s problems.

AI research has often investigated strategy generalization through the lens of integrating planning with analogy. Case-based planning looks at how plans stored in memory are retrieved at the appropriate juncture, modified, and applied to solve a new problem (89). The majority of this work has focused on agents that use propositional knowledge representations, and very little (if any) has applied these methods to address intelligence tests.

Research on strategy selection and adaptation would be enormously informative for studying not just how people approach a new type of intelligence test but also interproblem learning on intelligence tests, that is, learning from one problem (even without feedback) and using this knowledge to inform the solution of the next problem. In humans, one fascinating study gave each of two groups of children a different set of Raven’s-like problems to start with, and then the same final set of problems that had ambiguous answers (53). Depending on which set of starting problems they received, the children predictably gravitated toward one of two profiles of performance on the final problems. Modeling these phenomena remains an open challenge for AI research.

Learning the Problem Definition

Even with intelligent agents that generate their own problem-solving strategies or programs, the problem definition—that is, the problem template and goal—is still provided by the human system designer. Interactive task learning is an area of AI research that investigates how “an agent actively tries to learn the actual definition of a task through natural interaction with a human instructor, not just how to perform a task better” (90). Research in interactive task learning generally involves designing agents or robots that learn from both verbal and nonverbal information, that is, instructions along with examples or situated experiences (91, 92).

Such multimodal inputs are used all of the time in human learning, including on intelligence tests: Most tests combine verbal (spoken or written) instructions with simple example problems to teach the test-taker the point of each new task that is presented. For example, the Raven’s test typically begins with spoken instructions to select the answer choice that best fills in the matrix, together with a very simple example problem that the test administrator is supposed to show the test-taker, along with the correct answer.

Any Raven’s agent must contain information about the problem definition in order to parse new problems appropriately and to follow a procedure that attains the goal. Moreover, agents should be able to modify their problem definition to accommodate slight problem variations. For example, if a new problem is presented with two empty spots in the matrix, a robust agent should be able to infer that this problem requires two corresponding answer responses.

In all extant Raven’s agents, knowledge of the problem definition is manually provided by system designers. While these concepts may seem straightforward to a person, and indeed are usually trivial to program into an agent as static program elements, it is a challenging open question to consider where these concepts come from, and how they might be learned. For example, people gain extensive experience in taking multiple choice tests from a very early age, especially in modern societies, but we do not know precisely how this knowledge is represented, or the mechanisms by which it is generalized to new tasks.

The interesting subproblem of “nonverbal task learning” considers how the task definition can be learned purely through a small number of observed examples, without the use of explicit language-based information at all (93). While nonverbal mechanisms are undoubtedly at play in multimodal task learning for most people, nonverbal task learning in its pure form does also occur.

There are many clinical populations in which individuals have difficulties in using or understanding language, including acquired aphasias or developmental language disorders. Nonverbal intelligence tests are specifically designed for use with such populations, and they avoid verbal instructions altogether (94). In these tests, examiners initially show test-takers a simple example problem and its solution. Test-takers must learn the task definition (e.g., matching shapes, finding one shape in another, completing a visual pattern, etc.) by observing the example, and then use this knowledge to solve a series of more difficult test problems.

A small but intriguing set of converging research threads in AI have pinpointed the importance of nonverbal task learning. One recent study using robots looked at how abstract goals can be inferred from a small number of visual problem examples and applied to new problems, where the goal is represented in terms of a set of programs that meets it (95). Even more recently, a new Abstraction and Reasoning Corpus has been proposed for artificial agents, containing 1,000 visual tasks with distinct goals; agents must infer the goal for a given task from a few examples and then use this knowledge to solve new problems (96). Both of these tasks are similar to the Raven’s test in the sense that, even though the Raven’s test ostensibly only has a single goal (i.e., choose the answer that fits best), different Raven’s problems can be thought of as requiring different formulations of this overarching and extremely vague goal. These examples also pose interesting questions about the extent to which problem goals might be implicitly represented within an agent’s problem-solving strategy, instead of explicitly, and the pros and cons of each alternative.

Note that this discussion only considers goals that are well defined, at least in the minds of the problem creators. Intelligence tests are a rather odd social construct for this reason; in a way, the test-taker is trying to infer the intent of the test designer. How agents (or humans) represent and reason about their own goals might involve an extension of the processes described here, or they might be different modes of reasoning altogether.

Conclusion and Implications for Cognitive Science

We close by returning to the motivating questions from the Introduction. The cognitive science question is, what are the computations taking place in someone’s mind when they use visual imagery?

AI research alone cannot, of course, fully answer this question, and so we presented a second, more limited question: If you have an intelligent agent that uses visual imagery-based knowledge representations and reasoning operations, then what kinds of problem solving might be possible, and how would it all work?

In this paper, we have presented a review of AI research and open lines of inquiry related to answering this question in the context of imagery-based agents that solve problems from the Raven’s Progressive Matrices intelligence test. We discussed 1) why intelligence tests are such a good challenge for AI; 2) a framework for artificial problem-solving agents; 3) several imagery-based agents that solve Raven’s problems; and 4) how an imagery-based agent could learn its domain knowledge, problem-solving strategies, and problem definition, instead of these components being manually designed and programmed.

More generally, whether or not imagery-based AI agents are at all similar to humans, designing, implementing, and studying such agents contributes valuable information about what is possible in terms of computation and intelligence. AI research that develops different kinds of agents is helpful for sketching out different points in the space of what is possible, and AI research that enables such agents to learn is helpful for hypothesizing how and why various computational elements of intelligence might come to be. Then, further interdisciplinary inquiries can proceed to connect findings and hypotheses derived from these lines of AI research to corresponding lines of research about what humans do.

Data Availability.

There are no data underlying this work.