TABLE OF CONTENTS

PRINT November 2014

TECHNOLOGY

art and the algorithms of forgery

Artist unknown, Contraste de formes, n.d., oil on canvas, 35 × 28".

IN THE PROCESS of authentication—of verifying who made what when—every painting becomes a landscape painting. Pigments are harvests, geology, trade routes, chemistry: Scheele’s green and lead white, viridian and madder and chrome yellow. The board on which they’re painted can be dendrochronologically dated, analyzing the rings and grain that document dry years and bitter winters in an oak on the Ligurian coast. A portrait from a wall in a private home is a slow-developing accumulation piece about coal heat, gaslight, and lampblack. The cotton of the canvas of a fake Fernand Léger, supposedly painted in 1913, carries “bomb peak” levels of carbon 14, the residue of postwar atmospheric nuclear testing—making it a picture of the skies over New Mexico and Novaya Zemlya in the 1950s that also happens to have a knockoff “Contraste de formes” on it. Just as a city skyline has its mirror image in the negative space of quarries, mines, and pits, so a painting forensically analyzed is the reflection of its infrastructure and era.

A painting is also a probability landscape. A brushstroke, a drip, a line are decisions made against the backdrop of all the other possible marks not made—not just in the sense of pentimenti and underdrawing but as characteristic choices, as style. Or take prose: Every word is a step on a path through potential arrangements. Stylometric machine-learning tools are very good at finding these patterns, with which we can distinguish authors and identify collaborations and forgeries. Prose decomposed is data: lengths of sentences, choice and order of words, syntax and idiosyncrasies of usage, and elements whose predictive power escapes the human writer. For Gertrude Stein, a comma was like a butler, solicitously reminding you to breathe and helping you find your slippers; as it turns out, commas are also like butlers in that they can testify against their employers and rat them out.

In the case of words, stylistic analysis is fairly intuitive. Rhythm, cadence, a baffling preference for crepuscular—we can understand how inferences can be drawn from these elements. But how can this be applied to visual entities like paintings? Consider a signature: Sylvia Ann Howland’s, in 1867. A great fortune hangs on that scribble, among the most closely studied in history, the Zapruder film of handwriting. Howland had willed some of her estate in trust to her niece, Hetty Robinson; Robinson produced a second, secret will awarding her the whole thing and sued the executors. The important page was in Robinson’s handwriting, as she’d taken dictation from her elderly, infirm aunt. Only the signatures on the page were Howland’s. Or not. On this, millions rested.

The concern wasn’t that the signatures on the conflicting wills were too different; it was that they were too similar. They were identical, stroke by stroke, and even their placement and distance from the margins on their respective pages was the same, which made them look less like writing than tracing. “Such evanescent shadows of probability,” said the mathematician and astronomer Benjamin Peirce in his testimony, “cannot belong to actual life.” He and his son Charles Sanders Peirce, the philosopher, were attacking this problem with the mathematics of probability rather than the visual connoisseurship of signature experts. They identified the most salient visual features of the signature—thirty downstrokes—and went through dozens of examples, cataloguing the variations for each combination. They developed a statistical model of the (very remote) likelihood that the signatures on the contested page could precisely correspond. In the end, the case was dismissed and Robinson settled out of court. (She had already married by then and become Hetty Green, the “Witch of Wall Street,” the legendarily shrewd investor who was to become the country’s wealthiest woman.) The debate was never really settled: Were the signatures perfectly fine, or were they too perfect?

The case is pretty perfect, for our purposes. It was the young C. S. Peirce, after all—who would go on to found the discipline of semiotics in America and distinguish signs-referring-to-things from signs-that-are-things—being faced with distinguishing a signature from a picture of a signature. (Shades of Jasper Johns and his paintings of flags and targets, which are also, well, flags and targets.) How Peirce père et fils went about it was to take something visual and turn it into sets of probabilities. Which brings us back to paintings.

Metaphors begin to fail us here, but let’s synchronize our dive watches and jump in. A lot of the world we encounter, and the ways in which we encounter it, can be described as waveforms. And all waveforms, as simple as a heartbeat or complicated as a free-jazz ensemble meltdown, can be described as the sum—the superposition—of simple sine waves of different frequencies. This astonishing fact implies a technique. The process of turning a complex signal into a collection of simple constituent waves is called a Fourier transform, after the mathematician and physicist Joseph Fourier. This transformation is ubiquitously useful: If you can take the complexity of a human voice or a digital image and transform it into its underlying frequencies, you can weigh the frequencies in their effects on human perception (for understanding a voice or seeing an image) and discard those less important—processing and compressing, doing more with less bandwidth and storage space. Transforms, by the nature of their decompositions, can also reveal data hidden within a signal that might otherwise have been opaque, disassembling the puzzle of the underlying patterns to make sense of everything from from your MRI to the chord at the beginning of “A Hard Day’s Night.”

Good so far? The next step is transforming a signal into wavelets. Wavelets are fascinating and enormously difficult to describe simply: They break data up into different frequency components at different degrees of resolution so that you get finer- and coarser-grained samples across a scale simultaneously. You can comprehend the forest and the trees together, and over time. Think of it as cinematic deep focus for data, with that Gregg Toland shock of seeing everything clearly, all the way back to the last gallery of Kane’s Xanadu—and think of the new kinds of stories we were able to tell in cinema with deep focus. Rather than C. S. Peirce spending weeks laying onionskin transparencies over one another and counting thousands of discrepant downstrokes, wavelet transformation can decompose digital images into patterns with distinctive properties at all scales: whole, parts, and parts of parts.

Let’s scan a bunch of van Goghs and decompose the images. Forget crude pixel coordinates; with a “dual-tree complex wavelet transform” you get color patterns, orientations, and local components in a whole-picture context across different levels of detail. Now we can use the kind of stylometric machine-learning tools mentioned above to do “feature extraction,” identifying the most meaningful parts of the data for what we want to get from a collection of images. (Human faces? License-plate numbers? Background cats?) In this case, we want the most distinctive elements of van Gogh’s brush-handling technique—not just stroke by stroke, but how his marks vary from painting to painting. We want to quantify and infer that elusive variable of style hidden within everything that actually shows up in the scanned image, perturbing what’s visible like an unobserved planet wobbling the orbits of a sunlit system. You can, like Ingrid Daubechies, one of the most important wavelet theorists, and her collaborators, identify a painting as a van Gogh and date it in relation to other paintings in the training set—and in relation to a moment in the genesis of a style, and thus to a place and an approximate time. The process gets fine-grained enough that (in another Daubechies experiment) the system can spot copies through the “hesitancy index,” the way a stroke trying to reproduce another stroke has a different and measurable fluency in the texture of the details.

This is only the beginning. Rapid improvements in “deep learning” systems—the ones getting better and better at identifying your face across social-network posts, at translating analogies between languages, reading handwritten numbers, and finding distinct objects in videos—are supercharging the tools that can process and make sense of these new classes of visual data. Computers are learning how to look at paintings with their inhuman form of probabilistic connoisseurship.

Two interesting consequences follow from this, one to do with forgery and the other with creation. For the first: Duchamp was preoccupied with what he called the infrathin, a way to think through the smallest of differences, like the difference between two almost (nearly, virtually, practically) identical items. “The more identical they are,” he noted, the more they “move towards / this infra thin separative / difference.” In the feature vectors, predictive neighborhoods, “random forest” classifiers, and other components of data analysis, we draw asymptotically close to the mystery of the art object itself, to what the real thing has of realness—a theoretical subject of practical interest to curators and museum conservators, who are often the source of the data for training the systems. A hand capturing a glance caught in the mirror in the cicada-loud air of a day in high summer: The moment has a texture, as immediate and yet as faint and intangible as vapor on glass or the trace of warmth in a chair just vacated, to take two of Duchamp’s striking examples of the infrathin, and machine learning can find that trace, that grain in what is seen.

For the second: Imagine these technologies and algorithms functioning not as authenticators and supplements to art history but as collaborators, the way artificial sound decomposition and synthesis transformed music, from Delia Derbyshire making avant-garde sonic auroras to Giorgio Moroder reinventing the dance floor. Think of a painting—of the act of painting itself—as the starting point for riding the statistical dynamics of color, brushstroke, and line in new directions, the way you’d ride a wave. See a quondam still life through the eyes of a neural network that’s been feature trained on nothing but satellite photography and human handwriting. Extract the purest expression of the hidden variable around which the work of a given painter turns, and crank it up to uncanny, feverish degrees, making pictures more purely Leonora Carrington or Mark Rothko than either of them could have produced. (Then crossbreed Carrington with Bacon, and let their hybrid work evolve through a few million generations of modified genetic algorithms, competing for survival measured in whorls and steep color gradients.) Generate antistyles, parastyles, metastyles, and blow through a dozen future schools and genres in a month. Over the ridges and horizons of the probability landscape, waiting for us, are the unseen, unthought forests and deserts of the visible.

Finn Brunton is an assistant professor in media, culture, and communication at New York University.