Old Musicians Never Die. They Just Become Holograms.

https://www.nytimes.com/2020/01/07/magazine/hologram-musicians.html

Version 0 of 1.

In preparation for his first American tour in a decade, Ronnie James Dio spent months sequestered in a modest office suite in Marina del Rey, in Los Angeles. The office was on the second floor of a strip mall, above a vape shop and a massage parlor. I visited at the end of May, only a couple of days before the opening date of the tour, and among Dio’s team, there was a tangible air of anticipation. Dio never became a household name, but he is considered one of the great heavy-metal vocalists of all time, up there with Ozzy Osbourne (whom he replaced in Black Sabbath) and metal-adjacent rockers like Axl Rose and Robert Plant. Beginning in the 1970s, Dio took a lead role in codifying a number of his genre’s most ludicrous, yet utterly foundational, conventions. He sang of wolves and demons, toured with an animatronic dragon and supposedly introduced the splay-fingered “devil horns” headbanger’s salute, which he claimed his Italian grandmother used to flash as an old-world method of warding off the malocchio and other forms of bad luck.

Opinion among the Dio faithful, nonetheless, was divided on the subject of his “Dio Returns” comeback tour, largely because Dio has been dead for almost 10 years. The Marina del Rey office suite was the site of a visual-effects company creating a Dio hologram. The hologram would tour with a living backing group consisting, in large part, of former Dio bandmates.

If you missed the tour, you might want to take a moment here and call up one of the fan-shot videos posted on YouTube — say, “Rainbow in the Dark,” Dio’s 1983 hit, filmed at the Center Stage Theater in Atlanta on June 3, during which the Dio hologram prowls a central portion of the stage, bobbing, weaving, twirling his microphone cord to the monster riffs and occasionally using his free hand to air-conduct his most operatic vocal flourishes. (“His” — would “its” be more apt? Neither word feels quite right.) At one point, the bassist, Bjorn Englen, takes several very deliberate steps to his left, allowing the hologram to dance in front of him and adding to the illusion of a three-dimensional conjuring.

The hologram itself has an uneasy pallor, a brighter shade than the humans onstage but at the same time insubstantial, like a ghost struggling to fully materialize. One crucial decision that had faced the animators was choosing the right age for their creation. Dio in his MTV-era prime tempted them, of course, but then wouldn’t it be strange to watch him perform alongside band members who were roughed up by the ensuing years like the rest of us? Then again, Dio’s actual age in 2019, were he alive, would be 77, which is not ideal for a heavy-metal frontman. The creative team ultimately settled on a spry, middle-aged Dio, outfitting him in black leather pants, a studded leather wristband and a bell-sleeved white tunic embossed with a silver cross.

A start-up called Eyellusion produced “Dio Returns.” It’s one of a handful of companies looking to mold and ultimately monetize a new, hybrid category of entertainment — part concert, part technology-driven spectacle — centered, thus far, on the holographic afterlives of deceased musical stars. Eyellusion also toured a hologram of Frank Zappa in the spring, in a show overseen by Zappa’s son Ahmet. The tour kicked off in April at the Capitol Theater in Port Chester, N.Y., about an hour north of Manhattan in Westchester County. A few hours before the show, I talked to the owner of the venue, the 47-year-old concert promoter Peter Shapiro. In 2015, he was a producer of the Grateful Dead’s 50th-anniversary “Fare Thee Well” concerts. The five shows grossed more than $50 million, becoming, according to Billboard, “one of the most successful events in live-music history.” We met at the Capitol Theater bar, which is called Garcia’s and serves as a sort of secular reliquary devoted to the Dead’s frontman, Jerry Garcia. The décor included one of Garcia’s banjos and a Chuck Close-style portrait of Garcia made entirely of Lego bricks. Shapiro, who attended a preview of the Zappa concert, said, “What I just saw felt closer to seeing Zappa than seeing a cover band do it,” adding that, based on ticket sales alone, he would definitely book another hologram show. The theater, which holds 1,800 people, was close to sold out for opening night.

“But here’s the headline,” Shapiro went on. “Look at who’s gone, just in the last couple of years: Bowie, Prince, Petty. Now look who’s still going but who’s not going to be here in 10 years, probably, at least not touring: the Stones, the Who, the Eagles, Aerosmith, Billy Joel, Elton John, McCartney, Springsteen. That is the base not just of classic rock but of the live-music touring business. Yes, there’s Taylor Swift, there’s Ariana Grande. But the base is these guys.”

Shapiro’s calculation might be morbid, but he isn’t wrong. According to the trade publication Pollstar, roughly half of the 20 top-grossing North American touring acts of 2019 were led by artists who were at least 60 years old, among them Cher, Kiss, Fleetwood Mac, Paul McCartney, Dead & Company and Billy Joel; the Rolling Stones, Elton John and Bob Seger took the top three slots. Using technology to blur the line between the quick and the dead tends to be a recipe for dystopian science fiction, but in this case, it could also mean a lucrative new income stream for a music industry in flux, at a time when beloved entertainers can no longer count on CD or download revenues to support their loved ones after they’ve died. “If you’re an estate in the age of streaming and algorithms, you’re thinking: Where is our revenue coming from?” Brian Baumley, who handles publicity for Eyellusion, told me. Some of those estates, Baumley bets, will arrive at a reasonable conclusion about the dead artists whose legacies they hope to extend: “We have to put them back on the road.”

Tupac Shakur became one of the earliest test subjects for the new technology 15 years after his murder, when his hologram made a surprise appearance at the 2012 Coachella festival. To actually project a person-size holographic image into three-dimensional space, à la Princess Leia in “Star Wars,” would require powerful, prohibitively expensive lasers that would also burn human flesh. The Tupac hologram was created with a combination of C.G.I., a body double and a 19th-century theatrical trick known as Pepper’s Ghost, some variation of which has been used for almost all the hologram musical performances of recent years.

As the magician and magic historian Jim Steinmeyer recounts in his book “Hiding the Elephant,” John Henry Pepper, the director of the Royal Polytechnic Institution in London, popularized the technology with a dramatization of a scene from the Charles Dickens novella “The Haunted Man” on Christmas Eve 1862. To call up his ghosts, Pepper projected a bright light onto an actor in a hidden, cutout space beneath the stage, something like an orchestra pit, casting a reflection onto an angled pane of glass. The glass stood upright on the stage but remained invisible to the audience. The spectral image appeared slightly behind the glass, “moving in the same space with the actors and the scenery,” Steinmeyer writes. “If all the players were perfectly synchronized, the ghost could interact with the characters onstage, avoiding sword thrusts or walking through walls.” Pepper intended the original display, which took place at the Polytechnic Institution, as a scientific lecture, but the audience’s riotous response persuaded him to go the magician’s route, and soon he began touring the illusion in British and American theaters.

The Tupac hologram performed only two songs, shouting, “What the [expletive] is up, Coachella?” and rapping “2 of Amerikaz Most Wanted” alongside Snoop Dogg. But his digital resurrection worked as a proof of concept. A handful of one-off stunts involving other dead musicians followed: A Michael Jackson hologram performed at the 2014 Billboard Music Awards, and the Mexican pop superstar Juan Gabriel made a holographic appearance at his own memorial concert after his sudden death in 2016. Still-breathing musicians also made use of the technology, including the rapper Chief Keef, who in 2015, as a means of avoiding outstanding legal warrants, beamed a hologram performance from California to a music festival in Hammond, Ind. But the outstanding question remained: Would audiences turn out for an entire hologram concert?

Marty Tudor, chief executive of Base Hologram Productions, is an entertainment-industry veteran whose multifarious career has included, among other things, managing Paula Abdul and Jon Cryer, producing a series of exercise videos with a trainer from “The Biggest Loser” and running an independent record label. When he saw footage of the Tupac hologram at Coachella, Tudor had a hunch that there might be potential for the new technology beyond gimmicky festival cameos.

Tudor took the idea to Brian Becker, the former chief executive of Clear Channel Entertainment, which was the largest events promoter and venue operator in the country during Becker’s tenure. For Becker, live entertainment was a family business. In 1966, his father, Allen Becker, a life-insurance salesman from Houston, helped found a regional events-promotion company called Pace Entertainment that eventually became a major national promoter. When Brian joined the company after college, he helped to start Pace’s theatrical division, which soon came to dominate, and largely invent, a regional touring market for effects-laden Broadway spectacles like “Cats,” “Miss Saigon,” “Les Misérables” and “The Phantom of the Opera.” The technical innovations of those shows, Becker told me, “evened the score,” signaling to regional audiences that they would be seeing a production with all the same bells, whistles and helicopters as a show in New York or London. “We’re always cognizant of seams in our industry that might allow us to do things differently,” Becker said. After hearing out Tudor’s hologram pitch, Becker wondered if the technology might represent such a seam.

In the wake of the Tupac performance, a somewhat motley assortment of newly minted hologram companies were asking themselves the same question, and soon a scramble to lock down exclusive deals with music estates ensued. Digital Domain, the visual-effects house that created Tupac, wound up declaring bankruptcy not long after the Coachella performance, but one of its owners, a Florida investor named John Textor, quickly started a new company, Pulse Evolution, which produced the Jackson hologram and soon after announced that it had also cut hologram deals with the estates of Elvis Presley and Marilyn Monroe, as well as for the band Abba, which broke up in 1982. An eccentric British-Greek billionaire named Alki David, meanwhile, started a rival hologram company, Hologram USA. An heir to a Coca-Cola bottling fortune, David, along with his partners, announced that he would be producing holographic images of Patsy Cline, Billie Holiday and Jackie Wilson, among others. (In September, David and Hologram USA were charged by the Securities and Exchange Commission with “making false and misleading statements to investors and potential investors.” David has said he intends to countersue.)

Base Hologram, which was founded by Tudor and Becker, started out by securing rights to produce holograms of Maria Callas and Roy Orbison, debuting each show in 2018 with performances in Europe and America. Orbison’s estate, which is controlled by his three sons (via a company called Roy’s Boys), approached Base after a deal with another hologram producer fell through, Tudor told me. “Roy was a fairly static live performer — most of the movement you have onstage is him strumming his guitar — so he was the perfect first performer for our purposes,” Tudor said. (A 58-date Orbison-Buddy Holly hologram tour began in San Francisco in September.) The Callas hologram was necessarily more emotive. At a brief demonstration I attended at Sotheby’s in New York, the hologram wore a white gown and a long red shawl. After performing “Melons! Coupons!” from Act III of “Carmen,” a scene involving fortune telling, the hologram tossed a deck of cards in the air, which briefly froze alongside the music before drifting to the ground. “Though a melodramatic touch, it worked,” Anthony Tommasini wrote in his New York Times review of the Lincoln Center performance, in which he described the show as “amazing, yet also absurd; strangely captivating, yet also campy and ridiculous.” In February, Base will unveil the dead-celebrity-hologram sector’s biggest marquee name thus far, at least for a full concert: Whitney Houston, whose tragic, relatively recent death has made the planned tour the most controversial of any on the books. (Shortly after the announcement, Questlove tweeted: “& hell begins.”)

Deborah Speer, a features editor at Pollstar, which covers the live-entertainment industry, told me that based on the numbers she has seen for the Orbison and Zappa tours, “obviously, there’s a market” for hologram shows. According to the trade publication, the solo Orbison tour grossed nearly $1.7 million over 16 shows, selling 71 percent of the seats available, while Zappa sold an average of 973 seats per show, nearly selling out venues in Amsterdam and London. Whether such tours can cross over from clubs, theaters and performing-arts centers into arenas remains to be seen and will depend largely on the success of bigger-name stars like Houston.

Early one morning in May, I visited a soundstage in the Griffith Park neighborhood of Los Angeles to observe a motion-capture shoot for the Whitney Houston hologram. The soundstage was a cavernous, warehouselike space, moodily lit, aggressively air-conditioned. Several of the Angelenos on hand complained about the cold, including Tudor, who sat in a nearby director’s chair wearing a puffy vest over a striped dress shirt and jeans. Fatima Robinson, the director of the production, wore a head scarf and a winter jacket and cupped a rechargeable electronic hand-warming device between her palms. Robinson is a choreographer whose credits include Kendrick Lamar’s 2016 Grammys performance, the Weeknd’s 2016 Oscars performance, the film version of “Dreamgirls,” NBC’s live broadcast of “The Wiz” and music videos for Michael Jackson, Mary J. Blige and Aaliyah. Robinson also choreographed Houston herself — the living Houston — in 1993, for the “I’m Every Woman” video. “She was pregnant at the time and in a wonderful place,” Robinson told me.

Veterans of pedigreed Hollywood postproduction houses create the C.G.I. holograms in the same way they would make characters like Gollum or Thanos: Motion-capture photography records the performance of a body double, which becomes the basis for a three-dimensional digital model, a block of clay animators proceed to modify — in the case of celebrity holograms, most drastically by augmenting the body double’s features with a digitally sculpted likeness of the artist, which can lip-synch to an existing vocal track.

The Houston body double took the stage and began to run through the moves for the first song of the day: “Step by Step,” a jaunty, affirmational gospel-dance track from the 1996 soundtrack to “The Preacher’s Wife.” The double had freckles and wore her hair in dyed cornrows but possessed Houston’s approximate build. She wore black tights, a black T-shirt and a baggy white cardigan (costumes created by Houston’s former stylist would be worn in a subsequent shoot) and stood atop a sort of oversize lazy susan, which crouching tech guys, who referred to the device as a turntable, slowly spun as she lip-synched to the song.

Robinson sipped tea and watched the pantomime intently. After the first run-through, she said, “We need to go a little slower.” The body double had been chosen from a pool of 900 applicants, and she was clearly a talented performer in her own right. (Base requested that The Times not reveal her identity.) “Step by Step” remains an underappreciated Houston song, cloying but oddly irresistible, and as I watched it mock-sung over and over, I felt freshly reminded of Houston’s skill at putting over mediocre material, not just in the obvious way — that is, through the power of her voice — but with her presence, that way she had of conveying joy, supreme confidence and the ecstasy of the choir all at once, and at the same time letting us know, even back then, that she wasn’t as sweet as her songs’ lyrics might suggest. This complexity came through in the body double’s performance, in the way she worked her shoulders or flashed a hard look at the nonexistent audience. Houston wasn’t much of a dancer, but “she had a serious strut,” noted Robinson, who had studied her performances like game tapes.

Lit for the filming, the double cast a horror-movie shadow on the soundproofed wall of the otherwise darkened soundstage. There was something eerie about the way Houston’s voice and the mid-’90s dance beat echoed through the vast space — music being played at club volume to a nearly empty room, with no one dancing, not even the avatar pretending to sing. But despite the workaday setting and the unconcealed artifice, by the third or fourth time I heard the song, I couldn’t help feeling … something. Would I describe myself as moved? I’m not sure. But I also found myself wondering if, despite how fundamentally wrong the entire concept for this show felt, there might be some crazy way it could actually work. The future hologram moved her mouth around Houston’s voice:

Well there’s a bridgeAnd there’s a riverThat I still must crossAs I’m going on my journeyOh, I might be lost

In the final show, Tudor whispered to me, the turntable could be digitally removed or made to look like something else. The creative team hadn’t settled on anything yet. But if they wanted to, they could make Houston look as if she were floating on air, spinning, ascendant.

I met Ronnie James Dio once, when he was alive. Tenacious D, the parody band that gave Jack Black his start, had recorded a gently mocking tribute song called “Dio,” in which Black demands Dio’s cape and scepter and informs him that he’s too old to rock (“no more rockin’ for you!”). Dio had been a good sport about the whole thing and agreed to make a cameo in the Tenacious D movie, which premiered in 2006 at Grauman’s Chinese Theater. I remember standing around the after-party, nursing a drink and feeling awkward, when I spotted Dio, chatting in a corner of the ballroom with his wife. I decided to introduce myself. He was quite short, even for a celebrity, and exceedingly gracious. He told me Black had personally called to pitch the film, insisting that they wouldn’t make the movie unless he agreed to “play the part of Ronnie James Dio.” Smiling, Dio continued, “Then he said: ‘Well, we will make the movie. But it’ll be [expletive].’ ”

Across town in Marina del Rey 13 years later, I sat in the office of Eyellusion’s creative director, Chad Finnerty, as he digitally manipulated a photorealistic 3-D image of Dio’s face. Finnerty grew up in Pennsylvania with dreams of becoming a Disney animator — old-fashioned cell animation, like what they did on “Snow White and the Seven Dwarfs” — but by the time he graduated from college, the world had gone digital. He spent years working as a C.G.I. animator at Digital Domain, on movies like “Flags of Our Fathers” and “Pirates of the Caribbean: At World’s End.” When Jeff Pezzuti, a Westchester-based vice president of finance at a cloud-computing consulting firm, decided to start his own hologram company, Eyellusion, he reached out to Finnerty, asking if he wanted to talk. Pezzuti loved heavy metal — he wore a Dio T-shirt for his seventh-grade class picture — and after seeing the Tupac hologram, he wondered, “Can we do something like that in the rock world?” Eyellusion has since received a $2 million investment from Thomas Dolan, whose family owns controlling interests in Madison Square Garden and AMC Networks and whose father founded the New York-area cable-television giant Cablevision.

Finnerty supervised the creation of the Zappa and Dio holograms for Eyellusion. “I’m a bit rusty with this program,” he apologized, pecking at his desktop keyboard. Soon a hideously lifelike digital rendering of Dio’s face appeared on a large-screen monitor hanging on the wall. For a moment, it bobbed in front of a black backdrop, which made me think of the old “Charlie Rose” set. I briefly thought about pitching a “Black Mirror” episode in which a Charlie Rose-type character interviews the cryogenically preserved heads of rock stars. “We collected all of our data in 2017,” Finnerty explained. That’s when they filmed the body double and did the facial capture, is what he meant. During the facial capture, hundreds of eye, mouth and facial-muscle movements of a living subject (not necessarily the body double) are recorded. Imagine a puppeteer, Finnerty said, only with thousands of puppet strings to manipulate.

He clicked his mouse, manipulating a digital lever on the screen, and “Dio’s” eye suddenly, eerily shifted to the left. You couldn’t do this two years ago, Finnerty went on, moving another lever. “Dio’s” eyes shifted right, up, down. Finnerty said he had done lots of work on “The Walking Dead,” but that was forgiving, because it’s zombies. Having a person look real while performing a song for six minutes, with no cutting away or other editing assists that would be available in a film or television show, that was something else entirely.

“Dio” winked, puckered his lips, raised an eyebrow.

I stared at the image’s mottled skin, textured and painted with a level of detail down to the pore. “Hair simulation is the most difficult part of the entire process,” Finnerty said, adding, “My hair guy is also my fire, water and ice guy.” His lighting team had done the skin. Had Dio submitted himself to a full-body scan while alive, the process would have been much easier. Finnerty thought it would be great if more living musicians and actors were proactive about being scanned. Any actor who has starred in a movie involving significant amounts of C.G.I. has already been scanned, he pointed out.

The more bullish hologram boosters envision all sorts of uses beyond the second coming of music deities major and minor. Finnerty just made a hologram for the Ronald Reagan Presidential Library of the former president. Prime Minister Narendra Modi of India has campaigned holographically, and a circus in Germany uses holographic projections of elephants and horses instead of live animals. Base, meanwhile, has cut a deal with Jack Horner, the paleontologist who served as a scientific adviser for “Jurassic Park,” to create dinosaur holograms that will travel to natural-history museums. Imagine, Becker said, a dialogue between holograms of Abraham Lincoln and Martin Luther King Jr. Or a Julia Child hologram teaching a cooking class. Or a Derek Jeter hologram teaching you how to bat.

As for concerts, in the not very distant future, Finnerty predicted, the technology would evolve to the point at which a puppeteer sitting in the wings with a laptop could work the digital strings live — allowing the hologram to react to the crowd or to members of a live band. Imagining this future as he watched “Dio” on his screen, Finnerty referred to him as the “asset,” as in: “This asset is ready for any other adventure we want to put him on. We could beam him into a bar. A coffee house. Not that Dio would play a coffee house.”

Whenever I wondered aloud whether fans might find the shows unsettling or disrespectful, the hologram-industry representative I happened to be speaking to would grow defensive. It’s stagecraft, part of a larger production, the person would tell me. We respect these artists, and we take what we’re doing very seriously. And as these representatives point out, people see tribute acts all the time. An Australian Pink Floyd, Tudor said, just played in Los Angeles! Pollstar’s Speer told me that well over 175 tribute bands reported numbers to the magazine; one of the better performers, “Rain — a Tribute to the Beatles,” often turns up in the top half of the Concert Pulse chart, averaging 1,833 tickets and $95,955 per show over the past three years.

For what it’s worth, the crowd at the Zappa concert seemed utterly charmed — cheering when the hologram Zappa materialized in the center of the stage during the opening number, “Cosmik Debris.” I was sitting about eight rows from the front. It looked like Zappa up there, more or less, though his form radiated the paranormal brightness that holograms can’t help emitting. Eventually, “Frank” addressed the audience: “Good evening. You won’t believe it, but I’m as happy to see you guys as you are to see the show. I’m your resident buffoon, and my name is Frank.” The artificiality of the canned banter had a “Weekend at Bernie’s” aspect to it, making me hyperaware of the sunglasses covering the lifeless eyes of the corpse propped up between living people (in this case, a hot backing band composed predominantly of musicians who had toured with Zappa over the years).

In certain respects, Zappa’s psychedelic jams and goofy, satirical lyrics lent themselves perfectly to the experiment, allowing the creative team to deploy the Zappa hologram judiciously (“like the shark from ‘Jaws,’ ” someone backstage told me) in and around trippy visuals that reminded me of old screen-saver graphics: animated dental floss, a penguin being punished by a dominatrix, Zappa as a leisure-suit-wearing Ken doll.

As I watched the show, my mind drifted, and I began to imagine more dubious ways corporate entities might exploit their particular assets. With artificial intelligence and voice cloning, there would be no reason to limit the shows to recordings made when the artist was still alive. An Aretha Franklin hologram could shush a noisy audience member, banter with her drummer and cover “Shallow.” Chris Stapleton and Sturgill Simpson could form a supergroup with holograms of Merle Haggard and Waylon Jennings. Kurt Cobain, sporting the same faded green cardigan he wore on “MTV Unplugged,” might turn up at a surprise appearance with Billie Eilish at the Grammys. A one-off Beatles reunion in Hyde Park, live Paul and Ringo, hologram John and George. Hologram Biggie takes the Thomas Jefferson role in “Hamilton.” Bob Marley interrupts his performance of “Exodus” to plug the new season of “Curb Your Enthusiasm.”

On the stage of the Capitol Theater, a grotesque claymation version of Zappa had materialized, and the guy sitting next to me began air-drumming alongside the live percussionists. Before the concert, Ahmet Zappa had pointed me to a passage in his father’s 1989 autobiography in which he seemed to predict the technology that would allow him to return to Port Chester 26 years after his death: a digressive riff about his “idea for a new device, potentially worth several billion dollars,” one that would “generate free-standing 3-d images, in any size (on your coffee table at home, or on a larger scale for theatrical use).” So maybe Zappa would have appreciated his 2019 tour. And maybe holograms will make the leap from ridiculous-seeming technology to ubiquity, like podcasts or e-cigarettes.

Ahmet was 15 when his father received a diagnosis of prostate cancer and was given three months to live. One way to think about the show, he told me, is as “a very childlike way of dealing with loss.” For a couple of hours every night, Frank is up there onstage again, playing with his guys, and Ahmet can almost convince himself that he has his father back. You’d think there would be a market for something like that.

Mark Binelli is a contributing writer for the magazine. He last wrote a Letter of Recommendation column about badly dubbed movies.