TORONTO – When director Denis Villeneuve first approached Jake Gyllenhaal to star as a detective in his new dramatic thriller, “Prisoners,” the actor was skeptical. The movie contained plenty of genre elements — heinous crime, creepy suspect — that could have turned it into pulp cliche. And besides, Gyllenhaal, fresh off the LAPD street story “End of Watch,” had just played a cop.

“The initial conscious response was, ‘Another drama where I played a member of law enforcement? I don’t know,”‘ Gyllenhaal said. “But to me, this is a movie about the unconscious.”

Additional Photos

Jake Gyllenhaal as Detective Loki in "Prisoners." Warner Bros.

Jake Gyllenhaal as Detective Loki tries to get a grip on an enraged Keller Dover, played by Hugh Jackman.

Indeed, there is much going on beneath the surface of “Prisoners,” both with the detective and the man who plays him. Detective Loki is one of the more complicated police presences on the big screen recently. Moving beyond the conventions of the typical screen gumshoe, he’s a man brimming with confidence but boiling with rage, as firmly convinced of the need for justice as he is frustrated by his inability to bring it about.

The film, which opens Friday, is the result of another mental journey to a dark place for the actor, who has been submerging his consciousness in roles like this recently and in the process re-energizing his career. Eight years after he landed a supporting Oscar nomination for “Brokeback Mountain,” Gyllenhaal, at 32, looks to be making good on the promise he showed in that film and creating work to match his level of popularity.

CRITIC’S DARLING … MOSTLY

Gyllenhaal has an unusually high degree of association with cultural phenomena (“Brokeback,” “Donnie Darko”) as well as critical darlings (“Zodiac,” “The Good Girl”). But big studio movies haven’t always been kind to him. Two years after his turn as the naive love interest in 2002’s “The Good Girl,” there was the Roland Emmerich kitchen-sink effects tale “The Day After Tomorrow.” And his well-regarded role as the author Robert Graysmith in 2007’s “Zodiac” led to the maligned desert action-adventure “Prince of Persia: The Sands of Time” several years later. Some actors grow into themselves when they move to a bigger stage (think: Denzel Washington). But for Gyllenhaal, these movies didn’t play to his strengths, and he seemed dwarfed by his surroundings.

This has left him with a kind of restless questioning, a desire to do things grittier, more his own way, not simply to jump on the next sleek passing vessel.

He demonstrated the point recently when he opted out of a Disney adaptation of the big-budget fairy tale musical “Into the Woods” to sign on to something more moody — an indie thriller called “Nightcrawler” in which he’ll play a Los Angeles crime journalist.

It’s not an accident that he’s taken on the wounded masculinity of movies such as “End of Watch” and “Prisoners” just as he’s taken up boxing and mixed-martial arts, the ultimate sports for actors craving both swagger and pain. As with those endeavors, Gyllenhaal believes if he just picks his targets with enough foresight and engages in enough preparation, he can get to an optimal career place and land his punches.

“I can’t totally put it into words, but I feel like I’ve begun to prepare in strange ways in the past three or four movies, starting with ‘End of Watch,’” said the soft-spoken but intense actor. “I don’t think I’m sharp enough to not prepare and come on set and kill it. I’ve learned over the years that freedom is just the other side of discipline.”

To play an L.A. cop in David Ayer’s “End of Watch” last year, Gyllenhaal spent months on the beat with police observing “extraordinary and horrific things.” Soon after that, he began working with the French Canadian auteur Villeneuve, whose “Incendies” was nominated for 2011’s foreign-language Oscar and was notable for its brutal revelations.

Villeneuve first cast him in “Enemy,” an erotic thriller adapted from a Jose Saramago novel. Gyllenhaal plays both lead parts — a teacher and an unemployed actor who may be his doppelg?er. The movie, which does not yet have U.S. distribution, has a Jungian quality and evokes the brooding, paranoid spirit of other double-trouble movies such as “Dead Ringers” and “Black Swan.”

Shortly after they started working on “Enemy,” Villeneuve began pushing Gyllenhaal to take on “Prisoners” as well. Though it’s rare for a high-profile actor to make such a back-to-back commitment with a little-known director, Gyllenhaal believed it was a risk worth taking and signed on long before “Enemy” was completed.

“The most success I’ve had in my career comes from those shots, those gambles — ‘Brokeback,’ ‘Donnie Darko.’ You have to know what games you’re good at. You have to know what position you play,” Gyllenhaal said. “I love movies that are saying things that people might find odd at times. I don’t find them odd at all. They give me comfort.”

Both films played at the Toronto film festival that ended Sunday, and both received strong reviews. “Prisoners,” in particular, has been a favorite; Variety called the actor’s turn a “career best” and noted the “full breadth of his impressive range.”

Directed from a script by Aaron Guzikowski, the movie slyly explores questions of revenge, parental responsibility, individualism and the suspicions that rattle around the human brain. Gyllenhaal’s Loki swaggeringly tries to crack the difficult case of two missing girls on behalf of a contractor named Keller Dover (Hugh Jackman), whose daughter disappears along with a friend outside their small-town Pennsylvania home. Convinced Loki is not doing enough, Dover secretly kidnaps the man he believes abducted the girls (Paul Dano) and tortures him to find the truth. As Loki is thwarted by the case, he and Dover become mistrustful of each other.

Gyllenhaal plays the part with a mixture of sympathy and bravado; prominent tattoos and references to a tough childhood suggest a criminal past that further colors the character. The performance is rife with subtle contradictions. Loki is full of bravado about his ability to find the kidnapper but also is manifestly unsure of how to do it.

GROWING UP ON-SCREEN

Those who worked with him on the movie say Gyllenhaal seems to be growing out of a boyishness that marked many of his roles.

“When I knew him years before he seemed like such a boy,” said Maria Bello, who co-stars in “Prisoners” as Dover’s wife. “And as soon as I saw him at the reading (of the ‘Prisoners’ script), I sensed a depth, an anger, a sense of experience, a sense of having lived more than I ever had sensed, a fight within himself that made his work so interesting.”

In many scenes, he can also be seen blinking obsessively, a tic that adds a layer of humanity and that only came about after he talked Villeneuve into letting him try it.

“I remember meeting Denis before we shot the movie, and I said, ‘I think the character has some physical tic or attribute,”‘ Gyllenhaal said. “And I could see his reaction,” the actor recalled, laughing. “Directors can have a sense of terror when you suggest something like that.”

Many days on the set of “Prisoners,” Gyllenhaal, Villeneuve and the film’s decorated cinematographer Roger Deakins would huddle and plan shots that weren’t in the script. “It was just the three of us, calling a play. You go deep, you cut over the middle,” Gyllenhaal said.

Besides his turn to moodier fare, he has been branching out in other ways. After living in Los Angeles for most of his life, he moved to New York a few years ago. It was a welcome change of scenery for someone who was born into Hollywood. His father, Stephen Gyllenhaal, is a prolific director, and his mother, Naomi Foner, has written scripts for numerous movies. His sister, the actress Maggie Gyllenhaal, is married to Peter Sarsgaard.

“I heard about the movie business before I even knew what it was,” said the actor. “So I surround myself now with people who are like, ‘Can we not talk about movies for an hour?’” That is true — to an extent. Gyllenhaal, who is not married, has been romantically linked to a number of high-profile entertainers over the years, including Reese Witherspoon, Kirsten Dunst and Taylor Swift.

He’ll soon be returning to L.A. for a few months to shoot “Nightcrawler.” At Toronto, the actor was also coming off a hectic Labor Day weekend packing up his New York apartment in preparation for the move. Sipping coffee, he showed that, for all of the machismo on screen, he still carried a sentimental side.

“If it was a choice given to me, I would have been at Rosh Hashanah,” he said. “I would have been eating apple and honey with my nieces.”

Related Stories