Summers called President Obama on Sunday to notify him, according to Dow Jones Newswires, which first reported the news.
"I have reluctantly concluded that any possible confirmation process for me would be acrimonious and would not serve the interest of the Federal Reserve , the administration or, ultimately, the interests of the nation's ongoing economic recovery," Mr. Summers is quoted as saying in a letter to the president following the call.
President Obama said he had spoken with Summers earlier Sunday and accepted his resignation.
"Larry was a critical member of my team as we faced down the worst economic crisis since the Great Depression, and it was in no small part because of his expertise, wisdom, and leadership that we wrestled the economy back to growth and made the kind of progress we are seeing today," the president said in a statement. "I look forward to continuing to seek his guidance and counsel in the future."
Summers and Fed Vice Chair Janet Yellen were thought to be the top two contenders for the job and it was widely speculated that Summers was Obama's top choice.
Bernanke's term is up in January, so Obama is expected to make his pick soon.
Todd Schoenberger, managing partner at LandColt Capital, called Summers' exit "unfortunate."
"Larry's genius on monetary policy is already known; but his ability to go against consensus and choose a more challenging path to accomplish what he believes is in the best interest of the American economy is legendary. It is for this reason, a Fed Chairman Summers would have been in the best long-term interest for the country's monetary policy decisions, and continuing economic recovery," Schoenberger said.
Still, he said, "Wall Street will praise the idea that a decision on Yellen is imminent and that she is most likely to continue to stay with the current Bernanke thesis."
"Larry Summers will be missed," said Michael Barr, a law professor at the University of Michigan and a CNBC contributor. "Dovish candidates like Janet Yellen willcontinue the slides in the dollar and bond. Less monetary accommodation will beseen as positive. Tomorrow's markets will be jumpy."
Dow and S&P futures were up 1 percent on Sunday evening following the news. ( Click here to check the latest futures action .)
(Read more: Get ready for a Summers rally in the stock market )
Pimco's Tony Crescenzi said the news should help calm the bond market.
"[I]nvestors were braced for the likelihood of a Summers nomination, in particular for the notion that he might raise interest rates sooner than perhaps Janet Yellen might if she were selected, so this news should suppress forward rates and steepen the yield curve ahead of Wednesday's Federal Open Market Committee meeting," Crescenzi said.
"When the Fed on Wednesday delivers for the first time its projection of where it expects the federal-funds rate to be at the end of 2016, markets will be less skeptical about the projection and therefore the future path of the Fed's policy rate," he said.
The race for the next Fed chief has triggered a very unusual public debate about who should get the position.
In recent weeks, several key Democrats on the Senate Banking Committee have come out publicly opposing a possible Summers nomination, including Jon Tester on Friday.
And several hundred economist signed a letter to President Obama urging him to pick Yellen. Former Fed official Alan Blinder was among them. On CNBC last week, Blinder said it wasn't so much an anti-Summers campaign as it was a pro-Yellen campaign.
"[I]t's very unfortunate. It's led to a situation where people like myself and two or three hundred other economists...that list keeps growing by the way every minute ... they felt compelled to come out and take sides because it's like a campaign," Blinder said.
Blinder said Yellen's experience and judgment are the primary reasons he and other economists are pushing for her to succeed Bernanke.
"She was one of the early members of the Federal Open Market Committee to see that this [housing crisis] was not just a financial catastrophe. This was going to really damage the real economy and the Fed ought to do something about it," Blinder said.
Crescenzi said Yellen also has a similar style to Bernanke.
"Bernanke forms his views around the consensus, and Yellen likely would, too," Crescenzi said on CNBC a few months ago. "It's felt that Larry Summers, with his gravitas, might put the policy-making around his views rather than around the consensus. Some would dispute that idea, but that's a sense that some have."
One of the other criticisms of Summers had been that he was too cozy with Wall Street. But a CNBC survey this summer showed that even Wall Street thought Yellen should get the job.
CNBC polled 40 economists, traders and strategists polled and their top criteria for the job was monetary-policy expertise.
A whopping 70 percent said Obama should pick Yellen, while just 25 percent said Summers.
If Yellen is nominated and confirmed, she would be the first woman to lead the Federal Reserve.
Below are some of the reactions on Twitter to news of Summers taking himself out of the running:
[To view the Storify on a mobile device, click here .]