"Our colleague, Matt Zames, Chief Operating Officer, recently informed me of his decision to leave our firm following 13 years of outstanding service," Dimon said in the memo.
Zames was considered by many to be a potential successor to Jamie Dimon when the CEO and Chairman eventually decided to step down. The Wall Street Journal in 2014 called him a "potential favorite" for the position because his work in helping Dimon navigate the financial crisis.
Dimon said in the memo Zames "was an integral part of the team that helped manage the firm successfully through the financial crisis."
"It's very difficult for JPMorgan to hold onto anyone because Jamie Dimon has no intention of leaving," said Dick Bove of Rafferty Capital Markets, noting he didn't think Zames' departure indicated problems within the Wall Street giant.
"It is not that unusual for this to happen at JPMorgan," he said. The bank "has become the incubator for presidents and CEOs of other financial companies. That makes it seem very likely that Matt Zames will show up as the CEO of some other company really soon."
Sources familiar with the situation told CNBC that Zames wants to run his own company but is unlikely to end up at a rival firm. Zames has many offers but hasn't chosen a new firm yet, the sources said.
The memo did not make it immediately clear that there would be a replacement for Zames, instead laying out a new organizational structure dividing up the COO's roles.
JPMorgan did not immediately return a CNBC call for comment.
"There's a need on the part of the company to move its executives around in order to give them new challenges," Bove said. The long-time bank analyst said Wednesday it's time to break up the large Wall Street institutions in order for their stocks to do well.
The memo said Zames "will be here over the coming weeks to help ensure a smooth transition of his responsibilities." It does not mention any plans beyond that for Zames.
— CNBC's Wilfred Frost and John Melloy contributed to this report.