The move mirrors that of Japanese rivals Daiwa, Nomura, and Sumitomo Mitsui.
In a statement, Mizuho said its decision was made in response to the UK's decision to leave the EU and "against the background of continuing the effort to develop our network for servicing customers in Europe and around the world in a sustainable and reliable manner".
The Japanese lender will establish a German subsidiary that will lead securities operations across EU countries.
Earlier this month, Sumitomo Mitsui selected Frankfurt as its new EU base, with reports indicating some of the lender's 1,000 staff in London may need to move to Germany.
In its statement, Mizuho made no reference to whether or not jobs would be moved from London.
Frankfurt Main Finance, a lobby group championing the German city as a major benefactor of Brexit, hailed this evening's announcement.
"We see Mizuho’s decision as another show of trust in the Financial Centre Frankfurt, for which we are most grateful," said Frankfurt Main Finance managing director Hubertus Vath.