Financials, which finished as the third worst performing group on Thursday, will have trouble delivering the goods to investors, especially if 10-year treasury yields fall below 2.5 percent, he said.
"I think the banks continue to sell off. There's a lot of good news factored in these banks, mainly further growth down the road. The banks need a higher yield curve," said Miller on CNBC's " Fast Money ." "I think rates are going to stay lower than people think."
Miller expects Wells Fargo to weather the headwinds.
"What we like are the 'Wells Fargos' of the world. They keep hitting singles and doubles, and they keep on getting that earnings growth up there," added Miller. "On a relative basis, Wells is where you want to be in this group."
The bank reports its second-quarter results before the bell Friday. The stock finished down less than 1 percent at $51.81 per share.
-By CNBC's Stephanie Landsman