The 23-year-old spent last season on loan with Sunderland after being signed by Sam Allardyce - who has since been replaced by David Moyes - and did his bit as the Black Cats won their Premier League relegation battle last term.
He has returned to Spurs this summer, but Tottenham already boast two capable right-backs in Kyle Walker and Kieran Trippier and it remains to be seen what manager Mauricio Pochettino thinks about the USA international.
But former Bolton Wanderers midfielder Holden would like to see his compatriot leave the Lilywhites in favour of joining Sunderland on a permanent basis.
"I think if he wants to develop as a player he'll have to leave Tottenham," he told ESPN. "I think Sunderland's a good fit for him.
"We saw him come on leaps and bounds under Sam Allardyce. He became a much better player and sitting on the bench at Tottenham won't do him any good."
Yedlin himself admitted last month that speculation linking him with a permanent switch to Sunderland was "awesome", as quoted by the Seattle Sounders' official website, but time will tell whether that materialises.
He was not signed by current boss Moyes so it remains to be seen if the Scot would like to retain his services for next season and beyond.