Add the fact that manager Slaven Bilic said he's happy at West Ham recently and supporters might well feel a little less anxious about their hero's future than they would have otherwise.
But should they worry? The truth is that Payet's stock is rising pretty much every time he laces up his football boots, and the more it grows the harder it will be to keep him.
He had barely finished celebrating his last-minute wonder goal for France on Friday night when speculation linking him with Real Madrid resurfaced.
If rumours are to be believed, the European champions are readying a £40 million move for Payet (Le10 Sport) and, at 29 years of age, how many more chances is he going to get to join a club of that stature?
In an ideal sport, the Frenchman would tell Real where to go and acknowledge the fact that so much of his resurgence is down to West Ham and Bilic.
But it's a far from ideal sport and qualities such as loyalty belong to a forgotten era.
He owes West Ham another season - he absolutely does - but it is fanciful to suggest he would turn down Chelsea or Real Madrid to honour an intangible, moral commitment at the expense of a once-in-a-lifetime move.
It is a shame that such a talent had to wait so long to hit these sorts of heights, but he may feel it's a bigger shame that he has not won a single major title in football.
He has five more years on his contract, but it's hard to keep a player who does not want to be there. History is littered with examples, and that's when this love affair could turn ugly.
Given the mutual affection between Payet and the West Ham fans, that would be the biggest shame of all.