The Ostersunds boss has been linked with the Liberty Stadium job with reports suggesting the Swans have made Potter their first choice.
The 43-year-old is even reported to have held talks with the Premier League side.
Though he has rubbished those rumours as speculations and insists he remains focus on his job in hand in Sweden.
“It’s just speculation,” Potter said. “I said the same yesterday and before the last game. It’s the same thing now.
“We had to focus on today’s game and we did it really well. Now we have to focus on the game on Saturday.
“We’ve had a tough spring because of many reasons but we have the opportunity to finish the spring with 17 points if we win on Saturday. Let’s focus on that.”
Potter has had a successful time in Sweden having taken over at Ostersunds in 2011.
He led the club to the Swedish Cup in 2017 before reaching the last 32 of the Europa League this season where they were knocked out by Arsenal.
The Swansea job has been vacant since the departure of Carlos Carvalhal, who left the club following relegation to the Championship.
The Portuguese spent less than six months at the south Wales side having taken over from Paul Clement in December.
And Ostersunds chairman Daniel Kindberg echoes Potter’s words relating to the role.
He said: ”No one has contacted me about Graham, so it’s just speculation. I have not talked to Huw Jenkins (Swansea’s chairman).
“I see it just as speculation. It’s probably the third, fourth or fifth time there’s been speculation about it, so I sort it under speculation as usual.”