The Seagulls have been praised by observers such as Pep Guardiola for their attractive football this season but are still in danger of relegation as the season reaches its closing stages and Potter believes only survival will mean success.
He told Sky Sports: "You always have to think about things from your own perspective, you should not necessarily seek validation from outside, just because it is somebody's opinion does not necessarily mean it is the truth, it is just an opinion.
"And that is the same if it is positive or negative as there is a lot of negative opinion out there as well. But obviously from a weight of opinion and who you are going to listen to, Pep Guardiola in terms of understanding football, there is clearly a weight there as opposed to someone who has no experience or no knowledge of football and has an opinion."
Brighton have largely battled against relegation since being promoted to the top-flight in 2017 but Potter feels that the club could have an "exciting future".
He said: "We know it is a tough league, we know it is the Premier League, we know we have had to fight as a club to get to this point, so we are still fighting to try and take the next step.
"We have made progress and I think you can see an exciting future, but we have to also understand the reality of where are at - and that is fighting for points in every single game, starting against Everton."