Garry Monk's men are seven points clear of the bottom three following a 3-0 win over Norwich City last weekend and can move closer to safety with victory at Hull City on Saturday.
But captain Williams feels that the Welsh club should be aiming higher and has challenged his team-mates to claim maximum points in all of their final six games.
"What we don't want to focus on is getting to 37 or 38 points and then stopping," Williams told The South Wales Evening Post.
"We're trying not to lose again this season, to win as many as we can and to just keep going. It's in our hands.
"We're not saying: 'Let's go for two more wins'.
"Forget that. We've got six games left so why can't we try to win them all? If we can't win them, let's not lose.
"That's the way we're looking at it.
"It's a bit bullish to say that after we've just drawn one game and won one game, but that's the way we feel.
"We've got to set our standards high and get back to where we have been in the last two years.
"We want to finish the season strong and take the momentum into next season."
‘I don’t think Liverpool would look at Ollie Watkins, a striker isn’t a pressing issue for them – it’s Arsenal who need one’ Former Reds star explains why his old club don’t need an out-and-out forward this summer
‘He’ll still be playing at 40 at a good level because he’s in such good shape and looks after himself so well. He does everything to be a top professional’: Ex-Liverpool coach insists Mohamed Salah has plenty more miles in the tank
