Poyet, who was appointed in October, saw his managerial career in the Premier League start brightly as Sunderland posted home wins over Newcastle United and Manchester City.
But the former Brighton and Hove Albion boss has seen his side return to the foot of the table after a four-game winless run.
Sunderland are five points adrift of their nearest rivals and Poyet is unhappy with how things are progressing.
"Is it enjoyable? No, I can tell you it's not, but I don't think everybody gets a job and tries to enjoy the job," the 46-year-old said.
"That's a dream job. That's what I have said to the players: You want to play football? Yes. You want to earn money? Yes.
"Have you got a lovely place to train? Yes. Then you are lucky because you are enjoying your life.
"But at the moment, no, I am not really enjoying my life. But it's the way it is. I am trying to make sure I can enjoy it in a few weeks' time, that would be nice."
Poyet will hope his side return to winning ways when they visit West Ham on Saturday.
