His remarks come as he and fellow judge Prue Leith sparked controversy last night in the first episode of the new series by banishing Dan Chambers.
Paul, 53, said: “When it’s on, everyone is talking about what happened and invariably shouting at Prue and I for getting rid of the wrong person.”
He added: “I’m waiting for the hate to start hitting me on Twitter .
“They won’t go for Prue, they’ll go for me. Whenever somebody’s favourite leaves there’s always a backlash.”
Sandi Toksvig revealed she and fellow host Noel Fielding are also unhappy with the decision at times.
She said: “When they decide who’s going to leave, Noel and I are sometimes going ‘No, no, not them!’, because we have the most banter with that person… [It] is nothing to do with baking.”
Dan’s fruit cake was raw in the middle, but many viewers thought Jamie Finn should have been booted off instead.
His angel slices were woeful. They had only two out of three layers and were “rubbery”, according to Paul.
Jamie, 20, remained in the competition despite coming last in the technical challenge and then failing to remember a vital ingredient for his showstopper.
While making his dog-themed creation, the gap year student, from Surrey, cried: “Oh my goodness, I’ve forgotten the eggs. I’ll have to start again. No wonder it’s like mixing concrete.” The final result was not impressive. Prue said: “One mouthful is nice… but a whole slice would be overwhelmingly sweet.”
Paul said: “There’s not a lot of finesse. Your cake’s overbaked.” Dan’s effort was praised as “clever”.
After all three challenges, Jamie admitted he thought he would be sent packing. He said: “Wow, I really messed up. You don’t really know until you’re under pressure what it’s going to be like.”
He then ducked away from the camera, saying: “I don’t want to get filmed with tears in my eyes. It’s so cringe isn’t it?” Jamie, who has an identical twin brother, had been keeping his kitchen skills secret from pals.
He said previously: “All my mates don’t even know that I bake so they will get a shock to see me in an apron.” Dan, a support worker for people with learning disabilities, took his early exit from the Channel 4 show with good grace, having admitted his fruit cake was “a mess”.
Paul told him: “I think your flavours are good but your baking is way out.”
Dan, 32, from Rotherham, South Yorks, said last night: “Well, it wasn’t a piece of cake at all… I had a bad week, but I will definitely take away the friendships.
“I am not going to miss the baking under pressure, I’m just going to miss everyone in that tent. I felt it was the right decision that I should go.”