Boris Johnson faces pressure within the Tory party to re-establish his grip following a chaotic speech to business leaders and a revolt over social care.

Senior Tory Jeremy Hunt said it had “not been a great month” for the Government, “not just on trivial issues like speeches going wrong but on much more serious issues like parliamentary standards”.

Downing Street insisted the Prime Minister was physically “well” and was “focused on delivering for the public” following questions about his leadership.

But one Tory MP said the Prime Minister was “losing the confidence” of his backbenchers and should quit in the new year.

The Prime Minister’s address to the Confederation of British Industry (CBI) on Monday saw him lose his place in his notes, talk about Peppa Pig and impersonate a car.

The Government then survived a rebellion over its social care reforms, with 19 Tories opposing the plans and dozens more not voting at all in response to the cap on costs being less generous than expected.

It followed a bruising few weeks which have seen Mr Johnson’s judgment being questioned over his handling of the Owen Paterson row on parliamentary standards and Tory criticism of scaled-back plans for rail upgrades in the North.

Former Cabinet minister Mr Hunt, who stood against Mr Johnson for the Tory leadership in 2019, said the CBI speech “wasn’t a great moment and it hasn’t been a great month for the Government”.

In a sign of the questions being asked about Mr Johnson, a senior Downing Street source told the BBC “there is a lot of concern inside the building about the PM…. It’s just not working”.

“Cabinet needs to wake up and demand serious changes otherwise it’ll keep getting worse. If they don’t insist, he just won’t do anything about it,” the source said.

Mr Hunt acknowledged there were “noises off” about Mr Johnson’s leadership within the Tory party, but insisted the criticism was not on the same scale as that faced by David Cameron or Theresa May.

“I’m sure there are things that we can do better,” he said.

“But I was in the Cabinet for nine years from 2010 and frankly there’s never been a time when there weren’t noises off in Westminster, there weren’t backbench MPs with complaints about the way the Government is operating,” Mr Hunt told Times Radio.

On Sky News, he said: “We all have bad moments and yesterday was not a great one for the Prime Minister.”

A Tory backbencher told the PA news agency that support for Mr Johnson was waning, as shown by empty spaces behind him in the Commons at Prime Minister’s Questions last week.

“I think the Prime Minister really needs to consider his position,” the MP said. “He needs to announce in January that he is going.

“He is a lovable clown, the problem is no-one is laughing anymore, are they?”

The Prime Minister’s official spokesman sought to minimise the fallout from Monday’s events.

“The Prime Minister briefly lost his place in a speech,” the spokesman said. “He has given hundreds of speeches. I don’t think it’s unusual for people on rare occasions to lose their place in a speech.”

The spokesman added: “The Prime Minister is very much focused on delivering for the public.”

Asked whether Mr Johnson had a core group of ministers around him to offer advice, the spokesman said: “The Prime Minister has an entire Cabinet to draw on who provide advice, as you would expect.”

The spokesman said he “wants people to be able to speak freely and give their views” around the Cabinet table.

Mr Hunt, the Commons Health Committee chairman, was one of the Tories who abstained over the social care reforms in Parliament.

The Prime Minister narrowly succeeded in getting MPs to back his new policy to cap care costs in England on Monday evening.

The Government won the vote by 26, a major cut to the Prime Minister’s working majority of around 80 MPs, as 19 Conservatives including former Cabinet minister Esther McVey and ex-chief whip Mark Harper rebelled to oppose the plans – while 68 Tories did not vote for them, either because they abstained or could not attend.

Mr Hunt told the BBC: “I was conflicted, I actually ended up abstaining because it is a big disappointment that they changed the way the cap is calculated.”

Not including council support in calculating whether the cap on care costs has been reached means it “won’t protect the assets of as many people as we had hoped for”, he said.

The scale of the revolt could encourage peers to seek to amend the legislation when it reaches the Lords.

Crossbench peer Baroness Finlay of Llandaff said an impact assessment of the reforms – which was not available to MPs – will be “very important”.

She told the BBC’s Today programme: “It may be that we will say to the Commons, ‘can you think again?’, it may be that we come up with constructive amendments to improve what is on the table at the moment because, clearly, there’s a lot of disquiet.”