Mr Johnson’s party lost the Devon seat of Tiverton and Honiton to the Lib Dems, and Wakefield to Labour. The by-elections took place against a backdrop of public anger about Covid rule-breaking in Downing Street, along with concerns over rising costs and a squeeze on wages.
The Conservatives also suffered a loss in West Yorkshire where Labour secured a 12.7% swing to regain the Wakefield constituency by 4,925 votes.
Speaking to GB News, Mr Nelson claimed that the responsibility of the current loss in the by-elections must be borne by Mr Johnson.
He said: “It is very much Boris Johnson’s responsibility and I think the real problem about these by-elections is that it’s not so much now ‘Boris Johnson was fined for breaches of lockdown’, what we are seeing now is complete loss of trust amongst the voters.
“And where Tories once saw Boris Johnson as a vote winner, he is now looking like a vote loser.
Once he gets at that stage, they are asking themselves ‘what’s the point of keeping him?’”
On Friday, Mr Johnson admitted results from two by-elections were not “brilliant” but vowed to keep going.
Mr Johnson said he would “listen and learn” and focus on “delivering for the people of this country”.
The two defeats prompted Conservative party chair Oliver Dowden to quit saying: “We cannot carry on with business as usual.”
READ MORE: Mum ‘mortified’ after Asda security guard ‘asked her to leave’ store
Other cabinet ministers have rallied round the PM, who is currently in Rwanda attending a meeting of Commonwealth government leaders.
Taking questions at a press conference in Kigali, Mr Johnson said Mr Dowden did a “lot of good work” but added: “I genuinely, genuinely don’t think the way forward is to focus on issues of personality whether they are mine or others.”