The British fighters have been tipped to fight this summer - eight years after Joshua won the first bout between the pair - but Whyte thinks that his rival is worried about putting himself at risk again.
The fighter told The Sun: "Inactivity is killing the sport and it needs a shake-up. I am waiting for my next fight — and was meant to fight Joshua after Franklin. Joshua and his team told everybody, ‘The winner of Whyte and Franklin fights AJ’ — and then they boxed the loser.
"Joshua’s team are doing interviews saying he beats me in four rounds. But he can’t decide if I’m his dream opponent or his worst nightmare. I want the fight. I know that DAZN want the fight, too, because it would be an absolutely huge event. The only problem we all have is Joshua. He has confidence issues and is afraid to take the fight."
Whyte argued that Joshua is concerned that a defeat would put his proposed fight with Deontay Wilder later this year at risk.
He said: "He doesn’t want to risk his retirement payday in Saudi Arabia against Wilder — who is also past his sell-by date.
"Joshua isn’t the golden goose anymore, the sport doesn’t revolve around him. Him and his people don’t seem to realise it but duck me again and lose to Wilder in December and he and his team — that know so much but do so little — are finished."