The Tartan Tornado, 34, beat Jose Ramirez, 33, in 2021 and unified the WBO, WBA, WBC and IBF super-lightweight belts, with Bud, 38, holding the WBO championship in the division above.
Now, Taylor has admitted he wishes he moved up to the welterweight divison to face Crawford instead of honoring his promise to give Jack Catterall, 32, a chance to take his titles.
He told Sky Sports: “That's the only regret I've got from my career. I was awarded the 'super' title with the WBO after I beat Ramirez and it [allowed him] to go up to welterweight to get that fight.
“I shook [Catterall’s] hand. There were no contracts involved. I could have easily went, ‘See you later, go earn your chance.’
“But because he had stepped aside, because he was mandatory challenger and he allowed me to fight Ramirez first for all the marbles, so I made a deal with the guy, I shook the guy's hand, which everybody seems to forget, and said if I win you get first shot. And that's what happened.
“I stuck to my word. But everybody seems to forget that, especially him.
“That's the only little regret that I've got. But I don't lose any sleep over it.”
Taylor emphasized he would have wanted to fight Crawford - who recently defeated Saul ‘Canelo’ Alvarez, 35, to become the super-middeweight champion.
He explained: “I think he did a better job against Canelo than what [Floyd] Mayweather did. He had a fight with him.
“Mayweather just danced and moved and made him miss. Crawford made him miss and made him pay on multiple occasions and took the fight to him.
“I would say he beats Mayweather. I think Mayweather had the better opponents but you can only beat who's put in front of you and Crawford did that.”
Taylor - who retired in July after sustaining an eye injury - admitted life after boxing “has been tough”.
He explained: “I've had good days and bad days. The bad days have been bad, the good days have been brilliant. I've been so busy chasing something over the last 20 odd years, to all of a sudden have nothing to chase.
“You find yourself in the house by yourself, everyone's working, your friends are working, your wife's working. Your family's working and you find yourself just on your own. I found it quite hard to deal with.”
