The NFL icon - who is a part-owner of the Blues - questioned the former England captain's attitude during his disastrous spell as manager in 2023 in a new documentary and Rooney is "really disappointed" by the remarks.
Speaking on his new BBC podcast The Wayne Rooney Show, he said: "I think Tom came in once, which was the day before a game where the days are a little bit lighter anyway, and I don't think he really understood football that well.
"But what he does understand is, he's a hard worker, we know that.
"Football is not NFL – NFL works for three months a year. Players do need rest as well, so I think he's very unfair, the way he's come out and portrayed that."
Rooney was sacked by Birmingham after 83 days in charge in January 2024, but feels that the club's relegation to League One at the end of that season shows that the problems were ingrained.
The former Manchester United player said: "When I went into Birmingham, they were in a mess really.
"Hence the fact that the players weren't really the players who could take the club forward.
"You had Tony Mowbray, Gary Rowett after me, who also struggled as well."
Rooney stressed that he "massively" respects Brady and praised Birmingham for making an immediate return to the Championship under their latest coach Chris Davies.
He said of the seven-time Super Bowl winner: "Listen, I respect Tom Brady massively.
"He's one of the greatest, if not the greatest, athletes of all time and Birmingham do look like they're getting it right now, which is good and I think what they have done is got the players out that they needed to get out."