The Welsh region have endured a tough season, currently sitting second bottom of the Pro14's Conference A and saw their Champions Cup aspirations ended by Saturday's 24-7 defeat at Clermont Auvergne.
Tandy took charge in 2012, having played as a back-rower for 12 years with first Neath, and then the Ospreys when regional rugby took off in 2003.
Managing director Andrew Millward confirmed the news on Monday, admitting that recent results had forced the board's hand.
"Steve has been an outstanding servant to Ospreys Rugby - often in challenging circumstances," said Millward. "His dedication to the region and his hard work has been absolute.
"However, we're aware that results have not been good enough this season and that, regrettably, a change is necessary to keep the organization moving forward - consistent with the long-term expectations of our loyal and valued supporters and commercial partners."