Although the tech giant's current boss asked him to "stay on the board" when he handed over the reins, MacFarlane knew when it was the right time to step aside.
Spencer told The Vergecast: "John said, 'Look, you're the right person to take us into the future, and you're going to want to make a series of changes which I may not agree with and I will get in the way.
"'So I'm going to step off the board and I'm going to completely step out of the company. And I'll always be there should you need any advice or just want to use me as a sounding board, but you need to make the changes you think to get us to the next stage.'
"And so it started there. And he was so right, because even a month later we were going through a number of changes which I knew were pushing the organization in an uncomfortable way, and one of those things was around the acceleration of the product road map."
This year, the company is finally taking the plunge with its first Bluetooth speaker, and Spencer admitted it was difficult to pitch the idea to the company as a whole.
He added: "We have the most incredible product team in the world and they have amazing ideas, and what I tried to do was just create a culture and a system where it's okay to talk about these ideas, bring these ideas forward, and look at what's next.
"And culturally, there were some sacred cows around things like Bluetooth, and that was important to get us to the stage we were at at that point, but it was holding us back from the true potential of what I think we have for the future."