Three weeks on from suffering a 2-0 home loss to the big-spending Ligue 1 champions, the visitors upset the odds to seal progress from this Champions League last-16 clash in remarkable fashion.
Romelu Lukaku scored either side of Juan Bernat's effort in a frantic first half that made way for an edgy second period at the Parc des Princes, where the video assistant referee's late intervention proved the catalyst to a remarkable win.
Marcus Rashford slammed home a stoppage-time spot-kick after referee Damir Skomina adjudged Presnel Kimpembe to have handled as United ran out 3-1 victors to see them through to the quarter-finals on away goals after it ended 3-3 on aggregate.
"I've got some fantastic memories," the 1999 treble hero said. "Eh, football - that's what it gives you. It's special when you're a coach or a manager doing it with your staff and players.
"The dressing room in there was absolutely unbelievable, just to see the whole club together with the owners, ex-managers, ex-players. Great atmosphere."
Sir Alex Ferguson, Eric Cantona and co-owner Avram Glazer were among those to head to the dressing room to celebrate with the United players, who were heard chanting Solskjaer's name after surely sealing the permanent manager's position.
"I keep doing this job as best as I can every single day and let's see where it takes us," he said. "It's been a fantastic time with the players and the staff I've got working with me. I'm going to enjoy this job as long as I've got it.
"If it's another two or three months, however long it is, I'm going to enjoy it, I'm going to smile."