Eddie Jones' side lost their third straight game last weekend as Ireland recorded a 24-15 victory at Twickenham to wrap up a Grand Slam success.
England were the favorites to win the event and they now face a tough few months before they travel to South Africa this summer.
South Africa are expected to improve under new coach Rassie Erasmus and Robshaw admits his team need to improve before they take on the Springboks.
He told reporters: "Confidence is such a massive thing in sport.
"I wouldn't say we lack confidence but we probably don't have that swagger we had two years ago. We need to find a way to get that back.
"We're going to South Africa in the summer, which is going to be a tough tour so we need to bounce back and be stronger. We need to learn how to win again."