I love this scene because Dean was trying so hard to still seem pissed off at Cas, but he finally cracked and you could hear and see how worried he had actually been.
And I find it interesting how Dean says "WE were worried" because honestly, Sam didn't seem that worried. Curious. But not worried. He was actually trying to comfort Dean and was saying things like "It's Cas. He disappears all the time. I'm sure he's fine." Meanwhile Dean was the one calling Cas everyday and leaving him voice-mails and wringing his hands. "We were worried" no baby, YOU were worried.