This happens easily. It's actually almost inevitable. And shouldn't be resisted.
It's just that it's not just about you two. Remember the people that are left out. They aren't 'losers' because they got left out. Actually they may well not envy you atall. Its possible they hope you'll get it together pretty quickly and move out.