As long as you are both happy with it that's all that matters.
Personally I couldn't do it, part of the enjoyment for me is seeing the Mr enjoy himself, for him not to be there just wouldn't be as hot for me.
I'd possibly contemplate meeting another woman alone but no men or couples.
Mrs