Many people that visit but never return to a small group do so because it feels cliquish. When visitors walk in to your meeting, your top priority is to make them feel welcome. Greet them at the door; give them a hug or a handshake and look them in the eye when they talk.
If you fear your group gets cliquey, you can change your group atmosphere by having a brief introduction period during which each person shares one thing they love about the group. This is a great way for new people to hear good reasons to stay and join the group. Have visitors share how they heard about the group and what they hope to get out of the evening or study.
You can also ask everyone to introduce themselves briefly. It's worth taking time for this kind of thing in the first week, even if your Bible study needs to be shortened, because it creates a welcoming atmosphere.
If you have seekers in your group—people who don't even know if they want to believe in Jesus, let alone be in your group—be sensitive. Talk to them about something other than church, and avoid putting them on the spot (i.e., having them close or open with prayer).