Whatever you do, don't kill them. Those things are like hydras, two more will grow in their place!
Really though, unless they're poisonous, make spiders your bros. They'll help keep more pesky insects away. Or, if you still don't like them, just scoop 'em up with a tissue and release them...