It's relatively safe if you know that you and your neighbours don't use insecticides. Washing them won't help if they contain harmful chemicals.
So, if it's safe or not: You never know where the worm might have crawled through. If they're in a healthy ground with no chemicals, it's ok.