With worms, persistence is key, and sometimes it just won't work. My axolotl grew up on live black worms which he ate easily from when he was at the LFS (at 2.5 inches, all legs), and he stayed on them until 5 inches. Around that time I tried Hikari Carnivore Sinking pellets which he took easily if I held them in front of him or dropped them right on his face.
I tried swapping him to red wrigglers, but he hated them. He did the same thing yours are doing. Sometimes I would think he ate the 2 worms I fed him only to find out he threw them up overnight. I finally had to give up on them because he was losing weight, even though he was eating them he wasn't eating enough. I then tried Canadian night crawlers and he ate the first huge worm after some struggle, but he ate it and seemed happy. After that he took no more worms even though I tried for about half a week without feeding him anything else, he's now back to the pellets.
My axolotl is now almost 8 inches and quite chubby so he's doing quite well. He seems very healthy and exhibits the odd yet normal axolotl behaviors.
I would stay don't feed your axolotl but the red wrigglers and keep trying through out the day. I don't like the idea of not feeding them anything else, didn't do anything other than starve my axolotl as he refused anyways, so I would just feed him a bit of the old food the next day if he doesn't take it, then try a bit of worm at the end of the day.
You could try dropping them so the worm or any food sinks towards the tip of their mouth and they'll feel it and snatch it up. You could also try cutting up the larger worms to a smaller size to see if your axolotl prefers that. And if you're trying red wrigglers, some have mentioned blanching them quickly because it will kill the worm and cause it to release the foul tasting secretion. Then you can rinse him off in cool water and hopefully it would be more palatable for your axolotl.
As for how much they eat, it depends on the axolotl. My axolotl would eat 3 times a day if I fed him, but he only really needs 1 a day at most. If I can, I just feed him a smaller amount of pellets in the morning and then the usual amount during his usual night time feeding. The best way to know if they need more is by comparing the body size to head size, they need to be roughly the same width. Otherwise, your axolotls will tell you when they're hungry.