The problem with a tiny worm farm is the same as the problem with a tiny aquarium. In either case, any slight error or imbalance in the environment can quickly cause the whole thing to crash. A little too much food, or a little too dry or wet, and the worms are likely to escape - or die trying.
It's OK to start a farm with a small number of worms and build up slowly, but don't try to keep them in a very tiny space. I would guess that a 5-gallon bucket would be the smallest possible container that would work, and even there you'd have to be vigilant and change out the substrate fairly often. It works best if you have two containers, one with drainage holes and another container outside of it to catch the liquid.
My worms don't care much for potato peels. They are much more enthusiastic about fruit, such as banana peel, melon rind, mango peel, etc.
Have fun!