I would tackle it like this.  If some are painted and some aren't, and you want to get rid of the paint, start with those.  Put those in a plastic paint storage container with a lid, cover them with paint stripper, and seal the container.  Every so often, stir the contents so all the surfaces get exposed.  You might need to let it sit overnight so all the paint dissolves.  

When there's no more solid paint on the magnets, pour off the paint gunk into something disposable.  Cover the magnets with paint thinner, and stir them around to get all the residual paint gunk thinned out.  Pour that into the disposable container.  If there's still paint gunk on the magnets, repeat the paint thinner.

If you don't have access to an appropriate place to dispose of the chemicals, spread the residue on something and leave it outside to dry.  After the solvents have evaporated, you can dispose of any dry residue in the trash.

There's a possibility that what's getting on your hands is some kind of grease or oil.  Add the unpainted magnets to the container (if you decide not to strip the paint, start here with all of the magnets).  Pour in enough 90+% isopropyl alcohol to just cover them, and stir everything well.

Add a squirt of liquid dishwashing soap, like Dawn, and stir everything well.  Add some hot water (as hot as will come out of the faucet); roughly equal in volume to the alcohol and soap.  Stir the magnets to get all of the surfaces exposed and to agitate them.  Let it soak for awhile with regular agitation.  Then pour off the dirty water and see of the magnets feel clean, with no more residue.  

If not, repeat the soap and hot water.  When they feel clean, fill the container with hot water, agitate to rinse everything, then pour off the water.

Dump the magnets onto an old towel, flatten them into a layer, then wrap the towel over them and roll everything around inside the towel to dry them all off.  Then let the magnets air dry on a dry towel.