In the original author's case, the cause turned out to be what was intended by the manufacturer to be a wobble reducer. The tool's collet sleeve contained a stepped area to center the end of the bit shaft, and the shaft was catching on that, forcing it to an angle (the accepted self-answer). Many rotary tools don't have that stepped area. There is another common reason for the bit to wobble as described in the question.
Rotary tools spin at very high speed. The slightest imbalance acts like a vibrator (vibrator mechanisms are typically a motor with an unbalanced spinning weight). That imbalance can be from slight bending of the bit from excessive pressure, a minor manufacturing imperfection, uneven wear on the bit, crud stuck to the bit, etc. Here are some things that will minimize or eliminate the wobble.
Start with the tool and bit individually:
- Clean or blow out the collet sleeve, collet, nut, and threads. Debris on the contacting surfaces can prevent the bit from being held straight. Debris in the collet slots can prevent the collet from centering the bit. Debris in general can unbalance the spinning assembly.
- Inspect the bit. Clean off any debris stuck to it. Look for obvious signs of uneven wear.
- Insert the bit and run the tool at a very low speed (a few revolutions per second). Watch the head of the bit. If the bit isn't in straight, or is bent, or there is uneven wear, you can often see the head wobble from side to side at low speed.
Tool and bit in combination:
Make sure the bit is inserted as far as possible into the collet rather than trying to extend the bit for extra reach (but don't insert it past the uniform portion of the shaft; the shaft often tapers where the head is attached, and the collet won't properly grab that area). The farther out the head is from the collet, the more leverage the vibration will have.
Run it at normal speed and see if it wobbles. If it does after the previous steps, it's likely because it is unbalanced. There's a process to compensate for minor imbalances (won't help with something like a bent or damaged bit).
Use a marker like an ultra-fine Sharpie to put a mark on the visible portion of the bit shaft in line with one of the slots in the collet (you can also extend the alignment mark onto the collet). Loosen the collet nut, hold the collet, and rotate the bit in the collet. Then hold the collet so it doesn't rotate in the sleeve and tighten the nut. As a first pass, rotate the bit 180 degrees. Run the tool at normal speed and see if the wobble is better or worse. Repeat, trying 90 degrees in each direction from the original orientation. Adjust the rotation of the bit until you find the minimum wobble.
If this entire procedure doesn't eliminate the wobble, check another new or known-good bit. If you also can't eliminate the wobble on that one, it indicates the problem is with the tool. It could have worn bearings, it could be damaged, or it could be just a cheap tool that lacks precision. You may still be able to use it for awhile at a slower speed, but once it starts to wear out, the problem will accelerate. If, on the other hand, the new bit doesn't wobble, it means the old bit is bad and its problem isn't correctable.