It's hard to diagnose without seeing the bit you used, but here are my ideas:
The line in the center of your image has only one chipped edge on one side. That implies that the problem either arises at the side where the tool starts touching the glass or leaves the glass.
If you turn the glass (or the tool) so that the rotation of the bit is ...
Most common would be a rotary bit.
This one would be specifically for glass carving:
The bottom row would be the fine detail tips, would be beneficial to wanting to carve out small detail.
Brand wise, you really can't go wrong with a rotary bit. I have been in the tool industry for about five years. Yes, you can get really high quality rotary ...
Sand down the edges with 150-grit, 220-grit, 320-grit, then finally 400-grit sandpaper to make your glass even smoother. Then, use 1000-grit and 2000-grit sandpaper to polish the edge to perfection. Wipe the edge of the glass with a clean damp cloth once you've finished sanding to wipe away any leftover grit or dust.