If you want to make something your kids could color with pencils, crayons, or maybe markers, get tracing paper and draw stained-glass designs on it with a black permanent marker. (If you do the coloring with markers, either wait several days between doing the black parts and coloring them in, or flip the tracing paper around: do the black "leading" on one side of the paper, and the coloring on the other side. Otherwise, no matter how "permanent" you permanent marker, the colored markers can cause it to smudge.) The advantage of this method is that it's pretty cheap, and it will probably end up very similar to the coloring book you have, at least if your coloring book is similar to what I've encountered before. The disadvantage of this method is that it doesn't really look like stained glass.
The other option is to go to the craft store and visit their faux stained glass section. Yes, there is such a thing. :) It involves clear plastic sheets as a base, a faux leading that's basically black dimensional paint in a squeeze bottle (i.e. you pipe it on almost like cake frosting instead of actually painting with it), and clear acrylic paints to go between your leading lines. Again, the process is to paint/pipe the designs with the "leading", wait for that to dry, and then paint the designs. The advantage of this method is that it looks very, very similar to actual stained glass without involving cutting glass and soldering and working with actual lead (Pb). The disadvantage is that it gets pretty expensive. You can reduce the expense somewhat if you can find cheaper sources of clear plastic. For example, see if you can find plain clear plastic plates (like from the party goods store).