Yup, that's right, good ole' Lipton (or whatever other type of black tea you happen to have on hand). Diluted coffee can also work, but will smell like, well, coffee-stained paper.
Note that this shouldn't be used on inkjet printouts, as inkjet ink isn't waterproof. So I would suggest printing out the master map (in black and white) and heading out to your local office supply store to make copies.
After applying the tea, you can hang up the maps to dry. The paper will get crinkly, but that's actually a good thing for this application. If they get too crinkly, apply a cool iron (barely above its lowest setting) to smooth them back out.
As a final step, go out to your garage (or a similar well-ventilated but sheltered spot) and use a candle or a lighter to burn away the edges of the maps. Keep a bucket of water handy, just in case. Don't go overboard - just do the corners and a little bit of a few edges. After blowing out the flames, break off most of the black stuff, leaving only a browned, ragged edge.
Note that the results will not be archival: tea is acidic. But for a prop for a birthday party, that shouldn't be an issue.