If you don't want the artwork destroyed, then yes. Any contact with unfixed pastel (oil or chalk) will smudge it. This includes contact through the plastic bags you're dropping the art into.
An artist-grade, archival-quality spray fixative should cause minimal color changes to your drawing, and proper handling will prevent any health issues.
In selecting your fixative, you'll want to evaluate whether it's designed for use on (oil) pastels, whether you wish to use it only at the final layer or throughout the work ("workable" vs "final" fixatives), and whether you prefer a gloss or matte finish on the final artwork. You'll also want to test your choice of fixative on a sample piece using your same oil pastels before spraying it on a finished piece, to ensure that there are no unexpected interactions between the fixative and your specific brand of pastels. (If you change brands, you'll likely want to repeat this test.)
As far as safety goes, you can check for an ACMI label of either "AP" or "CL:"
Products bearing the AP seal of the Art & Creative Materials Institute, Inc. (ACMI) are certified non-toxic. A medical expert evaluates each product and its ingredients. A product can be certified non-toxic only if it contains no materials in sufficient quantities to be toxic or injurious to humans, or to cause acute or chronic health problems. AP certification is reviewed by ACMI's Toxicological Advisory Board. These products are certified by ACMI to be labeled in accordance with the chronic hazard labeling standard, ASTM D-4236 and federal law P.L. 100-695.
Products bearing the CL seal of the Art & Creative Materials Institute ("Caution Label") contain ingredients that are toxic or hazardous, but they can be used safely with appropriate caution. Materials that bear the CL seal should be used only by those persons who are able to read, understand, and follow suggested safety precautions for handling those materials. The Caution Label signifies that although the product contains a toxic element, it can be handled safely if the directions on the container or packaging are followed. Many such art products cannot be made non-hazardous, but are necessary for certain creative activities.
(source and more details: Dick Blick Health and Safety Info
The primary health hazards you'll encounter with an aerosol fixative will be particulate and organic vapors. If you plan to use sealant frequently, you'll want to get a half-face respirator mask rated for particulate (N95+ in US standards) and organic vapor protection. You'll need to ensure the fit is correct, keep the mask itself clean, and replace the filter cartridges regularly. You'll also need to fit-test it every time you wear it (done by putting the mask on, covering the filters and inhaling to ensure no air comes in, then covering the exhalation valve and exhaling to ensure no air escapes), to be sure you've got it properly sealed.
While the specific hobby focus is different, this page has some excellent overall information on how to select and maintain a half-face respirator, with more detail than I've gone into here: Respiratory Protection (Note: this page duplicates a separate one blocked behind a registration wall on a private forum.)
Two key hilights from the page are:
Regarding particulate protection:
For sanding applications (including sanding of resin, apoxie and other sculpting materials, and wood), you need a respirator that provides at least N95 level protection. It should say N95 somewhere on the mask itself. If it doesn't say, it is not good enough. There is a scale to protection levels and anything above N95 is also acceptable (of course).
The levels are:
N95, P95, N100, P100
Regarding organic vapor protection:
The process of spraying releases aerosols (and this is true whether or not you are using an aerosol spray can product. Airbrushing paint creates aerosols too!) for which an N95 filter is not effective.
At this point, you will need to move into a mask that protects you against Organic Vapours AND has an N95 filter. This will mean moving into a half-face respirator.
Your respirator is no longer disposable and can be used again and again without ill effect; the only thing that will need replacing is your filter and (if applicable) your prefilter. Some respirator brands have an N95 filter built right into the Organic Vapour cartridge, but I recommend looking for one that does not, for ease of replacement. Cartridges and filters have different life spans and it is more economical to replace only the part that requires replacing (more on this below!).
If you want your work to stay in good condition, not smudge, and not transfer to the skin or clothes of anyone who brushes up against it, you'll need to spray with a sealant, but in no way does that mean you are going to "ruin" either your artwork or your health.