These should have been just comments, but they ended up too long, so here I go again, partly repeating what others had already said...
That's some very concerning fading you got there. Just in case smudging somehow contributes to it as suggested by other people, I wanted to point out that for your chosen methodology — finishing up detailed work on parts before moving on to the next — it would be possible to organize your progress in a from-top-towards-bottom, and — if you are right-handed — from-left-towards-right directions. That could mostly eliminate the situations where already finished work ends up under your working hand.
Also: never allow sunlight to directly hit your artwork. Not during creating it, and not during exhibition, when hanging on a wall somewhere. (Long-term exhibition should happen only under the protection of UV-filtering glass, or similar varnish.)
Now you could test the lightfastness (or the lack thereof) of your pencils: create two, simple, identical test pieces: this should be just some plain, colored stripes applied thickly with the color pencils you use most often, and which you want to test. To keep it economic on the materials used, it doesn't even need to be a big area.
Keep one of the test pieces in a protective dark environment, like a desk drawer or something. The other one, tape it to your window, facing outwards, where the sun can shine on it as much as it wants.
After every week of passing time, you could bring those two pieces together for a quick comparison, to see how fast the sun-exposed one fades.
If you find the rate of change on the sun-exposed one shockingly fast, you could buy a few color pencils from another brand (look for indicators/markings of lightfastness up front), and — before investing more work into actually drawing with them — just repeat the fade comparison test with them too.
(Consider that the fading test will present different results in summer- versus wintertime, due to the difference in the intensity of the sunlight and in the amount of sunny hours.)
In the end, pick the pencil brand that performs best in the fading-in-the-sun test.