TL;DR: The RGB model isn't really a model of how light works. It's a model of how our vision system processes color. Red, green, and blue are like keys that manipulate our vision system; that's what makes them primary colors. The creation and use of RGB colors "hacks" our vision system to create optical illusions of color that we can't distinguish from real color. The answer explores the integral relationship between RGB and our vision system for an appreciation of why RGB are the primary colors. CMY is an extrapolation of RGB.
RGB
Color Vision
Color is how we perceive different light wavelengths. Virtually all of the characteristics of how colors interact are defined by how our vision system processes and perceives color. So let's start with color vision.
Visible light is a continuum of wavelengths. We need an efficient way to process the near-infinite color information that bombards us in real-time. Nature has come up with a solution. Our vision system performs some simplifications and abstractions. It reduces color to a "shorthand" representation, which is what our brain then uses.
How it works and why RGB are primary colors
Our eyes have three kinds of photoreceptors for color vision (cones). They react to short, medium, or long wavelengths of light, and their sensitivities overlap.
source
Notice that the sensitivity curves don't evenly divide the color spectrum, with one type sensitive to blue, one to green, and one to red. In fact, the medium and long wavelength cones almost (but not quite), completely overlap. At the far ends of the spectrum, it is mainly one kind of cone that is stimulated. For most of the spectrum, a given wavelength stimulates either two or three kinds of cones to some extent.
Each type of cone will react to a broad range of colors, but can't tell what color light stimulated it. Any wavelength it is sensitive to will stimulate it, but the closer the wavelength is to the cone's sensitivity peak, the more it will be stimulated.
Your vision system interprets this by comparing the output signals of the three kinds of cones to each other to deduce what color light would have produced that combination of reactions. The result is translated to a color representation, which is what we perceive.
Red, green, and blue are like code keys that trigger perception of a given color. Their ability to trigger a whole gamut of color perceptions that mirror our experience with real color is what makes them primary colors.
Could different primary colors work?
The short answer is no. A longer explanation:
Color (your experience of it), is what your vision system says it is, and it behaves as your vision system is designed. Any other primary color scheme won't produce the same result when processed by your vision system.
One reason is that, in part, externally created RGB output works through optical illusions, tricking your vision system based on how your cones work and how their output is interpreted (more explanation later). To do that, any color system needs to mirror how your vision system works, including the sensitivity profiles of the cones.
Another reason is that RGB is a simplification. Color information that isn't needed to perceive color is discarded (more explanation later). There are holes in the color information that don't usually affect you. A different primary color scheme will have a different set of holes, which won't align with how your vision system processes color.
RGB vs. Vision System
In nature, color is a linear progression of wavelengths, not a color wheel. Red, green, and blue are simply three colors that can be combined independent of the color spectrum, so the linear nature of the spectrum becomes irrelevant. That allows treating them as a color wheel, and representing colors that don't exist as a wavelength in the visible spectrum.
For example, blue light is at one end of the visible sprectrum, and red light is at the other. There is no wavelength for magenta. But magenta can be represented with the RGB information.
Our vision system normally provides good color perception if it is interpretting the color of real objects illuminated with broad-spectrum light. But if we artificially create colors or illumination using RGB, some strange things can happen.
RGB electronic displays don't really reproduce the colors reflected off the original, illuminated object. They produce optical illusions that your vision system interprets as the intended color. Your vision system can't differentiate whether it is seeing normal light or RGB. That allows coding of "misinformation" to trick your vision system. In this case, we're using it as a benefit. (I'll cover some examples below.)
If you illuminate with RGB light, it can mess up colors and illustrate the holes in the color information. Although the light can look white, the wavelengths needed to reflect the color of objects may be missing or limited, so objects can look miscolored or dark. That's why the color rendering index was created for light bulbs, as @ChrisH mentioned in a comment. This video linked by @eyeballfrog in a comment does a great job illustrating the difference between true white light and RGB white, and the effects.
Examples of how external RGB output can manipulate your brain
Your brain deals with an abstraction of reality. If necessary, it extrapolates what a color combination should look like, even if you have never seen that combination in real life. For example, yellow and blue light combine to produce white, so you can never see the (additive) color yellow-blue in real life. But if you view yellow light in one eye and blue light in the other, you can trick your brain into inventing the color yellow-blue (which looks like a combination of what yellow and blue each look like, not green).
So how can you create white light by combining only red, green, and blue light? You don't. It's an optical illusion. If you provide your eyes with red, green, and blue light, your vision system reacts with the same response as to white light, so your brain interprets it as white.
How can mixing red light and green light produce yellow light? It doesn't, it's another optical illusion. The medium and long wavelength cones are sensitive to ranges of color that straddle the yellow wavelength. Yellow light triggers both kinds of cones, and that ratio of activity is interpreted as having been caused by yellow light. But the raw signal that is being interpretted is that both the medium and long wavelength cones have been stimulated. Instead of yellow light, those two cones can be similarly stimulated by the right combination of red light plus green light. The cones don't know what part of the light spectrum did it, and your vision system interprets it as yellow.
What makes RGB the "cone code"?
I'll circle back to elaborate on what makes RGB the cone code.
The vision system deciphers color via how the cones respond to it. It is able to discern and differentiate every color in the visible spectrum because every color stimulates a unique pattern of cone responses. It also isn't a mix and match, where the response patterns are unique but"random", and the vision system matches them to a table of colors. As you move across the spectrum, the cone output patterns follow smooth, complex curves that the vision system is designed to interpret.
Colors that are close to each other on the spectrum produce output patterns that are similar, and the perceived colors look similar. The spectrum looks like a rainbow because as you move across the wavelengths, the color we perceive follows the pattern, with one color blending smoothly into the next to form a continuum of hues.
The vision system cares only about the net output of each cone, it doesn't care how the cones got stimulated to produce that output. Intense light of a wavelength near the tail of the sensitivity distribution can stimulate the cone to the same output as weak light of a wavelength near it's peak sensitivity.
This was adequate during evolutionary times because everything was illuminated by broad-spectrum light (the sun, fire, etc.). The vision system could assume that the cones were all responding to the same distribution of light wavelengths. It only needed to compare the cones to each other to interpret the output.
With modern technology, we can fool the vision system to perceive colors by artificially stimulating the cones with the right wavelengths and intensities on other parts of their sensitivity curves to produce the same output pattern.
The problem is that for most of the spectrum, two or three kinds of cones will be stimulated by whatever wavelength you use. For example, when using a combination that includes reddish light and greenish light, the reddish light will stimulate both the M and L cones to some extent, and the greenish light will also stimulate both the M and L cones to some extent.
The output of each cone reflects its total stimulation. It becomes a puzzle to pick the right combination of wavelengths and intensities so the net stimulation of each cone matches its response for the intended color.
For most of the spectrum, you could find multiple combinations of wavelength and intensity to evoke a particular color. But what if you want to evoke all possible colors using the smallest number of triggering wavelengths, varying only their intensity (i.e., primary colors)? One requirement is that you would need to use a set of wavelengths that allows each type of cone to have the strongest response on at least one.
Looking at the sensitivity curves, that would require at least a wavelength in the blue range for the S cone, a wavelength in the greenish range for the M cone, and a wavelength in the reddish range for the L cone.
But that's not sufficient, there are requirements for the response patterns of each cone compared to the others. Varying only the intensity of each fixed, component wavelength needs to produce a cumulative response matching not just a single color, but all of the patterns for the spectrum. The results also need to be a continuous progression of color as you increment the intensity of a component wavelength.
That's a lot of complex requirements and constraints. But there happen to be "Goldilocks" spots in those three color ranges that allow you to use just three wavelengths to simulate most of the spectrum.
Because each selected wavelength needs to influence multiple cones for this scheme to work, the selected wavelengths can't be at the extremes of the visible spectrum, where a single type of cone is responsive. The bluest blue this can evoke is the selected blueish wavelength at 100% intensity, and the reddist red it can evoke is the selected reddish wavelength at 100% intensity; this trick can't evoke colors farther out on the spectrum. Thus, the gamut of RGB primary colors can't include the entire visible spectrum.
CMY
Since RGB models how our brains process additive color, CMY (the color space), is an extrapolation of it for subtractive color. It just works in reverse using the RGB primary colors (note that CMYK is a different conversion).
In this chart, "nothing" is the absence of color being contributed. On an RGB device, "nothing" would appear as the background color of the device, which might not be pure black. On a CMY application, "nothing" would be the background color of the media, which might not be pure white.
Theoretically, every color in one model can be expressed in the other; one is just the inverse of the other. In real life, colors created from CMY primary colors are often muddy, not like the equivalent RGB color that the model suggests. The model isn't wrong.
The problem is with how the colors are implemented. There are issues like the colors being created by mixing pigments, or colorants that aren't a pure color, so you don't get pure subtractive color. When CMY colors are created, the conversion is much more complex than the chart above describes because of the need to compensate for real life considerations like available colorants and how they behave. But that's a subject for another question.