RGB (Red Green Blue), is the common system for representing primary colors for emitted light. That's the color system used for computer monitors. The hex codes are a way of specifying the RGB values. Each two-digit pair in the sequence represents a value between 0 and 255 (in hexadecimal, so only two characters are needed for each value), for red, green and blue light. That allows over 16 million possible colors, which is adequately close to what the human eye can distinguish.
CMY (Cyan Magenta Yellow), is a common system for representing primary colors of reflected light. It's typically the color system used for print colors. Additional colors are often added to the pallet because it is difficult to produce intense RGB primary colors by mixing CMY colors. So printers typically add black, and high-end photo printers often add one or more of the RGB primary colors.
To create precise mixed colors, you need to start with precise primary colors. For example, a nice, intense blue may not be an optically "pure" blue primary color, so any colors mixed from that will be off.
With paint, there is also the effect of how concentrated the pigment is in the paint. If you mix a color from two paints that are optically perfect primary colors but have different concentrations of pigment, the resulting color will be off in the direction of the paint containing a higher ratio of pigment. You create the color by mixing controlled amounts of pigment, not controlled amounts of paint.
Companies that sell mixed paint colors use precisely known colorants, in terms of both the color and the concentration of pigment. Their equipment is calibrated based on those standard source colorants.
You're starting with paint whose precise color and pigment concentration are unknown, so there is no basis for mixing colors from those without calibration equipment (other than by eye). You have the further problem that you aren't starting with a pallet that is primary colors for reflected light, so there will be ranges of color you can't reproduce from just those.
You can approximate a CMY pallet by mixing the colors you have, and that might get you close enough for the colors you need, but there are two big problems. One is that you can never mix pure magenta or cyan primary colors from other colors. The best you can do is to create very desaturated versions. The other problem is that you have no way to produce accurate primary hues as a mixing starting point without calibration equipment.
So unfortunately, you can't get there from here. You may be able to come adequately close for your needs if you buy a color calibration sensor and a good white light source with a high color rendering index. There are some relatively inexpensive ones for calibrating monitors. You may even be able to create something crude with a cellphone camera, a standard color target, and some calibration software.
Once you have a calibrated color sensor, you could precisely mix some colors and measure how far off they are from theoretical. That will indicate differences in pigment concentration, and you could adjust for that.
The reality, though, is that you will invest more time, effort, and money in trying to use your existing paints in this way than buying paints (or paint-like printer's inks) designed to be used in the way you want to use them.