Can you measure the amount of tooth that a piece of paper has? I'm imagining the paper analogue of grit for sandpaper.
In the sense you're asking the question, the answer is no. Sandpaper has a specific purpose that's directly related to the particle size of the grit. It's produced in a huge range of incremental values that allow you to polish a surface by using progressively finer grits to erase the scratches left by the previous, coarser grits. Theoretically, sandpaper could be made in any grit on a continuous scale if there was a need for it.
Paper tooth isn't graded in that way. "Tooth" is more of a descriptive term, like "roughness". Paper comes in a few ranges of roughness for different purposes. Within those ranges, there is some variation between similar products. But it's more the case that the paper is made optimized for the end use, rather than made to a specific "grit".
Can the amount of tooth be measured, allowing comparisons between different paper manufacturers?
The paper manufacturer has some form of measurement of the paper characteristics for quality control purposes, and you can see differences in the surface under a microscope. But there isn't a universal standard measurement of tooth, where you could look up any paper and see its comparable measure.
If not, is it possible for some cold pressed paper to have less texture than some hot pressed paper (i.e. an overlap between the two)?
Texture is different from tooth. Texture is a relatively large-scale pattern of irregularity in the surface. Paper can have a texture but low tooth and vice versa. Tooth is roughness at a near microscopic scale. Paper for a particular purpose will have an appropriate combination of texture and tooth.
Different production methods are used for different objectives. During paper manufacturing, different methods can be used to produce paper of different surface characteristics for different purposes. Very pronounced "geometric" texture (like laid lines or a weave pattern), is often done by making the paper in a mold. An unstructured irregular surface is often done using a felt blanket or belt with the desired surface texture.
If a smoother surface is needed, a subsequent step passes the paper through rollers or a press. Whether that is done cold or hot, how much pressure is used, and the smoothness of the surface applying the pressure, affects how smooth the paper's texture will be.
The resulting characteristics of these methods are distinctly different, and the method is selected based on the type of paper (end purpose). Hot pressed paper is smoother than cold-pressed, so this method is used when the smoothest surface is desired (there isn't overlap).
The level of tooth is affected by a number of things, such as the type of cellulose used, but it is largely determined by the characteristics of the sizing applied. "Sizing" is a treatment typically applied in a bath to the almost finished paper. It's primary purpose is to control the absorbency of the paper so that it doesn't simply act like a sponge.
Sources and further reading: