I would strongly consider what is usually used for covers in these types of books. Hardboard or millboard. They are both similar materials that I would suggest for this. Unfortunately their names are sometimes used interchangeably so I would like to clarify a little what they are. Then you should be able to make a informed choice.
Millboard is what you would find if you were to take a simple binder or traditional hardcover book apart. It is also heavier than millboard so keep that in mind. Side Note: Doing just that is a great way to get perfectly sized materials!
Hardboard is, like its name suggests, a very hard rigid and dense board. It is similar to both MDF and particle board. These come in thin sheets that would be perfect for binding. I talk about hardboard a little more over at Woodworking.SE
I would lean towards millboard as it is more forgiving to bending than hardboard. Hardboard will hardly move at all. Both are perfectly adequate for binding leather. I have made 5 books that were leather bound.
However, I want to bump up the quality of the binding
If you wanted to do something else to bump up your game I would suggest using actual leather for you book covers. It is really easy to come by if you are no picky about colours. Local secondhand stores are usually where I source my leathers. Jackets might be tempting for their size but they are usually too expensive. Skirts are the best best at being both cheap and containing large enough swatches to cover your book panels.