Ab initio random structure searching (AIRSS) and density functional theory methods are used to predict structures of calcium and magnesium carbonate (CaCO$3$ and MgCO$3$) at high pressures. We find a previously unknown CaCO$3$ structure which is more stable than the aragonite and "post aragonite" phases in the range 32--48 GPa. At pressures from 67 GPa to well over 100 GPa the most stable phase is a previously unknown CaCO$3$ structure of the pyroxene type with fourfold coordinated carbon atoms. We also predict a stable structure of MgCO$3$ in the range 85--101 GPa. Our results lead to a revision of the phase diagram of CaCO$3$ over more than half the pressure range encountered within the Earth's mantle, and smaller changes to the phase diagram of MgCO$3$. We predict CaCO$3$ to be more stable than MgCO$3$ in the Earth's mantle above 100 GPa, and that CO$2$ is not a thermodynamically stable compound under deep mantle conditions. Our results have significant implications for understanding the Earth's deep carbon cycle.