Square.
The widespread custom in synagogues to depict the Tablets as rectangles with rounded tops is not historically correct. The Talmud in Baba Batra 14a states that the measurements were 6x6x3 handbreadths each.
Michelangelo’s sculpture “Moses” (c. 1515) shows the Tablets as being square – in accordance with the Talmudic description – although his sizes and proportions are not exactly as the Talmud states.
Nevertheless, the widespread synagogue custom is to show the Tablets as rectangles with rounded tops. Why?
The Ten Commandments and the Torah that God gave us are more than words on a stone or a parchment. They are our wisdom and the essence of our life. We are meant to make them part of us, as the verse states, "Write them (the words of Torah) on the tablets of your heart" (Proverbs 3:3).
This is the basis of the depiction of the Tablets as rectangles with rounded tops. The Tablets appear in the shape of a heart since they are one with the heart.