It was definitely not Hat’ḥor , for the simple reason that a number of statues of Hat’ḥor at the turquoise mining site now known as Serabit el-Khadim were defaced with Hebrew graffiti—not something that worshippers would do:

- The Egyptian goddess Hatḥor (a.k.a. Astarte) at Serabit-el-Khadim in the western Sinai Desert

This occurred at a time corresponding to the expulsion of Canaanites and other western Asians from Egypt by the founders of the New Kingdom around 1550 BCE—which is the most likely inspiration of the Exodus legend.
The most famous of these inscriptions (above), says, in the Paleo-Hebrew script (upside down—i.e. inscribed by someone sitting on top), Mat[tat] le-Ba’alat = ‘A gift for the goddess’—a sarcastic reference to something undoubtedly unspeakable that they left on the statue, as further desecration.
Baalat means ‘Mrs. Baal’—in recognition of the fact that the goddess Hatḥor was in fact the Egyptian incarnation of the Canaanite goddess Astarte (a.k.a. Ashera, Ishtar, etc.), who was the wife of Baal, the chief Canaanite god.
Since the inscription is in Canaanite, and the Hatḥor/Astarte was worshipped by all Canaanites except the Israelites in their new, Moses-led religion that we now know as Judaism, it follows that the vandals in this case must have been Israelites. The casual, slipshod manner of the inscription (upside down, and not on a straight line) suggests that it was made by young hooligans, as well, not at an official desecration ceremony—likely with the encouragement of Moses himself or one of his priests.
As for the ‘golden calf’, this was more likely not a calf, but a bull, representing not Hatḥor, but Baal himself (the Bible’s reference to it as a ‘calf’ was the attempt by the biblical compilers and editors to denigrate it, and to obscure the fact that the incident was a case of Baal-worship). It happened because when Moses went up Mt. Sinai to receive the Commandments and failed to return after one, two, three, four, five weeks and counting (he returned only after 40 days), the Israelites quite naturally got cold feet and fell back on their ancestral religion.
Long after Moses’ return, however, and the resettlement of Canaan, worship of Baal (and of Astarte) remained a deeply entrenched and persistent feature in Israelite religion and tradition—not least because all other Canaanites continued to worship him—so much so, that even centuries after the Exodus, in the late First Temple period (ca. 950–585 BCE), Judean and Israelite prophets and righteous kings still struggled to root it out.
Amazing what one can tell from a simple graffiti inscription, isn’t it?