My wife, who reverted to Islam prior to marrying me, told me that some non-Muslims think that Muslims worship the Kaabah in Mecca.
Many also find it odd that Muslims say that they are against idol worship, but then pray facing the Kaabah. And images during the Haj where Muslims pray in front of the Kaabah and circumambulate around it reinforces this misconception.
Muslims do not worship the Kaabah but only worship towards it for unity. All Muslims aroung the world pray towards one direction, that is the Kaabah From any given point, the direction facing the Kaaba is called the Qiblat.
Based on Islamic tradition, the Kaabah was rebuilt by Prophet Abraham PBUH, to worship Allah, the one god, but subsequently humans started placing and worshipping idols in it
During the pre-Islamic days, the Kaabah was filled with idols, and pagans at that time used to worship them, but they were removed when Prophet Muhammad PBUH successfully reentered Mecca in December 629.
For a clearer answer on this, watch Dr Zakir Naik's reply to a question about why Muslims bow down towards and circumambulate around the Kaabah.
PBUH - Peace Be Upon Him