1. The quarry marks in the relieving chambers are not the only ones. There are marks including Khufu's Horus name on the outside of the pyramid on the 5th course.
2. The ivory statue is no longer thought by many Egyptologists to be the only remaining statue of Khufu. It is now believed that some statues previously attributed to Sneferu are Khufu.
No matter how much you do not wish to admit it, a lack of evidence is indeed evidence of absense.