Dictionary definition of the word "denier":
One that denies: a denier of harsh realities.
I don't see anything in the dictionary definition to do with holocaust denial. If you both are feeling that you're being accused of holocaust denial, then that is surely in your own minds and no one else's.
Holocaust denial is an entirely different issue altogether.