You've absolutely hit the nail on the head on this one, imo.
This point came up in relation to a brief exchange I had with Thanos on this board recently, in relation to Christianity.
We too often presume, when going into conversations, that we know what certain labels mean. I am a Christian, but what does that imply, really? Am I an evangelical,a Catholic, liberal Catholic, a conservative, an gnostic...? My point there was that, while their may be a common denominator truth to what being a Christian means, the more obvious thing is that what's most obvious are how Christians all see what that means slightly differently, from limited points of view that are shaped by limited and limiting influences. And the exact same thing applies to non Christians and anti-Christians whose points of view are also a construction of various limited perspectives -- never the whole picture, because frankly nobody on either side of the discussion can really say what it 'all' means.
I find it quite fascinating, that as adults we all know this to be true, but so often put it aside when entering discussions. But this tendency also applies to this topic and it is very significant.
You are entirely correct. We do tend to insist that aliens do or do not exist because they may or may not fart around in flying machines like the ones we've only been employing for one century in, what?, 200,000 years of being homo sapiens? I find it bizarre that this premise doesn't get brought up more often, so well done.
At the end of the day I presume that there is and has been alien interaction, by species that are much more advanced than us, and that they choose to operate *very* discreetly in our sphere almost all of the time. One very good way to do that is to use this propensity of ours to their maximum advantage, to operate in ways that we tend to least suspect as we assist them in this process by fixating in areas that we tend to presume 'must' be the only way they can appear and/or interact. It's a principle that our best magicians have been using for thousands of years by now. Few seem to embrace the lesson's wider applications.