That phrase sounds like something I would say in order to problematize and draw out the very fact that, as a White person, I have (well, had( the privilege of passing through the border unnoticed.

But, I guess, it sounds like this person actually meant it as an honest question? I would probably not start out with an unpacking-your-backpack sort of discussion, because that tends to blow people's minds and the fallout is unpredictable.