The vote was to leave the EU. We did. We can have endless debates on why people voted to leave, but the responses will vary. Some will say immigration, others sovereignty, others because they think there’s better trade deals elsewhere.
The referendum wasn’t to select a selectorate for future democratic decisions; it was just to leave the EU. If you think the argument was for immigration control, the problem is that not every Brexiter voted based on this, and as the referendum wasn’t on this, it’s perfectly reasonable for it to be debated and voted on against the will of the select Brexiters who were motivated by it. 2016 wasn’t a referendum to forever prevent debate and people putting forward their ideas as to how the country goes.