Thats not true.People say land holding class in Swat are pashtuns and others who are called swatis are non-pashtuns.
Everyone living in Swat is a Swati. The same way anyone living in Pakistan is a Pakistani. Land owning or not.
The other Swatis are the ones who left when the Yousafzais moved then.