In classical physics you would be right, but in modern physics there is no standard frame of reference. It’s equally correct to think that the people are still and portal is moving as it is to think that portal is still and people go in it immn fast speed.

Regardless, people and portal have large speed difference going in, so there will be large speed difference going out.