I'm not sure with what you're trying to say, but about networking, here's what happened:
ServerSocket (in the server) will have all of the clients' socket, and each time ServerSocket accept a connection to a client, that client should get a listener (which you will create with multi threading), that way there will be more than one client can connect simultaneously.
port is not assigned to the client, port is opened from the server, and the client will connect to that specific port.
If you're not sure with multi threading, feel free to see my previous tutorial about multi threading, I think I made several of them

(they're all in Java forum)