Right now, the only work-arounds I can think of are:
- Use an IP address for one port and a name for a 2nd one
- Add aliases to DNS or your hosts file for additional ports
- Manage your own key file swapping (e.g. copy "host-123.key" to "host.key" right before connecting)