How to match IPv6 addresses as a host pattern in ssh_config

It's possible you may be confused on where to configure how your SSH server listens for connections, and where to configure how your SSH client initiates connections.

When I put these entries in /etc/ssh/sshd_config:

Port 222
#ListenAddress 0.0.0.0
#ListenAddress ::

(in other words, a configured port number, but default values for IPv4 and IPv6 listening addresses) and if I configure my public interface with these IPs:

inet 10.10.11.35 netmask 0xffffff00 broadcast 10.10.11.255
inet6 fdad:cafe:f00d::1 prefixlen 48

and restart my sshd service to look for which sockets it's listening on, I see both TCPv4 port 222, and TCPv6 port 222:

root     sshd        8722 3   tcp6   *:222           *:*
root     sshd        8722 4   tcp4   *:222           *:*

Now if I edit ~/.ssh/config to instruct my SSH client to differentiate between the IPv6 host jimsdesk and the IPv4 host jimsdeskv4:

$ grep -B4 -A3 jimsdeskv4 ~/.ssh/config
Host jimsdesk fdad:cafe:f00d::1
  Hostname fdad:cafe:f00d::1
  Port 222

Host jimsdeskv4 10.10.11.35
  Hostname 10.10.11.35
  Port 222

Notice there are no brackets around the IPv6 addresses. Also, those entries could equally well be put in /etc/ssh/ssh_config.

Then by choosing the right host nickname, I can connect via IPv6:

$ ssh jimsdesk set | grep ^SSH
SSH_CLIENT='fdad:cafe:f00d::1 52341 222'
SSH_CONNECTION='fdad:cafe:f00d::1 52341 fdad:cafe:f00d::1 222'

or via IPv4:

$ ssh jimsdeskv4 set | grep ^SSH
SSH_CLIENT='10.10.11.35 29742 222'
SSH_CONNECTION='10.10.11.35 29742 10.10.11.35 222'

I can also SSH directly to the IPv6 address, and my .ssh/config figures it out just fine.

$ ssh fdad:cafe:f00d::1 set | grep ^SSH
SSH_CLIENT='fdad:cafe:f00d::1 47360 222'
SSH_CONNECTION='fdad:cafe:f00d::1 47360 fdad:cafe:f00d::1 222'

Update: As @Muru points out in the comments below, if you have trusted clients who need to connect but cannot accommodate a non-standard TCP port, you will need to use two Port lines in /etc/ssh/sshd_config:

Port 22
Port 222

Again, this needs to be in /etc/ssh/sshd_config, not /etc/ssh/ssh_config for the SSHD server to react to it.

If listening on port 22 is a deal-breaker for you, then you'll need to implement firewall rules sufficient to protect TCP4 and TCP6 ports 22 from any clients who are not authorized to access your SSH server. Alternatively, you could implement firewall rules which selectively redirect ports 22 to ports 222 but only for permitted source IP numbers, but you'd also need to similarly re-map reverse traffic. It's likely that simply permitting TCP ports 22 traffic only from known clients is the easier (and more secure) way to go.