Secure Shell (SSH)

(Back to Home)

Table of Contents

• Introduction
• Authentication Methods
• The Need for SSH
• SSH Commands
  • ssh
  • ssh-keygen
  • ssh-copy-id
  • ssh-import-id
  • ssh-agent
  • ssh-add
• Generating Keys
  • Sharing Keys with the Server
• Building Blocks of SSH
• A SSH Connection
  • Initialization
  • Algorithm Negotiation
  • Key Exchange Phase
  • End of Key Exchange
  • Subsequent Encrypted Communication
  • Connection Termination
• Resources

Introduction

• Secure SHell (SSH) is a protocol used to securely connect with remote machines and establish secure tunnels to communicate with them.
• SSH is a Layer 7 (Application Layer) protocol that operates on port no. 22 by default.
• ssh (using applications like OpenSSH or PuTTY) is the client and sshd (OpenSSH Daemon) is the server.
• Basic syntax: ssh <username>@<hostname_or_IP>
  • If the SSH port is not the default (22), then the -p flag can be used to specify the port number. Eg: ssh [email protected] -p 8000
• SSH is the secure connection and tunnel for protocols such as SFTP (Secure/SSH File Transfer Protocol) and SCP (Secure Copy).

Authentication Methods

• Password

  

• Public-private key pair

  

  • The server usually sends the client a challenge ciphertext encrypted under the client’s public key that the client has to decrypt using their secret key, do some processing and send back to the server to authenticate itself.
  • Refer to the Sharing Keys with the Server section to figure out how the server knows the client’s public key.

• There are many more authentication methods, but the above are the most common ones.

The Need for SSH

• Telnet, which was an older way to connect to remote machines, exchanges data in plain text, which is easier for an attacker to snoop on a compromised connection and is vulnerable to MITM (Monkey-in-the-Middle) attacks. SSH is a secure replacement for Telnet, as it encrypts the connection.
• As mentioned before SSH has the option to use public and private key pairs to communicate. These Cryptographically-generated keys are much more secure than passwords, which can turn out to be relatively simple. (Rainbow Tables)
  • Also, once connections have been made using the public and private keys, the client automatically chooses the appropriate key for a particular connection and the user might not have to enter their password as well, which makes it convenient to connect to the machine, as they don’t have to remember their password.
• Over time SSH has been used for tunnelling, forwarding TCP ports, creating X11 connections, being used as the underlying security protocol for SFTP and SCP, etc.
• OpenSSH features
• A brief history of SSH and remote access (Telnet, rlogin, rsh, rcp, etc.)

SSH Commands

ssh

Connect to a remote machine and forward ports. For more on port forwarding and its types, look at the ‘Tunneling and port forwarding’ bullet point in the Resources section below.

Usually provided by the openssh-client package.

ssh man(ual) page

$ ssh username@hostname
$ ssh username@hostname -p 22
$ ssh username@hostname -i ~/.ssh/private_key_file
$ ssh username@hostname -o PasswordAuthentication=yes -o PubkeyAuthentication=no -o PreferredAuthentications=password

# Local port forwarding
$ ssh -L <local_ip_optional>:<local_port>:<remote_ip>:<remote_port> username@hostname

# Remote port forwarding
$ ssh -R <remote_ip_optional>:<remote_port>:<local_or_destination_ip>:<local_or_destination_port>

ssh-keygen

Generate keypairs (public and private keys) and get the fingerprint of keys.

ssh-keygen man(ual) page

$ ssh-keygen

# `-b` is the bit-length of the key (This is NOT the length of the generated text output of the key.)
$ ssh-keygen -t rsa -b 4096 -C "comment/e-mail"

# `-a` is the number of rounds
$ ssh-keygen -t ed25519 -a 32

# Get public key from private key
$ ssh-keygen -y -f /path/to/private/key

# Remove a key from the known_hosts file (which stores public keys of machines previously connected to)
$ ssh-keygen -R "<hostname_or_ip>:<optional_port>" -f /path/to/known_hosts/file

# Get the fingerprint of a key
# NOTE: In a keypair, both keys (the public and private keys) will produce the same fingerprint
$ ssh-keygen -l -E sha256 -f /path/to/public/key
$ ssh-keygen -l -E sha256 -f /path/to/private/key

ssh-copy-id

Appends public keys to the ~/.ssh/authorized_keys file on the server. The client can then log in without a password.

NOTE: Unless a public key was installed on the server during OS installation and that key pair is being used to add more public keys to the server using this command, a password (or some other form of authentication) will be required.

ssh-copy-id man(ual) page

# Copying all local public keys to the server
$ ssh-copy-id username@hostname

# Copying only a specific local public key to the server
$ ssh-copy-id -i /path/to/public/key.pub username@hostname

ssh-import-id

Import public keys from GitHub (gh) or Launchpad (lp) and append them to the ~/.ssh/authorized_keys file on the server. The client can then log in without a password.

Adding a new SSH key to your GitHub account

ssh-import-id man(ual) page

$ ssh-import-id gh:<github_username>
$ ssh-import-id lp:<launchpad_username>

ssh-agent

Stores SSH private keys and if keys have a passphrase, i.e., are password-protected, it remembers that and doesn’t prompt the user for it after the first time.

ssh-agent man(ual) page

# To populate the SSH_AUTH_SOCK env var, so that ssh-add can communicate with ssh-agent
$ eval "$(ssh-agent)"

ssh-add

Adds, lists and removes private keys from the ssh-agent utility. Removing/deleting a key does not delete the actual key file from the system.

ssh-add man(ual) page

# To populate the SSH_AUTH_SOCK env var, so that ssh-add can communicate with ssh-agent
$ eval "$(ssh-agent)"

# List private keys managed by ssh-agent
$ ssh-add -l

# Add a private key to ssh-agent
$ ssh-add /path/to/private/key

# Delete a private key from ssh-agent
# Does not delete private key file from `~/.ssh`
$ ssh-add -d /path/to/private/key

Generating Keys

• ssh-keygen is the utility that generates a public-private key pair.
• Location of the generated key pair:
  • Windows: C:/Users/<username>/.ssh
  • Linux: /users/<username>/.ssh
• key_name is the Private Key file and the key_name.pub is the Public Key file.
• The Public Key should be put in the server in the authorized_keys file in the .ssh directory on the server.
• On connection, if there is a publickey error, then use ssh-add <path_to_private_key> to add the keys to ssh-agent and try connecting again. (More details)
  • The path to the appropriate private key file can be linked to during connection as well, using the -i flag. Eg: ssh [email protected] -p 22 -i ./.ssh/private_key_file
• Further SSH config, like the sshd_config file can be found in /etc/ssh.
• Keys can themselves be password protected as well.

Sharing Keys with the Server

The authorized_keys file in the .ssh directory on the server usually holds all the public keys from clients that are allowed to connect with the server. How do those public keys get there, though?

There are usually two ways to do this:

• Log in with a pre-configured username and password and manually add the public key (or copy it over directly using something like SCP).
  • After the initial use of the password to transfer the public key to the remote server/instance, the password authentication method can be turned off in the remote server/instance settings to improve security and prevent brute force attacks, but if the user loses their private key, they permanently lose access to the remote server/instance.
• A service pre-configures it for the user and just hands them the private key to directly connect with their remote instance.
  • Eg: AWS, GENI, etc.
  • The service obviously uses the first method to configure it for the user to reduce the hassle for them and to improve the service’s security as well, so that their infrastructure is not vulnerable due to the user’s miscalculations (if any).
  • If these pre-configured keys are lost though, then it usually results in a permanent loss.

Building Blocks of SSH

SSH uses an underlying reliable connection protocol or service over which it enables secure communication and other services. The underlying connection protocol is almost always TCP, but other protocols like WebSocket can theoretically be used as well.

On top of TCP, SSH has three parts, namely the SSH Transport Layer Protocol, the SSH User Authentication Protocol and the SSH Connection Protocol

• SSH Transport Layer Protocol
  • It provides server authentication (Authenticates the server to the client), and data confidentiality and integrity.
  • It also provides Perfect Forward Secrecy (FS or PFS) due to the use of the Diffie-Hellman Key Exchange Algorithm.
  • It usually runs on top of TCP, but other reliable data streams can be used as well.
• SSH User Authentication Protocol
  • It provides user authentication (Authenticates the user/client to the server).
  • It runs on top of the SSH Transport Layer Protocol.
  • Why is user/client authentication separate and after server authentication?
• SSH Connection Protocol
  • It multiplexes a single SSH encrypted tunnel into multiple logical communication channels (session channel, Port Forwarding channel, etc.).
  • It runs on top of the SSH User Authentication Protocol.

SSH Config Files

• SSH client
  • In order of overriding behaviour from top to bottom
    • Command-line options
      • Eg: ssh -o PasswordAuthentication=yes -o PubkeyAuthentication=no -o PreferredAuthentications=password username@hostname
    • User-specific config: ~/.ssh/config
    • If present (and included above all config lines in the global config mentioned in the next point): /etc/ssh/ssh_config.d/*.conf
    • Global config: /etc/ssh/ssh_config
• SSH server
  • The SSH server runs a daemon called sshd.
  • In order of overriding behaviour from top to bottom
    • If present (and included above all config lines in the global config mentioned in the next point): /etc/ssh/sshd_config.d/*.conf
    • Global config: /etc/ssh/sshd_config
      • Note the difference in the SSH client and server config files: ssh_config vs sshd_config (d = daemon)

NOTE: If any changes are made to any configuration files, please restart the SSH and/or SSHD service or reboot the machine.

A SSH Connection

Legend:

• C = Client
• S = Server
• -> = Arrow indicating the direction of communication
• ACK = TCP Acknowledgement flag
• PSH = TCP Push flag

• A TCP handshake takes place between the client and the server to establish an underlying connection.

  

Initialization

• C -> S: Identification String Exchange

  • Eg: SSH-2.0-OpenSSH_for_Windows_8.6 (Packet no. 24 in the image below.), SSH-2.0-PuTTY_Release_0.73, etc.
  • Contents of packet
    • The SSH Protocol version used by the client.
    • The SSH initiating software name and version used by the client.
  • The server sends a TCP ACK packet to acknowledge that. (Packet no. 25 in the image below.)

  
  

• S -> C: Identification String Message

  • Eg: SSH-2.0-OpenSSH_8.0 (Packet no. 26 in the image below.), SSH-2.0-OpenSSH_7.1p2 Debian-1, etc.
  • Contents of packet
    • The SSH Protocol version used by the server.
    • The SSH initiating software name and version used by the server.
  • The client sends a TCP ACK packet to acknowledge that. (Packet no. 27 in the image below.)

  
  

Algorithm Negotiation

• Decisions for the following types of algorithms are made in this stage:
  • Key Exchange (KEX) Algorithms
    • To be able to establish a secure channel when the server and client have no pre-decided keys and to facilitate generation of other keys.
    • Eg: Diffie-Hellman, Elliptic Curve Diffie-Hellman (ECDH), etc.
  • Server Host Key Algorithms
    • Asymmetric/Public Key Cryptography
    • For Digital Signatures or Digital Certificates to authenticate the server’s identity to the client and later is one of the ways for clients to authenticate themselves to the server.
      • Both C -> S and S -> C communications use different key pairs.
    • Eg: RSA SHA2 512, RSA SHA2 256, SSH RSA, Elliptic Curve Digital Signature Algorithm (ECDSA), etc.
  • Encryption Algorithms
    • Symmetric/Secret/Private Key Cryptography
    • Eg: ChaCha20-Poly1305, AES 256 GCM, AES 128 CTR, etc.
    • Fastest for encryption, so arriving to this key through Key exchange and other processes is carried out.
  • Message Authentication Code (MAC) Algorithms
    • Proves integrity (and implicit authentication) of message.
    • Also provides protection against Replay Attacks, as the MAC will not be the same for repeated requests. (Every request will have a different sequence number.)
    • Eg: HMAC SHA2 256, UMAC 64 ETM, etc.
  • Compression Algorithms
    • Eg: Zlib, etc.

• C -> S: Key Exchange Initialization (SSH_MSG_KEXINIT)

  • The client sends the server all the algorithms it supports. (Packet nos. 27 and 28 in the image below.)
  • The list is in order of preference. The ones at the start/top of the list are more preferred by the client than the ones below it.

  
  

• S -> C: Key Exchange Initialization (SSH_MSG_KEXINIT)

  • The server sends the client all the algorithms it supports. (Packet no. 29 in the image below.)
  • The list is in order of preference. The ones at the start/top of the list are more preferred by the client than the ones below it.
  • The server acknowledges the previous C -> S SSH_MSG_KEXINIT message (ACK flag set) along with sending its own SSH_MSG_KEXINIT message in the same TCP segment (PSH push flag set). (This can be verified using the Wireshark Trace files given in the ‘Resources’ section below.)

  
  

• If both, the client and the server, are able to find common grounds for each type of algorithm, the connection can move to the next phase or else it will fail.

Key Exchange Phase

Learn about Elliptic Curve Diffie-Hellman (ECDH).

• C -> S: Elliptic Curve Diffie-Hellman Key Exchange Initialization (SSH_MSG_KEX_ECDH_INIT)

  • The client sends its ECDH ephemeral public key to the server. (Packet no. 30 in the image below.)
    • The first half of the Key Exchange process.
  • NOTE: This ECDH key exchange is carried out so that keys of an encryption algorithm can be generated/exchanged securely.
  • Again, this same TCP segment acknowledges the S -> C SSH_MSG_KEXINIT message and sends the SSH_MSG_KEX_ECDH_INIT data of this message along with it. (ACK flag is set along with the PSH flag that indicates data being sent.)

  
  

• S -> C: Elliptic Curve Diffie-Hellman Key Exchange Reply (SSH_MSG_KEX_ECDH_REPLY)

  • The server sends multiple things to the client. (Packet no. 31 in the image below.)

    • The server’s Host (Digital Signature/Certificate) public key
      • The authenticity of this public key has to be verified either manually (the fingerprint that SSH asks the user to verify when connecting to a host for the first time) or through some other certificates.
        • This fingerprinting process to authenticate a server is a big security problem, as most users don’t actually check the fingerprints that the SSH client prompts the user with, which opens up the MITM (Monkey-in-the-Middle) Attack vector. Having automated checks like maybe DNSSEC DNS queries to verify the public key belonging to the server or Digital Certificates to verify the public key belonging to the server would be nice to have.
    • The server’s ECDH ephemeral public key
      • The second half of the Key Exchange process.

    • A signature of a key exchange hash (The server signs it with its private host key)

      • The public key that the server gave, if successfully verified, is used to verify this signature with the hash that the client also generates.

      Explanation:

      On server:
          -   Compute `server_hash`
              -   Computation includes the final computed ECDH shared secret.
          -   `sign(server_hash, server_private_key)` = `signature`
          -   Send (signature, server_public_key, ecdh_ephemeral_key) to client
      
      On client:
          -   `verify(server_public_key)` = `valid` or `invalid`
              -   If successfully verified, then the `server_public_key` actually belongs to the server that it initially claimed to be.
          -   Compute `client_hash`
              -   Computation includes the final computed ECDH shared secret.
          -   `verify(client_hash, signature, server_public_key)` = `valid` or `invalid`
              -   If successfully verified, then the `ecdh_ephemeral_key` can be trusted to be coming from the server and it can also be confirmed that both, the client and the server, have the same ECDH shared secret.
      

  • This step also proves to the client that the server is actually who it claims to be and there isn’t a MITM (Monkey-in-the-Middle) Attack going on. Thus, the server authenticates itself to the client.
    • The client authenticates itself to the server in the subsequent encrypted communication phase.
  • It is very interesting to explore the generation of the hash and the consequence of the Diffie Hellman shared key.
  • As before, this TCP segment has both ACK and PSH flags set. This is a common pattern across the entire communication, where there is an ACK for every packet from C -> S and S -> C, and instead of sending it separately, it is often combined with another data containing packet (which will have the PSH push flag set).

  
  

End of Key Exchange

• The end of the key exchange phase is signalled by SSH_MSG_NEWKEYS from S -> C and C -> S.
• It also indicates that all communication after this will be encrypted.

• S -> C (Packet no. 31 in the image below.)

  

• C -> S (Packet no. 32 in the image below.)

  

Subsequent Encrypted Communication

• After this, all the communication between the client and server are encrypted. (Thus data payload is not visible in the encrypted packets that Wireshark captures.)

• The server sends its fingerprint for the user to verify its identity.

  • The SSH client will ask the user to manually verify the identity of the server by presenting the fingerprint of one of the server’s public key located at /etc/ssh/*.pub.
    • These are generally not manually-generated, but are initially set up by SSH itself.
  • If the user approves of the fingerprint, then SSH will store the public key(s) of the server in the ~/.ssh/known_hosts file.
  • The advantage of storing the server’s public key in the file is that the next time the user connects to the server, the SSH client will not ask the user to approve the server’s fingerprint, but will automatically verify it with the server’s public key stored in the known_hosts file.

  • Fingerprinting demo: The server presents its fingerprint to the client.

    • On the server

      • The server’s public key(s) for identifying itself can be found at /etc/ssh

        $ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
        256 SHA256:EPvxjioRUt/Non2BGcrwIgN5i19uvpuOQ7WbpU8HzPU root@Vostro-3525 (ED25519)
        

      • The output of the above command is what is presented to the client. This is the fingerprint of the public key.

    • On the client

      • When connecting to the server for the first time, the above fingerprint is what is presented to the client.
      • If the client wants to check the fingerprint, they can do so in a few ways (non-exhaustive list):
        • Have physical access to the server they want to connect to via SSH and run the above fingerprinting command and manually compare the fingerprints.
        • If the client does not have physical access to the server, some out-of-band system that is trusted by the user (e.g. the automation software that installed the OS on the serber) should provide the fingerprint to the client after recording it by running the above ssh-keygen command.
• Now that the user knows that they are talking to the correct server, the SSH client authenticates itself to the server here. (SSH_MSG_USERAUTH_REQUEST)
  • The client authenticates itself during this encrypted communication phase rather than before so that it is able to securely transmit passwords and other sensitive data to the server securely.
  • The client has different authentication methods that it can choose from.
• What can be interesting about packets of information that are encrypted? Well, SSH does something clever here and a similar behaviour is observed in Telnet as well.
• After connection with the remote instance, one’s intuition expects that SSH sends the user-typed command to the remote instance after the Enter key is hit, but SSH actually sends a request to the remote instance every time the user hits a key on their keyboard. The remote instance acknowledges the receipt of the keystroke, checks what happens when that keystroke is executed on the terminal and sends back the action to the client. The client then displays whatever the remote instance responds with, on the user-facing terminal.
  • Why send every keystroke to the remote instance? This might not make sense for commands such as ls (because they can be communicated to the remote instance after the Enter key is hit), but imagine using something like the Vim editor which requires the use of the Esc key to do certain actions. One does not hit the Enter key after the Esc key as that might mean something else to the program, so communicating every keystroke and getting back the reaction of that keystroke is necessary.
  • Article: SSH uses four TCP segments for each character you type
    • Interesting sub-discussion on this on Hacker News

• In the image below, a connection with csa3.bu.edu is established, the ls command is run on the remote instance and then the exit command is run on the remote instance to gracefully end the connection.

  

• The above exchange was analysed using Wireshark.

  • The 2nd Wireshark Trace file as linked in the ‘Resources’ section below.
  • Connection to the remote instance in csa3.bu.edu has already been established.

  
  

• Breaking down the exchange for the ls command. (As shown in the two images above.)
  • On hitting the l key on the client, the client sends a request to the server. (Packet no. 60 in the image above.)
  • The server responds with a combined TCP segment containing the ACK flag for the receipt of the keystroke and the data with the reaction of the keystroke (PSH flag set) so that the client can draw it on the user-facing terminal. (Packet no. 61 in the image above.)
  • The client acknowledges that with a TCP ACK segment. (Packet no. 62 in the image above.)
  • The same goes for the s key. (Packet nos. 63 to 65 in the image above.)
  • The user then hits the Enter key after typing the ls command (Packet no. 66 in the image above.) and the server acknowledges this key press and replies with the reaction of pressing the Enter key in subsequent TCP segments (Packet nos. 67 to 70 in the image above.).
  • The client then acknowledges the receipt of the server’s replies with a TCP ACK segment. (Packet no. 71 in the image above.)

Connection Termination

• Connection either times out if left idle for too long or can be gracefully terminated using the exit command.

• A TCP FIN segment is exchanged to terminate the connection to the remote instance. (Packet no. 253 in the image below.)

  

Resources

• SSH Crash Course
• An Excruciatingly Detailed Guide To SSH (But Only The Things I Actually Find Useful)
• How Secure Shell Works
• Tunneling and port forwarding
  • Visual guide to SSH tunneling and port forwarding
  • A Visual Guide to SSH Tunnels: Local and Remote Port Forwarding (bastion hosts, etc.)
  • How to Set up SSH Tunneling (Port Forwarding)
• Virtual Network Computing (VNC) over SSH
  • How To Install and Configure TigerVNC server on Ubuntu
  • Connecting using VNC from a Windows computer to a Linux system
• How SSH Password-less Key-based Authentication Works
• known_hosts and authorized_keys
  • What is the difference between authorized_keys and known_hosts file for SSH?
  • What is actually in known_hosts?
  • SSH known_hosts file format
• Cryptography Basics - SSH Protocol Explained
• Understanding the SSH Encryption and Connection Process
• A brief history of SSH and remote access
• Wiresharking Secure Shell
• How SSH port became 22
• RFC 4251: The Secure Shell (SSH) Protocol Architecture
• SSH uses four TCP segments for each character you type
  • Interesting sub-discussion on this on Hacker News
• OpenSSH
  • OpenSSH utilities
• Wireshark SSH Trace files
  • These are the Wireshark Network Trace files used to analyse the SSH protocol packets.
  • Please use the filter ip.addr == 128.197.11.45 to view the entire SSH communication being talked about in this article.
  • The first Wireshark SSH Trace file (.pcapng)
  • The second Wireshark SSH Trace file (.pcapng)
• SSH Keys
• GitHub SSH key is Leaked - How bad is this?
• Running SSH service on a server
• How To Set up SSH Keys on a Linux/Unix System
• How to Remotely Open a GUI Application With PuTTY
• Cryptography
• CLI commands & info
• Turning a Laptop into a Server
• SSH file permissions: Why am I still getting a password prompt with ssh with public key authentication?
• Securing SSH service
  • How do I disable remote SSH login as root from a server?
  • The Best Ways to Secure Your SSH Server
  • A good explanation of Port Forwarding and general SSH security practices
  • Fail2Ban
    • How To Protect SSH with Fail2Ban on Ubuntu 20.04
    • How Fail2Ban Works to Protect Services on a Linux Server
• SSH obfuscation
  • Chaff
  • SSH Keystroke Obfuscation Bypass
  • Random chaff insertion in ssh session