SSH SCP SFTP Daily Work Summary

Posted on | Edited on | In

This blog used to walk through some ssh(secure shell), scp and sftp use cases.

Aside: difference between OpenSSH vs OpenSSL The file format is different but they both encode the same kind of keys. Moreover, they are both generated with the same code.

Notice that restart sshd daemon will not disconnect current ssh connection, even if you stop the sshd daemon for a short time or restart the network daemon(don’t stop it!), the current ssh session is still working, see this issue: The reason is, sshd fork a child process on each connection, the child process will not die if either sshd or the whole network is restarted. sshd listens on port 22 for incoming connections. When someone connects it spawns a new process for that connection and goes back to listening.

1
2
3
4
5
6
7
8
9
# check pid and ppid of current ssh child process
ps -ef | grep -v grep | grep ssh

# restart some daemon
systemctl restart sshd
systemctl restart network

# the old child ssh session ppid was changed to 1
ps -ef | grep -v grep | grep ssh

Install SSH (SCP SFTP)

Notice that ssh, scp and sftp are all installed from openssh-clients package. 操作的目标机器的用户以及密码就是目标机器上在/etc/passwd中对应的用户及其密码。

First, understand that there are openssh client and openssh server: Installing and Enabling OpenSSH on CentOS 7, this article briefly introduces openssh configuration and firewall setting for it.

1
yum –y install openssh-server openssh-clients

If only install openssh-clients, you can ssh to others but others cannot ssh to you, since you don’t have ssh server listening at port 22.

After installing openssh-server, enable and start the sshd daemon

1
2
3
4
systemctl enable sshd
systemctl start sshd
# check status
systemctl status sshd

The system OpenSSH server configuration file is /etc/ssh/sshd_config, the custom configuration file is ~/.ssh/config. The /etc/ssh/ssh_config is for system-wide client behavior.

Restricted configuration you may need on server side:

1
2
3
4
5
Port 22
PermitRootLogin prohibit-password
PubkeyAuthentication yes
# after copy the public key in
PasswordAuthentication no

After making changes, restart sshd daemon.

Firewall setting for ssh is file /etc/sysconfig/iptables.

SSHFS

This is a remote mount implemented by SSH, handy if NFS is not workable, search my blog <<Linux Storage System>>.

SSH Tunnel

Forward a local port to remote port, one on one mapping. 用在比如database or web servers 没有对外开放端口，我们可以通过SSH穿过防火墙(SSH的端口是开 放的)去远程映射它们的端口到本地，然后通过localhost访问。

1
2
3
4
5
6
7
# -L: port forward
# 10003: local port
# 8000: remote port of a web server, for example a python simple http server.
# -N: Do not execute a remote command. This is useful for just forwarding ports.
# If the 8000 port is blocked by firewall remotely, but after mapping,
# we can access it locally with the local port 1234.
ssh -L [127.0.0.1:]10003:remotehost:8000 user@remotehost -N

Then go to localhost:10003 on browser to see the web page.

The port forwarding approach is limited on that single port mapping, for unlimited access, you need SOCKS proxy tunneling, see next section.

SSH SOCKS Proxy Tunnel

Introduction to SOCKS proxy

Although by default SOCKS proxy does not provide encryption, but we run it over SSH, so the traffic is encrypted.

How To Route Web Traffic Securely Without a VPN Using a SOCKS Tunnel: A SOCKS proxy is basically an SSH tunnel in which specific applications forward their traffic down the tunnel to the server, and then on the server end, the proxy forwards the traffic out to the general Internet. Unlike a VPN, a SOCKS proxy has to be configured on an app by app basis on the client machine, but can be set up without any specialty client agents.

The remote host must has ssh server running.

1
2
3
4
5
6
7
8
9
10
# -D: dynamic application-level port forwarding, see curl man for more
# explanation about SOCKS support.
# [127.0.0.1:]11000: local mapping port.
# -N: Do not execute a remote command. This is useful for just forwarding ports.
# -C: Compresses the data before sending it
# -q: quiet

# -f: Forks the process in the background
# don't like tunnel, on the remote host it is dynamic forwarding
ssh -D [127.0.0.1:]11000 -f -C -N -q user@remotehost

This is actaully a SOCKS5 proxy created by SSH, after it is established, you can check by:

1
2
# Now you can access the web that original can only access by remote host.
curl -ILk -x socks5://localhost:11000 "https://web_can_only_access_by_remotehost"

Or configuring the web browser to use this SOCKS5 proxy: localhost:11000. On Firefox FoxyProxy plugin, set and use it. Now we can access whatever the remotehost can access.

Manually kill the tunnel process if you use -f.

SSH X11 Forwarding

Similar to VNC, but VNC transmits whole desktop which is more expensive. Linux has good support to X11, on Mac, need to install XQuartz(still not work on Mac).

1
2
3
4
5
# -X: X11 forwarding
ssh -X user@remotehost

# gedit is running on remotehost but reflect GUI locally
> gedit

SSH Agent

很久之前看书的时候没明白这个概念. 一个常见的用处就是保护originating host的private key. A handy program called ssh-agent simplifies working with SSH private keys.

In Mac, ssh-agent is running by default, but in Linux, start by yourself (ensure only one instance of ssh-agent is running).

1
2
3
4
5
6
7
# Don't need do this if you are Mac, or your company laptop has agent running by
# default you can check by:
ssh-add -l

# First check if only one instance is running
ps aux | grep ssh-agent
# if it is there but cannot work, kill it.

If you run ssh-agent, it will output environment vars you need to set, for example, you can also manually export these instead of using eval:

1
2
3
4
5
6
7
8
9
ssh-agent

# export these manually is OK
SSH_AUTH_SOCK=/tmp/ssh-YI7PBGlkOteo/agent.2547; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2548; export SSH_AGENT_PID;
echo Agent pid 2548;

# Start it.
eval $(ssh-agent)

Add your private key to ssh-agent, sometimes git ssh clone failed, you may need to add private key to agent:

1
2
3
4
5
6
7
8
9
10
11
# default path ~/.ssh/id-rsa
ssh-add
ssh-add <other private key path>

# list all identities 
ssh-add -l

# delete all identities
ssh-add -D
# delete specified identity
ssh-add -d <private key path>

How to start ssh-agent on login: https://stackoverflow.com/questions/18880024/start-ssh-agent-on-login Add below to your .bash_profile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
SSH_ENV="$HOME/.ssh/env"

function start_agent {
    echo "Initialising new SSH agent..."
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo 'succeeded'
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    # add private key ~/.ssh/id_rsa.pub
    /usr/bin/ssh-add;
}

# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
    # need resource, but ssh-agent is there
    . "${SSH_ENV}" > /dev/null
    # ps ${SSH_AGENT_PID} doesn't work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        # statement block
        start_agent;
    }
else
    start_agent;
fi

When you try to make a connection to a remote host, and you have ssh-agent running, the SSH client will automatically use the keys stored in ssh-agent to authenticate with the host.

Advantages:

  1. For encrypted SSH private keys，只有第一次加入ssh-agent的时候要求输入password 如果 不使用ssh-agent，每次SSH都会要求输入password
  2. If you are using Ansible to manage hosts that use different SSH keys, using an SSH agent simplifies your Ansible configuration files.
  3. ssh-agent forwarding, see below

ssh-agent 还解决了一个问题，比如你有personal and work git account各一个，但各自是不同的 ssh key pairs，在git clone的时候如何指定用哪个private key呢? see this link.

SSH Agent Forwarding

If you are cloning a Git repository on remote host via SSH, you’ll need to use an SSH private key recognized by your Git server. I like to avoid copying private SSH keys to my remote host (for example, a EC2 instance), in order to limit the damage in case a host ever gets compromised.

1
2
3
4
5
6
7
8
# The example.xx.com does not have the private key to access git repo but the
# local host has.
# -A: agent forwarding
ssh -A root@example.xxx.com

# git clone via ssh mechanism on remote host with the private key provided by
# agent from local host.
git clone git@github.com:lorin/mezzanine-example.git

Here -A limit the agent forwarding in this session only, you can have ssh config to set agent forwarding on a broad view.

ProxyJump

现在想想，当时登录openshift or softlayer 的master时，也需要经过堡垒机，所以应该可以配置 proxyjump. 这个和ssh-agent没有必然联系，如果没用ssh-agent, 则应该可以在配置config file中 指定key的位置.

Using OpenSSH ProxyJump It uses vagrant VMs to demonstrate. 但是我觉得不需要再指定port, user, identifykey对 target server了，这些应该在bastion上已经配置好了。

SSH agent and ProxyJump explained Talking risk of SSH agent forwarding, access internal hosts through a bastion, ProxyJump is much safer. 也谈到了SSH是怎么handshake, 传输中用的是新的对称钥匙.

JumpBox or Bastion Host: Notice that you need to generate key and copy the public key to bastion host first.

Baston hosts are usually public-facing, hardened systems that serve as an entrypoint to systems behind a firewall or other restricted location, and they are especially popular with the rise of cloud computing.

The ssh command has an easy way to make use of bastion hosts to connect to a remote host with a single command. Instead of first SSHing to the bastion host and then using ssh on the bastion to connect to the remote host, ssh can create the initial and second connections itself by using ProxyJump.

1
2
3
4
5
# -J specify the jumphost
ssh -J <bastion-host> <remote-host> [-l <remote login user>] [-i <pem file>]
ssh -J user@<bastion:port> <user@remote:port>
# Jump through a series of hosts
ssh -J <bastion1>,<bastion2> <remote>

最主要的还是配置~/.ssh/config 文件, basic settings: 注意，之前遇到过奇怪的问题，用同样的config file，别人一切正常，但我就连不上，简化了一下config file后就好了，当时的解决办法是把Match host模块移到匹配的Host 下面，其实Match host不要也行 很多可以合并的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# May have more options, for example, User, Port, AgentForward, etc.
# refer `man ssh_config`

# The `Host` sections are read in order and the options matched will get
# accumulated

# The Bastion Host
Host <jump-host-nickname>
  User <user name>
  # default is no
  ProxyUseFdpass no
  # jumpbox port
  Port 22
  # jumpbox IP
  HostName <hostname or IP>

# The Remote Host
Host <remote-host-nickname>
  Hostname <remote-hostname or ip address for example: 172.12.234.12>
  User <user name>
  AddKeysToAgent yes
  IdentitiesOnly yes
  # may need pem file, the private key
  IdentityFile ~/.ssh/file.pem
  StrictHostKeyChecking no
  ServerAliveInterval 60

# The remote host match this IP will use jumpbox
# 这个可以不要，就是用来match Host的
Match host 172.??.*
  ProxyJump <jump-host-nickname>

# Or can specify jumpbox directly
Host <remote-host-nickname>
  HostName < remote-hostname or ip address>
  ProxyJump bastion-host-nickname

Then you can ssh directly: ssh remote-host-nickname

ProxyCommand is an alternative of ProxyJump, but it is old.

1
ssh -o ProxyCommand="ssh -W %h:%p bastion-host" remote-host

Force SSH Password Login

Usually SSH password authentication is disabled, that is, you can only log in over SSH using public key authentication, to enable password login:

1
2
3
4
5
6
7
# /etc/ssh/sshd_config
# set to yes
PasswordAuthentication yes
# you may also need to allow root login
PermitRootLogin yes
# restart sshd
systemctl restart sshd

Create new user to test:

1
2
useradd alice
passwd alice

Logout and try:

1
2
3
4
5
6
7
# PubkeyAuthentication may be needed
# then input the password
ssh -p 2222 \
    -o StrictHostKeyChecking=no \
    -o UserKnownHostsFile=/dev/null \
    -o PubkeyAuthentication=no \
    alice@127.0.0.1

Debug

  1. Use -v, -vv, -vvv flag in ssh command (比如之前pem file 权限和格式没设置对都 有提示)
  2. Wireshark capture ssh traffic on that interface, you should see SSHv2 protocol and more details
  3. Check system log, journalctl | grep sshd.
  4. Launch sshd on another port in debug mode: sudo /usr/bin/sshd -d -p 2020, then ssh to this port 2020 from client ssh -p 2020 user@remote_server.
  5. Possibly restriced by firewall

Usage Summary

1
2
3
4
5
6
7
8
9
10
11
12
13
10/01/2018 ssh send command
11/14/2018 ssh run shell script
12/19/2018 ssh-copy-id
01/06/2019 ssh-kenscan
01/08/2019 ECDSA host key changed
01/22/2019 no prompt first time
01/23/2019 sshpass
02/21/2019 scp folder or files
03/11/2019 ssh -i option
03/12/2019 recover public key
09/05/2020 sftp
01/20/2021 ssh config
03/17/2022 ssh config permission

10/01/2018

use ssh send commands to execute on remote machine: -t flag allow you to interact with remote machine:

11/14/2018

use ssh run shell script in remote machine

12/19/2018

use ssh-copy-id to copy local machine public key to remote machine’s ~/.ssh/authorized_keys file, so next time when you ssh, scp or sftp again, no prompt to require password:

Sometimes I see people use ~/.ssh/id_rsa with ssh-copy-id, that confused me because that is private key, OK, man tells me why:

1
2
3
-i identity_file
        ...If the filename does not end in .pub this is added.  If the filename
        is omitted, the default_ID_file is used.

01/06/2019

use ssh-keyscan to get remote machine ecdsa identity, you can put this item into local known_hosts file, so when first time ssh login, there is no prompt to input yes:

Actually better to use -o StrictHostKeyChecking=no flag.

01/08/2019

I create a new cluster with the same master hostname as the deleted one, so when I try to ssh to it, interesting thing happens:

go to ~/.ssh/known_hosts file and delete the corresponding ECDSA line

01/22/2019

when you first time ssh or scp, sftp to remote machine, it will prompt to add remote machine to ~/.ssh/known_hosts file, this may interrupt ansible or shell script running, so I want to skip it. For example:

use -o StrictHostKeyChecking=no option, it will silently add remote host name to ~/.ssh/known_host file.

1
2
ssh-copy-id -i .ssh/id_dsa.pub -o StrictHostKeyChecking=no root@example.com
scp -o StrictHostKeyChecking=no -r ./source root@example.com:~

if you don’t want to add the host name, -o UserKnownHostsFile=/dev/null option can save you.

01/23/2019

scp or ssh without prompt input password

1
2
3
yum install -y sshpass
# Explicitly input password
sshpass -p <password> scp/ssh ...

It’s useful to set password-less at first time, combine all of these, no prompt will show up:

1
sshpass -p <password> ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no ...

02/21/2019

scp source directory and it’s content recursively to root user home directory in example.com.

1
scp -o StrictHostKeyChecking=no -r ~/source root@example.com:~

scp all files in source directory to target directory in example.com.

1
scp -o StrictHostKeyChecking=no ./source/* root@example.com:~/target

03/11/2019

The ssh command has -i option, you associate private key with this flags:

1
ssh -i ~/.ssh/id_rsa xxx

Note that SSH never send private key over the network, -i merely used to answer challenge that is generated using the corresponding public key from target machine, you don’t need to explicitly use -i if you use default private key in right location.

03/12/2019

If public key is lost, you can use existing private key to generate one:

1
ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

Or just create new key pair

1
echo "yes" | ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

09/05/2020

sftp server is up when install openssh-server package, and can be configured in /etc/ssh/sshd_config file: https://www.techrepublic.com/article/how-to-set-up-an-sftp-server-on-linux/

For the interactive commands, see man sftp, there are logs of regular commands can be used in sftp, for example: cd, chmod, ln, rm, etc.

The online free ftp server for testing purpose. Also, the mmnt.net can be used to find free ftp servers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Use password or ssh-public key to login to sftp server
sftp -o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
demo@test.rebex.net[:path]

# print local working directory, the default place to hold download file
sftp> lpwd
# change local working directory
sftp> lcd [path]
# escape to local shell, type `exit` to back
sftp> !
# secape to local command
sftp> ![command]

# enable/disable progress meter
sftp> progress
# download file to local working directory
sftp> get <filename>
# download file to specified directory
sftp> get <filename> <local file path>

# upload file
# default file is in local working directory and upload to sftp current folder
# if no path is specified
sftp> put [local file path] [remote file path]

# quit sftp
sftp> bye

For non-interactive download/upload file:

1
2
3
4
5
# download
sftp user@hostname[:path] <local file path>
# upload, tricky
echo "put <local file path>" | sftp user@hostname[:path]
sftp user@hostname[:path] <<< $'put <local file path>'

Used in shell script:

1
2
3
4
5
6
sftp user@hostname <<EOF
cd /xxx/yyy/zzz
cd /aaa/bbb/ccc
put file.tgz
bye
EOF

01/20/2021

When the network connection is poor and ruining your SSH session, you can adjust the connection settings with larger interval probing and retry times:

1
2
3
4
5
6
7
8
9
10
11
12
Host myhostshortcut
    User XXX
    Port 22
    # or ip address
    HostName myhost.com
    User barthelemy
    # no-op probe to server interval 60s
    ServerAliveInterval 60
    # probe conunt max 10 times if noresponse
    ServerAliveCountMax 10
    # no tcp no-op probe
    TCPKeepAlive no

08/13/2021

When I ran git pull on Gitlab local repo from my cloudtop, I got error output:

1
2
3
4
5
6
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This was resolved by adding IdentitiesOnly yes in ssh config file under gitlab config block, which instructs ssh to only use the authentication identity files specified on the command line or the configured in the ssh_config file.

Reference: https://www.tecmint.com/fix-ssh-too-many-authentication-failures-error/

03/17/2022

The Ansible playbook failed to ssh target VM and reported error:

1
2
3
4
5
module.backups-v2.null_resource.setup_cluster (local-exec): [WARNING]: Unhandled
error in Python interpreter discovery for host
module.backups-v2.null_resource.setup_cluster (local-exec): 10.10.16.205: Failed
to connect to the host via ssh: Bad owner or permissions
module.backups-v2.null_resource.setup_cluster (local-exec): on /root/.ssh/config

It turns out the permission and owner issue on /root/.ssh/config file, see this ticket for detail, the fix is to set 600 as chmod and chown by owner.

0%