Outsourced Call Center Solutions

31West Global Services | Blog  ⁄  Data Security  ⁄  How to copy files between Linux Servers using Shell Script?

How to copy files between Linux Servers using Shell Script?


  • How to copy files between Linux Servers using Shell Script?

While there are many ways to do this, the easiest (and most secure) is to setup password-less certificate based ssh access and then use scp or rsync to copy the files.

In this tutorial, we assume you want to copy files from user1 on server1 to user2 on server2.

First, let’s setup up certificate based ssh access so that the shell script can use ssh without requiring passwords.

Step 1:
On server1, check if you already have SSH Keys generated. While logged in as user1, enter the command:

ls -a ~/.ssh/

or as root user, you can use:

ls -a /home/user1/.ssh/

Check if the file “id_rsa.pub” is listed by the commands above (see screenshot below).
If yes, proceed directly to step 3. If not, move to Step 2.

SSH1

Step 2:
So you don’t have the file “id_rsa.pub”. No sweat, we can quickly create it.
Again, logged in as user1, enter the following command:

ssh-keygen -t rsa

The command will ask you for a filename into which the key should be saved. The default location and filename will be shown within braces, which is perfect, so justĀ hit enter to use the default.
Next, the command will ask for a passphrase. We don’t want to use any passphrase, so simply hit enter again (and again for the confirmation)!
You will see output similar to the one shown below, showing which files have been as “Your identification” and “Your public key”.

SSH2

 

Step 3:
You should find two files in the .ssh directory of your home directory that look like

id_rsa

and

id_rsa.pub

We will use the second file.
Use the ‘cat’ command below to output the contents of the public key:

cat ~/.ssh/id_rsa.pub

The command output will look similar to the following (just an example – don’t try matching the key string!):

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2+uXM7uSXqlh3Mz7id9lzrsG7juf5WomsTn+OHGBO3GSAAXkNEGzR+0pxxiRubp5Tn6mJhOOORsff
+kjxfb8bJLZrdFydZ/PK/M1niYvfVzMAUjv/IqiWNPQezrqj6aR5dlBhmwy8GrShuFhPMfGj+nvSSjoUywN7s3IlfVQycBSEgMWrwxbE4tkhT2BzdXWvpaWYZPllzgdhJCEU+aNyNMY5
+5dFwBxsfq5J0xkYyPGpi3BRuEeMFtw9Bzd8s415VM+sVLBVKMBWmH0fUZet+StKMip2dABFFH/kmkNeq0okfwN/RYjt7a1HlOFQCSG4fGFrEhvbfRo/xGTeBQ== user1@server1.31west.net

SSH3

 

Simply copy the output to your clipboard – you will need to paste into a file on server2 as described in the next step.

Step 4:
Logged in as user2 on server2, open the file ~/.ssh/authorized_keys (or /home/user2/.ssh/authorized_keys) using vi or any other file editor.
Then paste (append) the content copied from Step 3 on the last line of this file.

If the directory and/or file does not exist, create it first: the content copied from Step 3 can be pasted into the first line of this file.

Save the file and voila, password-less ssh key based access has been set up!

SSH4

Try it by using the following command (logged in as user1 on server 1)

ssh user2@server2

if server2 is not a DNS searchable domain name, then use the IP address instead, so your command could be:

ssh user2@xx.xx.xx.xx

where xx.xx.xx.xx is the IP address of server2.

Also, if you are running the ssh daemon on server2 on the non-standard port of 22, then you must specify the port to the ssh command. For example, if ssh is running on
port 2323, then the command will be:

ssh -p2323 user2@server2

or:

ssh -p2323 user2@xx.xx.xx.xx

SSH5

 

OKAY, so password-less key based ssh access is working … but how to copy files using a shell script now?
Answer: There are two common methods – using scp (secure copy) or rsync (remote sync)… but which one should you use?

Method A: scp (secure copy)

Advantages:

  • Simple command, easy to use
  • suitable if you need to copy less number of large files as a one-time exercise
  • preferable if you need to copy large number of small files

Disadvantages:

  • not suitable of you make small changes to large files and need to copy them to remote server often
  • not preferable if you want to copy large files to the remote server.

Method B: rsync

Advantages:

  • suitable for copying same files often with minor changes
  • faster copying to remote server as incremental file copy can be used.
  • suitable to sync multiple directories with lots of large files as only the files which have changed since the last copy will be copied again.

Disadvantages:

  • not suitable for small sized files
  • not beneficial over scp if the same files are not required to be copied / updated often.

Let’s try these two methods in shell scripts now….

A: shell script example with scp

#!/bin/bash
#
# All examples below assume that ssh is running on port 2323 on the remote server (server2)
# If ssh is running on standard port 22 on the remote server (server2), then “-P 2323” can be omitted.
#
# replace server2 with FQDN of the remote server or its IP address
#
# the next line will copy one single large file to the remote server
scp -P 2323 ~/mybigfile user2@server2:/path/to/copy/on/remote/mybigfile
#
# the next line will copy two large files to the home directory of user2 on remote server
scp -P 2323 ~/mybigfile ~/mysecondbigfile user2@server2:~/
#
# the next line will copy all files recursively within the “MyLargeDirectory” to the home directory of user2 on remote server
scp -P 2323 -r ~/MyLargeDirectory user2@server2:~/
#
# the -r option specifies that scp should recursively copy entire directories.

The screnshot images below show how the script will quietly copy the files for you !

SSH6

SSH7

SSH8

SSH9
B: shell script example with rsync

#!/bin/bash
#
# All examples below assume that ssh is running on port 2323 on the remote server (server2)
# If ssh is running on standard port 22 on the remote server (server2), then “-p 2323” can be omitted.
#
# replace server2 with FQDN of the remote server or its IP address
#
# the next line will copy one single large file to the remote server
rsync -e “ssh -p 2323” -avz ~/mybigfile user2@server2:/path/to/copy/on/remote/mybigfile
#
# the next line will copy two large files to the home directory of user2 on remote server
rsync -e “ssh -p 2323” -avz ~/mybigfile ~/mysecondbigfile user2@server2:~/
#
# the next line will copy all files recursively within the “MyLargeDirectory” to the home directory of user2 on remote server
rsync -e “ssh -p 2323” -avz ~/MyLargeDirectory user2@server2:~/
#
# rsync options used above are explained as follows:
# -e = specify the remote shell to use (in our example, the remote shell to use is ssh with the port 2323
# -a = archive mode – it means that : (a) files will be copied recursively; (b) symlinks will be copied as symlinks;
# (c) file permissions will be preserved; (d) file times will be preserved;
# (e) file group ownership will be preserved; (f) file ownership will be preserved;
# (h) device files will be preserved.
# -v = verbose output
# -z = use compression to reduce bandwidth usage over the network and speed up the file copying process.
#
# rsync offers many more options which can be useful in different situations.
# Explore the available options of rsync by reading the rsync manual using the command “man rsync”

The screnshot images below show how the script using rsync provide verbose output so you know whats happening at each step of the file copying/incremental copying.

SSH10

SSH11

SSH12

I hope you enjoyed reading this tutorial… until next time… keep learning !



Related Posts