Joe Poniatowski

I recently ran a training session teaching some Configuration Management (CM) personnel some of the basic UNIX/linux shell commands, along with some of the common ways Secure Shell (SSH) utilities can be used to move application code around during deployments. I created an outline for the class, which is reproduced below. In no way is this outline a complete reference for using shell commands and SSH for CM, but it introduces some of the basic utilities and commands that can be a part of a comprehensive CM architecture.

I) bash – a *nix shell
A) A shell is a command-line interface to an OS. There are lots of shells available in *nix (korne, bourne, ‘C’…). bash tries to include the best features of each. Shells are related to DOS.
B) cygwin makes it work in Windows, along with most other ‘POSIX’ compliant programs & utilities (including OpenSSH).
C) Some common shell commands (all of these work in the other shells as well):

i) cd : Change working directory.

(a) cd : By itself, cd puts you in your own home directory.
(b) cd /tmp/ftp_files : puts you in the /tmp/ftp_files directory. The leading ‘/’ means ‘start at the root or base of the file system, and traverse from there.
(c) cd myfiles : puts you in a subdirectory from your current location called myfiles. You could be anywhere in the file system – this form of the command will only look there for the named sub-directory.

ii) ls: List files.

(a) ls : lists files in current directory
(b) ls /usr/bin : lists files in sub-directory /usr/bin
(c) ls –l : lists files in ‘long’ format, showing owner, permissions, sizes, etc.
(d) ls BAM* : Lists all files in current directory whose names start ‘BAM’. The ‘*’ is a wild-card.

iii) cp : Copies files from place to place, optionally with new name.

(a) cp thisfile.txt thatfile.txt : makes copy of thisfile.txt with name thatfile.txt.
(b) cp /tmp/sales.wks /home/jp : Copies file named sales.wks from directory /tmp to directory /home/jp (assuming this directory exists).
(c) cp /var/news/daily/* ~ : Copies all files (using a wild-card again) from directory /var/news/daily to the user’s home directory. The ‘~’ by itself means current user’s home.

iv) grep : Matches a string with some source of text, often the contents of a file.

(a) grep error: *.log : Searches all files in the current directory whose names end in .log for any lines containing the text “error:”. If it finds any, it lists the file name along with the actual line of matching text.
(b) grep –i virus ~brian/* : Searches all the files in user Brian’s home directory for any file which includes the term “virus”. The ‘-i’ switch makes the search case-insensitive. The ‘~’ followed immediately by a user name is short-hand meaning the named user’s home directory.

D) Pipes and re-direction

i) ‘>’ redirects standard output., usually to a file but can be to other devices.

(a) ls –l /sales > sales_files.txt : Lists all files in the /sales directory and puts the results in a file in the current directory named sales_files.txt, over-writing the file if it exists.
(b) ls –l /sales >> sales_files.txt : Same as above, but two >>’s together means ‘append to’ instead of ‘over-write’. If the output file exists, this will stick the output of the ls command at the end of the file.

ii) ‘<’ redirects standard input, often from a file.
iii) '<< word' redirects standard input from whatever text follows until word appears on a line by itself. This usually appears in a shell script:
mailx –s “New Notifications” user1 <<EOF
The process finished normally
EOF The previous commands appearing in a script would cause a mail to be sent to user1, and take the text in between the EOFs as input. ‘mailx’ usually takes input from the keyboard. Note – ‘date’ is a bash command which returns the current date and time. Including anything inside of back-ticks like this tells bash to run the command(s) inside them and put the output in at that location.
iv) ‘|’ is the pipe symbol. It means “take the standard output of one command and make it the standard input of another command.”

(a) ls –l /tmp | grep –i junk : List all files in the temp directory, pipe the output into grep, and show only the lines containing the word ‘junk’ (upper or lower case).

E) Parameters: Inside a shell script, arguments passed in from the command-line are referenced with $1, $2, etc.

F) Permissions: All files in a *nix file system have permissions associated. You can see them with the ‘ls –l’ command. They appear at the beginning of the line for each file listed. The very first character is special. It will be ‘d’ if the file is a sub-directory. Otherwise, it’s usually a ‘-‘. The next 9 characters are actually 3 groups of 3 – one for the file’s owner, one for the file’s group, and one for everyone else. Within each group of 3, the first character is for ‘read’, the second character is for ‘write’, and the 3rd character is for ‘execute’.

i) –rwxr-xr– : This would mean that the file is not a directory, the owner has read, write, and execute permissions, the group the file belongs to (actually, anyone belonging to the file’s group) has read and execute permissions, and everyone else has only read.

ii) chmod is the command to change permissions on a file, which you can generally only do if you are the ‘root’ user (in *nix) or in the ‘Administrators’ group (in Windows).

(a) chmod u+rwx myfile.doc : this would add read, write, and execute permissions to the owner (‘u’ is for ‘user’) of this file.
(b) chmod g-rx,o-rx *.prv : This would remove (note the ‘-‘ instead of ‘+’) read and execute permissions for both group members and everyone else not the owner on all files in the current directory ending in ‘.prv’. Note that the ‘g’ is for group and the ‘o’ is for ‘other’ (not ‘owner’).

(c) There is a numeric form of the chmod command, where the bits in each triad are used to compute a binary number (just FYI).

II) SSH – Secure Shell – in general and as implemented in OpenSSH
A) Uses key pairs for authentication and encryption. You keep tight control of your private key. You distribute your public key. Although they use different implementations, the same underlying technology is used for security in SSH, HTTPS (secure website access), and e-mail. SSH is a communications protocol which encrypts all traffic between a client and a host. The client can be you on your PC, you on a server, or a process running on a PC or server. The host will be some other machine* to which the client connects. SSH has 3 main implementations (there are others, but less frequently used):

i) ssh : This is a terminal emulation program like telnet, which gives you a shell interface (bash in our case) on the target server. Once connected, for all intents and purposes the server acts as if you are logged on to a terminal that is connected directly to the server. The basic sytax looks like this inside a shell like bash:

(a) ssh username@servername : this would log you on the the remote server. If you’re using the same username that you are using on your local system (eg; your PC), and you’ve established keys and set up for automatic authentication, you will not be asked for a password. Otherwise, you’ll be prompted for a password, but it – like all SSH traffic – will be encrypted before being sent over the network (unlike telnet).

(b) ssh –i PrivateKey username@servername : With this form of the command you can automate logging on to the remote host with a different username than you are using locally, and tell ssh where to find the private key to use for authentication. ‘PrivateKey’ in this case refers to a file that contains the remote user’s private key.

(c) ssh joe@myserver.com ls -l : ssh can be used to invoke a command on the remote host. In this example, the command would connect to myserver.com as user joe, and list the files in the user’s default (home) directory. NOTE: you may have to ‘escape’ any special constructs if you want the remote shell to evaluate them, otherwise the local shell will expand them first, then pass the results to the remote shell.

ii) sftp : Secure File Transfer Protocol. This looks very much like ftp and basically uses the same commands, but all traffic is encrypted via SSH. sftp (like ftp) is most suited for interactive file transfer sessions, where you can navigate around on the host, picking and choosing the files you wish to transfer, although it is possible to script sftp sessions.

iii) scp : Secure Copy. This is another method of transferring files. Not as interactive as sftp but more easily scripted. The same authentication mechanisms exist for scp and sftp as for ssh – keys will be used if configured, and the default private key can be over-ridden with the ‘-i’ flag. Note that the sytax is very much like the ‘cp’ shell command mentioned above.

(a) scp ~batchuser/daily* johndoe@server1.domain:/batch/daily_jobs : This will copy all the files from the user ‘batchuser’s home directory whose names start with ‘daily’, log on to remote server ‘server1.domain’ as user ‘johndoe’, and put them in /batch/daily_jobs. Files are moving from the client machine to the remote host in this example.

(b) scp johndoe@server1.domain:/var/log/httpd/error_log . : This will log on to server1.domain, copy the error_log file from directoy /var/log/httpd, and put in in the user’s current directory on the client machine.

(c) Common command-line switches include –p (preserve date/time stamps), and –r (recurse sub-directories when transferring multiple files). In bash, most command-line switches can be combined:

(1) scp –rp /deployments/* cm_user@prod.server.org:/BigApplication/ This will copy all files including any in sub-directories from the /deployments directory to the /BigApplication directory on the target server, preserving any date and time stamps.

(d) If using PuTTY from the command-line, you pre-pend a ‘p’ to the file transfer commands – so ‘sftp’ becomces ‘psftp’, and ‘scp’ becomes ‘pscp’.

B) Creating the keys

i) ssh-keygen –trsa : This will generate a pair of keys that work together. If you accept the defaults, the private key will be called ‘id_rsa’ and the public key will be called ‘id_rsa.pub’, and both will appear in a subdirectory under the user’s home directory called .ssh. NOTE: directories and files beginning with a period in bash and other *nix shells are considered ‘hidden’, and won’t appear in normal file listings. These are the names that SSH will try to use by default. If you name them something else, then you have to use the –i flag on the command line to specify the private key. During creation of the keys, you will be asked to supply a pass phrase. This can be any text, including numbers, most non-special symbols (like ‘#’), and spaces. If you supply one, you’ll have to supply it every time you use your private key for authentication.* If you leave it blank, you won’t have to supply the pass phrase when your keys are accessed. PROs and CONs!
ii) Copy the public key to any host servers you will be connecting to from this box, and append it to the correct ‘authorized_keys’ file on that remote host.

(a) scp .ssh/id_rsa.pub doej@business.server.net:.authorized_keys/my.pub
(b) ssh doej@business.server.net
(c) cd .ssh
(d) cat my.pub >> authorized_keys (the ‘cat’ command dumps the contents of the file, in this case appending it to authorized_keys)
(e) chmod 600 authorized_keys
(f) chmod 700 .
(g) exit (you should now be able to re-connect without being prompted for a password, although you might be prompted for a pass-phrase by ssh)

III) Using SSH with PuTTY: Anyone can install bash and OpenSSH on their Windows workstations by installing cygwin, in which case you don’t need PuTTY. (PROs and CONs). If you do use PuTTY however, there are some things you can set up to enhance the experience, including automatic log-on.

A) Configure PuTTY. When you run it, you’ll see a configuration screen. Along the left side of the window there will be a navigation tree. On the main screen, under ‘Host Name’ enter the DNS name or IP address of the remote server you want to connect to. In the ‘Saved Sessions’ field, enter a short name for this connection, and click on the [Save] button on the right. Don’t use spaces in this field to make things easier later. This is all you need for a connection, but you will be prompted for a username and password when you click on the [Open] button at the bottom.

i) With PuTTY running (the configuration screen), select one of your saved sessions and click on the [Load] button. In the navigation tree, under ‘Connection’ you’ll see an entry for ‘Data’ – click once on that and fill in your user name for the remote host in the ‘Auto-login username’ field. Click on ‘Session’ at the top of the navigation tree, and then click the [Save] button again. Now if you click [Open], you won’t be prompted for a user name, but you’ll still be prompted for a password.

ii) Generate keys. In the folder where the PuTTY executable is located, there is another executable called PUTTYGEN.EXE. Run that program, and click on the [Generate] button. Move the mouse pointer around on the window to generate some randomness, which puttygen will use to create the keys. When it’s done you’ll see something like this:
iii) If you want to use a pass phrase, enter it into the ‘Key passphrase’ and ‘Confirm passphrase’ fields.
iv) Save the public and private keys by clicking on the corresponding buttons. I tend to put them in a separate folder. Use names that describe the connection(s) you will use these keys with.
v) Select all the text in the field marked ‘Public key for pasting into OpenSSH authorized_keys file.’ Right-click and select ‘Copy’ to put this text in your clipboard.
vi) Back in the PuTTY configuration screen, load your saved connection (if not already loaded), then in the navigation tree select Connection->SSH->Auth.
vii) Click the [Browse] button, and navigate to the place where you saved your private key generated previously, select the key and click [Open].
viii) In the navigation tree, scroll up and pick ‘Session’, then click [Save].
ix) Click [Open] to connect to the remote host. You will still need your password at this point.
x) cd .ssh
xi) cat >>authorized_keys
xii) click the right mouse button. This will paste the public key text onto the screen. Hit , then -D (control-D) . This will append the text to the ‘authorized_keys’ file.
xiii) chmod 600 authorized_keys
xiv) chmod 700 .
xv) exit (You should now be able to run PuTTY again, load and open your saved session, and not be prompted for a password. You may be prompted by PuTTY for a pass phrase.)

B) Using pscp and psftp (PuTTY’s implementations of scp and sftp). These run in a DOS command window. These work just like their OpenSSH counterparts, but can take advantage of the configuration settings stored in PuTTY. Basically, on the command line, substitute the ‘Saved Sessions’ name from PuTTY for the server name. So if you have a PuTTY session named ‘BuildServer’, these examples would use the same keys and other settings for file transfers:

i) pscp myScript.sh poniatowskij@BuildServer:scripts
ii) pscp poniatowskij@BuildServer:/var/log/errors.txt .

IV) Miscellaneous - if there’s time and interest
A) Pros and Cons of pass phrases. (More secure, less handy for automation and scripting)
B) Authentication Agents (PuTTY offers Pageant). Keep the pass phrase but don’t leave your workstation onlocked.
C) WinSCP – GUI based alternative to pscp and psftp.
D) Numeric mode of ‘chmod’