That’s wizardry for you.   How many times have you typed grep ‘^$’?

Do you grep?

grep, the Unix command that searches files for patterns, is one of the most useful Unix utilities.   This example here grep ‘^$’ passes a regular expression with two characters:  a carrot (^) which means the beginning of line, and a dollar sign ($) which means, end of line.  Taken together they say, return me ever blank line – that is, has nothing between beginning and end of line.

Here it is in action.  Show me everything that is meaningful in my apache configuration file – that which is not a comment and is not a blank line.  The -e allows multiple patterns.  The -v of course reverses things, it shows everything not matching the pattern.

grep -ve '^#' -e '^$' apache2.conf | more

vbin has a very useful command called sav.

You always need to make a backup copy of a file before messing with it.  A simple

   $ cp book.xml -p book.xml.sav

will do.  The -p preserves the original date and ownership of the file.

sav does this for you. In other words:

   $ sav book.xml
   cp -p book.xml book.xml.sav

Its that’s simple.  Its just a convenient way to make a quick backup before you edit something.  If you want to see the changes since you backed it up, you can use savdiff:

   $ savdiff book.xml
   diff book.xml.sav book.xml

If you want to revert back you can use unsav:

   $ unsav book.xml
   cp -p book.xml.sav book.xml

vbin is a collection of very useful Unix Bash utilities.

Available on github: https://github.com/dlink/vbin

It is easy to install – See instructions in the README

Here are some more examples of beautiful things you can do in Unix.

1.) Execute an SQL script against a database called db3, translate the output into CSV and report how long it took to execute.

$ time cat usage.sql | db3 |tr '\t' ',' > usage.csv


real 0m0.605s
user 0m0.004s
sys 0m0.001s


2. Compare the output with yesterdays output

$ diff usage_20141022.csv usage.csv > usage.diff

3.) Show a frequency histogram of changes since yesterday on columns 6 and 7.  (See future posts on histogram, and csv )


$ grep '^>' usage.diff |cut -c3- |csv - -c6,7 | histogram

| 172403 CONED,CONEDG
|   5000 NYSEC,NYSECG
|   4308 LPH,LPHG
|     15 NULL,CGO2
|      1 orig_util,new_util

4.) Look for all cases of ‘MATCH’ and for columns containing GAS, ignoring case, display to the screen and also capture to a file.

$ grep -ie match -e ",gas," usage.csv |tee gas_matches.csv


Unix with the Bash shell is beautiful because you can string a list of simple commands together to instruct computers to do complex things.  Folks sometimes refer to this as “sed, grep, awk”.  Those are Unix commands with cryptic names – they sound like mysterious incantations of wizardry.  And in effect they are.

Here are some examples:

1. What users are on this box?

The /etc/passwd file contains a list of all users the box.  To see the last 5 people added to system you can type:

$ cat /etc/passwd | tail – 5

The dollar sign is the prompt (you don’t type that)

the cat command lists out the contents of a file.

Piping that (|) into tail -5 lists the last 5 rows.

The results may look like this:

dlink:x:2000:2000:David Link:/home/dlink:/bin/bash
mysql:x:104:111:MySQL Server,,,:/nonexistent:/bin/false
jzlink:x:2001:2000:Julia Link:/home/jzlink:/bin/bash
ftp:x:105:112:ftp daemon,,,:/srv/ftp:/bin/false
postfix:x:106:113::/var/spool/postfix:/bin/false

This shows  name, x, user id, group id, description, home directory and default shell for each user.  x is a place holder for the password.

The Unix Operating System, born 45 years ago, and the Bash Shell, born 25 years ago – are the two most powerful programming tools ever created.

Unix was the brain child of Ken Thompson and Dennis Ritchie while working in the mid-1960s.  Today most of us use a Linux, Linus Torvalds’ variant of Unix release in 1991.

Bash was developed in 1977.   It is published under the GNU Public license.

Combined, these two pieces of software make the most formidable tool in any developers arsenal.   Unix, is a multi-user, mult-tasking, file system centric core that talks to the computers CPU (its brain), which in turn controls all the computer resources.    Bash is a command line shell wrapped around Unix giving humans the ability to interact with the Unix core in a pseudo English way.

This example, typed after the bash prompt ($) reads ‘List files in long format’, or simply ‘L S minus L’:

$ ls -l

The results might look like this:

drwxr-xr-x 2 dlink dev 4096 Sep 28 20:03 200px
-rw-r–r– 1 dlink dev 230678 Feb 8 2014 Angel.jpg
-rw-r–r– 1 dlink dev 46252 Dec 1 2013 Black_Horse.jpg
-rw-r–r– 1 dlink dev 1379310 Apr 25 13:20 Buddha_watercolor.png
-rw-r–r– 1 dlink dev 22673 Sep 28 19:45 Chacmool.jpg
-rw-r–r– 1 dlink dev 52911 Dec 1 2013 Flowers_3-29-2013.jpeg
-rw-r–r– 1 dlink dev 32078 Sep 28 19:45 Good Nature.jpg

…

The Machintosh Operating System is a Unix variant.   The Windows Operating System is not, and should be avoided for all serious development work.