grep ‘^$’

That’s wizardry for you.   How many times have you typed grep ‘^$’?

Do you grep?

grep, the Unix command that searches files for patterns, is one of the most useful Unix utilities.   This example here grep ‘^$’ passes a regular expression with two characters:  a carrot (^) which means the beginning of line, and a dollar sign ($) which means, end of line.  Taken together they say, return me ever blank line – that is, has nothing between beginning and end of line.

Here it is in action.  Show me everything that is meaningful in my apache configuration file – that which is not a comment and is not a blank line.  The -e allows multiple patterns.  The -v of course reverses things, it shows everything not matching the pattern.

grep -ve '^#' -e '^$' apache2.conf | more

 

False Optimization

So often I hear tech folks talking about some implementation as being better than another because “It is more efficient“.   Efficient?   Efficient for who?  The computer?  Are we shaving milliseconds off processing time?   That’s ridiculous.

We should be in the business of optimizing human time, not computer time.   Human time is more expensive – while computer time is cheap.

We want code to be laid out clearly and easy to read.  We want code to be like a poem.  We want something that we can maintain over time without having to scratch our heads every time we come back to.

A simple example, have a look at,

select
   id, process_date, qty
from
   usage
where
   coalesce(units, '') not in ('K1', 'K2', 'K3')
;

What the heck is that coalesce doing in the where clause? If you think about it for a minute it make sense.  Oh, it’s more efficient for the computer that way. Who cares?  How about

select
   id, process_date, qty
from
   usage
where
   units is not null and
   units not in in ('K1', 'K2', 'K3')
;

Here the where clause is immediately easy to understand and to maintain.  It speaks to the essence of the problem in clear pseudo English.

PHP vs Python

$What is $easier to $read?  {A $document of $commands in a $pseudo $English $language $strewed with ($dollar $sign $symbols ($$), $French $Braces ({), and $semi-colons)?  Or a $document $without all $that?}

What is easier to read?  A document of commands in a pseudo English language strewed with dollar sign symbols ($), Flower Braces ({), and semi-colons?  Or a document without all that?

PHP

$logger = new Logger();
if ($error) {
   $logger->info('Problemo');
}

4 lines.  70 characters.

Python

logger = Logger()
if error:
   logger.info('Problemo')

3 lines. 50 characters

sav, savdiff, and unsav

vbin has a very useful command called sav.

You always need to make a backup copy of a file before messing with it.  A simple

   $ cp book.xml -p book.xml.sav

will do.  The -p preserves the original date and ownership of the file.

sav does this for you. In other words:

   $ sav book.xml
   cp -p book.xml book.xml.sav

Its that’s simple.  Its just a convenient way to make a quick backup before you edit something.  If you want to see the changes since you backed it up, you can use savdiff:

   $ savdiff book.xml
   diff book.xml.sav book.xml

If you want to revert back you can use unsav:

   $ unsav book.xml
   cp -p book.xml.sav book.xml

Why is Unix Beautiful 2

Here are some more examples of beautiful things you can do in Unix.

1.) Execute an SQL script against a database called db3, translate the output into CSV and report how long it took to execute.

$ time cat usage.sql | db3 |tr '\t' ',' > usage.csv

real 0m0.605s
user 0m0.004s
sys 0m0.001s

2. Compare the output with yesterdays output

$ diff usage_20141022.csv usage.csv > usage.diff

3.) Show a frequency histogram of changes since yesterday on columns 6 and 7.  (See future posts on histogram, and csv )


$ grep '^>' usage.diff |cut -c3- |csv - -c6,7 | histogram

| 172403 CONED,CONEDG
|   5000 NYSEC,NYSECG
|   4308 LPH,LPHG
|     15 NULL,CGO2
|      1 orig_util,new_util

4.) Look for all cases of ‘MATCH’ and for columns containing GAS, ignoring case, display to the screen and also capture to a file.

$ grep -ie match -e ",gas," usage.csv |tee gas_matches.csv

Why Unix is Beautiful

Unix with the Bash shell is beautiful because you can string a list of simple commands together to instruct computers to do complex things.  Folks sometimes refer to this as “sedgrepawk”.  Those are Unix commands with cryptic names – they sound like mysterious incantations of wizardry.  And in effect they are.

Here are some examples:

1. What users are on this box?

The /etc/passwd file contains a list of all users the box.  To see the last 5 people added to system you can type:

$ cat /etc/passwd | tail – 5

The dollar sign is the prompt (you don’t type that)

the cat command lists out the contents of a file.

Piping that (|) into tail -5 lists the last 5 rows.

The results may look like this:

dlink:x:2000:2000:David Link:/home/dlink:/bin/bash
mysql:x:104:111:MySQL Server,,,:/nonexistent:/bin/false
jzlink:x:2001:2000:Julia Link:/home/jzlink:/bin/bash
ftp:x:105:112:ftp daemon,,,:/srv/ftp:/bin/false
postfix:x:106:113::/var/spool/postfix:/bin/false

This shows  name, x, user id, group id, description, home directory and default shell for each user.  x is a place holder for the password.

What is Unix?

The Unix Operating System, born 45 years ago, and the Bash Shell, born 25 years ago – are the two most powerful programming tools ever created.

Unix was the brain child of Ken Thompson and Dennis Ritchie while working in the mid-1960s.  Today most of us use a Linux, Linus Torvalds’ variant of Unix release in 1991.

Bash was developed in 1977.   It is published under the GNU Public license.

Combined, these two pieces of software make the most formidable tool in any developers arsenal.   Unix, is a multi-user, mult-tasking, file system centric core that talks to the computers CPU (its brain), which in turn controls all the computer resources.    Bash is a command line shell wrapped around Unix giving humans the ability to interact with the Unix core in a pseudo English way.

This example, typed after the bash prompt ($) reads ‘List files in long format’, or simply ‘L S minus L’:

$ ls -l

The results might look like this:

drwxr-xr-x 2 dlink dev 4096 Sep 28 20:03 200px
-rw-r–r– 1 dlink dev 230678 Feb 8 2014 Angel.jpg
-rw-r–r– 1 dlink dev 46252 Dec 1 2013 Black_Horse.jpg
-rw-r–r– 1 dlink dev 1379310 Apr 25 13:20 Buddha_watercolor.png
-rw-r–r– 1 dlink dev 22673 Sep 28 19:45 Chacmool.jpg
-rw-r–r– 1 dlink dev 52911 Dec 1 2013 Flowers_3-29-2013.jpeg
-rw-r–r– 1 dlink dev 32078 Sep 28 19:45 Good Nature.jpg

The Machintosh Operating System is a Unix variant.   The Windows Operating System is not, and should be avoided for all serious development work.

Elegance in Programming

Ask yourself, is it Elegant?   Is it a thing of beauty?

If this is some important piece of code that others will be using then it should be written well.

I read once in an old programming book, “Born to Code in C”, that first you need to make it work, then make it beautiful.   I code by this rule.   So often I find folks don’t spend enough time on the second part.