New Line command

Once in a while, I come across a task that i need to perform, may be using a python script and then it amazes me to find a tool built in to Linux / Unix.

I had a series of numbers (stored as a column) representing a physical quantity measured with respect to time. The data was stored as a text file. I had to add the time (in seconds) column before the column in the file. The first row had time of 1 second, the second row had a time of 2 seconds etc. In short, I just needed to add the line number to each row.

Ordinarily, one could use python to open the file, read each row and add a incremental number in front of each row. But with Linux all that I had to do was

nl file1.txt > file2.txt

where 'nl' is the new line command. The file 'file1.txt' contained the one column of numbers, the measured values and 'file2.txt' will contain two columns, the seconds column and the datacolumn.

The new line command can also work across pages, can number headers, footers etc. Please check the man page for more details.

Dec 28, 2009
I found a second method to add line number:

To add line number including blank line cat -n file1.txt > file2.txt. If you wish to add line numbers only for non-blank lines, use cat -b file1.txt > file2.txt

xargs - taking output of one command and making input for another

If you would like to take the output of one command and pass it on to another UNIX program, you are in luck. Like many things in Unix/Linux, there are many ways to perform this operation.

For example if I need to find all jobs that are running in the queue and get complete details, I could perform either

qstat -f `qstat | grep R | awk '{print $1}'`

or

qstat | grep R | awk '{print $1}' | xargs -n1 qstat -f

In the first version, the command within `backticks` is evaluated to obtain the list of jobs that are "running". In this command, 'qstat | grep R' will give a list of rows for jobs running. The awk command splits that string and obtains the list of job names.

In the second version, 'qstat | grep R | awk '{print $1}'' is evaluated and the output is passed to xargs command. Both these commands are equivalent, except that the second option is more robust in handling whitespace and null character in output before it becomes an input to the next command.

We will continue further with the use of xargs and calculate the total time all the currently running jobs would need.

qstat | grep R | awk '{print $1}' | xargs -n1 qstat -f | grep Resource_List.walltime | awk '{split($3,a,":");sum+=a[1]}END{print sum}'

As seen earlier, the output of xargs command gives the input to 'grep Resource_List.walltime' command. The output of this command is the rows that contain information about the wallclock time for each of these jobs. These rows are then parsed to obtain just the 3rd column ($3 in the command) which contains wallclock time formatted as hh:mm:ss.

This string is split across ":" and the first number, namely the hour, is obtained as "a[1]" in the command. This process is repeated for each of the job and at end the total sum is printed.

I will keep posting more of such tidbits in the future ....

Configuring Make file

Any of us who have used Linux will eventually end up installing softwares from source. The most common method for installing softwares written is C, C++, Fortran etc is using Makefile.

The Makefile contains the list of commands that will be used to create the various libraries, binaries etc.

In the most simplest of the scenario, the installation will involve

./configure
make
make install
make clean


The first step prepares the makefile with all relavant configuration depending on the system on which it is being installed. This could include, the location where the files will be stored after installation, the type of CPU etc. The second step "make" compiles and "make install" builds and places the program in the appropriate locations. "make clean" clears any temporary files that have been created.

Depending on the different scenarios and type of software being installed, different configuration may have to be set. In the example below, we will configure the installation of magick++ (a C++ library for ImageMagick) so that it is installed in /usr/local/magick++/magick++/ instead of the default location /usr/local/

./configure --with-quantum-depth=8 --prefix=/usr/local/magick++/magick++/ --exec-prefix=/usr/local/magick++/magick++/

--prefix is the location where the lib and include files will be stored after compiling. If not specified, it will be assumed as /usr/local

--exec-prefix is the location where bin files will be located.

By default, --prefix = --exec-prefix

There are many other configuration parameters that can be set, which again depends on the software. I will keep posting more such configuration.

Setting automatic jobs in Linux

I was working on setting up a cron job for monitoring the machines under my control and so the natural thing was to set a cron job in Linux. I was new to crontab command in Linux but as you will soon see, the command is very simple and hence should be easy to master. There are some nuances that I learnt during the course of the work and I will discuss it at the end.

The crontab command begins with setting the time, date at which a particular job needs to be run. These parameters are set using a crontab file, which can be edited by calling the command crontab -l. In the editor, type the following command

5 * * * * command-to-execute > logfile.log

The number 5 represents the minute at which the command has to be executed. The subsequent values represents the hour, day, month and day of the week (Sunday = 0) respectively. The value * represents all the possible combination. The > pipes the output of the command to a logfile. Once the crontab file has been set, it can be verified using crontab -l. The crontab file can also be written to a text file and loaded using crontab cronfile.txt. The content of the cronfile.txt is same as above.

If you wish that you receive a mail every time the cron runs, add the following line first

MAILTO: johndoe@email.com (Of course, replace it with the correct email address)

So far, everything is easy and good but be careful with the following,

1.Ensure that there is an empty line after the last line of crontab file.
2.For some reason, crontab cannot use the PATH stored by Linux. So ensure that all commands in the crontab file and all the commands and scripts called by crontab are referred with full path.

So instead of calling
* 2 * * * python /path/to/file/pythonfile.py

call
* 2 * * * /usr/bin/python /path/to/file/pythonfile.py

Also make sure that all the path in the python file are also referred using full path.