New Line command

Once in a while, I come across a task that i need to perform, may be using a python script and then it amazes me to find a tool built in to Linux / Unix.

I had a series of numbers (stored as a column) representing a physical quantity measured with respect to time. The data was stored as a text file. I had to add the time (in seconds) column before the column in the file. The first row had time of 1 second, the second row had a time of 2 seconds etc. In short, I just needed to add the line number to each row.

Ordinarily, one could use python to open the file, read each row and add a incremental number in front of each row. But with Linux all that I had to do was

nl file1.txt > file2.txt

where 'nl' is the new line command. The file 'file1.txt' contained the one column of numbers, the measured values and 'file2.txt' will contain two columns, the seconds column and the datacolumn.

The new line command can also work across pages, can number headers, footers etc. Please check the man page for more details.

Dec 28, 2009
I found a second method to add line number:

To add line number including blank line cat -n file1.txt > file2.txt. If you wish to add line numbers only for non-blank lines, use cat -b file1.txt > file2.txt

xargs - taking output of one command and making input for another

If you would like to take the output of one command and pass it on to another UNIX program, you are in luck. Like many things in Unix/Linux, there are many ways to perform this operation.

For example if I need to find all jobs that are running in the queue and get complete details, I could perform either

qstat -f `qstat | grep R | awk '{print $1}'`

or

qstat | grep R | awk '{print $1}' | xargs -n1 qstat -f

In the first version, the command within `backticks` is evaluated to obtain the list of jobs that are "running". In this command, 'qstat | grep R' will give a list of rows for jobs running. The awk command splits that string and obtains the list of job names.

In the second version, 'qstat | grep R | awk '{print $1}'' is evaluated and the output is passed to xargs command. Both these commands are equivalent, except that the second option is more robust in handling whitespace and null character in output before it becomes an input to the next command.

We will continue further with the use of xargs and calculate the total time all the currently running jobs would need.

qstat | grep R | awk '{print $1}' | xargs -n1 qstat -f | grep Resource_List.walltime | awk '{split($3,a,":");sum+=a[1]}END{print sum}'

As seen earlier, the output of xargs command gives the input to 'grep Resource_List.walltime' command. The output of this command is the rows that contain information about the wallclock time for each of these jobs. These rows are then parsed to obtain just the 3rd column ($3 in the command) which contains wallclock time formatted as hh:mm:ss.

This string is split across ":" and the first number, namely the hour, is obtained as "a[1]" in the command. This process is repeated for each of the job and at end the total sum is printed.

I will keep posting more of such tidbits in the future ....

Creating module files

A complex program in Linux will generally be installed in multiple locations. For example, a C++ library like Magick++ after installation will require include files, library files and binary files. Each of these is located in different folders and may be in different parent directories aw well.

In one of our installation, Magick++ is installed in /usr/local/magick++/magick++/. Under this directory, the include files are in include, the library files are in lib and binary files are in bin.

For a user defined program using the Magick++ library, the path to all three have to be in their environment variable. This is set using a module file.

A module file is a TCL script. An example of the module file for adding path to magick++ is given below.


#%Module######################################################################
##
## Magick++ Module
##
proc ModulesHelp { } {
puts stderr "\tThis module adds PATH that allow you to compile Magick++
}

set MAGICK_LIB_HOME "/usr/local/magick++/magick++/lib"
set MAGICK_BIN_HOME "/usr/local/magick++/magick++/bin"
set MAGICK_INCLUDE_HOME "/usr/local/magick++/magick++/include"

append-path PATH $MAGICK_BIN_HOME
append-path LD_LIBRARY_PATH $MAGICK_LIB_HOME
append-path LD_INCLUDE_PATH $MAGICK_INCLUDE_HOME


The module file begins with #%Module, which helps to identify a module file. The proc ModulesHelp prints a helpful message whenever "module help magick++" is typed in linux command prompt. The next three lines create variables that store the location of lib, bin and include direcory. Finally these paths are appended to the environment variables PATH, LD_LIBRARY_PATH and LD_INCLUDE_PATH respectively.

To invoke this module file and attach all these path to environment variables, type "module load magick++" or "module add magick++" at the Linux command line.

To unload these path and environment variables, type "module unload magick++"

In addition to appending path, we can also prepend paths, set and unset environment variables, set and unset aliases etc. Refer to the manpage for more details.