tag:blogger.com,1999:blog-11490887742161687642024-02-07T20:36:06.214-06:00Images & VisualizationBlog on multiple topics in Image Processing and Visualization and anything else not connected.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.comBlogger36125tag:blogger.com,1999:blog-1149088774216168764.post-12927799100678641132014-05-10T13:50:00.001-05:002014-05-10T13:54:04.372-05:00My new textbook and job<div dir="ltr" style="text-align: left;" trbidi="on">
So many things have happened in the last few months and I have not had time to add new entries to my blog. The first is the textbook that I co-authored with Sridevi Pudipeddi which was released at the end of February. It was marathon run to complete and get the book ready for publishing. I also moved to a new job in California in April.<br />
<br />
<br />
During my work as image processing consultant at the Minnesota Supercomputing Institute, I have worked with students in various disciplines of science. In all these cases, images were acquired using x-ray, CT, MRI, Electron microscope and Optical microscope. It is important that the students have knowledge of both the physical methods of obtaining images and the analytical processing methods to understand the science behind the images. Thus, a course in image acquisition and processing has broad appeal across the STEM disciplines and is useful for transforming undergraduate and graduate curriculum to better prepare students for their future.<br />
<br />
There are books that discusses image acquisition alone and there are books on image processing alone. The image processing algorithms depend on the image acquisition method. We wrote a book that discusses both, so that students can learn from one source. You can check out sample chapter of the book at <a href="http://www.reedwith.us/content/be34053073" target="_blank">reedwith.us</a>. You can buy the book at Amazon by clicking on the image below.<br />
<br />
<div style="text-align: center;">
<a href="http://www.amazon.com/Processing-Acquisition-Mathematical-Computational-Sciences/dp/1466583754/ref=la_B00H3V8T9S_1_1/188-5982510-3365956?s=books&ie=UTF8&qid=1399744879&sr=1-1" target="_blank"><img alt="" src="http://ecx.images-amazon.com/images/I/51eU5virDaL.jpg" width="200px" /></a>
</div>
<br />
I also changed job. I started working as Senior Engineer at Elekta in Sunnyvale, CA. I will be focussing mostly on x-ray and CT during my tenure at Elekta.</div>
Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-76142498695189509812013-09-09T10:08:00.003-05:002013-09-09T10:13:50.303-05:00Installing large number of packages in R - method 2<div dir="ltr" style="text-align: left;" trbidi="on">
In one my <a href="http://imageviz.blogspot.com/2012/10/installing-large-number-of-packages-in-r.html" target="_blank">previous blog</a>, I discussed a method for installing multiple R packages from one version in another. In the post, I used a combination of R and Python. In this post, I will present a method that uses only R. <br />
<br />
There are two functions in the file (listed below), listmultipack and installmultipack. You can copy the content of the listing below and name it 'packageinstall.r'.<br />
<br />
The function, listmultipack reads the list of all packages in a given version and writes them to a file. The default file name is 'requirements.txt'. The name has been chosen to follow the Python convention. Alternately, you can name it to any other file name.<br />
<br />
Once the new R version is installed on the same machine or a different machine, the installmultipack function can be used to install all the packages in the 'requirements.txt' or the file name that you chose. Finally the function prints all warnings that were generated during the installation.<br />
<br />
To use the file, at the R command prompt, you need to load the file using<br />
<br />
<i>>> source("packageinstall.r")</i><br />
<i><br /></i>
To obtain the list of packages on to file 'mylist.txt', type<br />
<i><br /></i>
<i>>> </i>listmultipack('<i>mylist.txt</i>')<br />
<br />
To install all the packages in the mylist.txt file, type<br />
<br />
<i>>> installmultipack('mylist.txt')</i><br />
<i><br /></i>
<br />
<script src="https://gist.github.com/anonymous/6496774.js"></script><br />
<br />
Alternately, if you do not have a previous installation of R and would like to install multiple R packages using one command, you can also create a requirements file by listing the package names in a text file. Each package name should be in a line by itself. <br />
<div class="p2">
<br /></div>
</div>
Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-70850270596628191512013-02-14T16:31:00.002-06:002013-02-15T11:44:18.905-06:00Checkpointing in Python<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
Scientific programs are known to be computationally expensive. They generally require significant amount of time for processing. For example, a program might run for several hours or days. It is not always possible to guarantee that the machine in which the program is running will be available to the user over a long period. In such cases, Checkpointing can be used to store the state of the program at different times, so that the program can be restarted without the need to restart the computation from the beginning. In this post, we will discuss a method for checkpointing Python programs.</div>
<div>
<br /></div>
<div>
The first program shown below performs the addition of all numbers from 1 to 200 and prints the sum. This program might look too simplistic for scientific computation but it provides a good platform for discussing checkpointing. Instead of computing the sum from 1 to 200, imagine computing the same to a really high value such as trillions. In such case, the computation might run for hours. If the program is interrupted, the computation needs to be started from the beginning.</div>
<br />
<script src="https://gist.github.com/anonymous/4956752.js"></script>The second program is the check pointed version of the first program. In checkpointing, the current state of the program is stored as a file. Whenever a program runs, it looks for the checkpoint file, so that it can restore the program state. If it does not exist, the program assumes that it is being run for for the first time. During the computation, the program outputs its state to a file at regular interval. The exact time interval is dependent on the program being solved. If the program runs successfully, the content of the checkpoint file is no longer needed and hence removed from the disk.<script src="https://gist.github.com/anonymous/4956761.js"></script><br />
<br />
In line 7-12, the program checks if the checkpoint exist and if it does, reads its content and stores it in th<b>e </b>variable start and total_sum. If the checkpoint file does not exist, it applies a default value. In the process of computation, the current state of the program is output to the checkpoint file (checkpt_file) every 5th iteration. The sleep statement is added to slow the execution. In this program, the state of the program is stored at every 5th iteration. If the program successfully completes, the checkpoint file is removed using os.unlink method (line 28).<br />
<br />
The checkpoint file is written and read using Python's pickle. Thus, any Python datatype that can be pickled can be stored. Alternate formats such as hdf5, csv, xls etc can also be used to store the file. pickle was chosen as it is built in to Python and also due to the data stored being a picklable dictionary.<br />
<br />
The image below is a snapshot after running the second program. The program was interrupted at the 7th iteration. When the program was restarted, using the command line, it begins with iteration 6 as the state of the program up to iteration 5 was stored in the checkpoint file.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnWU2yaeU5NlLo7g335GiG6Op98X3EBx9QX_q4szWz6BGkDwPLDCl09t5mUewHO9ZfyprrfRg_rBmCE9DTsXQSe3vKrfYdvTchYvmaFjb-9UGD7skjPxaj7Dzcdred0pNNvnj5K7tNjhE/s1600/Screen+shot+2013-02-14+at+3.46.10+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="313" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnWU2yaeU5NlLo7g335GiG6Op98X3EBx9QX_q4szWz6BGkDwPLDCl09t5mUewHO9ZfyprrfRg_rBmCE9DTsXQSe3vKrfYdvTchYvmaFjb-9UGD7skjPxaj7Dzcdred0pNNvnj5K7tNjhE/s400/Screen+shot+2013-02-14+at+3.46.10+PM.png" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br /></div>
Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-68042869054475118082012-10-12T09:18:00.000-05:002012-10-12T15:43:32.347-05:00Installing large number of packages in R<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<br />
I have been learning R, more specifically install and maintain R for users. Recently, I had to install the latest version 2.15.1 from source. Once it was installed, the next step was to install the packages from the older version (2.15.0) in the latest version. Although installing packages in R is as simple as invoking the function install.packages(), it quickly becomes cumbersome when you have to install more than 400 packages. <br />
<br />
Instead I resorted to a combination of R and Python to complete this process.<br />
<br />
First, determine the list of all packages in the older version of R using the following commands<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">packs <- installed.packages()</span><br />
<span style="font-family: Courier New, Courier, monospace;">exc <- names(packs[,'Package'])</span><br />
<br />
Here, exc contains a column of package names.<br />
<br />
Then, store this list in a text file, 'test.Rdata', so that it can be processed using Python.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">write(exc,'test.Rdata')</span><br />
<br />
To install, more than one package, a command similar to the one below can be used. This command will install the three packages, BiocInstaller, coda and DEGseq. However, the aim of this blog post is to describe a method to install much more than these three packages.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">install.packages(c("BiocInstaller", "coda", "DEGseq"),dependencies=TRUE) </span><br />
<br />
The Python script that generates this command is given below. This program reads the column of data and concatenates them after adding " and , as appropriate. <br />
<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">fp = open("test.RData","r")</span><br />
<span style="font-family: Courier New, Courier, monospace;">s = 'install.packages(c('</span><br />
<span style="font-family: Courier New, Courier, monospace;">for i in fp.readlines():</span><br />
<span style="font-family: Courier New, Courier, monospace;"> s = s+'"'+i.strip()+'",'</span><br />
<span style="font-family: Courier New, Courier, monospace;">s = s+'),dependencies=TRUE)'</span><br />
<span style="font-family: Courier New, Courier, monospace;">print s</span><br />
<br />
<br />
Finally, copy the output of the Python program in to the R command line and wait a few hours to finish the installation. <br />
<br />
<b>I am assuming this is a common problem. How do you handle it? You can give your advice in the form of comments.</b><br />
<br />
PS: The concatenation can be performed using any other scripting language like perl, php, bash etc.</div>
Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-23861866831763181572012-03-03T18:46:00.001-06:002012-03-14T14:24:32.992-05:00Comparing Matlab and Python for large image processing problems<div dir="ltr" style="text-align: left;" trbidi="on">
Matlab is a popular scripting language for numerical computing. It is popular and powerful due to its various toolbox. Since Matlab is developed commercially, dedicated programmers are working in adding new features and enhancing the existing ones all the time. Large image data sets created using latest imaging modalities typically take a long time to process. The typical approach is to run the processing in parallel on many cores either on a desktop, a cluster or a supercomputer. Commercial software like Matlab require dedicated license such as Distributed Computing Server for running such parallel jobs, which has an added cost. <br />
<br />
Graphical Processing Unit (GPU) programming is becoming popular and easier than ever. The license cost for GPU programming in Matlab is typically lower than that for parallel programming. GPU programming is useful, if there is large amount of computation and fewer data transfer. This limit exists because of smaller bandwidth between the CPU and GPU, the typical path that data has to take for processing in the GPU. Large data set image processing typically involves large amount of I/O and data transfer between CPU and GPU and hence may not provide enough scalability.<br />
<br />
<b>Advantage Python</b><br />
<br />
Python is a free and open source scripting language. It can be scaled to large number of processors. It has been shown that there is not a significant difference in computational time for a program written in Matlab vs the one written in <a href="https://modelingguru.nasa.gov/docs/DOC-1762">python using numpy</a>. The author found that numpy run time is in the same order of magnitude as Fortan and C programs with optimization enabled. Although, the processing time are for slightly older versions of the various languages, my experience has shown that the range of processing time remains similar. The processing time will also vary depending on the nature of the algorithm and the expertise of the programmer.<br />
<br />
Python has parallel processing modules like <a href="http://mpi4py.scipy.org/">mpi4py</a>, that allow scaling the application to large number of cores. In a supercomputing environment with large number of cores, python can be run on many of the cores. Matlab on the other hand can only scale to the extent of the number of license. <br />
<br />
Python also has GPU programming capability through <a href="http://mathema.tician.de/software/pycuda">pycuda</a>, in case if GPU programming suits the application.<br />
<br />
Python is a more general purpose language compared to Matlab. Hence integrating various databases, servers, file handling, string handling in to the program is easy. <br />
<br />
<b>Disadvantage Python</b><br />
<br />
Since Python is an open-source package, it does not have a wide variety of canned functions and toolboxes as Matlab. Hence, the user has to work on developing some of them. I hope that over time this issue will be resolved.<br />
<br /></div>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0Minneapolis, MN 55421, USA45.0563001 -93.259945945.0338666 -93.299427900000012 45.0787336 -93.2204639tag:blogger.com,1999:blog-1149088774216168764.post-87012421830535638662012-02-17T19:14:00.001-06:002012-02-17T19:21:39.954-06:00Under-graduate education in Image processing<div dir="ltr" style="text-align: left;" trbidi="on">
<b><span style="font-size: large;">SURVEY URL: <a href="http://goo.gl/ORDDz"> http://goo.gl/ORDDz</a></span></b><br />
<br />
<div class="MsoNormal" style="text-align: justify;">
Image acquisition and processing
have become a standard method for qualifying and quantifying experimental
measurements in various Science Technology Engineering and Mathematics (STEM)
disciplines. Discoveries have been
made possible in medical sciences by advances in diagnostic imaging such as
x-ray based computed tomography (CT) and magnetic resonance imaging (MRI). Biological and cellular functions have
been revealed with new imaging techniques in light based microscopy. Advancements in material sciences have
been aided by electron microscopy analysis of nanoparticles. All these examples and many
more require both knowledge of the physical methods to obtain images and the
analytical processing methods to understand the science behind the images. </div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Imaging technology continues to advance
with new modalities and methods available to students and researchers in STEM
disciplines. Thus, a course in image acquisition and processing would have
broad appeal across the STEM disciplines and be useful for transforming
undergraduate and graduate curriculum to better prepare students for their
future.</div>
<div class="MsoNormal" style="text-align: justify;">
<br />
<div class="MsoNormal">
Image analysis is an
extraordinarily practical technique that need not be limited to highly
analytical individuals with a math and engineering background. Since researchers in biology, medicine,
and chemistry along with students and scientists from mathematics, physics and
various engineering fields use these techniques regularly; there is a need for
a course that provides a gradual introduction to both acquisition and processing. </div>
<br /></div>
Such a course will prepare students in the common image acquisition techniques like CT, MRI, light microscope and electron microscope. It will also introduce the practical aspects of image acquisition such as noise, resolution etc. The students will also be programming image processing using<a href="http://python.org/"> Python </a>as a part of the curriculum. They will be introduced to python modules such as <a href="http://numpy.scipy.org/">numpy</a> and <a href="http://scipy.org/">scipy</a>. They will learn the various image processing operations like segmentation, morphological operations, measurements and visualization.<br />
<br />
We wanted our understanding of this need with the data collected from surveying people interested in image processing or people who wish that they had such a course during their senior year in under-graduate or in graduate school. We created a <a href="http://goo.gl/ORDDz">survey</a> to obtain your feedback. It will take only a minute of your time. We request that you fill as much information as you can. Please forward this URL or this blogpost to your friends as well.<br />
<br />
<br /></div>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0Minneapolis, MN 55421, USA45.0563001 -93.259945945.0338666 -93.299427900000012 45.0787336 -93.2204639tag:blogger.com,1999:blog-1149088774216168764.post-25946327321302395192012-02-12T21:37:00.003-06:002012-02-13T10:34:23.199-06:00Python modules for scientific image processing<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><br />
<div>
<span id="internal-source-marker_0.9473421797156334" style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Recently my friend, Nick Labello working at University of Chicago performed a large scale, large data image processing. It took one week to process the data acquired over 12 hours. The program was written in Matlab and was run on a desktop. If the processing time needs to be reduced, the best course of action is to parallelize the program and run it on multiple cores / nodes. Matlab parallelization can be expensive as it is closed-source commercial software. Python on the other hand is free and open-source and hence can be parallelized to large number of cores. Python also has parallelization modules like mpi4py that eases the task. With a clear choice of programming language, Nick worked on evaluating the various python modules. The chosen python module was used to rewrite the Matlab program. Specifically, he reviewed the following</span><br /><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<ol style="font-weight: bold;"><span id="internal-source-marker_0.9473421797156334" style="font-family: Arial, Helvetica, sans-serif;">
<li style="color: #222222; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><span style="text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Numpy</span></li>
<li style="color: #222222; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><span style="text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Scipy</span></li>
<li style="color: #222222; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><span style="white-space: pre-wrap;">Pymorph</span></li>
<li style="color: #222222; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><span style="text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Mahotas</span></li>
<li style="color: #222222; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><span style="text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Scikits-image </span></li>
<li style="color: #222222; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><span style="text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Python Imaging Library (PIL)</span></li>
</span></ol>
<span id="internal-source-marker_0.9473421797156334" style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Nick’s view on the various modules along with some of my view is given below. </span><br /><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #222222; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b><a href="http://numpy.scipy.org/">Numpy</a></b></span></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It </span><span style="color: #222222; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">do</span><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">esn't actually give you any image processing capabilities, but all of the image processing libraries rely on Numpy arrays to store the image data. As a happy result, it is trivial to bounce between all of the different image processing libraries in the code because they all read and write the same datatype. </span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222;"><span style="white-space: pre-wrap;"><br /></span></span><a href="http://scipy.org/" style="font-weight: bold; white-space: pre-wrap;">Scipy NDImage </a><b style="color: #222222; white-space: pre-wrap;"> </b><span style="white-space: pre-wrap;"><b></b></span></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;">It provides a stable, solid, easy to use very basic functionality (erosion, dilation, etc..). It is missing a lot of the more advanced functions you might find in Matlab such as functions to find endpoints, perimeter pixels, etc. These must be pieced together from elementary operations. The documentation for NDImage is NOT very good for new users. I had a hard time with it at first. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #222222; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="http://www.mmorph.com/pymorph/">PyMorph</a></span><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="http://luispedro.org/software/pymorph"> </a> </span></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It has lots of image processing functions that do not rely on anything but Numpy. It does not depend on Scipy. It provides a python only library. A second interesting thing about PyMorph is that it has quite a lot of functionality compared to the other libraries. Unfortunately, since it is written in Python, it is hundreds of times slower than Scipy and the other libraries. This will become an issue for advanced image processing users. </span><br /><a href="http://luispedro.org/software/pymorph" style="font-weight: bold;"><span style="color: #1155cc; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span></a><br /><a href="http://luispedro.org/software/pymorph" style="font-weight: bold;"><span style="color: #1155cc; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span></a><br /><span style="color: #222222; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="http://luispedro.org/software/mahotas">Mahotas</a> </span></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 18px; text-align: -webkit-auto; white-space: pre-wrap;">It provides only the most basic functions, but blazing fast, and, in my experience, twice as fast as the equivalent functions in Scipy.</span><br />
<a href="http://luispedro.org/software/mahotas" style="font-weight: bold;"><span style="color: #1155cc; font-family: Arial, Helvetica, sans-serif; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span></a><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="http://scikits-image.org/">Scikits-Image</a></span><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It picks up where Scipy leaves off and offers quite a few functions not found in Scipy.</span><br /><a href="http://scikits-image.org/" style="font-weight: bold;"><span style="color: #1155cc; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;"></span></a><br /><span style="color: #222222; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="http://www.pythonware.com/products/pil/">Python Imaging Library</a> (PIL) </span><span style="color: #222222; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: #222222; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is free to use but is not open-source. It has very few image processing algorithms and not necessarily useful for scientific imaging.</span><br />
<span style="color: #222222;"><span style="font-family: Arial, Helvetica, sans-serif; white-space: pre-wrap;"><br /></span></span><br />
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="http://enthought.com/products/epd.php">Enthought</a> </span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is __not__ an image processing library. It is a python distribution ready for scientific programming. It has over 100 scientific packages pre-installed, and is compiled against fast MKL/BLAS. My tests were a little bit faster in Enthought-python than the regular Python by 2-3%. Your mileage may vary. The big advantage, though, is that it has everything I need except Mahotas and PyMorph. Also, Mahotas installs easily on Enthought, whereas it was difficult on the original python installation, due to various dependencies. Enthought is free for students and employees of Universities and Colleges. Others need to pay for the service.</span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><i>Ravi's view: </i> Personally, I do not have much problem installing python modules to regular python interpreter. The only time it becomes difficult is installing scientific packages like Scipy that have numerous dependencies like Boost libraries. Readymade binary packages do eliminate the installation steps but are not necessarily well tuned for optimal performance. Enthought is a great alternative that is as easy to install and yet optimized for performance.</span></div>
</div>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-70573390510941187852012-02-08T22:58:00.001-06:002012-02-13T09:27:10.444-06:00Plotting three variable graph using Matlab<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcP9JKiVnqZnPyNAzyjR2Q8hoG6yGi-QADCr2mVW9T13xy1aag60rsBh-raYDeJAIFqdNuenhDOVwcoGZhhO4aERdv7CKrqfYUQOhvK3zPF_4n3mWh3IJMrobsthFtBZ2gaxf-BHxKvFo/s1600/Picture+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
<span style="font-family: inherit;">Recently,</span> a user wanted to visualize the effect of four different test conditions causing changes in three different parameters. This visualization will help understand the effect of change in one parameter on others.<br />
<br />
The user suggested to plot the three parameters along three different axis. For example, the three parameters with values of [95.0, 1.2, 4.5] will correspond to the co-ordinates of [95.0, 0.0, 0.0], [0, 1.2, 0] and [0, 0, 4.5]. Using these coordinates, one can form a triangle. The shape of the triangle will be different for the various test conditions and it will be easier to visualize the effect of the test conditions on the parameters. <br />
<br />
I was not initially sure whether I could accomplish this using a standard Matlab plots. I did a search on google but was not successful, as I did not have a good search term. I resorted to creating a OpenGL program using GL_TRIANGLES. I later found that Matlab has similar functionality. Triangles and other polygons can be easily constructed in Matlab using the "<a href="http://www.mathworks.com/help/techdoc/ref/patch.html">patch</a>" function.<br />
<br />
<b><i>The program</i></b><br />
<br />
The vals contains the value that needs to be plotted. Each column is one test condition and the rows contain the parameters that need to be plotted along the axis. The "for loop" runs for each column and creates the x, y and z coordinates and stores them in a, b and c. The patch command will create a triangle using the three coordinates. The last parameter in the patch command is the color of the patch. By default, patch is rendered with all surface opaque and the color specified in the patch command. Since there are too many surfaces, the patch was made transparent and the edges were given different line style and thickness using "plottools". The resulting plot can be seen below.<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">% The variables vals has three rows and four columns. The columns contain co-ordinate values along x,y,z axis respectively. The four rows will result in four triangular surfaces.</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">vals = [11.11,3.55,4.97,2.14;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> 2.61,4.63,4.53,4.66;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> 91.2,79,93.4,56.3];</span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">figure;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">hold on;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">for i = 1:size(vals,2) % For each column in vals</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> a = [vals(1,i) 0 0]</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> b = [0 vals(2,i) 0]</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> c = [0 0 vals(3,i)]</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> patch(a,b,c,10*i)</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">end</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">grid on;</span></blockquote>
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisih7_DOijuNQiLWCN0xkFPzsSzBOsCe9TEMgrWYmqAwaOCRSPzXonutkSn2HQ4djPW8SKpAVGnzZTeCvC7qcq3uVfxwl54xRR84t6zT95DvvlwfAFnWz2G36m-R2Pj7qlITUPxCr3sZk/s1600/blogplot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="395" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisih7_DOijuNQiLWCN0xkFPzsSzBOsCe9TEMgrWYmqAwaOCRSPzXonutkSn2HQ4djPW8SKpAVGnzZTeCvC7qcq3uVfxwl54xRR84t6zT95DvvlwfAFnWz2G36m-R2Pj7qlITUPxCr3sZk/s400/blogplot.png" width="400" /></a></div>
<br /></div>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0Minneapolis, MN 55421, USA45.0563001 -93.259945945.0338666 -93.299427900000012 45.0787336 -93.2204639tag:blogger.com,1999:blog-1149088774216168764.post-62302703279696636452010-10-22T14:23:00.023-05:002010-10-22T21:13:02.940-05:00Calculating the number of repeating objects in an imageRecently I was asked to help solve a problem determining the number of seats in an airline. An example of such an airline layout is shown below (click to view the bigger image.)<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgycdHxJsESMgL_jCEw_0SXofnrl-NPePkU1krNg9sQc9bvcbuvMw8pqvckGIewFo4JFEROdrx3KYsRy0-kuOBWRQheSdRteeTMrO_LJ38s5C0hEvUIztYz6Y47n8c6FlphNEU1MXpcxo4/s1600/airline_seating.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 70px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgycdHxJsESMgL_jCEw_0SXofnrl-NPePkU1krNg9sQc9bvcbuvMw8pqvckGIewFo4JFEROdrx3KYsRy0-kuOBWRQheSdRteeTMrO_LJ38s5C0hEvUIztYz6Y47n8c6FlphNEU1MXpcxo4/s400/airline_seating.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5530956170467145810" /></a><br /><br />I decided to use the method I knew best, Cross Correlation. The idea is to cross correlate the template of the seat (i.e., an image of a single seat) with every pixel in the airline layout image. The template (coordinate origin is its center) is moved over to a particular pixel in the image. The cross correlation coefficient is calculated and the value is used as the intensity of a new image. This is repeated by moving the template to every pixel in the airline layout image. The pixels for which the template matches perfectly with the airline layout image, will have the correlation coefficient close to 1. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6UpL_XyCtyf4h8DmCXZzMVIgJgziKzIVgmNR6izPOubFRHu8SxU738iD-E17q-rIx7s4JlHTAnE0SfHo6GhW0hek93yfRJWJZP9IsgRxbJf5eOmGiYOHECn3wnXr0iqtE_3GFe2I1iKs/s1600/template.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 48px; height: 45px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6UpL_XyCtyf4h8DmCXZzMVIgJgziKzIVgmNR6izPOubFRHu8SxU738iD-E17q-rIx7s4JlHTAnE0SfHo6GhW0hek93yfRJWJZP9IsgRxbJf5eOmGiYOHECn3wnXr0iqtE_3GFe2I1iKs/s400/template.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5530956110219946578" /></a><br />Above: A hard to see template of the seat.<br /><br />The results of the cross correlation is shown below (click to view the bigger image.)<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG2cvXyhkNcqtHgnU4CtTqkVr0-xpaMHm8WdBQu-0tZOdUalU5j_COOX24stXobK2Zjyy22jLakwxfE2mbniqoYxTy1cXAxNdo_J-KsuFZHTVWmMDJVlvZT6zMoGDz6aysCMq25YyqJ0c/s1600/correlated_image.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 123px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG2cvXyhkNcqtHgnU4CtTqkVr0-xpaMHm8WdBQu-0tZOdUalU5j_COOX24stXobK2Zjyy22jLakwxfE2mbniqoYxTy1cXAxNdo_J-KsuFZHTVWmMDJVlvZT6zMoGDz6aysCMq25YyqJ0c/s400/correlated_image.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5530956023730202386" /></a><br />The bright spots in this image are the points with the highest correlation. It then becomes a simple process of segmenting the high intensity pixel. <br /><br />To perform these operations, my natural choice was Matlab and its Image processing toolbox. <br /><code><br />im = imread('airline_seating.jpg');<br />im = rgb2gray(im);<br /><br />im_template = imread('template.jpg');<br /> im_template= rgb2gray(im_template);<br /><br />C = normxcorr2(im_template, im);<br />C1 = C>0.7;<br /><br />stat = regionprops(C1);<br />noofseats = size(stat,1);<br />disp(['Number of seats = ',num2str(noofseats)]);<br /></code><br /><br />In the first 4 lines of code, we read the airline layout image and the template image. To obtain the correlation image, I did not have to write my own correlation function instead Matlab has one already ready to be used. This function, normxcorr2 needs the airline layout and the template matrix. Once the correlation image is obtained, we segment it based on the logic that any pixel with value more than 0.7 is considered as pixels corresponding to the center of seat. Since the center of the seat did not segment as a single pixel, I could not count the number of pixels as the number of seats. Instead I calculated the number of regions using regionprops and store it as a structure. The number of elements of the structure is the number of seats.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-6041518768862535202010-10-06T13:22:00.010-05:002010-10-17T21:09:04.487-05:00Spamming of HTML forms - one caseRecently I found that a newspaper in its online edition switched from image based <a href="http://en.wikipedia.org/wiki/CAPTCHA">CAPTCHA system</a> to solving a mathematical puzzle, in-order to prevent spamming of their comment section using a computer program. A screen capture of the same can be found below.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAc7JLABm2T-BVBVflvVHgAIy9gL4EUZI24WExsGS2821l1M_OHPixnfvD4K5Ws_qmTP9OTB1B9U2qoLyStwLOHYfZNoseZUXKWW31gLcDxkh80gzG-6xuBaUDBLF3v86HU5x8qp_Q28E/s1600/Picture+1.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 222px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAc7JLABm2T-BVBVflvVHgAIy9gL4EUZI24WExsGS2821l1M_OHPixnfvD4K5Ws_qmTP9OTB1B9U2qoLyStwLOHYfZNoseZUXKWW31gLcDxkh80gzG-6xuBaUDBLF3v86HU5x8qp_Q28E/s400/Picture+1.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5525000283202249538" /></a><br /><br />The problem with such a system is that they can be easily solved using a computer, which defeats the purpose of using it to differentiate human and computers apart. To test my own skill, I wanted to write a program that can download the page, read it and solve the puzzle as well. Using the information I obtain, I could then post comments without human intervention. <br /><br />To accomplish this task, I used the usual suspects like python and and the HTML parser, <a href="http://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a>. BeautifulSoup reads a string of html or xml and converts it to a tree. Using the tree, it is easy to navigate through the tags or search for a particular one based on id or name. It is also powerful enough to differentiate tags based on CSS class in html tags.<br /><br /><code><br />1. import urllib<br />2. from BeautifulSoup import BeautifulSoup<br />3. import string,re<br /><br />4. doc = urllib.urlopen('http://www.somesight.com/comment/reply/1854565').read()<br />5. soup = BeautifulSoup(''.join(doc))<br /><br />6. a = soup.findAll("span",{"class":"field-prefix"})<br />7. b = a[0].contents[0].split("=")[0].split("+")<br />8. c = [int(bs) for bs in b]<br />9. captcha_response = sum(c)<br />10. print a,captcha_response<br /><br />11. token1 = soup.findAll("input",id="edit-captcha-token")<br />12. token1_val = token1[0]['value']<br />13. print token1,token1_val<br /></code><br /><br />The two important information that I need to calculate are the captcha_response which is the solution to the mathematical problem and the captcha_token, a hidden html field in the webpage. Line #6 searches for a class, field-prefix in span tag. This tag contains the string for the mathematical puzzle that needs to be solved. I obtain the contents of this string and split it in-order to obtain the individual numbers in a list. Finally I convert those numbers from string to integer in Line #8 and sum them using line #9. <br /><br />Line #11 searches the hidden captcha token, stored in the input tag with id="edit-captcha-token".<br /><br />Armed with these two information, we can post any name and comment to the form. The comments were moderated but it would still require lot of human intervention to clear the spams. <br /><br />I informed the webmaster of this issue. They have since moved to a image based system. I removed all reference to the site in this blog post and program in-order to keep their anonymity.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-72409432811935372242010-09-21T14:16:00.026-05:002010-09-22T10:00:08.952-05:00Ajax in djangoAsynchronous Javascript and XML (Ajax) has become the most important tool for web developers in improving the look and feel of their website. A normal HTML request is sent by a client with request for a particular page. The server prepares the page and sends it to the client browser that renders the page. This approach is still valid for most pages. But the problem is that the whole page gets rendered every time the server processes a request. This might be acceptable for some websites but sites like google maps, stock ticker sites would rather modify a small portion of the page rather than reload them completely. Ajax plays an important role in such applications.<br /><br />Django provides support for Ajax and my google-fu turned up a few of them that were really good. The problem I had with them was that the sites focused more on how syntactically Ajax can be executed in Django rather than the overall principle. So, in this post I will focus just on the basic principles of Ajax in Django. The anatomy of a typical django request can be summarized by the image below. The request comes in from the client, the view function processes it and the response is returned using a call to HttpResponse or render_to_response etc. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcgbcR8CMpFlRy99pw2UwoEcjCyDZw1ERZ80W1urYtvNXuSazZqlNJyMz645ay_YQKIoy8Ljz2d57HF5iQCnerJhB5apKDIDfkmi7XKJY_bpoavd8XsGR3keEq2iji7N5ln-pEHK9vjZ0/s1600/Picture+1.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 368px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcgbcR8CMpFlRy99pw2UwoEcjCyDZw1ERZ80W1urYtvNXuSazZqlNJyMz645ay_YQKIoy8Ljz2d57HF5iQCnerJhB5apKDIDfkmi7XKJY_bpoavd8XsGR3keEq2iji7N5ln-pEHK9vjZ0/s400/Picture+1.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5519454077391765282" /></a><br /><br /><span style="font-weight:bold;">In a normal django request, the response is in the form of HTML or CSV or some such file. Ajax needs JSON or XML instead. Hence django is almost ready for Ajax. The only extra step is the need to serialize the data in to JSON or XML format. Django makes it easy by providing serializers that are builtin. </span><br /><br />During the process of serializing, django takes two approaches. <span style="font-weight:bold;">The first approach is for serializing list of django model objects and the second approach is for everything else.</span> The everything else could include any dictionary of values that you create but need to pass through serialization. Lets look at each approach separately. Both approach are serialized using simplejson built in to django. In the former, django provides a separate module and hence you do not have to make an explicit simplejson call. In the latter, you need to make explicit calls to simplejson functions. <br /><br />In the first approach, we pass the list of django model objects to the serialize function in the serializers module. <br /><br /><pre><code><br /> from django.core import serializers<br /> <br /> def search(request, query):<br /> return serializers.serialize("xml",User.objects.filter(lastname__startswith=query))<br /></code></pre><br /><br />In the second approach, we need to make a explicit call to simplejson. In the program below, we first load the simplejson module. We then create our dictionary using the values, number and image_loc. We then return the json value using simplejson.dumps() function and django's HttpResponse function. The javascript on the django template will then modify the HTML page according to the values in the json object. If you use the first approach to the dictionary that you created, it will give an error, "'str' object has no attribute '_meta'"<br /><br /><br /><pre><code><br />from django.utils import simplejson<br /><br />def servevalues(request):<br /> number = 10000<br /> image_loc = 'http://www.google.com/images/logos/ps_logo2.png'<br /> data = {"number":number,"image_loc":image_loc}<br /> return HttpResponse(simplejson.dumps(data),mimetype='application/javascript')<br /></code></pre><br /><br /><br />Remember:<br />1. Ajax calls to django are very similar to http request calls. Both return some form of textual information. Hence syntactically they look alike<br /><br />2. There are 2 approach to serializing. The first approach is for django model objects and the second approach is for any other object / dictionary that you create. In either case, the code that you have to write is minimal and simple, the django way!Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-84479861478988562352010-07-16T11:34:00.011-05:002010-10-05T09:46:01.594-05:00Keep tab on items you lent using borrow-err.comOn my free time, I have been working on a small but useful and interesting project. It resulted in the site <a href="http://www.borrow-err.com/">http://www.borrow-err.com/</a><br /><br />The reason for creating the site was my forgetfulness. I lend books to others and I forget about it. So, I decided to create a website where I can keep track of items I lent to others. If you would like to try it, just key in the details of the items and the name of the borrower in to the home page. The website will then send you a reminder email every month for the items you lent. If you are lucky and you get your items back, you can remove them from the list using the links provided in the email. <br /><br />You can access it using any browser. The page is also light enough that it loads fine in mobile browsers as well. So whether you are at home, office or on road, you can use borrow-err.com to keep track of items that you lent. Try it and give me your feedback.<br /><br />Remember: <span style="font-style:italic;">borrower's err, so you need borrow-err.com</span><br /><br />October 5th, 2010: <span style="font-style:italic;">The previous version had the bare minimum styling. Hence I made some changes and I believe the new version is more pleasing to your eye.</span>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-61836989684788930332009-11-21T10:48:00.006-06:002009-11-24T13:18:00.449-06:00Did you know that ... Part 21. Digital Imaging received this years Nobel Prize in Physics. Willard<br />Boyle and George Smith received their recognition for work on Charge<br />Couple Devices (CCD) used in range of scientific instruments including the telescope.<br /><br />2. There have been many Nobel Prize awarded to people who used imaging as one of their tools of investigation<br /><br />3. Hough Transform used in the detection of geometrical shapes was initially <a href="http://www.google.com/patents?vid=3069654">patented</a> by Paul Hough in 1962<br /><br />4. <a href="http://stardust.jpl.nasa.gov/tech/aerogel.html">NASA Aerogel</a> was used to collect the interstellar and comet particles. The aerogel slows the particles otherwise traveling at high velocity to a halt without heating. NASA collected images of these tracks along which the particle translated and requested help from public in determining the location of them.<br /><br />5. <a href="http://www.openshotvideo.com/">Open shot</a> is a free non-linear movie editor under Linux. You can clip, resize and trim movies, add transition between scenes, change speed of clips and even "<a href=http://en.wikipedia.org/wiki/Ken_Burns_Effect>Ken Burns effect</a>".Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-51523449799527275082009-07-08T09:43:00.003-05:002009-07-08T10:24:55.881-05:00Matlab alternativesMatlab is a powerful programming language for scientific computing. The popularity of Matlab stems from the fact that it is a high level programming language focusing more on faster development and less on syntax. It also has many tool boxes that extend the functionality that is offered by basic Matlab on to other scientific and engineering domains. In spite of all these advantage, it comes with a price, the fact that we need to pay for purchasing, installing and maintaining it. Although the price we pay might be easily offset by the benefits, many Matlab like software are in existence and perform similar even if not the same set of functionality. In this blog, we will look at some of the alternatives. In all these alternatives, the program written to be run on Matlab can be run directly, within the constrains of functionality offered by that particular software. <br /><br /><span style="font-weight:bold;">FreeMat</span><br /><a href="http://freemat.sourceforge.net/">FreeMat</a> is a open source software released under GPL. It is available for most of the common OS like Windows, Linux and Mac. Installing the software on a Windows machine was a piece of cake. Its repertoire of functionality is large but does not have many of the toolboxes offered by Matlab.<br /><br /><span style="font-weight:bold;">Octave</span><br /><a href="http://www.gnu.org/software/octave/">Octave</a> is another software available under GPL. The syntax of Octave commands are similar to Matlab (<a href="http://www.gnu.org/software/octave/FAQ.html#MATLAB-compatibility">and not the same</a>) and is also highly extensible using C, C++ and Fortran. Unlike FreeMat, Octave comes with many toolboxes for Image processing, Signal processing, Statistics etc. Installing the software on a Windows machine using <a href="http://www.cygwin.com/">cygwin</a> is easy but installing from source on Linux machine requires expertise due to many dependencies. <br /><br /><span style="font-weight:bold;">Numpy and Scipy </span><br /><a href="http://numpy.scipy.org/">Numpy and Scipy</a> are Python modules that can be used to perform high level scientific computation like Matlab. Since these modules are based in python, their syntax is not similar to Matlab but due to its focus on high level programming, the syntax is generally simple. With installation of dependencies like python imaging library (PIL), we can perform image and signal processing. Scipy can also be used to perform optimization and statistics. Prebuilt binary packages are easy to install but installation from source is cumbersome due to its dependencies.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-24465392397913734012009-02-19T13:19:00.015-06:002009-03-03T16:43:50.909-06:00Python and AbaqusRecently I had the opportunity to work with a student who needed to perform a Finite Element Analysis on roughly 400 files using Abaqus. Processing such large number of images using the graphical user interface would have been impractical. We were happy to learn that Abaqus has a python scripting module which could help us with the automation.<br /><br />We set about writing the python program from the individual function calls given in their manuals. But considering the scope of the software and its complexity, the approach quickly became difficult. Instead, we resorted to creating "macros" and modifying it to our purpose.<br /><br /><span style="font-weight: bold; color: rgb(102, 0, 0);">Creating Macros</span><br /><br />The macros in Abaqus lets you perform a series of operations and record them as python scripts. The scripts by default are stored in "abaqusMacros.py" with each macro recorded as a function. Since the macro was created for a particular model, we modified the names in the function " Entire_Work_Flow" to be generic so that other models can be loaded. We then added other functions that will call the function created using macro.<br /><br />The other function created were "getvalues" that obtains the relavant von mises stress values from the ODB file. The main function reads each of the solid model (.sat file) in a given directory and passes the filename to the macro function. It then calls getvalues function and stores the result in a CSV file for further analysis.<br /><br />To run the script, type "abaqus cae -noGUI scriptname.py".<br /><br />The Entire_Work_Flow has been trimmed to show only the relevant lines that corresponds to creation of the name of parts, instance and job name.<br /><br /><object id="_ds_4637202" name="_ds_4637202" type="application/x-shockwave-flash" data="http://viewer.docstoc.com/" width="670" height="550"><param name="FlashVars" value="doc_id=4637202&mem_id=14095&doc_type=doc&fullscreen=0"><param name="movie" value="http://viewer.docstoc.com/"><param name="allowScriptAccess" value="always"><param name="allowFullScreen" value="true"></object><br /><span style="font-size:78%;"><a href="http://www.blogger.com/docs/4637202/Abaqus_python">Abaqus_python</a> - <a href="http://www.docstoc.com/">Free Legal Forms</a></span>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com1tag:blogger.com,1999:blog-1149088774216168764.post-21590265667580307982009-01-26T16:04:00.014-06:002009-02-03T10:35:25.617-06:00New Line commandOnce in a while, I come across a task that i need to perform, may be using a python script and then it amazes me to find a tool built in to Linux / Unix.<br /><br />I had a series of numbers (stored as a column) representing a physical quantity measured with respect to time. The data was stored as a text file. I had to add the time (in seconds) column before the column in the file. The first row had time of 1 second, the second row had a time of 2 seconds etc. In short, I just needed to add the line number to each row.<br /><br />Ordinarily, one could use python to open the file, read each row and add a incremental number in front of each row. But with Linux all that I had to do was <br /><br /><code>nl file1.txt > file2.txt</code><br /><br />where 'nl' is the new line command. The file 'file1.txt' contained the one column of numbers, the measured values and 'file2.txt' will contain two columns, the seconds column and the datacolumn. <br /><br />The new line command can also work across pages, can number headers, footers etc. Please check the <a href="http://linux.die.net/man/1/nl">man page</a> for more details.<br /><br /><span style="font-style:italic;">Dec 28, 2009</span><br />I found a second method to add line number:<br /><br />To add line number including blank line <code>cat -n file1.txt > file2.txt</code>. If you wish to add line numbers only for non-blank lines, use <code>cat -b file1.txt > file2.txt</code>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-59848880482061612542009-01-23T09:52:00.007-06:002009-01-23T15:20:10.100-06:00xargs - taking output of one command and making input for anotherIf you would like to take the output of one command and pass it on to another UNIX program, you are in luck. Like many things in Unix/Linux, there are many ways to perform this operation. <br /><br />For example if I need to find all jobs that are running in the queue and get complete details, I could perform either<br /><br /><code>qstat -f `qstat | grep R | awk '{print $1}'` </code> <br /><br />or <br /><br /><code>qstat | grep R | awk '{print $1}' | xargs -n1 qstat -f </code><br /><br />In the first version, the command within `backticks` is evaluated to obtain the list of jobs that are "running". In this command, 'qstat | grep R' will give a list of rows for jobs running. The awk command splits that string and obtains the list of job names. <br /><br />In the second version, 'qstat | grep R | awk '{print $1}'' is evaluated and the output is passed to xargs command. Both these commands are equivalent, except that the second option is more robust in handling whitespace and null character in output before it becomes an input to the next command.<br /><br />We will continue further with the use of xargs and calculate the total time all the currently running jobs would need. <br /><br /><code>qstat | grep R | awk '{print $1}' | xargs -n1 qstat -f | grep Resource_List.walltime | awk '{split($3,a,":");sum+=a[1]}END{print sum}'</code><br /><br />As seen earlier, the output of xargs command gives the input to 'grep Resource_List.walltime' command. The output of this command is the rows that contain information about the wallclock time for each of these jobs. These rows are then parsed to obtain just the 3rd column ($3 in the command) which contains wallclock time formatted as hh:mm:ss. <br /><br />This string is split across ":" and the first number, namely the hour, is obtained as "a[1]" in the command. This process is repeated for each of the job and at end the total sum is printed. <br /><br />I will keep posting more of such tidbits in the future ....Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-84842988551147538052009-01-22T08:53:00.007-06:002009-01-22T11:09:44.829-06:00Creating module filesA complex program in Linux will generally be installed in multiple locations. For example, a C++ library like Magick++ after installation will require include files, library files and binary files. Each of these is located in different folders and may be in different parent directories aw well. <br /><br />In one of our installation, Magick++ is installed in /usr/local/magick++/magick++/. Under this directory, the include files are in <span style="font-style:italic;">include</span>, the library files are in <span style="font-style:italic;">lib</span> and binary files are in <span style="font-style:italic;">bin</span>.<br /><br />For a user defined program using the Magick++ library, the path to all three have to be in their environment variable. This is set using a module file.<br /><br />A module file is a TCL script. An example of the module file for adding path to magick++ is given below. <br /><br /><code><br />#%Module######################################################################<br />##<br />## Magick++ Module<br />##<br />proc ModulesHelp { } {<br /> puts stderr "\tThis module adds PATH that allow you to compile Magick++<br />}<br /><br />set MAGICK_LIB_HOME "/usr/local/magick++/magick++/lib"<br />set MAGICK_BIN_HOME "/usr/local/magick++/magick++/bin"<br />set MAGICK_INCLUDE_HOME "/usr/local/magick++/magick++/include"<br /><br />append-path PATH $MAGICK_BIN_HOME<br />append-path LD_LIBRARY_PATH $MAGICK_LIB_HOME<br />append-path LD_INCLUDE_PATH $MAGICK_INCLUDE_HOME<br /></code><br /><br />The module file begins with #%Module, which helps to identify a module file. The proc ModulesHelp prints a helpful message whenever "<span style="font-style:italic;">module help magick++</span>" is typed in linux command prompt. The next three lines create variables that store the location of lib, bin and include direcory. Finally these paths are appended to the environment variables PATH, LD_LIBRARY_PATH and LD_INCLUDE_PATH respectively.<br /><br />To invoke this module file and attach all these path to environment variables, type "<span style="font-style:italic;">module load magick++</span>" or "<span style="font-style:italic;">module add magick++</span>" at the Linux command line.<br /><br />To unload these path and environment variables, type "<span style="font-style:italic;">module unload magick++</span>"<br /><br />In addition to appending path, we can also prepend paths, set and unset environment variables, set and unset aliases etc. Refer to the <a href="http://modules.sourceforge.net/man/modulefile4.html">manpage</a> for more details.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-88152692552816804152009-01-22T08:30:00.008-06:002009-01-22T08:53:31.377-06:00Configuring Make fileAny of us who have used Linux will eventually end up installing softwares from source. The most common method for installing softwares written is C, C++, Fortran etc is using Makefile. <br /><br />The Makefile contains the list of commands that will be used to create the various libraries, binaries etc. <br /><br />In the most simplest of the scenario, the installation will involve<br /><br /><span style="font-style:italic;">./configure<br />make<br />make install<br />make clean<br /></span><br /><br />The first step prepares the makefile with all relavant configuration depending on the system on which it is being installed. This could include, the location where the files will be stored after installation, the type of CPU etc. The second step "make" compiles and "make install" builds and places the program in the appropriate locations. "make clean" clears any temporary files that have been created. <br /><br />Depending on the different scenarios and type of software being installed, different configuration may have to be set. In the example below, we will configure the installation of magick++ (a C++ library for ImageMagick) so that it is installed in /usr/local/magick++/magick++/ instead of the default location /usr/local/<br /><br /><span style="font-style:italic;">./configure --with-quantum-depth=8 --prefix=/usr/local/magick++/magick++/ --exec-prefix=/usr/local/magick++/magick++/<br /></span><br />--prefix is the location where the lib and include files will be stored after compiling. If not specified, it will be assumed as /usr/local<br /><br />--exec-prefix is the location where bin files will be located. <br /><br />By default, --prefix = --exec-prefix<br /><br />There are many other configuration parameters that can be set, which again depends on the software. I will keep posting more such configuration.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-80226182697698716232009-01-03T17:38:00.025-06:002009-01-03T22:42:49.104-06:00Spell checker in Unicode using PythonI have a project where I had to perform spell check on characters recognized using an optical character recognition program (OCR). My first choice was to search for an existing program written preferably in python, my favorite choice for such work. You can download the complete file <a href="http://www.msi.umn.edu/%7Echityala/spellcheck/spellchecker.py">here</a>.<br /><br />Amazingly I found this work by <a href="http://norvig.com/spell-correct.html">Peter Norvig </a>. It was very well documented and well written piece of code.<br /><br />But I had few issues that I needed to fix and so I could not use it directly.<br /><br />1. In my program, unicode characters need to defined as the default character for all input and output unlike peter's program which works on ascii.<br /><br />This is performed in the following code<br /><br /><br />#!/usr/bin/python -Wall<br /><font style="color: rgb(255, 0, 0);"># -*- coding: utf-8 -*- </font><br /><br />import re, collections, pprint,os<br /><red style="color: rgb(255, 0, 0);">import sys </red><br />import codecs<br /><br />if __name__ == '__main__':<br />...<br /><font style="color: rgb(255, 0, 0);">reload(sys)</font> <font style="color: rgb(255, 0, 0);"> sys.setdefaultencoding('iso8859-1') </font><br /></code><br /><br />2. The list of alphabets will also include the unicode characters applicable in my situation like <br /><font style="color: rgb(255, 0, 0);">alphabet = u'abcdefghijklmnopqrstuvwxyzàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿß'</font><br /><br />3. The unicode feature of python is smart enough to recognise the right characters for conversion from upper case to lower case. All that needs to be done is to call the .lower() function on any unicode characters in the following function.<br /><br /><br /><font style="color: rgb(255, 0, 0);">def words(text):</font> <font style="color: rgb(255, 0, 0);">return re.findall(u'[abcdefghijklmnopqrstuvwxyzàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿß]+',text.lower())</font><br /><br /><br />4. Peter's program trains different words by determining the probability of its occurence. In simple terms, it counts the number of time a word appears in a standard piece of text. The larger the piece of text, the more representative it is to the real world. This scenario was not true in my case, as I do not have a piece of text where a word gets repeated multiple times.<br /><br />In my case, I have a list of words in a text file. Almost all the word gets repeated only once and not any more. So the rank of a word was not in frequency but its ordinality.<br /><br />The ord function in python returns the unicode position of a character input. In the function below, I first determine the ordinality of each word in the possible candidates (i.e., the original set of words). Then the ordinality of the word to be spell checked is also found. The difference between the two ordinalities is determined and the location of the lowest value gives the location of the correct word in the candidates.<br /><br /><font style="color: rgb(255, 0, 0);"><br />def best_candidate(candidates,word):<br /> clist = list(candidates)<br /> #Find ordinality for the complete list<br /> so = []<br /> for cl in clist:<br /> sum_ord = 0<br /> for c in cl:<br /> sum_ord = sum_ord+ord(c)<br /> so.append(sum_ord)<br /> <br /> #Find ordinality of the given word<br /> sum_ord = 0<br /> for c in word:<br /> sum_ord = sum_ord+ord(c)<br /><p>#Find difference in ordinality and also lowest value location<br /> so_item_l = []<br /> for so_item in so:<br /> so_item_l.append(abs(so_item-sum_ord))<br /> min_loc = so_item_l.index(min(so_item_l))<br /> <br /> return clist[min_loc]<br /></font><br /><br /><br /><br /></codealphabet>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-58186684706672633292008-07-08T10:12:00.008-05:002008-07-08T10:21:22.048-05:00Convert a series of Matlab figure files in to jpgThis program converts a set of fig files located in a directory in to jpg files. The program runs through the directory and gets a list of fig file names. These files are then read using the ‘openfig’ command and the output is written using ‘saveas’ command. For example, a file named ‘image.fig’ will be read and saved as ‘image.fig.jpg’. The only variable that needs to be changed is the figdirectory<p></p> <span style="font-weight: bold;">Bonus: </span>You can use the parsing directory structure to read any kind of files located in a directory.<br /><br /><p><br /><code><br />%%Matlab program - fig2jpg<br />figdirectory = 'C:\Documents and Settings\username\Desktop'<br /><br />fullpath = sprintf('%s/*.fig',figdirectory)<br />d = dir(fullpath);<br />length_d = length(d)<br />if(length_d == 0)<br /> disp('couldnt read the directory details\n');<br /> disp('check if your files are in correct directory\n');<br />end<br /><br />startfig = 1<br />endfig = length_d<br /><br />for i = startfig:endfig<br /> fname = d(i).name;<br /> fname_input = sprintf('%s/%s',figdirectory,fname)<br /> fname_output = sprintf('%s/%s.jpg',figdirectory,fname)<br /> saveas(openfig(fname_input),fname_output,'jpg');<br />endRavihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com5tag:blogger.com,1999:blog-1149088774216168764.post-89989564582332778702008-05-13T21:53:00.003-05:002008-07-08T10:16:40.857-05:00Setting automatic jobs in LinuxI was working on setting up a cron job for monitoring the machines under my control and so the natural thing was to set a cron job in Linux. I was new to <code>crontab</code> command in Linux but as you will soon see, the command is very simple and hence should be easy to master. There are some nuances that I learnt during the course of the work and I will discuss it at the end.<br /><br />The crontab command begins with setting the time, date at which a particular job needs to be run. These parameters are set using a crontab file, which can be edited by calling the command <code style="font-style: italic;">crontab -l</code>. In the editor, type the following command<br /><br /><span style="font-style: italic;">5 * * * * command-to-execute > logfile.log </span><br /><br />The number 5 represents the minute at which the command has to be executed. The subsequent values represents the hour, day, month and day of the week (Sunday = 0) respectively. The value * represents all the possible combination. The > pipes the output of the command to a logfile. Once the crontab file has been set, it can be verified using <code>crontab -l</code>. The crontab file can also be written to a text file and loaded using <code>crontab cronfile.txt</code>. The content of the cronfile.txt is same as above.<br /><br />If you wish that you receive a mail every time the cron runs, add the following line first<br /><br />MAILTO: johndoe@email.com (Of course, replace it with the correct email address)<br /><br />So far, everything is easy and good but be careful with the following,<br /><br />1.Ensure that there is an empty line after the last line of crontab file.<br />2.For some reason, crontab cannot use the PATH stored by Linux. So ensure that all commands in the crontab file and all the commands and scripts called by crontab are referred with full path.<br /> <br /> So instead of calling<br /> <code style="font-style: italic;">* 2 * * * python /path/to/file/pythonfile.py</code><code style="font-style: italic;"><br /></code><code><br />call</code><code style="font-style: italic;"><br /> * 2 * * * /usr/bin/python /path/to/file/pythonfile.py</code><br /><br />Also make sure that all the path in the python file are also referred using full path.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-47791037310729600292008-04-09T15:37:00.001-05:002008-04-12T23:58:37.586-05:00Optical Character Recoginition (OCR) using TesseractRecently I had a very interesting project from client who wanted to extract the text (in my case numbers) embedded on thousands of images. My natural choice was to use Tesseract, as it can be scripted and applied to many images in sequenence. <br /><br />Tesseract is an OCR software, originally developed by Hewlett Packard and currently developed by Google. It is a open source software released under Apache license. Since it is open source, you can get your hands on it and install it on pretty much any operating system. I installed it on a Windows machine under Cygwin and the installation was a breeze. <br /><br />Tesseract does not have any segmentation methods, no document layout and can only output the recognised text to a file. But its accuracy is good enough for many applications. It was ranked among top 3rd OCR software for the year 1995. Making a call to tesseract is also easy<br /><br /><span style="font-style:italic;">tesseract data.bmp text.dat [-l langid]</span><br /><br />The values within [] are optional. The <span style="font-style:italic;">langid </span>is the the language being recognized. The default language is English. But it also supports French, Italian, German, Spanish. <br /><br />Since Tesseract does not have any segmentation methods to separate the text from background, the user have to apply these methods using other softwares like ImageMagick, ITK etc. The most common segmentation technique for scanned documents is the <span style="font-style:italic;">Local Adaptive Thresholding</span>. It takes in to account the variation in background intensity across the scanned image and thresholds accordingly. But the right technique has to be chosen depending on the type of image being recognized. <br /><br /><span style="font-weight:bold;">Possible additions to Tesseract</span><br />All the text that I needed to decipher in my images were numbers but Tesseract does not have a langid for numbers. Since there are no langid for numbers, Tesseract deciphered some of these numbers as alphabets. If I have time in the future, I will work on creating the langid for numbers as it will be helpful for many people. If you find that it might be helpful for you, I encourage you to create one or contact me and we can work togther.Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com5tag:blogger.com,1999:blog-1149088774216168764.post-72629509540748801392008-03-27T16:06:00.002-05:002008-03-29T23:39:55.312-05:00Summary of all chaptersCurrently I am busy with my other projects and I am not able to continue working on chapters on Draw and Pixel class and also on other useful small projects that you could perform. So meanwhile, I am uploading a file which has all my writing thus far. Enjoy reading.<br /><br /><object width="670" height="550"><param name="movie" value="http://www.docstoc.com/docs/wrapper.ashx?doc_id=429912&swf_url=http%3A//content1.docstoc.com.s3.amazonaws.com/chapter1to6.pdf.swf&enableFullScreen=1"/><param name="allowScriptAccess" value="always" /><param name="allowFullScreen" value="true" /><embed src="http://www.docstoc.com/docs/wrapper.ashx?doc_id=429912&swf_url=http%3A//content1.docstoc.com.s3.amazonaws.com/chapter1to6.pdf.swf&enableFullScreen=1" width="670" height="550" allowScriptAccess="always" allowFullScreen="true" type="application/x-shockwave-flash"></embed></object><br /><font size="1"><a href="http://www.docstoc.com/docs/429912/Magick-chapter-1-to-6">Magick++ chapter 1 to 6 </a> - Get more <a href="http://www.docstoc.com/">free documents</a></font>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0tag:blogger.com,1999:blog-1149088774216168764.post-71130636353454948552008-03-24T08:59:00.001-05:002008-03-27T16:13:11.695-05:00Chapter 6: Image class – Part 2In chapter 4, we looked at the overview of the Image class and discussed a few functions, for filtering images, in detail using some example program. In this chapter, we will discuss a higher order image processing function like segmentation. We will first demonstrate a program for performing adaptive thresholding and then a program for performing fuzzy c mean clustering. Each of these algorithms will be explained and then the program will be illustrated to ensure complete understanding. <br /><br /><br />Continue reading the pdf file ...<br /><br /><object width="670" height="550"><param name="movie" value="http://www.docstoc.com/docs/wrapper.ashx?doc_id=429892&swf_url=http%3A//content1.docstoc.com.s3.amazonaws.com/chapter6%255b2%255d.pdf.swf&enableFullScreen=1"/><param name="allowScriptAccess" value="always" /><param name="allowFullScreen" value="true" /><embed src="http://www.docstoc.com/docs/wrapper.ashx?doc_id=429892&swf_url=http%3A//content1.docstoc.com.s3.amazonaws.com/chapter6%255b2%255d.pdf.swf&enableFullScreen=1" width="670" height="550" allowScriptAccess="always" allowFullScreen="true" type="application/x-shockwave-flash"></embed></object><br /><font size="1"><a href="http://www.docstoc.com/docs/429892/Chapter-6-Image-class-–-Part-2">Chapter 6: Image class – Part 2</a> - Get more <a href="http://www.docstoc.com/">free documents</a></font>Ravihttp://www.blogger.com/profile/13707779317135150795noreply@blogger.com0