Am I Autodidact?

Basics: Lucene and Search

2008-05-24T15:09:00.003+05:30

Basics about Lucene ranking -- here

Search Internals by user rainman on wikipedia

This might also be helpful -- a thesis

I/O Redirection in shell or shell scripting

2008-04-06T17:48:00.002+05:30

You will come across this situation when you want to run a process in background. Mostly, you want to capture the output or the errors thrown by the process.

As you must be already knowing, three file descriptors are available by default in any programming language. Similar is the case for a shell or shell scripting. The three are standard input (stdin), standard output (stdout) and standard error (stderr). Their corresponding file descriptors are 0, 1 and 2.

You can use these file descriptors while redirecting your input/output.

Syntax for redirection is as follows

M>N

# "M" is a file descriptor, which defaults to 1, if not explicitly set.
# "N" is a filename.
# File descriptor "M" is redirect to file "N."

Example: To redirect stdout to filename

1>filename

To redirect stderr to errorlog

2>errorlog

Another format is

M>&N

# "M" is a file descriptor, which defaults to 1, if not set.
# "N" is another file descriptor.

Example: To redirect both stderr and stdout to the same file

1>somefile 2>&1

More about I/O redirection here.

MySQL: Purge Master Logs

2008-04-06T15:17:00.002+05:30

Binary logs sometime take up too much space on the server. I wanted space so I had to delete the database logs. They were taking up around 2GB space.

http://dev.mysql.com/doc/refman/5.0/en/purge-master-logs.html --> this link will tell you how to delete the logs.

If you want to turn off the logs, then edit /var/mysql/my.cnf on Ubuntu and comment out the following lines

line starting with log_bin
line starting with expire_log_days
line starting with max_binlog_size

If you want to save space and also want the logs then you can ignore commenting log_bin line and try reducing the values for expire_log_days and max_binlog_size.

expire_log_days will tell you the number of days for which the logs will be kept. If it's value is 2. Then all logs older than 2 days will be deleted.

max_binlog_size is the maximum size of the log file. If a log file reaches this limit, then a new log file will be created and appended to. Note, the old log file will not be deleted unless expire_log_days condition satisfies.

Move running process to background in Linux

2008-04-01T23:35:00.005+05:30

Use the following sequence of keystrokes if you want to move a running process to background in Linux.

CTRL+Z

bg

CTRL+Z will suspend the process and the command prompt will return to you. Type bg and press enter to send the process to background.

If you want to logout but keep the process running even after logging out then type disown -a. This will detach the process and it will continue to run even if you log out.

To start a process in detached mode, use nohup command.

nohup <command to start the process>

man nohup for more details.

Ignoring files in subversion

2008-02-27T22:34:00.003+05:30

Two lessons for today

How to ignore files in subversion
How to create a transcript

1. How to ignore files in subversion

To ignore a directory: svn propedit svn:ignore

The above command will open an editor, add the directory you want to ignore in the editor and exit from the editor. Note that the directory you are trying to ignore is not added to the repository or else it won't work.

To ignore a file: svn propedit svn:ignore

Again in the editor, add the files you want to ignore and exit. You can even use wildcards when you enter the file names. Ex: '*' to ignore all the files in the directory.

I have shown a method to ignore files using propedit. The same can be done using propset also. Details are here and here.

2. How to create a transcript

You will most probably be asked for a transcript while you are asking for help on IRC channels. Transcript is a copy of the text printed on the terminal. Whatever you have typed and whatever has been printed to the terminal by the commands that you executed.

The simplest way to create a transcript in Linux is to run the script command. After running the script command, run/type whatever commands you want to and when done, type 'exit' to stop the script. A file with the name typescript is created which will have the transcript of whatever you have done on the terminal.

Use your GPRS phone as modem in Ubuntu

2008-02-19T17:41:00.001+05:30

'Bunch of me: Linux Redux: Use your GPRS phone as modem in Ubuntu tells how to do it.

Configuring Bluetooth on Ubuntu

2008-02-19T16:56:00.002+05:30

This article on LinuxQuestions.org explains clearly how to configure bluetooth on Ubuntu to send/receive files to/from bluetooth devices and also how to set up bluetooth keyboard and mouse.

Few useful linux commands

2008-02-19T16:48:00.002+05:30

- To see how long your machine has been running
$ uptime
- To see some computer stats
$ lspci
- To see system temperature and battery charge
$ acpi -t
- To see a list of all running processes
$ ps aux
- To view the current date, time and year
$ date
- For a simple calendar
$ cal
- To see what programs are running with path names
$ ps -aux
- To see your current IP address
$ ifconfig -a
- To see what your system is doing at startup
$ dmesg
- To see information about the computer users
$ finger -l

More commands later...

Social Graph API

2008-02-14T15:37:00.003+05:30

Social Graph API is something to check out. Here is a blog from Google Code Blog. I haven't checked it out yet but plan to do so soon. I hope it does what it claims to do and does that well.

If you are wondering what this API does, here is a snippet from Google's blog entry.

we crawl the Web to find publicly declared relationships between people's accounts, just like Google crawls the Web for links between pages. But instead of returning links to HTML documents, the API returns JSON data structures representing the social relationships we discovered from all the XFN and FOAF. When a user signs up for your app, you can use the API to remind them who they've said they're friends with on other sites and ask them if they want to be friends on your new site.

Social application for Orkut

2008-02-13T14:29:00.004+05:30

I have checked out Open Social, a common set of APIs which can be used to build social applications that work across multiple websites. I think it is really cool and at the same time not so good.

I think it is cool because, there will be a lot of applications that will be created for Orkut now, just like those you find on Facebook. But at the same time, the saddest part is that there will be many useless applications which will clutter the site and confuse everybody.

Google has announced that Open Social applications will be available to users now. So, gear up and be ready to see it on your Orkut login sometime soon. Earlier, these were available only in the sandbox.

Well, I am looking forward for an application which could allow you to tag and share photos from your friends' albums. I really like few photos from my friend's photo album and would like to tag them so that I can easily see them when I want and also would like to allow our common friends to see what I have tagged. Hoping to see if something similar would be available.

Grand Challenges for Engineering

2008-02-13T13:37:00.003+05:30

National Academy of Engineering has a special site where in it will publish the grand challenges for engineering for the 21st century. The committee members are selected from varied fields. I am really looking forward to see what the grand challenges will be.

The challenges identified will be published to the website on 15th Feb, so don't forget to check them out.

Accessing ext2 or ext3 file systems on windows

2008-02-09T01:00:00.000+05:30

Ext2 or Ext3 file systems are commonly used by the Linux operating system. Support for these file systems is not by default built in to windows operating systems.

I remember trying to access windows file systems (FAT and NTFS) from linux and linux file systems (ext2 and ext3) from windows while I was in college. Back then, there wasn't a stable driver for NTFS on linux. Drivers could read from NTFS partitions but weren't stable enought for writing to it. But now, there are drivers that can read/write to both FAT/NTFS and are pretty stable. And you can find these drivers included in linux distributions by default.

I was surprised to see the windows drives mounted automatically on ubuntu and delighted to find out that I could write to those drives. This solved half my problems of sharing data between my two operating systems - windows vista and ubuntu linux. I was further delighted after seeing the search results for "accessing ext3 from windows". I found three results useful. The first one was for Explore2fs - a GUI tool for accessing ext2/ext3 file systems from windows. I had already used this while I was in college and know it's limitations. Windows programs cannot access those file systems, and files had to be copied to a windows drive to be accessed from programs. The other two results (Ext2 file system drivers) were of much interest to me this time. I chose Stephan Schreiber's implementation and installed it on vista. This implementation comes with an installer. Can be uninstalled from Add/Remove programs. Provides the ability to mount the file systems in read only mode along with support for large files (files larger than 2GB). My linux partition was mounted as a new drive and I could access it from all windows programs just as other windows drives. Now, I no longer have to worry about where to put my data so that it can be easily accessed from both the operating systems.

FYI, the ext2 file system driver I didn't mention above was Ext2Fsd, an open source project available on sourgeforge.

Customizing search in Windows XP to index files with unknown file extensions

2008-02-07T16:15:00.000+05:30

First of all, why would you want to customize search in Windows XP to index files with unknown file extensions?

Assume you have a java file (a file with a .java extension) and when you perform a search, text in this file is not searched. Why? Because it is an unknown file extension for Windows XP and it will not index this file. I know it is a plain text file and should be searched but it is not.

Perform the following to enable windows to index files with unknown extensions

Goto Start -> Search -> Change preferences
Click on With Indexing Service (for faster local searches) or Without Indexing Service.
Select Yes, enable Indexing Service. It might already be selected if it was Without Indexing Service in the previous step.
Click on Change Indexing Service settings (Advanced).
In the Indexing service window goto view menu and click on customize.
In the Customize View window check Console tree check box under the MMC section and click on ok.
Right click on Indexing Service on Local Machine and select properties.
On the generation tab, check Index files with unknown extensions check box and click on ok.
Close the Indexing Service window and click on ok in the search window.
Click on All files and folders and start searching. It will find any word in any file with any extension.

Responding to DOM Events using Javascript

2008-02-06T14:12:00.000+05:30

Responding to events occurring on a web page using javascript is a necessity these days rather than a preference. Look at the various popular websites today and you will realize that. So, I set out to explore this a little and the following is what I have learned for the first phase, if you want to say so.

A simple example of an event that can occur on a web page is a mousemove event that occurs when the mouse is moved. Events can be organized into three major categories -

UI events - generated by user interaction through an external device (mouse, keyboard, etc.)
UI Logical events - device independent user interface events such as focus change messages or element triggering notifications.
Mutation events - caused by any action which modifies the structure of the document.

Event Propagation

Let us look at the event flow before we jump into the details of event listener registration.

Once an event originates, it is passed through the Document Object Model in three phases -

The capturing phase - the event is first sent to the window, then to the document, followed by each ancestor of the DOM element where the event occurred downwards until it reaches that element.
The target phase - the event is sent to the target DOM element. Target element is the one on which the event has occurred. For example, the target element for a click event is the element which is clicked.
The bubbling phase - the event is sent to each element back upwards until it reaches the window again.

The below image shows how the event is propagated through the three phases. Let us assume that a mouse click event has occurred on the table element. Shaded elements show the order in which the event is propagated. The propagation of the event will start with the capturing phase. Any event listener registered on the window will be called to handle the event followed by document, html, body and so on until the parent of the target element is reached. The target element in this case is the table. This ends the capturing phase after which the target phase starts. In this phase, any click event listener registered on the table element will be called. In the bubbling phase, the event propagates back to the top calling any event listeners registered for the mouse click event.

An event listener being registered on an element may choose to have that listener capture events by specifying the useCapture parameter of the addEventListener method to be true.

Any event handler may choose to prevent further event propagation by calling the stopPropagation method of the event object. If any event listener calls this method, all additional listeners on the current element will be triggered but bubbling will cease at that level. Only one call to stopPropagation is required to prevent further bubbling.

Some events are cancelable. These events have a default action associated with them. An example of this is a hyperlink in a web browser. When the user clicks on the hyperlink the default action is generally to activate that hyperlink. Event listeners have the option of canceling the implementation's default action or allowing the default action to proceed. Cancellation is accomplished by calling the event object's preventDefault method. If one or more event listeners call preventDefault during any phase of event flow the default action will be canceled.

Registering Event Listeners

There are two ways to attach an event listener to an element. First, by using an attribute with script as its value. Second, by calling an element's addEventListener method. The former may only handle bubbling events but tends to be simpler to write. The latter can handle events at any phase and may also be used to attach multiple listeners for an event to an element.

Attribute Event Listeners

To use the attribute form, place an attribute on the element where you want the event listener to be, the name of which should be the event name preceded by the word 'on'. For example, the corresponding attribute for the 'click' event is 'onclick'. The value of the attribute should be some script that should be executed when the event occurs. Typically, this code will be short and just call a function defined in a separate script. An example of responding to a button being pressed:

<input type="button" label="OK" onclick="alert('Button was pressed!');" />

Since the click event will bubble, it is also possible to place the event listener on an enclosing element. In the example below, the listener has been placed on a div and will receive events for both elements.

<div onclick="alert(event.target.tagName);">
<input type="button" label="OK" />
<p>This is a paragraph</p>
</div>

In this example, the click event will bubble up from the button or paragraph to the div, where it is handled. If a second listener (the onclick attribute) were placed on the button, its code will be called first, followed by the handler on the div. Event handlers are passed the event object as an implied argument called 'event'. This is used to get specific information about the event. One commonly used property is the 'target' property of the event, which holds the element where the event actually occured. In the example we display an alert containing the target's tag name. The target is useful when using a bubbling event so that you could have a set of buttons which are all handled by a single script.

DOM Event Listeners

The second way to add an event handler is to call an element's addEventListener method. This allows you to attach an event listener dynamically and listen for events during the capturing phase. The syntax is as follows:

<input type="button" id="okbutton" label="OK"/>
<script>
function buttonPressed(event){
alert('Button was pressed!');
}
var button = document.getElementById("okbutton");
button.addEventListener('click', buttonPressed, true);
</script>

The getElementById() function returns the element with a given id, in this case the button. The addEventListener() function is called to add a new capturing event listener. The first argument is the name of the event to listen to. The second argument is the event listener function which will be called when the event occurs. Finally, the last argument should be true for capturing listeners. You can also listen during the bubbling phase by setting the last argument to false. The event listener function passed as the second argument should take one argument, the event object, as shown in the declaration for the buttonPressed function above.

Note: addEventListener will not work with Internet Explorer, you need to use attachEvent instead.

References:

Document Object Model Events
Adding Event Handler from the XUL tutorial

Registering event handlers section is taken from XUL tutorial mentioned as the second reference above.

How to Debug PHP with Vim and XDebug on Linux

2008-02-04T23:41:00.000+05:30

XDebug is definitely a wonderful tool for debugging PHP. I tried using Eclipse IDE for remote debugging but it had some version conflicts and so I searched for some other IDE but I didn't imagine that Vim will turn out to be the one.

This Box.net blog tells you how to do debug PHP with Vim and XDebug on Linux.

Java on Ubuntu

2007-11-10T16:02:00.000+05:30

Ubuntu has GCJ installed by default. A different version of Java can be installed using the Synaptic Package Manager. Sun's Java 1.5 and 1.6 are available through the package manager.

The problem is, even after installing Sun's Java, GCJ is the default.

The default version of java can be changed using the update-java-alternatives command.

update-java-alternatives -l will give the list of all installed java packages.
example output:


   java-1.5.0-sun 53 /usr/lib/jvm/java-1.5.0-sun
   java-gcj 1042 /usr/lib/jvm/java-gcj

update-java-alternatives -s java-1.5.0-sun will set Sun's java as the default.

For some applications like Ant, Tomcat to run, you nee to set the JAVA_HOME environment variable like below
export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun

The above export can be put in /etc/bash.bashrc (for all users) or /home/<user>/.bashrc (for a particular user) so that JAVA_HOME is available on logon.

A Roundup of Free Operating Systems

2007-08-08T12:52:00.001+05:30

By David Chisnall

Whenever people think of a free operating system (if they do), the odds are that they will think of Linux, a clone of UNIX from 1991. They also might think of one or more of the BSDs, UNIX-derivatives dating back to the early ’80s. Let’s face it, though—UNIX is pretty boring. It was great for a PDP-11, and it gets the job done now, but it’s not exactly exciting.
Fortunately, the Free Software community has some much more interesting projects which, even if they might not be as useful, are much more fun.

Complete article is here

(c) Article is provided courtesy of Prentice Hall PTR.

Search for Selected Text

2006-06-28T05:00:00.000+05:30

Highlight text in any web page and search for it on Google with the click of a button.

Start by creating a new button on your browser's links bar and naming it "Search Google" or something similar.

Next, if you’re using Mozilla, Firefox, or Netscape, place this JavaScript code (all on one line) into the button’s location field:

javascript: s = document.getSelection( );
for(i = 0; i < frames.length; i++) {
s = frames[i].document.getSelection( );
}
location.href = 'http://www.google.com/search?q='+escape(s);

If you’re using Internet Explorer, you’ll need this code:

javascript: s = (document.frames.length ? ' ':
document.selection.createRange( ).text);
for(i = 0; i < document.frames.length; i++) {
s=document.frames[i].document.selection.createRange( ).text;
}
location.href = 'http://www.google.com/search?q='+escape(s);

Next, open any web page, highlight some text, and click your new search button. The next thing you’ll see is a search results page with items matching your selected text.

Bug: This doesn't work when there are frames in the webpage.

Converting FAT volumes to NTFS

2006-05-01T12:58:00.000+05:30

The convert command converts FAT volumes to NTFS. This command is run from the command prompt.

ex: convert D: /FS:NTFS - will convert D: from FAT to NTFS

Getting rid of java.lang.OutOfMemoryError

2006-04-22T21:45:00.000+05:30

Most JVMs allocate memory on the heap for almost everything, except for reflective data. That is put in a separate location which is a section of the heap that is reserved for permanent generation. This gets easily filled up when you dynamically load an unload classes, or have a large number of classes. Simply add the following options to the java executable and all worries will be gone:

-XX:PermSize=256M -XX:MaxPermSize=256M

The entire command looks like this:

java -server -Djava.awt.headless=true -Xms1024M -Xmx1024M -XX:PermSize=256M -XX:MaxPermSize=256M

Read This

Print system information under Linux

2006-03-29T15:59:00.000+05:30

'uname' prints information about the machine and operating system it is run on.

% uname -a

Linux localhost.localdomain 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 i686 athlon i386 GNU/Linux

Use this command if you want to know any of the following information

kernel name
network node hostname
kernel release
kernel version
machine hardware name
processor type
hardware platform
operating system

If you are interested in knowing information about the kernel distribution then use `cat /etc/issue` The output of which will be something as below (on Red Hat ofcourse)

Red Hat Linux release 9 (Shrike)
Kernel \r on an \m

Proxy Settings - Enviroment Variables

2006-03-09T16:29:00.000+05:30

Proxy can be set up using environment variables automatically for all users by creating the following file,

/etc/profile.d/proxyenv.sh

http_proxy="192.168.36.204:8080"
https_proxy="192.168.36.204:8080"
ftp_proxy="192.168.36.204:8080"

export http_proxy, https_proxy, ftp_proxy

If you are using csh instead of bash then create the following file,

/etc/profile.d/proxyenv.csh

setenv http_proxy="192.168.36.204:8080"
setenv https_proxy="192.168.36.204:8080"
setenv ftp_proxy="192.168.36.204:8080"

Reference: Unofficial ISS Linux Web Pages

Converting HTML to text

2006-03-09T14:06:00.000+05:30

I have used the following two scripts and found them to be not very impressive.

html2text (Python script)
Html2text (Perl script)

The Python script converts a HTML page into Markdown (a text-to-HTML format) which I don't want. I want text only.

The Perl script requires the input to be "normalized" by a program such as sgmlnorm before it could process it. Apart from this, the script doesn't work well for all the documents. It is limited to certain tags and has to be modified to get it to work for other tags. The text between the tags that are not handled just vanish from the output. Lets see if I can modify it to work for my documents atleast.

Why is there a size limit on file uploads?

2006-02-12T08:43:00.000+05:30

A potential problem with form processing scripts is that, by default, they attempt to process form POSTings no matter how large they are. A wily hacker could attack your site by sending a huge POST of many megabytes. The script will attempt to read the entire POST thus growing hugely in size until it runs out of memory. While the script attempts to allocate the memory the system may slow down dramatically. This is a form of denial fo service attack.

Another possible attack is for the remote user to force the script to accept a huge file upload. The script will accept the upload and store it in a temporary directory even if your script doesn't expect to receive an uploaded file. The file will be deleted automatically when the script terminates, but in the meantime the remote user may have filled up the server's disk space, causing problems for other programs.

The best way to avoid denail of service attacks is to limit the amount of memory, CPU time and disk space that the scripts can use. Some Web servers come with built-in facilities to accomplish this. In other cases, you can should use commands to put ceilings on resource usage.

Most servers try to avoid denial of service attacks by limiting resource usage and so there is a size limit on file uploads.

Building dictionary from the web

2006-02-11T12:48:00.000+05:30

Initially a small collection of "seed" texts are fed to the crawler (a few hundred words of running text have been sufficient in practice). Queries combining words from these texts are generated and passed to the Google API which returns a list of documents potentially written in the target language. These are downloaded, processed into plain text, and formatted. A combination of statistical techniques bootstrapped from the initial seed texts (and refined as more texts are added to the database) is used to determine which documents (or sections thereof) are written in the target language. The crawler then recursively follows links contained within documents that are in the target language. When these run out, the entire process is repeated, with a new set of Google queries generated from the new, larger corpus.

ref: http://borel.slu.edu/crubadan/