Sunday, December 26, 2010

Serving static files in Django

This is how to serve static files from the default Django python web server

1. add the following lines to your urls.py file


#!/usr/bin/python

urlpatterns = patterns('',
...

(r'^your_dir/(?P.*)$', 'django.views.static.serve', {'document_root': '/path/to/your/static/files/'}),
#(r'^data/(?P.*)$', 'django.views.static.serve', {'document_root': '/home/django/testproject/data/'}),
...

)

Wednesday, November 10, 2010

sample configuration xml files for Hadoop 0.20.x

In Hadoop 0.19.x or earlier, there were only one xml file to modify - hadoop-site.xml.
From Hadoop 0.20.x, there are 3 xml files that you have to configure.
They are (1) core-site.xml (2) mapred-site.xml (3) hdfs-site.xml.
Here are sample xml files that set only the minimal and required settings.


NOTE : they are found in your HADOOP_HOME/conf directory.


1. core-site.xml






hadoop.tmp.dir
/home/hadoop/hadoop-0.20.2/hdfs-tmp
A base for other temporary directories.



fs.default.name
hdfs://203.235.211.195:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.







2. mapred-site.xml






mapred.local.dir
/home/hadoop/hadoop-0.20.2/mapred-tmp
Comma-separated list of paths on the local
filesystem where temporary Map/Reduce data is written.




mapred.job.tracker
203.235.211.195:54311
The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.









3. hdfs-site.xml






hadoop.tmp.dir
/home/hadoop/hadoop-0.20.2/tmp
A base for other temporary directories.



dfs.data.dir
/home/hadoop/hadoop-0.20.2/dfs_blk/${user.name}
Comma separated list of paths on the local filesystem of a
DataNode where it should store its blocks.




dfs.default.name
hdfs://203.235.211.195:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.




dfs.replication
3
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.






Monday, October 18, 2010

running a simple embedded servlet container with Jetty

1. Install MySQL for Python

a. Download and install Jetty from http://dist.codehaus.org/jetty/jetty-6.1.25/jetty-6.1.25.zip

b. unzip the pkg.
[root@skcc-nebdap37 src]# yum install mysql mysql-devel

c. include the following jar in Java build path
- jetty-6.1.25.jar
- jetty-util-6.1.25.jar
- servlet-api-2.5-20081211.jar


2. Java code for running a servlet on a Jetty


a. main()

package tkang.test;

import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.jetty.servlet.ServletHolder;

public class JettyTest {

public static void main(String args[]) throws Exception {
Server server = new Server(8080);
Context root = new Context(server,"/",Context.SESSIONS);
ServletHolder sh = new ServletHolder(HelloWorldServlet.class);
root.addServlet(sh, "/helloworld");
server.start();
}
}



b. HelloWorld servlet code : HelloWorldServlet.java

package tkang.test;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class HelloWorldServlet extends HttpServlet {

public void service(HttpServletRequest request,
HttpServletResponse response) throws ServletException,
IOException {
response.setContentType("text/html");
response.setCharacterEncoding("utf-8");
response.setStatus(HttpServletResponse.SC_OK);
response.getWriter().println("

Hello

");
}

}

Sunday, October 17, 2010

mysql client with python

1. Install MySQL for Python

a. install mysql and mysql-devel pkg

[root@skcc-nebdap37 src]# yum install mysql mysql-devel


b. install python setuptools from http://pypi.python.org/pypi/setuptools#files


[root@skcc-nebdap37 src]# wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e

[root@skcc-nebdap37 src]# gunzip setuptools-0.6c11.tar.gz

[root@skcc-nebdap37 src]# tar xvf setuptools-0.6c11.tar

[root@skcc-nebdap37 src]# cd setuptools-0.6c11

[root@skcc-nebdap37 src]# python setup.py build

[root@skcc-nebdap37 src]# python setup.py install


c. Download and install MySQL for Python from http://sourceforge.net/projects/mysql-python


2. Sample MySQL for Python code


#!/usr/bin/python

import MySQLdb

# Establich a connection
db = MySQLdb.connection(host="localhost",
user="nebula",
passwd="nebula",
db="nebula")

# Run a MySQL query from Python and get the result set

db.query("""select * from USERS""")
r = db.store_result()

# Iterate through the result set
# Example calls back up to 100 rows
for row in r.fetch_row(100):
print row

Thursday, October 14, 2010

ssh client with python



1. Install paramiko & paracrypto
- paramiko : http://www.lag.net/paramiko/download/paramiko-1.7.6.tar.gz
- paracrypto : http://www.amk.ca/files/python/crypto/pycrypto-2.0.1.tar.gz
NOTE : they all work on a python version >= 2.4




2. sample code for using paramiko
- the following python code checks if sshd is alive at a remote server.

import paramiko
import sys
import socket
import os

addr = sys.argv[1]

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(hostname=addr, username="root", password="rootpw")

stdin, stdout, stderr = client.exec_command("ps -ef | grep sshd")

found = False
for line in stdout.read().splitlines():
if line.find("/usr/bin/sshd"):
found = True

if found:
print "sshd is alive!"
else:
print "sshd is dead!"

client.close()

Tuesday, September 28, 2010

Sending a HTTP Request with VBA




a. Sending a HTTP Post Request


Dim result As String
Dim myURL As String, postData As String
Dim winHttpReq As Object
Set winHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")

myURL = "http://192.168.10.101:80/your_web_service"
postData = "parameter=hello¶meter2=hi"

winHttpReq.Open "POST", myURL, False
winHttpReq.SetRequestHeader "Content-Type", "application/x-www-form-urlencoded"
winHttpReq.Send (postData)

result = winHttpReq.responseText






b. Sending a HTTP Get Request


Dim result As String
Dim myURL As String
Dim winHttpReq As Object
Set winHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")

myURL = "http://192.168.10.101:80/your_web_service?parameter=hello¶meter2=hi"

winHttpReq.Open "GET", myURL, False
winHttpReq.Send

result = winHttpReq.responseText

Concurrently running Excel VBA/Macro?

Short Answer : IT IS NOT POSSIBLE in Excel 2003 and 2007.

Excel runs built-in functions (e.g. SUM, AVERAGE) concurrently when possible,
but Excel does not run user-defined custom VBA/Macro concurrently.

Monday, August 30, 2010

Calling a function in a DLL from C# code

1. Analyze DLL's function entry point

a. Use "dumpbin.exe" to peek inside the dll's structure.



- We will use command "dumpbin.exe /EXPORT your_dll.dll"
e.g. dumpbin.exe /EXPORTS D:\myDLL\PiEstimatorDLL.dll

- It may be a good idea to write the output into a file
e.g. umpbin.exe /EXPORTS D:\myDLL\PiEstimatorDLL.dll >> dll_export.txt

- NOTE : dumpbin.exe should be installed with your Visual Studio by default.

b. Find function entry point from the output.



- Assume that we want to call PiEstimator.estimatePi() method in the given dll,
we should use "?estimatePi@PiEstimator@@SANH@Z" as the entry point for the function, as underlined in the above screen shot.

2. Use the DLL in your C# code

In your C# code, you should add "DllImport" to import and use your dll functions, such as the following.

...
[DllImport("PiEstimatorDll.dll", EntryPoint = "?estimatePi@PiEstimator@@SANH@Z"]
public static extern double estimatePi(int numIter);

....

double piEstimation = estimatePi(numIter);

...


- NOTE : you should set your PATH environment variable so that the compiler can look up your dll (e.g. PiEstimatorDLL.dll.) Otherwise, you shall give a full path to your dll in DllImport command.

Friday, August 27, 2010

Pi Estimation using Monte Carlo Method in C++ (Visual C++)


Here is C++ implementation of Pi Estimation using Monte Carlo Method.

Generating a uniformly distributed random number in C++ was quite a task. I think it's better to use a well-made library for random number generation. (e.g. Boost library)

a. header file

public class PiEstimator {
public:
PiEstimator(void);
~PiEstimator(void);
double estimatePi(int numIter);
};


b. cpp implementation file

#include "PiEstimator.h"
#include
#include
#include

unsigned time_seed() {
time_t now = time ( NULL );
unsigned char *p = (unsigned char *)&now;
unsigned seed = 0;
size_t i;

for ( i = 0; i < sizeof now; i++ )
seed = seed * ( UCHAR_MAX + 2U ) + p[i];

return seed;
}

PiEstimator::PiEstimator(void) {
srand ((unsigned)time_seed());
}

PiEstimator::~PiEstimator(void) {
}

double randomNumber() {
return (double)rand()/(double)RAND_MAX;
}

double getRandDistance() {
double x = randomNumber();
double y = randomNumber();
return sqrt(pow(x,2) + pow(y,2));
}

double PiEstimator::estimatePi(int numIter) {
int numHits = 0;
for (int i = 0; i < numIter; i++) {
if (getRandDistance() <= 1) {
numHits++;
}
}
double pi = 4 * (numHits / (double)numIter);
return pi;
}

Wednesday, August 25, 2010

Using DLL in Excel VBA macro

1. Hook up DLL with VBA

a. Start Excel and open up VBA Editor
- After starting your Excel file, press "Alt + F11" to start VBA editor

b. Add a module
- "Insert" -> "Module" to add a new module.

c. Write declaration code to hookup dll with VBA
- Declaring a VBA function to call a DLL goes like this.
[Public | Private] Declare Function name Lib "libname" [Alias "aliasname"] [([arglist])] [As type]

- In your Module file, add the following.


'Declare all the API-specific items Private to the module
Private Declare Function GetSystemMetrics Lib "user32" _
(ByVal nIndex As Long) As Long

Private Const SM_CXSCREEN = 0 'Screen width
Private Const SM_CYSCREEN = 1 'Screen height

'The width of the screen, in pixels
Public Function ScreenWidth() As Long
ScreenWidth = GetSystemMetrics(SM_CXSCREEN)
End Function

'The height of the screen, in pixels
Public Function ScreenHeight() As Long
ScreenHeight = GetSystemMetrics(SM_CYSCREEN)
End Function




2. Using VBA code in your excel.

Monday, August 02, 2010

Pi estimation with Monte Carlo Method in Python


The following page explains what Monte Carlo Method is in a very simple yet understandable way. It wasn't very difficult for me to figure it out so I guess it's not going to be too difficult for you, either!



The following is my implementation of pi estimation with Monte Carlo Method in Python.


#!/usr/bin/python

import random
import math

def get_dist ():
x = random.uniform(0,1)
y = random.uniform(0,1)
dist = math.sqrt(math.pow(x, 2) + math.pow(y, 2))
return dist


# number of hits / number of misses = ((1/4) * pi * sqrt(radius)) / sqrt(radius)
# pi = 4 * number of hits / number of misses

num_hits = 0
num_throws = 0
while 1:
num_throws += 1
# if distance from origin is <= 1, it's within the quadrant.
if (get_dist() <= 1):
num_hits += 1
pi = 4 * (num_hits / float(num_throws))
print "iteration #", num_throws, ":", pi

Monday, July 26, 2010

KVM configuration file example & explanation

1. KVM configuration file

KVM configuration file is normally placed under /etc/libvirt/qemu




osmgmt02
c533fb5c-71b7-2e4c-8b9a-7742ba3ada03
20971520
20971520
4

hvm








destroy
restart
restart

/usr/libexec/qemu-kvm





































Tuesday, July 20, 2010

Setting up a network bridge to enable NIC sharing between VM's in KVM

1. Configure a bridge device

Here we will setup a br1 device and have it connected to eth1 device.

a. Set up eth1

[root@nebskcc001 local]# cd /etc/sysconfig/network-scripts/
[root@nebskcc001 local]# vi ifcfg-eth1

DEVICE=eth1
BOOTPROTO=static
HWADDR=F4:CE:46:87:D1:B6 # YOUR MAC ADDRESS MAY VARY
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
BRIDGE=br1 # Bridge name to be connected to.


b. Set up br1

[root@nebskcc001 local]# vi ifcfg-br1

DEVICE=br1
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes



c. Restart the network service

[root@nebskcc001 local]# service network restart



2. Modify KVM configuration file.

a. change kvm configuration file.



Here we add the bridge device and have it used by the VM.
[root@nebskcc008 imgs]# cd /etc/libvirt/qemu
[root@nebskcc008 qemu]# vim bpm.xml



b. start vm.

Tuesday, July 13, 2010

Thrift Python Module Path Problem

By default, Thrift python module is installed under /usr/lib/python2.x/site-pacakges, if you went through the typical "./configure -> make -> make install" steps to install Thrift.

If your python cannot locate thrift module (e.g. you installed a newer version of Python like Python 2.7), you might add "export PYTHONPATH=/path/to/python/site-packages".

[tkang@neb005 py-impl]$ vi /etc/profile



# add following lines
export PYTHONPATH=/usr/lib/python2.7/site-packages

Thrift Server-Client in Python

1. Write Thrift stub code

[tkang@neb005 thrift]$ vi helloworld.thrift


const string HELLO_IN_KOREAN = "an-nyoung-ha-se-yo"
const string HELLO_IN_FRENCH = "bonjour!"
const string HELLO_IN_JAPANESE = "konichiwa!"

service HelloWorld {
void ping(),
i32 sayHello(),
i32 sayMsg(1:string msg)
}


2. Generate python code

[tkang@neb005 thrift]$ thrift --gen py helloworld.thrift

Generated codes will be saved under "gen-py" directory.
[tkang@neb005 thrift]$ ls
gen-py helloworld.thrift

3. Fill in Server code

[tkang@neb005 thrift]$ mkdir py-impl
[tkang@neb005 thrift]$ cd py-impl
[tkang@neb005 py-impl]$ vi PythonServer.py


#!/usr/bin/env python

import sys
sys.path.append('../gen-py')

from helloworld import HelloWorld
from helloworld.ttypes import *

from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer

import socket

class HelloWorldHandler:
def __init__(self):
self.log = {}

def ping(self):
print "ping()"

def sayHello(self):
print "sayHello()"
return "say hello from " + socket.gethostbyname(socket.gethostname())

def sayMsg(self, msg):
print "sayMsg(" + msg + ")"
return "say " + msg + " from " + socket.gethostbyname(socket.gethostname())

handler = HelloWorldHandler()
processor = HelloWorld.Processor(handler)
transport = TSocket.TServerSocket(30303)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()

server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)

print "Starting python server..."
server.serve()
print "done!"


4. Write Client code to connect to server

[tkang@neb005 py-impl]$ vi PythonClient.py


#!/usr/bin/env python

import sys
sys.path.append('../gen-py')

from helloworld import HelloWorld
from helloworld.ttypes import *
from helloworld.constants import *

from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol

try:
# Make socket
transport = TSocket.TSocket('localhost', 30303)

# Buffering is critical. Raw sockets are very slow
transport = TTransport.TBufferedTransport(transport)

# Wrap in a protocol
protocol = TBinaryProtocol.TBinaryProtocol(transport)

# Create a client to use the protocol encoder
client = HelloWorld.Client(protocol)

# Connect!
transport.open()

client.ping()
print "ping()"

msg = client.sayHello()
print msg
msg = client.sayMsg(HELLO_IN_KOREAN)
print msg

transport.close()

except Thrift.TException, tx:
print "%s" % (tx.message)

Monday, July 12, 2010

Thrift Installation


1. Download Source
- Download Tar From : http://incubator.apache.org/thrift/download/


2. Install all the required components

- Install pkgs with yum
[root@neb005 thrift-0.2.0]# yum install automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel

- Install autoconf version > 2.61 from : http://ftp.gnu.org/gnu/autoconf/autoconf-2.66.tar.gz

- build it and copy "libtool" binary to thrift build directory.

3. Install Thrift
- To install under /usr/local/thrift/
[root@localhost src]# gunzip thrift-0.2.0-incubating.tar.gz
[root@localhost src]# tar xvf thrift-0.2.0-incubating.tar
.....
[root@localhost src]# cd thrift-0.2.0
[root@neb005 thrift-0.2.0]# ./bootstrap.sh
[root@neb005 thrift-0.2.0]# ./configure
[root@neb005 thrift-0.2.0]# make
[root@neb005 thrift-0.2.0]# make install