Sunday, May 29, 2011

Using Django functionalities in a stand-alone application

1. Set django settings.py file

2. Run syncdb to create DB tables
python manage.py syncdb

3. Create a shell script to set environment variables (NOTE : setting required variables to run django somehow cannot be done by python.)



export PYTHONPATH=$PYTHONPATH:/home/djangoprojects
export DJANGO_SETTINGS_MODULE="legacytest.settings"

python ./access_db.py



4. Create a python file with django functions.

from newapp.models import *

# Pull all data from old DB
aa = Aa.objects.all()

Friday, April 15, 2011

Creating an XML document in Python



root = xml.Element('bestsellers')
root.attrib['yyyymm'] = "201012"
root.attrib['topN'] = str(1000)
idx = 0
for i in top1000_list:
idx += 1
child = xml.Element('item')
root.append(child)
child.attrib['rank'] = str(idx)
for k,v in i.items():
child.attrib['isbn'] = k
child.attrib['count'] = v
print xml.tostring(root)



Result is...


































String to JSON in Python



jsonstr = "[ {\"20101231\" : [ {\"ID1234\":\"ISBN1234\"} , {\"ID1234\":\"ISBN1234\"} ] } ]"
j = json.JSONDecoder().decode(jsonstr)

Sunday, January 16, 2011

tweepy (twitter API) Status object structure



{
'contributors': None,
'truncated': False,
'text': 'My Top Followers in 2010: @tkang1 @serin23 @uhrunland @aliassculptor @kor0307 @yunki62. Find yours @ http://mytopfollowersin2010.com',
'in_reply_to_status_id': None,
'id': 21041793667694593,
'_api': ,
'author': ,
'retweeted': False,
'coordinates': None,
'source': 'My Top Followers in 2010',
'in_reply_to_screen_name': None,
'id_str': '21041793667694593',
'retweet_count': 0,
'in_reply_to_user_id': None,
'favorited': False,
'retweeted_status': ,
'source_url': 'http://mytopfollowersin2010.com',
'user': ,
'geo': None,
'in_reply_to_user_id_str': None,
'created_at': datetime.datetime(2011, 1, 1, 3, 15, 29),
'in_reply_to_status_id_str': None,
'place': None
}


tweepy (twitter API) User object structure

1. tweepy User object has following fields.



{
'follow_request_sent': False,
'profile_use_background_image': True,
'id': 132728535,
'_api': ,
'verified': False,
'profile_sidebar_fill_color': 'C0DFEC',
'profile_text_color': '333333',
'followers_count': 80,
'protected': False,
'location': 'Seoul Korea',
'profile_background_color': '022330',
'id_str': '132728535',
'utc_offset': 32400,
'statuses_count': 742,
'description': "Cars, Musics, Games, Electronics, toys, food, etc... I'm just a typical boy!",
'friends_count': 133,
'profile_link_color': '0084B4',
'profile_image_url': 'http://a1.twimg.com/profile_images/1213351752/_2_2__normal.jpg',
'notifications': False,
'show_all_inline_media': False,
'geo_enabled': True,
'profile_background_image_url': 'http://a2.twimg.com/a/1294785484/images/themes/theme15/bg.png',
'screen_name': 'jaeeeee',
'lang': 'en',
'following': True,
'profile_background_tile': False,
'favourites_count': 2,
'name': 'Jae Jung Chung',
'url': 'http://www.carbonize.co.kr',
'created_at': datetime.datetime(2010, 4, 14, 1, 20, 45),
'contributors_enabled': False,
'time_zone': 'Seoul',
'profile_sidebar_border_color': 'a8c7f7',
'is_translator': False,
'listed_count': 2
}

Sunday, December 26, 2010

Serving static files in Django

This is how to serve static files from the default Django python web server

1. add the following lines to your urls.py file


#!/usr/bin/python

urlpatterns = patterns('',
...

(r'^your_dir/(?P.*)$', 'django.views.static.serve', {'document_root': '/path/to/your/static/files/'}),
#(r'^data/(?P.*)$', 'django.views.static.serve', {'document_root': '/home/django/testproject/data/'}),
...

)

Wednesday, November 10, 2010

sample configuration xml files for Hadoop 0.20.x

In Hadoop 0.19.x or earlier, there were only one xml file to modify - hadoop-site.xml.
From Hadoop 0.20.x, there are 3 xml files that you have to configure.
They are (1) core-site.xml (2) mapred-site.xml (3) hdfs-site.xml.
Here are sample xml files that set only the minimal and required settings.


NOTE : they are found in your HADOOP_HOME/conf directory.


1. core-site.xml






hadoop.tmp.dir
/home/hadoop/hadoop-0.20.2/hdfs-tmp
A base for other temporary directories.



fs.default.name
hdfs://203.235.211.195:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.







2. mapred-site.xml






mapred.local.dir
/home/hadoop/hadoop-0.20.2/mapred-tmp
Comma-separated list of paths on the local
filesystem where temporary Map/Reduce data is written.




mapred.job.tracker
203.235.211.195:54311
The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.









3. hdfs-site.xml






hadoop.tmp.dir
/home/hadoop/hadoop-0.20.2/tmp
A base for other temporary directories.



dfs.data.dir
/home/hadoop/hadoop-0.20.2/dfs_blk/${user.name}
Comma separated list of paths on the local filesystem of a
DataNode where it should store its blocks.




dfs.default.name
hdfs://203.235.211.195:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.




dfs.replication
3
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.






Monday, October 18, 2010

running a simple embedded servlet container with Jetty

1. Install MySQL for Python

a. Download and install Jetty from http://dist.codehaus.org/jetty/jetty-6.1.25/jetty-6.1.25.zip

b. unzip the pkg.
[root@skcc-nebdap37 src]# yum install mysql mysql-devel

c. include the following jar in Java build path
- jetty-6.1.25.jar
- jetty-util-6.1.25.jar
- servlet-api-2.5-20081211.jar


2. Java code for running a servlet on a Jetty


a. main()

package tkang.test;

import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.jetty.servlet.ServletHolder;

public class JettyTest {

public static void main(String args[]) throws Exception {
Server server = new Server(8080);
Context root = new Context(server,"/",Context.SESSIONS);
ServletHolder sh = new ServletHolder(HelloWorldServlet.class);
root.addServlet(sh, "/helloworld");
server.start();
}
}



b. HelloWorld servlet code : HelloWorldServlet.java

package tkang.test;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class HelloWorldServlet extends HttpServlet {

public void service(HttpServletRequest request,
HttpServletResponse response) throws ServletException,
IOException {
response.setContentType("text/html");
response.setCharacterEncoding("utf-8");
response.setStatus(HttpServletResponse.SC_OK);
response.getWriter().println("

Hello

");
}

}

Sunday, October 17, 2010

mysql client with python

1. Install MySQL for Python

a. install mysql and mysql-devel pkg

[root@skcc-nebdap37 src]# yum install mysql mysql-devel


b. install python setuptools from http://pypi.python.org/pypi/setuptools#files


[root@skcc-nebdap37 src]# wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e

[root@skcc-nebdap37 src]# gunzip setuptools-0.6c11.tar.gz

[root@skcc-nebdap37 src]# tar xvf setuptools-0.6c11.tar

[root@skcc-nebdap37 src]# cd setuptools-0.6c11

[root@skcc-nebdap37 src]# python setup.py build

[root@skcc-nebdap37 src]# python setup.py install


c. Download and install MySQL for Python from http://sourceforge.net/projects/mysql-python


2. Sample MySQL for Python code


#!/usr/bin/python

import MySQLdb

# Establich a connection
db = MySQLdb.connection(host="localhost",
user="nebula",
passwd="nebula",
db="nebula")

# Run a MySQL query from Python and get the result set

db.query("""select * from USERS""")
r = db.store_result()

# Iterate through the result set
# Example calls back up to 100 rows
for row in r.fetch_row(100):
print row