There are times when you do several indirections in a dictionary. That is, the key is also a dictionary with a key which also a dictionary like this:
>>> c['x'] = 'y'
>>> b['y'] = 'z'
>>> a['z'] = [1, 2, 3]
>>> a[b[c['x']]]
[1, 2, 3]
However, you don't want it throwing exception when the innermost key do not exists. In the example above, the innermost key is 'x' in c['x'], and instead of throwing an exception, you want it to return an empty list. This is what you do using 'get' method in dict:
>>> r = a.get(b.get(c.get('x','nothing here'),'nothing here'),[])
>>> r
[1, 2, 3]
>>> r = a.get(b.get(c.get('xx','nothing here'),'nothing here'),[])
>>> r
[]
'nothing here' should not match any keys for this to work. Starting at the innermost dict, 'xx' key does not exist in c, so, it returns 'nothing here' which also do not exist in b and therefore returned also 'nothing here', and finally, since 'nothing here' key does not exist in a, it returned an empty list.
A similar scenario might happen for a dict of a dict of a dict like this:
>>> u = {}
>>> u['xxx'] = {}
>>> u['xxx']['yyy'] = {}
>>> u['xxx']['yyy']['zzz'] = [2, 3,4]
Instead of throwing an exception if no such key exist in this dict, you just want it to return an empty list. Here is the way to do it.
>>> u.get('xxx',{}).get('yyy',{}).get('zzz',[])
[2, 3, 4]
>>> u.get('xxx',{}).get('yyy',{}).get('ooo',[])
[]
>>> u.get('xxx',{}).get('ooo',{}).get('zzz',[])
[]
>>> u.get('ooo',{}).get('yyy',{}).get('zzz',[])
[]
>>> u.get('ooo',{}).get('ooo',{}).get('zzz',[])
[]
>>> u.get('ooo',{}).get('ooo',{}).get('ooo',[])
[]
Ultimately, the default value at the last 'get' is the one that will be returned when any of the 3 keys do not exists in the dict. The first instance of get must return an empty dict because the 'get' method will be applied against it. Try using an empty list on that 'get' to see what I mean.
Thursday, January 19, 2012
Thursday, May 19, 2011
Creating Combinations
You should have python2.6 installed for this to work.
>>> import itertools
>>> lx = ['none','P1','P2','P3']
>>> lx
['none', 'P1', 'P2', 'P3']
>>> l2 = itertools.combinations(lx,2) # combination of 2 objects
>>> for x in l2:
... if 'P1' in x:
... p1.append(list(x))
...
>>> p1
[['none', 'P1'], ['P1', 'P2'], ['P1', 'P3']]
The above example created a list that contains 2 combinations that has 'P1' in it.
>>> import itertools
>>> lx = ['none','P1','P2','P3']
>>> lx
['none', 'P1', 'P2', 'P3']
>>> l2 = itertools.combinations(lx,2) # combination of 2 objects
>>> for x in l2:
... if 'P1' in x:
... p1.append(list(x))
...
>>> p1
[['none', 'P1'], ['P1', 'P2'], ['P1', 'P3']]
The above example created a list that contains 2 combinations that has 'P1' in it.
Sunday, February 13, 2011
Regular Expression
Regular expression is pattern matching and optionally, extracting data from those that matched. It is like 'grep' or 'findstr' command, only better.
For example, a station name might be an 'A', followed by numbers, followed by 'S'. Below are examples of station name.
A50S
A234S
A3S
A5425S
You might want to know whether a line of text contain a station name in it. Optionally, you might want to pull, say, '50' out of 'A50S'. Regular expression or regex is your tool. Below are the most common regex I use. For more complete discussion, read this.
Use the symbol ------------ To match
below
\d any decimal digit
\D any non-digit
\s any white space character like space and tab
\S any non-white space character
[a-z,A-Z] any alphabet letter
[a, b, c] either 'a' or 'b' or 'c'
. any character except newline
The pattern above would only match one item. For example, \d would only match 1 digit. If you want to match 2 digits, you need to place \d\d. If there could be 2 or 3 or 4 or more digits but you don't know on what particular instance it would be, this approach may not work. The way to approach that is to use the following 'special characters'
Use this --------- If you want to match, with respect
Special Character to the previous symbol (see above)
* zero or more instance of same thing
+ one or more instance of same thing
So, if you say '\d*', it would match even if there is no digit. '\d+' would match if there is at least one digit.
Now, the regex formed above would match anywhere in the line. There are times when you want a match only if it is at the beginning or end. To accomplish this, use the following 'anchors' at the beginning (use ^) or end (use $) of your regex
^regex - if you want a match only at the very beginning of the line
regex$ - If you want it at the end only
The example below should make it clear.
You want to find a match for station name which could either be:
A50S
A234S
A3S
A5425S
The pattern that you could see for station name is.
'A' followed by a number, followed by 'S' . The regex is
^ ^ ^
| ________| |
| | |
'A\d+S' ---------------------------------------------------
You would recall that '\d' is a decimal digit and so, one or more of that (the '+' symbol), meaning, one or more decimal digit, would catch any digits following the first digit. Had you used '\d*' instead, it would match even though there is no digit, like 'AS' because zero digit is allowed. Having formed the regex, you could use it as follows.
>>> import re
>>> pattern = re.compile('A\d+S')
>>> s = 'A50S'
>>> if pattern.search(s):
... print 'got a match'
... else:
... print 'no match'
...
got a match
First you import the regex module (re), then compile the regex into a pattern, then use the 'search' method of the pattern with the string to be searched as argument. This 'search' method will return nonzero value if it found a match and none otherwise.
>>> s = 'I went inside A698S and shouted'
>>> if pattern.search(s):
... print 'got a match'
... else:
... print 'no match'
...
got a match
It still got a match even though the station name is in the middle of the string.
>>> s = 'AB453S'
>>> if pattern.search(s):
... print 'got a match'
... else:
... print 'no match'
...
no match
It did not match this time because after 'A', you got 'B' which is not a digit.
So, if you got a file 'test.txt' which contains the following:
A432S is the first station - a station
Next to that is A53L - not a station
Third is A432222S which is large! - a station
Fourth is A5422CS - not a station
And you want to print those lines which contain names of station. Then the script reg.py below could do the trick.
import re
pattern = re.compile('A\d+S')
fin = open('test.txt', 'r')
for line in fin:
if pattern.search(line):
print line
fin.close()
Now, let me run this script...
[postgres@postgres-desktop:~]$ python reg.py (02-07 00:09)
A432S is the first station - a station
Third is A432222S which is large! - a station
If you want a match only if station name is at the beginning of the line, your regex would be:
'^A\d+S'
You need to 'compile' this again and assign to a pattern variable before doing a 'search'. Here is another example of regex expression.
To match:
xa aDfd455ddD
xa XdG4442sDt
xa cCC123Xt
The pattern here is 'xa' followed by space, followed by group of letters, followed by digits, followed by letters. Regex is
'xa [a-z,A-Z]+\d+[a-z,A-Z]+'
Again, keep in mind that the '+' sign means 'one or more instance of the same thing' and not concatenation of regular expression.
----------------------------------------------------------------------
EXTRACTING DATA USING REGEX
From our station example at the beginning, suppose, say for a station name of A42S, you want to extract the numerical part at the center ('42' in this station), how do you do it? Well, all you need to do is to enclose the regex that you want to extract inside a pair of parentheses and use the 'group' command.
The regex for station name is: 'A\d+S'
You want the digit portion which is '\d+'. So, enclose this by parentheses and assign it to a new pattern:
>>> pattern1 = re.compile('A(\d+)S')
Then, do a search followed by 'group'
>>> s = 'I am going to A4326S today'
>>> m = pattern1.search(s)
>>> m.group(1)
'4326'
Now, what if there are parentheses are part of the string to be searched? All you need to do is 'escape' it using '\'.
For example, you might be trying to match 'origin(23 45)'. Your regex would be:
'origin\(\d+ \d+\)'
If '23' and '45' are x and y coordinates, respectively and you want to extract them, it will be as follows. Enclose the digits that you want to extract with parentheses, do a search, followed by group:
>>> pattern2 = re.compile('origin\((\d+) (\d+)\)')
>>> s = 'display picture origin(23 45)'
>>> m = pattern2.search(s)
>>> x, y = m.group(1,2)
>>> x
'23'
>>> y
'45'
For example, a station name might be an 'A', followed by numbers, followed by 'S'. Below are examples of station name.
A50S
A234S
A3S
A5425S
You might want to know whether a line of text contain a station name in it. Optionally, you might want to pull, say, '50' out of 'A50S'. Regular expression or regex is your tool. Below are the most common regex I use. For more complete discussion, read this.
Use the symbol ------------ To match
below
\d any decimal digit
\D any non-digit
\s any white space character like space and tab
\S any non-white space character
[a-z,A-Z] any alphabet letter
[a, b, c] either 'a' or 'b' or 'c'
. any character except newline
The pattern above would only match one item. For example, \d would only match 1 digit. If you want to match 2 digits, you need to place \d\d. If there could be 2 or 3 or 4 or more digits but you don't know on what particular instance it would be, this approach may not work. The way to approach that is to use the following 'special characters'
Use this --------- If you want to match, with respect
Special Character to the previous symbol (see above)
* zero or more instance of same thing
+ one or more instance of same thing
So, if you say '\d*', it would match even if there is no digit. '\d+' would match if there is at least one digit.
Now, the regex formed above would match anywhere in the line. There are times when you want a match only if it is at the beginning or end. To accomplish this, use the following 'anchors' at the beginning (use ^) or end (use $) of your regex
^regex - if you want a match only at the very beginning of the line
regex$ - If you want it at the end only
The example below should make it clear.
You want to find a match for station name which could either be:
A50S
A234S
A3S
A5425S
The pattern that you could see for station name is.
'A' followed by a number, followed by 'S' . The regex is
^ ^ ^
| ________| |
| | |
'A\d+S' ---------------------------------------------------
You would recall that '\d' is a decimal digit and so, one or more of that (the '+' symbol), meaning, one or more decimal digit, would catch any digits following the first digit. Had you used '\d*' instead, it would match even though there is no digit, like 'AS' because zero digit is allowed. Having formed the regex, you could use it as follows.
>>> import re
>>> pattern = re.compile('A\d+S')
>>> s = 'A50S'
>>> if pattern.search(s):
... print 'got a match'
... else:
... print 'no match'
...
got a match
First you import the regex module (re), then compile the regex into a pattern, then use the 'search' method of the pattern with the string to be searched as argument. This 'search' method will return nonzero value if it found a match and none otherwise.
>>> s = 'I went inside A698S and shouted'
>>> if pattern.search(s):
... print 'got a match'
... else:
... print 'no match'
...
got a match
It still got a match even though the station name is in the middle of the string.
>>> s = 'AB453S'
>>> if pattern.search(s):
... print 'got a match'
... else:
... print 'no match'
...
no match
It did not match this time because after 'A', you got 'B' which is not a digit.
So, if you got a file 'test.txt' which contains the following:
A432S is the first station - a station
Next to that is A53L - not a station
Third is A432222S which is large! - a station
Fourth is A5422CS - not a station
And you want to print those lines which contain names of station. Then the script reg.py below could do the trick.
import re
pattern = re.compile('A\d+S')
fin = open('test.txt', 'r')
for line in fin:
if pattern.search(line):
print line
fin.close()
Now, let me run this script...
[postgres@postgres-desktop:~]$ python reg.py (02-07 00:09)
A432S is the first station - a station
Third is A432222S which is large! - a station
If you want a match only if station name is at the beginning of the line, your regex would be:
'^A\d+S'
You need to 'compile' this again and assign to a pattern variable before doing a 'search'. Here is another example of regex expression.
To match:
xa aDfd455ddD
xa XdG4442sDt
xa cCC123Xt
The pattern here is 'xa' followed by space, followed by group of letters, followed by digits, followed by letters. Regex is
'xa [a-z,A-Z]+\d+[a-z,A-Z]+'
Again, keep in mind that the '+' sign means 'one or more instance of the same thing' and not concatenation of regular expression.
----------------------------------------------------------------------
EXTRACTING DATA USING REGEX
From our station example at the beginning, suppose, say for a station name of A42S, you want to extract the numerical part at the center ('42' in this station), how do you do it? Well, all you need to do is to enclose the regex that you want to extract inside a pair of parentheses and use the 'group' command.
The regex for station name is: 'A\d+S'
You want the digit portion which is '\d+'. So, enclose this by parentheses and assign it to a new pattern:
>>> pattern1 = re.compile('A(\d+)S')
Then, do a search followed by 'group'
>>> s = 'I am going to A4326S today'
>>> m = pattern1.search(s)
>>> m.group(1)
'4326'
Now, what if there are parentheses are part of the string to be searched? All you need to do is 'escape' it using '\'.
For example, you might be trying to match 'origin(23 45)'. Your regex would be:
'origin\(\d+ \d+\)'
If '23' and '45' are x and y coordinates, respectively and you want to extract them, it will be as follows. Enclose the digits that you want to extract with parentheses, do a search, followed by group:
>>> pattern2 = re.compile('origin\((\d+) (\d+)\)')
>>> s = 'display picture origin(23 45)'
>>> m = pattern2.search(s)
>>> x, y = m.group(1,2)
>>> x
'23'
>>> y
'45'
Wednesday, January 19, 2011
Functions
Functions must be defined (or coded earlier in the script) before they can be called as shown below:
[postgres@postgres-desktop:~]$ python myfunction.py (01-18 00:06)
Traceback (most recent call last):
File "myfunction.py", line 1, in <module>
a = square(10)
NameError: name 'square' is not defined
Below is a listing of myfunction.py
a = square(10)
print a
def square(x):
return x*x
You use the keyword "def" to define a function. Just like in C, the function arguments goes inside the parentheses and are parsed in the same order that they are passed. You then terminate the "def" statement with a colon. You then indent the contents of the function. A function may, or may not return a value, pretty much like in C.
The reason why an exception was thrown when the code was run is that the function was defined after it was called and hence, the interpreter is not yet aware of its existence when the call was made. Remember, the way python works is that it reads the py file in sequence, starting at the beginning, interpreting it, and then executing the interpreted line.
To correct this, place the function definition ahead of its call like this:
def square(x):
return x*x
a = square(10)
print a
When you run it, the result is as expected:
[postgres@postgres-desktop:~]$ python myfunction.py (01-18 00:18)
100
Python knew the end of function definition when the indentation ends. One thing with python is that you could mix the function definition with the the main body and still works fine. In C, you can't do this. However, this could easily make yourself confused. So, avoid it. Below is an example:
# my function.py
y = 10
def square(x):
return x*x
a = square(y)
print a
Below is the result when you run it:
[postgres@postgres-desktop:~]$ python myfunction.py (01-18 00:23)
100
As you could see, it still runs without error.
You could define an empty function definition like this:
def square(x):
pass
which is equivalent to this C function
float square(float x)
{
}
DEFAULT VALUES
It is very easy to set default values in function definition. To illustrate:
>>> def whoAmI(name, age, sex='male'):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
>>> whoAmI('John', 33)
I am John, 33 yrs old, male
>>> whoAmI('Ann', 24, 'female')
I am Ann, 24 yrs old, female
In the first function call, only the name and age were passed. Since nothing was supplied for sex (i.e., only 2 arguments were supplied), it used the default value of sex, which is "male".
In the second call, all 3 arguments were supplied and the supplied value for sex was used.
Remember, the order by which the arguments were passed, will be the order they will be assigned on the variables at the receiving function.
Another thing to remember for assigning default values is that they should be at the end and not to be followed by non-default argument as shown below:
>>> def whoAmI(name, sex='male', age):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
File "<stdin>", line 1
SyntaxError: non-default argument follows default argument
NAMED ARGUMENTS
Use of named arguments is illustrated below:
>>> def whoAmI(name, age, sex):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
>>> whoAmI(age=20, sex='female', name='Mary')
I am Mary, 20 yrs old, female
As you could see, using named argument when you make a function call enables you to pass the arguments in any order you want and still get the desired result. The added advantages are: 1. It makes your code clearer because it reminds you which values goes to which variables when you make the call, and 2. It is easier to handle default values as illustrated below:
>>> def whoAmI(name, sex='male', age=25):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
>>> whoAmI('Kim', age=44)
I am Kim, 44 yrs old, male
>>> whoAmI(sex='female', name='Mary')
I am Mary, 25 yrs old, female
VARIABLE NUMBER OF ARGUMENTS
Remember the printf function in C? You pass any number of arguments and it works fine. Here is how you do it in python:
>>> def f(*v):
... for u in v:
... print u
...
>>> f(10, 20, 30)
10
20
30
>>> f(1234)
1234
>>> f('are','you','human',12.5)
are
you
human
12.5
Notice the asterisk (*) before the variable 'v' in the function argument. It is like saying that this variable will be a list to hold all parameters that were passed. You access the function arguments by iterating this 'list' variable.
You could actually combine the positional arguments with variable arguments like this:
>>> def g(positional, *variable):
... print 'positional argument is: ' + str(positional)
... print 'variable arguments are: ' + str(variable)
...
>>> g(10, 20, 30, 40)
positional argument is: 10
variable arguments are: (20, 30, 40)
Here is another example:
>>> def h(a, b, c, *d):
... print 'a=' + str(a)
... print 'b=' + str(b)
... print 'c=' + str(c)
... print 'variable argument d=' + str(d)
...
>>> h(11, 22, 33, 44, 55, 66, 77, 88)
a=11
b=22
c=33
variable argument d=(44, 55, 66, 77, 88)
Positional arguments should come first before variable arguments or there will be an exception.
KEYWORD ARGUMENTS
Here is how you handle a variable number of keyword arguments - you place double asterisk before the variable name on the receiving function and then, the keyword arguments are converted into a dictionary. The variable names that were passed were converted into a string.
>>> def f(**k):
... print k
...
>>> f(a='apple', b='banana')
{'a': 'apple', 'b': 'banana'}
>>> f(x=1, y='yarn',z=2)
{'y': 'yarn', 'x': 1, 'z': 2}
GENERAL FORMAT OF FUNCTION ARGUMENTS
The general format is that you have the positional arguments first, followed by variable arguments, then finally, the variable keyword arguments. Here is an example:
>>> def g(a,b, *args, **kwargs):
... print 'positional arguments are: a=' + str(a) + ', b=' + str(b)
... print 'variable arguments args=' + str(args)
... print 'keyword arguments kwargs=' + str(kwargs)
...
>>> g(10, 20, 30, 40, 50, 60, m='model', c='cat', x=100)
positional arguments are: a=10, b=20
variable arguments args=(30, 40, 50, 60)
keyword arguments kwargs={'x': 100, 'c': 'cat', 'm': 'model'}
FUNCTIONS AS FIRST CLASS OBJECTS IN PYTHON
In python, functions are treated just like strings or integers in the sense that you could pass them around as if they were data. The examples below illustrates the important points:
>>> def add(x,y):
... print 'inside add function'
... return x+y
...
>>> def sub(x,y):
... print 'inside sub function'
... return x-y
...
>>> def multiply(x,y):
... print 'inside multiply function'
... return x*y
...
>>> z = add # you could assign the function to a variable
>>> z(3,4) # and then invoke it through that variable
inside add function
7
>>> def doThis(func,x,y): # invokes the function passed to this function
... print 'inside doThis function, about to call another function'
... return func(x,y)
...
>>> doThis(sub,7,5)
inside doThis function, about to call another function
inside sub function
2
>>> doThis(multiply,4,5)
inside doThis function, about to call another function
inside multiply function
20
You could also create a dictionary of function and invoke it.
>>> switch = {'add':add, 'sub':sub, 'mul':multiply}
>>> switch['add'](4,3)
inside add function
7
>>> x = 'mul'
>>> switch[x](3,5)
inside multiply function
15
So, if you miss C's or VB's 'switch' statement, the above example demonstrates how to elegantly handle it in python.
[postgres@postgres-desktop:~]$ python myfunction.py (01-18 00:06)
Traceback (most recent call last):
File "myfunction.py", line 1, in <module>
a = square(10)
NameError: name 'square' is not defined
Below is a listing of myfunction.py
a = square(10)
print a
def square(x):
return x*x
You use the keyword "def" to define a function. Just like in C, the function arguments goes inside the parentheses and are parsed in the same order that they are passed. You then terminate the "def" statement with a colon. You then indent the contents of the function. A function may, or may not return a value, pretty much like in C.
The reason why an exception was thrown when the code was run is that the function was defined after it was called and hence, the interpreter is not yet aware of its existence when the call was made. Remember, the way python works is that it reads the py file in sequence, starting at the beginning, interpreting it, and then executing the interpreted line.
To correct this, place the function definition ahead of its call like this:
def square(x):
return x*x
a = square(10)
print a
When you run it, the result is as expected:
[postgres@postgres-desktop:~]$ python myfunction.py (01-18 00:18)
100
Python knew the end of function definition when the indentation ends. One thing with python is that you could mix the function definition with the the main body and still works fine. In C, you can't do this. However, this could easily make yourself confused. So, avoid it. Below is an example:
# my function.py
y = 10
def square(x):
return x*x
a = square(y)
print a
Below is the result when you run it:
[postgres@postgres-desktop:~]$ python myfunction.py (01-18 00:23)
100
As you could see, it still runs without error.
You could define an empty function definition like this:
def square(x):
pass
which is equivalent to this C function
float square(float x)
{
}
DEFAULT VALUES
It is very easy to set default values in function definition. To illustrate:
>>> def whoAmI(name, age, sex='male'):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
>>> whoAmI('John', 33)
I am John, 33 yrs old, male
>>> whoAmI('Ann', 24, 'female')
I am Ann, 24 yrs old, female
In the first function call, only the name and age were passed. Since nothing was supplied for sex (i.e., only 2 arguments were supplied), it used the default value of sex, which is "male".
In the second call, all 3 arguments were supplied and the supplied value for sex was used.
Remember, the order by which the arguments were passed, will be the order they will be assigned on the variables at the receiving function.
Another thing to remember for assigning default values is that they should be at the end and not to be followed by non-default argument as shown below:
>>> def whoAmI(name, sex='male', age):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
File "<stdin>", line 1
SyntaxError: non-default argument follows default argument
NAMED ARGUMENTS
Use of named arguments is illustrated below:
>>> def whoAmI(name, age, sex):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
>>> whoAmI(age=20, sex='female', name='Mary')
I am Mary, 20 yrs old, female
As you could see, using named argument when you make a function call enables you to pass the arguments in any order you want and still get the desired result. The added advantages are: 1. It makes your code clearer because it reminds you which values goes to which variables when you make the call, and 2. It is easier to handle default values as illustrated below:
>>> def whoAmI(name, sex='male', age=25):
... print 'I am ' + name + ', ' + str(age) + ' yrs old, ' + sex
...
>>> whoAmI('Kim', age=44)
I am Kim, 44 yrs old, male
>>> whoAmI(sex='female', name='Mary')
I am Mary, 25 yrs old, female
VARIABLE NUMBER OF ARGUMENTS
Remember the printf function in C? You pass any number of arguments and it works fine. Here is how you do it in python:
>>> def f(*v):
... for u in v:
... print u
...
>>> f(10, 20, 30)
10
20
30
>>> f(1234)
1234
>>> f('are','you','human',12.5)
are
you
human
12.5
Notice the asterisk (*) before the variable 'v' in the function argument. It is like saying that this variable will be a list to hold all parameters that were passed. You access the function arguments by iterating this 'list' variable.
You could actually combine the positional arguments with variable arguments like this:
>>> def g(positional, *variable):
... print 'positional argument is: ' + str(positional)
... print 'variable arguments are: ' + str(variable)
...
>>> g(10, 20, 30, 40)
positional argument is: 10
variable arguments are: (20, 30, 40)
Here is another example:
>>> def h(a, b, c, *d):
... print 'a=' + str(a)
... print 'b=' + str(b)
... print 'c=' + str(c)
... print 'variable argument d=' + str(d)
...
>>> h(11, 22, 33, 44, 55, 66, 77, 88)
a=11
b=22
c=33
variable argument d=(44, 55, 66, 77, 88)
Positional arguments should come first before variable arguments or there will be an exception.
KEYWORD ARGUMENTS
Here is how you handle a variable number of keyword arguments - you place double asterisk before the variable name on the receiving function and then, the keyword arguments are converted into a dictionary. The variable names that were passed were converted into a string.
>>> def f(**k):
... print k
...
>>> f(a='apple', b='banana')
{'a': 'apple', 'b': 'banana'}
>>> f(x=1, y='yarn',z=2)
{'y': 'yarn', 'x': 1, 'z': 2}
GENERAL FORMAT OF FUNCTION ARGUMENTS
The general format is that you have the positional arguments first, followed by variable arguments, then finally, the variable keyword arguments. Here is an example:
>>> def g(a,b, *args, **kwargs):
... print 'positional arguments are: a=' + str(a) + ', b=' + str(b)
... print 'variable arguments args=' + str(args)
... print 'keyword arguments kwargs=' + str(kwargs)
...
>>> g(10, 20, 30, 40, 50, 60, m='model', c='cat', x=100)
positional arguments are: a=10, b=20
variable arguments args=(30, 40, 50, 60)
keyword arguments kwargs={'x': 100, 'c': 'cat', 'm': 'model'}
FUNCTIONS AS FIRST CLASS OBJECTS IN PYTHON
In python, functions are treated just like strings or integers in the sense that you could pass them around as if they were data. The examples below illustrates the important points:
>>> def add(x,y):
... print 'inside add function'
... return x+y
...
>>> def sub(x,y):
... print 'inside sub function'
... return x-y
...
>>> def multiply(x,y):
... print 'inside multiply function'
... return x*y
...
>>> z = add # you could assign the function to a variable
>>> z(3,4) # and then invoke it through that variable
inside add function
7
>>> def doThis(func,x,y): # invokes the function passed to this function
... print 'inside doThis function, about to call another function'
... return func(x,y)
...
>>> doThis(sub,7,5)
inside doThis function, about to call another function
inside sub function
2
>>> doThis(multiply,4,5)
inside doThis function, about to call another function
inside multiply function
20
You could also create a dictionary of function and invoke it.
>>> switch = {'add':add, 'sub':sub, 'mul':multiply}
>>> switch['add'](4,3)
inside add function
7
>>> x = 'mul'
>>> switch[x](3,5)
inside multiply function
15
So, if you miss C's or VB's 'switch' statement, the above example demonstrates how to elegantly handle it in python.
Monday, January 10, 2011
Using Python to Access OSI Soft's PI Data
If you are not using PI historian (or don't know what it is), move on. This post is not for you.
First of all, for this to work, you will need pisdk installed. I think you already have it if you could access PI data using excel. You will then call it via COM service. Code snippet below:
import time
from datetime import datetime, timedelta
from win32com.client import Dispatch
tdMax = timedelta(minutes=15)
pisdk = Dispatch('PISDK.PISDK')
myServer = pisdk.Servers('PI_ServerName')
con =Dispatch('PISDKDlg.Connections')
con.Login(myServer,'username','password',1,0)
piTimeStart = Dispatch('PITimeServer.PITimeFormat')
piTimeEnd = Dispatch('PITimeServer.PITimeFormat')
piTimeStart.InputString = '08-25-2009 00:00:00'
piTimeEnd.InputString = '08-25-2009 23:59:59'
sampleAsynchStatus = Dispatch('PISDKCommon.PIAsynchStatus')
samplePoint = myServer.PIPoints['tagname']
sampleValues = samplePoint.Data.Summaries2(piTimeStart,piTimeEnd,'1h',5,0,sampleAsynchStatus)
t0 = datetime.now()
while True:
try:
valsum = sampleValues("Average").Value # retrieve the value
break # it will get out of infinite loop when there is no error
except: # it failed because server needs more time
td = datetime.now() - t0
if td > tdMax:
print "It is taking so long to get PI data ! Exiting now ...."
exit(1)
time.sleep(3)
i = 1
while i < valsum.Count+1:
print valsum(i).Value, valsum(i).ValueAttributes('Percentgood').Value
i += 1
...
61.4011819388 100.0
61.3148685875 100.0
61.4948477524 100.0
61.2000007629 100.0
61.2000007629 100.0
61.2000007629 100.0
63.8442935978 100.0
64.2453963685 100.0
66.205138361 100.0
71.4747024728 100.0
63.8786754349 100.0
64.3719732268 100.0
64.6296242787 100.0
61.8953031561 100.0
57.0552419993 100.0
57.2999992371 100.0
58.2445955483 100.0
57.5679091366 100.0
59.6997264523 100.0
60.9096459478 100.0
59.7756373596 100.0
55.1742569927 100.0
59.7151307987 100.0
For my officemates. That is just too much work, right? Actually, if you will write the application in VB, your code will look just like that. For this reason, I wrote a convenience class to hide all those warts and simplify everything as follows:
import sys
sys.path.append(r'M:\EMS\pyhomebrew')
from osipi import osipi
# with time stamp
b = osipi('username','password') # username, then, password, and optionally, PI server
for x, y in b.getAverage('pi-tag here','04-01-2010 00:00','04-01-2010 08:00','1h',True):
print x, y # x is time, y is value
# without time stamp
b = osipi('username','password') # username, then, password, and optionally, PI server
for x in b.getAverage('pi-tag here','04-01-2010 00:00','04-01-2010 08:00','1h'):
print x
# to get the compressed values
for x, y in b.getCompressed('pi-tag here','04-01-2010 00:00','04-01-2010 00:01',True):
print x, y # x is time, y is value
Why not use excel? You use excel when it is appropriate but there are instances when it is not. For one, excel has limitations on the number of rows. If you go back too far in time, excel could choke. I was told that the latest version of excel has virtually no limit on the number of rows - not true! I dumped the OAG database via rio and it could not read the whole thing!
First of all, for this to work, you will need pisdk installed. I think you already have it if you could access PI data using excel. You will then call it via COM service. Code snippet below:
import time
from datetime import datetime, timedelta
from win32com.client import Dispatch
tdMax = timedelta(minutes=15)
pisdk = Dispatch('PISDK.PISDK')
myServer = pisdk.Servers('PI_ServerName')
con =Dispatch('PISDKDlg.Connections')
con.Login(myServer,'username','password',1,0)
piTimeStart = Dispatch('PITimeServer.PITimeFormat')
piTimeEnd = Dispatch('PITimeServer.PITimeFormat')
piTimeStart.InputString = '08-25-2009 00:00:00'
piTimeEnd.InputString = '08-25-2009 23:59:59'
sampleAsynchStatus = Dispatch('PISDKCommon.PIAsynchStatus')
samplePoint = myServer.PIPoints['tagname']
sampleValues = samplePoint.Data.Summaries2(piTimeStart,piTimeEnd,'1h',5,0,sampleAsynchStatus)
t0 = datetime.now()
while True:
try:
valsum = sampleValues("Average").Value # retrieve the value
break # it will get out of infinite loop when there is no error
except: # it failed because server needs more time
td = datetime.now() - t0
if td > tdMax:
print "It is taking so long to get PI data ! Exiting now ...."
exit(1)
time.sleep(3)
i = 1
while i < valsum.Count+1:
print valsum(i).Value, valsum(i).ValueAttributes('Percentgood').Value
i += 1
...
61.4011819388 100.0
61.3148685875 100.0
61.4948477524 100.0
61.2000007629 100.0
61.2000007629 100.0
61.2000007629 100.0
63.8442935978 100.0
64.2453963685 100.0
66.205138361 100.0
71.4747024728 100.0
63.8786754349 100.0
64.3719732268 100.0
64.6296242787 100.0
61.8953031561 100.0
57.0552419993 100.0
57.2999992371 100.0
58.2445955483 100.0
57.5679091366 100.0
59.6997264523 100.0
60.9096459478 100.0
59.7756373596 100.0
55.1742569927 100.0
59.7151307987 100.0
For my officemates. That is just too much work, right? Actually, if you will write the application in VB, your code will look just like that. For this reason, I wrote a convenience class to hide all those warts and simplify everything as follows:
import sys
sys.path.append(r'M:\EMS\pyhomebrew')
from osipi import osipi
# with time stamp
b = osipi('username','password') # username, then, password, and optionally, PI server
for x, y in b.getAverage('pi-tag here','04-01-2010 00:00','04-01-2010 08:00','1h',True):
print x, y # x is time, y is value
# without time stamp
b = osipi('username','password') # username, then, password, and optionally, PI server
for x in b.getAverage('pi-tag here','04-01-2010 00:00','04-01-2010 08:00','1h'):
print x
# to get the compressed values
for x, y in b.getCompressed('pi-tag here','04-01-2010 00:00','04-01-2010 00:01',True):
print x, y # x is time, y is value
Why not use excel? You use excel when it is appropriate but there are instances when it is not. For one, excel has limitations on the number of rows. If you go back too far in time, excel could choke. I was told that the latest version of excel has virtually no limit on the number of rows - not true! I dumped the OAG database via rio and it could not read the whole thing!
Thursday, January 6, 2011
File Handling
Here are common problems and how they are handled in python. If I have a custom-made module that makes it easier, I will label it as "Using my custom module". This custom module is not available except for my office mates.
DELETING ALL FILES AND SUBFOLDERS IN A DIRECTORY named directoryName
import shutil
shutil.rmtree('directoryName')
DELETING A SINGLE FILE
import os
if os.path.exists('nameOfFileToDeleteIncludingCompletePath'):
os.remove('nameOfFileToDeleteIncludingCompletePath')
You need to check first if file exist before deleting or it may throw an exception. Alternatively, you could do it like below and accomplish the same thing.
import os
try:
os.remove('nameOfFileToDeleteIncludingCompletePath')
except:
pass
COPYING FILES
Though this could also be done indirectly through a system call (see last topic of this post), python has a module called shutil that handles this.
import shutil
shutil.copy('sourcefileName','destination')
'destination' could either be another filename or a directory.
GETTING THE FILES WHOSE NAMES FOLLOW A PATTERN
You can use the same pattern matching rule when you use the command line for directory listing. As an example, when you want the files which starts with "abc", followed by anything plus a txt extension, you would do "dir abc*.txt". To get the list of these files in python,
import glob
fileNameList = glob.glob('abc*.txt')
That is, you supply "glob" with same patterns as you would supply the "dir" or "ls" command. If the files are not in the current working directory, you do
import glob
fileNameList = glob.glob('\\path\\to\\file\\abc*.txt')
GETTING THE LATEST FILE IN A DIRECTORY
You most probably would want to do this in conjunction with the file matching capability above. First of all, some background. You can get the time of most recent modification of a file by doing this:
import os
fileAge = os.stat('filename')[8]
os.stat actually gives you the file attributes in a list and the 8th (relative to 0) element gives you the time of most recent modification in milliseconds. The larger this number, the more recent it is.
However, often times, you have a group of files and you want to use the latest. For example your files may look like. alarm_235.txt, alarm_111.txt, alarm_241.txt, and so on. I would do it this way:
import os
import glob
candidateFiles = glob.glob('\\path\\to\\alarm_*.txt')
fileModificationDate = [os.stat(f)[8] for f in candidateFiles]
latestFile = candidateFiles[fileModificationDate.index(max(fileModificationDate)]
"fileModificationDate" is the list containing the "modification date" of the files and in same order as the corresponding files in candidateFiles - i.e. - the first element in fileModificationDate the modification date of the first element in candidateFiles. Therefore, if the largest number (a.k.a. most recent) in fileModificationDate is the 3rd element, then, the corresponding file is the 3rd element in candidateFiles.
Now, max(fileModificationDate) would give you the largest number in fileModificationDate list. fileModificationDate.index(max(fileModificationDate)) will give you the index of this largest number in fileModificationDate. This index, therefore, when used in candidateFiles would give you the filename corresponding to the latest file.
For my officemates: I use this quite a lot and so, I decided to create a module for this. It could actually give you a list with the 1st element being the latest, 2nd element the next latest, and so on. Here is how you use it:
import sys
sys.path.append('M:\\EMS\\pyhomebrew')
from qFileUtils2 import getLatestFiles
latestFile = getLatestFiles('filePattern')[0]
filePattern is the same argument that you supply to glob.
If you want the top 3 latest files:
import sys
sys.path.append('M:\\EMS\\pyhomebrew')
from qFileUtils2 import getLatestFiles
latestFiles = getLatestFiles('filePattern', 3)
The latest file would be latestFiles[0], followed by latestFiles[1], followed by latestFiles[2]. Notice that getLatestFiles may have 1 or 2 arguments. If you supply only 1 argument, it will assume that you need the top 2. In other words, these 2 are identical:
getLatestFiles('filePattern')
getLatestFiles('filePattern', 2)
CHECKING IF A FILE IS UPDATED
Often times, you run a script which uses a data file and you forgot to update the data file. So, it used the old data file and all along, you thought you are done. It would be nice if the script could check the modification/creation time, compare that to the current time, and decide that the data file is not updated and tell you so. For example, you might have a new database release every 2 weeks. Therefore, if I will be running the script right now and the time difference between now and when the data file was last modified is longer than 10 hours, it is possible that the data file was indeed old! I have probably finished with my data file in the morning and running my script in the afternoon. This interval is certainly less than 10 hours.
This is how you do it:
import os
ageInHours = 10 # let us use 10 hrs for now
t1 = os.path.getmtime(fileName)
t2 = time.time()
ageInSeconds = 3600 * ageInHours
if t2-t1 > ageInSeconds: # file is older than 10 hrs
print 'data file is not updated!'
else:
print 'data file is updated!'
For my officemates: I have a module that does this - the function isUpdated.
import sys
sys.path.append('M:\\EMS\\pyhomebrew')
from qFileUtils2 import isUpdated
if not isUpdated('dataFile'):
print 'Your data file is not updated. Exiting now...'
exit(1)
# continue with your script
isUpdated may take 2 arguments like this:
isUpdated('filename',48)
where 48 hours is used as basis for saying that the file is not updated; 10 hours is the default if you don't supply the 2nd argument.
RUNNING SYSTEM COMMANDS
If you want to run system command from python:
import os
os.system('notepad test.txt') # runs notepad and open test.txt
So, whatever you would type in the command line, you simply place inside the parentheses. The only downside is that the result is not echoed back to your script; only the return status (0 if no errors are encountered, nonzero otherwise). For example, "dir *.*" will not return the name of the files on the current working directory. It will just tell you that the command executed and terminated normally via the return value of zero.
How do you pass environment variable to the system command? It is like this:
import os
os.environ['PYTHONPATH'] = r'D:\path\to\other\modules'
os.system('python yourScript.py') # this invokes another python script; PYTHONPATH is passed
Take note that changing the environment variable in this way is not permanent. It is unset again after your script terminates.
DELETING ALL FILES AND SUBFOLDERS IN A DIRECTORY named directoryName
import shutil
shutil.rmtree('directoryName')
DELETING A SINGLE FILE
import os
if os.path.exists('nameOfFileToDeleteIncludingCompletePath'):
os.remove('nameOfFileToDeleteIncludingCompletePath')
You need to check first if file exist before deleting or it may throw an exception. Alternatively, you could do it like below and accomplish the same thing.
import os
try:
os.remove('nameOfFileToDeleteIncludingCompletePath')
except:
pass
COPYING FILES
Though this could also be done indirectly through a system call (see last topic of this post), python has a module called shutil that handles this.
import shutil
shutil.copy('sourcefileName','destination')
'destination' could either be another filename or a directory.
GETTING THE FILES WHOSE NAMES FOLLOW A PATTERN
You can use the same pattern matching rule when you use the command line for directory listing. As an example, when you want the files which starts with "abc", followed by anything plus a txt extension, you would do "dir abc*.txt". To get the list of these files in python,
import glob
fileNameList = glob.glob('abc*.txt')
That is, you supply "glob" with same patterns as you would supply the "dir" or "ls" command. If the files are not in the current working directory, you do
import glob
fileNameList = glob.glob('\\path\\to\\file\\abc*.txt')
GETTING THE LATEST FILE IN A DIRECTORY
You most probably would want to do this in conjunction with the file matching capability above. First of all, some background. You can get the time of most recent modification of a file by doing this:
import os
fileAge = os.stat('filename')[8]
os.stat actually gives you the file attributes in a list and the 8th (relative to 0) element gives you the time of most recent modification in milliseconds. The larger this number, the more recent it is.
However, often times, you have a group of files and you want to use the latest. For example your files may look like. alarm_235.txt, alarm_111.txt, alarm_241.txt, and so on. I would do it this way:
import os
import glob
candidateFiles = glob.glob('\\path\\to\\alarm_*.txt')
fileModificationDate = [os.stat(f)[8] for f in candidateFiles]
latestFile = candidateFiles[fileModificationDate.index(max(fileModificationDate)]
"fileModificationDate" is the list containing the "modification date" of the files and in same order as the corresponding files in candidateFiles - i.e. - the first element in fileModificationDate the modification date of the first element in candidateFiles. Therefore, if the largest number (a.k.a. most recent) in fileModificationDate is the 3rd element, then, the corresponding file is the 3rd element in candidateFiles.
Now, max(fileModificationDate) would give you the largest number in fileModificationDate list. fileModificationDate.index(max(fileModificationDate)) will give you the index of this largest number in fileModificationDate. This index, therefore, when used in candidateFiles would give you the filename corresponding to the latest file.
For my officemates: I use this quite a lot and so, I decided to create a module for this. It could actually give you a list with the 1st element being the latest, 2nd element the next latest, and so on. Here is how you use it:
import sys
sys.path.append('M:\\EMS\\pyhomebrew')
from qFileUtils2 import getLatestFiles
latestFile = getLatestFiles('filePattern')[0]
filePattern is the same argument that you supply to glob.
If you want the top 3 latest files:
import sys
sys.path.append('M:\\EMS\\pyhomebrew')
from qFileUtils2 import getLatestFiles
latestFiles = getLatestFiles('filePattern', 3)
The latest file would be latestFiles[0], followed by latestFiles[1], followed by latestFiles[2]. Notice that getLatestFiles may have 1 or 2 arguments. If you supply only 1 argument, it will assume that you need the top 2. In other words, these 2 are identical:
getLatestFiles('filePattern')
getLatestFiles('filePattern', 2)
CHECKING IF A FILE IS UPDATED
Often times, you run a script which uses a data file and you forgot to update the data file. So, it used the old data file and all along, you thought you are done. It would be nice if the script could check the modification/creation time, compare that to the current time, and decide that the data file is not updated and tell you so. For example, you might have a new database release every 2 weeks. Therefore, if I will be running the script right now and the time difference between now and when the data file was last modified is longer than 10 hours, it is possible that the data file was indeed old! I have probably finished with my data file in the morning and running my script in the afternoon. This interval is certainly less than 10 hours.
This is how you do it:
import os
ageInHours = 10 # let us use 10 hrs for now
t1 = os.path.getmtime(fileName)
t2 = time.time()
ageInSeconds = 3600 * ageInHours
if t2-t1 > ageInSeconds: # file is older than 10 hrs
print 'data file is not updated!'
else:
print 'data file is updated!'
For my officemates: I have a module that does this - the function isUpdated.
import sys
sys.path.append('M:\\EMS\\pyhomebrew')
from qFileUtils2 import isUpdated
if not isUpdated('dataFile'):
print 'Your data file is not updated. Exiting now...'
exit(1)
# continue with your script
isUpdated may take 2 arguments like this:
isUpdated('filename',48)
where 48 hours is used as basis for saying that the file is not updated; 10 hours is the default if you don't supply the 2nd argument.
RUNNING SYSTEM COMMANDS
If you want to run system command from python:
import os
os.system('notepad test.txt') # runs notepad and open test.txt
So, whatever you would type in the command line, you simply place inside the parentheses. The only downside is that the result is not echoed back to your script; only the return status (0 if no errors are encountered, nonzero otherwise). For example, "dir *.*" will not return the name of the files on the current working directory. It will just tell you that the command executed and terminated normally via the return value of zero.
How do you pass environment variable to the system command? It is like this:
import os
os.environ['PYTHONPATH'] = r'D:\path\to\other\modules'
os.system('python yourScript.py') # this invokes another python script; PYTHONPATH is passed
Take note that changing the environment variable in this way is not permanent. It is unset again after your script terminates.
Tuesday, January 4, 2011
Exception Handling
If something happened and python could not go on it will throw an exception. Like for instance, you want to open a non-existing file. You could either do extra processing when the exception happen or ignore it. I won't get into too much detail here and discuss only that which I normally do. Below is how you handle an exception: you place the code inside the except-try block.
try:
# statements that could potentially throw an exception
except:
# Now, you have an exception. What do you want to do?
There are instances where you don't want the script to quit when it encounter an exception and just move on. In this case, you just place a 'pass' stetement in the except clause like this:
try:
# statements that could potentially throw an exception
except:
pass
If there are specific exceptions that you might wish to catch, place it in the except statement as shown below. Without qualifier, the except clause catches everything.
>>> s = 'abc'
>>> b = float(s)
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
ValueError: invalid literal for float(): abc
In the example above, we want to convert a string value to a floating point. Since you don't have numbers in string s, it throws an exception ValueError. You can handle this exception as shown below. As you could see, the exception ValueError was catched and the code inside the except clause was executed. Since the exception was caught in your code, the script went to completion and no 'Traceback' was shown.
>>> try:
... b = float(s)
... except ValueError:
... print 's is not a floating point string'
...
s is not a floating point string
If I have a script that runs via scheduler, I would place statements in the except clause to send me e-mail telling me what went wrong. In that way, I am notified in real-time and need not look on lengthy logs. Below is how I did it.
Another area where I find exception handling useful is when you have a backup and would want to shift to that when the primary resource is down. For example, say, you have an Oracle server Ora1 which is your primary source of data for your script and then there is backup server Ora2, and a third backup Ora3, this is how I use exception handling to take advantage of this redundancy:
except:
print 'All Oracle servers are down!!! Exiting now ...'
exit(1)
cur = connection.cursor()
cur.execute("Select * from userTable")
Now, if there is at least one Oracle server up, the script may continue without any problems. Better yet, use email to notify yourself if any of the Oracle server is down using the previous example.
try:
# statements that could potentially throw an exception
except:
# Now, you have an exception. What do you want to do?
There are instances where you don't want the script to quit when it encounter an exception and just move on. In this case, you just place a 'pass' stetement in the except clause like this:
try:
# statements that could potentially throw an exception
except:
pass
If there are specific exceptions that you might wish to catch, place it in the except statement as shown below. Without qualifier, the except clause catches everything.
>>> s = 'abc'
>>> b = float(s)
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
ValueError: invalid literal for float(): abc
In the example above, we want to convert a string value to a floating point. Since you don't have numbers in string s, it throws an exception ValueError. You can handle this exception as shown below. As you could see, the exception ValueError was catched and the code inside the except clause was executed. Since the exception was caught in your code, the script went to completion and no 'Traceback' was shown.
>>> try:
... b = float(s)
... except ValueError:
... print 's is not a floating point string'
...
s is not a floating point string
If I have a script that runs via scheduler, I would place statements in the except clause to send me e-mail telling me what went wrong. In that way, I am notified in real-time and need not look on lengthy logs. Below is how I did it.
import smtplib
def emailError(errMsg):
host='mailServerName'
s=smtplib.SMTP(host)
recepients = ['emailAddress1','emailAddress2','emailAddres3'
msg = "Subject: Application1 Had some Problems !!!\n\n" + errMsg
s.sendmail('dummyReplyEmailAddress',recepients,msg)
exit(1)
host='mailServerName'
s=smtplib.SMTP(host)
recepients = ['emailAddress1','emailAddress2','emailAddres3'
msg = "Subject: Application1 Had some Problems !!!\n\n" + errMsg
s.sendmail('dummyReplyEmailAddress',recepients,msg)
exit(1)
try:
fin = file("D:\\somewhere\\sysgenid","r")
except:
emailError("Can't open sysgenid file!\nWas it deleted?\n")
fin = file("D:\\somewhere\\sysgenid","r")
except:
emailError("Can't open sysgenid file!\nWas it deleted?\n")
Another area where I find exception handling useful is when you have a backup and would want to shift to that when the primary resource is down. For example, say, you have an Oracle server Ora1 which is your primary source of data for your script and then there is backup server Ora2, and a third backup Ora3, this is how I use exception handling to take advantage of this redundancy:
import cx_Oracle
try:
connection = cx_Oracle.connect('uid/pwd@Ora1')
except:
try:
connection = cx_Oracle.connect('uid/pwd@Ora2')
except:
try:
connection = cx_Oracle.connect('uid/pwd@Ora3')try:
connection = cx_Oracle.connect('uid/pwd@Ora1')
except:
try:
connection = cx_Oracle.connect('uid/pwd@Ora2')
except:
try:
except:
print 'All Oracle servers are down!!! Exiting now ...'
exit(1)
cur = connection.cursor()
cur.execute("Select * from userTable")
Now, if there is at least one Oracle server up, the script may continue without any problems. Better yet, use email to notify yourself if any of the Oracle server is down using the previous example.
Modules and PYTHONPATH
What makes python so useful are the wealth of modules written to simplify your tasks at hand. You just 'import' these modules and you are ready to go. Normally, the import statements are at the start of your script. The previous post has just shown you a module which allowed you to access window's COM objects.
Normally, if I need to do something, I just google it on the internet. Let me give you a concrete example. There was a time when I want to access an Oracle database using python. I typed "python Oracle" on the google search engine and this is the result:
-----------------------------------------
Normally, if I need to do something, I just google it on the internet. Let me give you a concrete example. There was a time when I want to access an Oracle database using python. I typed "python Oracle" on the google search engine and this is the result:
-----------------------------------------
Python - Oracle Wiki
2 posts - 1 author - Last post: 3 Feb 2010 Getting Started: #!/usr/bin/python import cx_Oracle connstr='scott/tiger' conn = cx_Oracle.connect(connstr) curs = conn.cursor() ...wiki.oracle.com/page/Python |
cx_Oracle
cx_Oracle is a Python extension module that allows access to Oracle databases and conforms to the ... Windows amd64 Installer (Oracle 10g, Python 2.5) ... cx-oracle.sourceforge.net/ |
Using Python With Oracle Database 11g
With the rise of Frameworks, Python is also becoming common for Web application development. If you want to use Python and an Oracle database, this tutorial ... www.oracle.com/technetwork/articles/dsl/python-091105.html |
Python - Oracle FAQ
5 Nov 2010 ... cx_Oracle is a Python extension module that allows access to Oracle databases and conforms to the Python database API specification. ... www.orafaq.com/wiki/Python ------------------------------------------------------------------- #!/usr/bin/python import cx_Oracle connstr='scott/tiger' conn = cx_Oracle.connect(connstr) curs = conn.cursor() curs.arraysize=50 curs.execute('select 2+2 "aaa" ,3*3 from dual') print curs.description print curs.fetchone() conn.close() Resources: - Python Programming Language - Official Website - cx_Oracle module for accessing Oracle Database from Python (DB API 2.0 conformant, updated for 11g) - Pydev, a superb IDE for Python built on top of Eclipse - Mod_python, Apache/Python Integration, introducing Python Server Pages technology - Python Package Index - official repository for 3rd party modules - comp.lang.python - active, helpful community of Python experts - Essbase.py - port of Essbase perl module to ----------------------------------------------------------------------------- This tells me that I need the cx_Oracle module (because of the 'import' statement at the start). So I checked via the python interpreter if I have this module. >>> import cx_Oracle Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named cx_Oracle Ok, I don't have it; I need to get this module and install it. Again, from the first site that I clicked, it has a link on cx_Oracle under "Resources" (please see above). So, I clicked on it and saw several windows installer for different versions of Oracle. I selected the one compatible to our Oracle server and installed it. Now, importing that module from the python interpreter no longer issue any error. Now, I need more documentation. Going back to google result, I selected "Python-Oracle FAQ" and saw these code snippets: --------------------------------------------------------------------------- python
>>> import cx_Oracle
>>> connection = cx_Oracle.connect('scott/tiger@orcl')
>>> # do some stuff here
>>> connection.close()connection = cx_Oracle.connect("uid/pwd@database")
cursor = connection.cursor()
cursor.execute("SELECT COUNT(*) FROM User_Tables")
count = cursor.fetchall()[0][0]
cursor.close()
connection.close()connection = cx_Oracle.connect("uid/pwd@database")
cursor = connection.cursor()
cursor.arraysize = 50
cursor.execute("""
select Col1, Col2, Col3
from SomeTable
where Col4 = :arg_1
and Col5 between :arg_2 and :arg_3""",
arg_1 = "VALUE",
arg_2 = 5,
arg_3 = 15)
for column_1, column_2, column_3 in cursor.fetchall():
print "Values from DB:", column_1, column_2, column_3
cursor.close()
connection.close()--------------------------------------------------Well, there are a lot more but you get the idea. If you don't know SQL, these code snippets may not be enough for you. Fortunately, in my case, I know SQL and these snippets are enough to keep me going. Now, if you don't have administrative rights, you would not be able to install the module at the default location where python expects to find it. To solve this problem, you have to include the installation folder in your PYTHONPATH so that python would find it. To change your PYTHONPATH, you could either convince your administrator to append the installation folder at the end of this environment variable (good luck with that!) or dynamically change PYTHONPATH in your script as follows: import sys sys.path.append(r'/path/to/cx_Oracle') import cx_Oracle sys.path is actually a list containing folders where python would search for modules |
Sunday, January 2, 2011
Manipulating Excel Sheets using Python
There are 2 ways to handle excel sheets using python:
1. Using the xlrd and xlwt
2. Using windows COM object
The first method is very fast and would not need excel application to be installed at the target machine. However, the drawback is that you could only either read an existing spreadsheet or write to NEW spreadsheet. You can't modify an existing spreadsheet. If you really want to modify, you open it for reading, then write your modifications to a temporary file, then overwrite the original file with the temporary file. I don't intend to cover this but here is a tutorial on how to use this module for those who are interested.
The second method is slower and you need to have excel application installed. But then, you have so much more flexibility and modification of existing spreadsheet is possible. I will discuss only this method.
Below is a code snippet on how you do it.
# import the module that you need for handling COM objects
The succeeding discussions are for my officemates only. However, you are welcome to read it to get some idea and perhaps roll out your own convenience class.
As you could see, syntax is identical to that in VB. Actually all spreadsheet manipulation that you could do in VB, you could also do in python and the syntax is exactly the same! To me, it looks ugly. So, I decided to create my own convenience class that hides the ugliness and it works like this:
# import the convenience class. It should be in PYTHONPATH
import sys
sys.path.append(r'M:\EMS\pyhomebrew')
from dxls import dxls
# open a workbook
wb = dxls(filename)
# open a worksheet from that workbook
ws = wb.ws(sheetName)
# read a cell value
cellValue = ws.read(row,col)
# write to a cell
ws.write(row,col,newValue)
# close the workbook and save changes
wb.close()
I wrote this class even before I learned enough to know that there is a convention to capitalize the first letter of the class name. Now, I have used this a lot in my other scripts and correcting it would break them.
I also added some convenience feature like instead of:
cellValue = ws.read(row,10)
you could just say:
cellValue = ws. read(row,'j') # cell column is 'J' which is 10th col
Or if you have a table and table heading for this 10th column is 'Address'
cellValue = ws. read(row,'Address')
If you have row heading, you could also use that as well. All of these are automatically done for you with no extra coding on your part. The only requirement is that the table on your worksheet should be well behaved which means:
1. Table has only 2 lines of title.
2. After table column headings, values immediately follow - i.e., no blank line
3. The next 4 lines after the table column headings have no blanks
It also detects the table boundaries: ws.minRow, ws.maxRow, ws.minCol and ws.maxCol
To iterate through the entire table,
for row in xrange(ws.minRow,ws.maxRow):
for col in xrange(ws.minCol, ws.maxCol):
# do something like multiply the cell contents by 2
# and then write the result on the same row but 100
# columns away from current cell position
ws.write(row,col+100,ws.read(row,col)*2)
As you could see, there is no need to continually check for an empty cell which marks the end of the table - the convenience class has already done it for you.
I mentioned 2 new concepts here: module and PYTHONPATH. That will be the next topic.
1. Using the xlrd and xlwt
2. Using windows COM object
The first method is very fast and would not need excel application to be installed at the target machine. However, the drawback is that you could only either read an existing spreadsheet or write to NEW spreadsheet. You can't modify an existing spreadsheet. If you really want to modify, you open it for reading, then write your modifications to a temporary file, then overwrite the original file with the temporary file. I don't intend to cover this but here is a tutorial on how to use this module for those who are interested.
The second method is slower and you need to have excel application installed. But then, you have so much more flexibility and modification of existing spreadsheet is possible. I will discuss only this method.
Below is a code snippet on how you do it.
# import the module that you need for handling COM objects
from win32com.client import Dispatch # Open a workbook for a given excel file wb = Dispatch('Excel.Application').Workbooks.Open(filename) # Open a worksheet from that workbook ws = self.wb.Sheets(sheetName)
# read a cell valuecellValue = ws.Cells(row,col).Text
# write to a cellws.Cells(row,col).Value = newValue
The succeeding discussions are for my officemates only. However, you are welcome to read it to get some idea and perhaps roll out your own convenience class.
As you could see, syntax is identical to that in VB. Actually all spreadsheet manipulation that you could do in VB, you could also do in python and the syntax is exactly the same! To me, it looks ugly. So, I decided to create my own convenience class that hides the ugliness and it works like this:
# import the convenience class. It should be in PYTHONPATH
import sys
sys.path.append(r'M:\EMS\pyhomebrew')
from dxls import dxls
# open a workbook
wb = dxls(filename)
# open a worksheet from that workbook
ws = wb.ws(sheetName)
# read a cell value
cellValue = ws.read(row,col)
# write to a cell
ws.write(row,col,newValue)
# close the workbook and save changes
wb.close()
I wrote this class even before I learned enough to know that there is a convention to capitalize the first letter of the class name. Now, I have used this a lot in my other scripts and correcting it would break them.
I also added some convenience feature like instead of:
cellValue = ws.read(row,10)
you could just say:
cellValue = ws. read(row,'j') # cell column is 'J' which is 10th col
Or if you have a table and table heading for this 10th column is 'Address'
cellValue = ws. read(row,'Address')
If you have row heading, you could also use that as well. All of these are automatically done for you with no extra coding on your part. The only requirement is that the table on your worksheet should be well behaved which means:
1. Table has only 2 lines of title.
2. After table column headings, values immediately follow - i.e., no blank line
3. The next 4 lines after the table column headings have no blanks
It also detects the table boundaries: ws.minRow, ws.maxRow, ws.minCol and ws.maxCol
To iterate through the entire table,
for row in xrange(ws.minRow,ws.maxRow):
for col in xrange(ws.minCol, ws.maxCol):
# do something like multiply the cell contents by 2
# and then write the result on the same row but 100
# columns away from current cell position
ws.write(row,col+100,ws.read(row,col)*2)
As you could see, there is no need to continually check for an empty cell which marks the end of the table - the convenience class has already done it for you.
I mentioned 2 new concepts here: module and PYTHONPATH. That will be the next topic.
Saturday, January 1, 2011
List Comprehension
Suppose you have a list of integers and you want to create another list whose elements are twice larger than the first list, how would you do it? Here are typical ways of doing it.
>>> a = [10, 20, 30, 40]
>>> b = []
>>> for i in a:
... b.append(2*i)
...
>>> b
[20, 40, 60, 80]
Those new to python may not easily understand what is going on in the above example; perhaps they would approach it like below:
>>> c = []
>>> for i in range(len(a)):
... c[i] = 2*a[i]
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
IndexError: list assignment index out of range
The reason why you have an error above is that array c has no elements and yet you are trying to assign its ith element (there is no ith element) to a new value.
For this approach to be feasible, create a list c which has the same number of elements as list a. The easiest way is to copy list a into list c using the idiom below
>>> c = a[:]
And then, proceed as usual.
>>> for i in range(len(a)):
... c[i] = 2*a[i]
...
>>> c
[20, 40, 60, 80]
If you used
c = a
instead of
c = a[:]
c and a would be referring to the same array and changing 'c' changes 'a' as well. On the other hand, if you used slices. you will get just a copy. Recall that if you leave the left side of ':' blank, it means all the way to the start and leaving the right side blank means all the way to the end.
Now, if you used list comprehension, the code would be:
>>> a = [10, 20, 30, 40]
>>> b = [2*x for x in a]
>>> b
[20, 40, 60, 80]
The way you read this is that, you pick the first element in list 'a', assign it to x, then multiply it by 2, and that is the first element. Then, you pick the second element in 'a', assign it to x, multiply it by 2, and that is the second element, and so on. It is pretty much like:
for x in a:
2*x
except that the result of each loop automatically become an element of a new list. Not only is list comprehension very compact - it is also very fast compared to the ordinary 'for' loop. In the 'for' loop, each line is interpreted as many times as there are elements in the loop; whereas in list comprehension, it is interpreted only once regardless of the list size.
If you want a new array whose elements is the square of each elements of another array, it will be like this:
>>> a
[10, 20, 30, 40]
>>> squared = [x*x for x in a]
>>> squared
[100, 400, 900, 1600]
I guess you now know what is going on. It is possible that instead of 2*x or x*x, you use function like this.
>>> def square(u):
... return u*u
...
>>> a
[10, 20, 30, 40]
>>> c = [square(x) for x in a]
>>> c
[100, 400, 900, 1600]
You could also use conditions. For example, you might want the square of each element only if the element is greater than 20.
>>> a
[10, 20, 30, 40]
>>> d = [square(x) for x in a if x > 20]
>>> d
[900, 1600]
Only 30 and 40 are greater than 20 and so, you only have 2 elements on the new array.
The elements of the new array need not be a function of the elements of the old array at all. For example, I could do something like this:
>>> a
[10, 20, 30, 40]
>>> divisibleBy20 = [1 for x in a if x % 20 == 0]
>>> divisibleBy20
[1, 1]
The above code sets the element of the new array equal to 1 if the old array element is exactly divisible by 20 (i.e., modulo is equal to zero). Only 2 elements are exactly divisible by 20. That is why you only have 2 elements with 1 in it.
Suppose you want to know if a string has 'abc' and 'xyz' in it, what would you do?
>>> s = 'Do you know your abcd?'
>>> if 'abc' in s or 'xyz' in s:
... print 'It is there!'
...
It is there!
Now, suppose you are given a list of strings to search instead of just 2? Maybe you would do it this way:
It is there!
>>> s
'Do you know your abcd?'
>>> searchThis = ['abc','xyz','uuu']
>>> found = False
>>> for x in searchThis:
... if x in s:
... found = True
... break
...
>>> if found:
... print 'found it!'
... else:
... print 'It is not there'
found it!
Well, there is a better way - use for-else:
>>> s
'Do you know your abcd?'
>>> for x in searchThis:
... if x in s:
... print 'found it!'
... break
... else:
... print 'It is not there'
...
found it!
Please note that the 'else' clause is NOT for the 'if' statement. It is for the 'for' statement and is executed if you exhausted the list without issuing a break statement
I think I have found a much better way using list comprehension:
>>> if sum([1 for x in searchThis if x in s]) > 0:
... print 'found it!'
... else:
... print 'It is not there'
...
found it!
To understand what is going on, let us see what we have inside the sum function.
>>> [1 for x in searchThis if x in s]
[1]
It assign a value from searchThis into x and whenever it can find that substring from 's', it places '1' in the new array. So, if none of the elements of searchThis matches, there will be nothing there. Now the sum function adds all elements of a list. If it is an empty list, sum will be zero and greater than zero (there is a match) otherwise.
I just found out that there is still another better way in doing the above using "any" . It is not really list comprehension but the syntax is very similar.
>>> s = 'Do you know your abcd?'
>>> searchThis = ['abc','xyz','uuu']
>>> any(x in s for x in searchThis)
True
"any" actually test if at least an element of a list satisfies a condition and bails out (i.e., do not continue testing the rest of the list) once it found one. Here is
another example of "any". It checks if any element of the list has a square greater than 78.
>>> lx = [3, 4, 5]
>>> any(x*x > 78 for x in lx)
False
>>> lx = [3, 4, 10]
>>> any(x*x > 78 for x in lx)
True
Another useful construct is "all" - it checks whether all elements of a list satisfies the condition. The example below checks if all elements has a square larger than 78.
>>> lx = [10, 11, 12]
>>> all(x*x > 78 for x in lx)
True
>>> lx = [3, 4, 10]
>>> all(x*x > 78 for x in lx)
False
Another common use that I find with list comprehension is trimming the elements of a list of spaces. If you use the Areva ODBC driver to extract data from the database, the returned values will be padded with spaces to fill up the exact size allocation for the data like this.
>>> substn = ['74S ','102S ','286S ']
>>> mySubstn = [x.strip() for x in substn]
>>> mySubstn
['74S', '102S', '286S']
In my later posts, I will discuss how to use python to extract data from within habitat
>>> a = [10, 20, 30, 40]
>>> b = []
>>> for i in a:
... b.append(2*i)
...
>>> b
[20, 40, 60, 80]
Those new to python may not easily understand what is going on in the above example; perhaps they would approach it like below:
>>> c = []
>>> for i in range(len(a)):
... c[i] = 2*a[i]
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
IndexError: list assignment index out of range
The reason why you have an error above is that array c has no elements and yet you are trying to assign its ith element (there is no ith element) to a new value.
For this approach to be feasible, create a list c which has the same number of elements as list a. The easiest way is to copy list a into list c using the idiom below
>>> c = a[:]
And then, proceed as usual.
>>> for i in range(len(a)):
... c[i] = 2*a[i]
...
>>> c
[20, 40, 60, 80]
If you used
c = a
instead of
c = a[:]
c and a would be referring to the same array and changing 'c' changes 'a' as well. On the other hand, if you used slices. you will get just a copy. Recall that if you leave the left side of ':' blank, it means all the way to the start and leaving the right side blank means all the way to the end.
Now, if you used list comprehension, the code would be:
>>> a = [10, 20, 30, 40]
>>> b = [2*x for x in a]
>>> b
[20, 40, 60, 80]
The way you read this is that, you pick the first element in list 'a', assign it to x, then multiply it by 2, and that is the first element. Then, you pick the second element in 'a', assign it to x, multiply it by 2, and that is the second element, and so on. It is pretty much like:
for x in a:
2*x
except that the result of each loop automatically become an element of a new list. Not only is list comprehension very compact - it is also very fast compared to the ordinary 'for' loop. In the 'for' loop, each line is interpreted as many times as there are elements in the loop; whereas in list comprehension, it is interpreted only once regardless of the list size.
If you want a new array whose elements is the square of each elements of another array, it will be like this:
>>> a
[10, 20, 30, 40]
>>> squared = [x*x for x in a]
>>> squared
[100, 400, 900, 1600]
I guess you now know what is going on. It is possible that instead of 2*x or x*x, you use function like this.
>>> def square(u):
... return u*u
...
>>> a
[10, 20, 30, 40]
>>> c = [square(x) for x in a]
>>> c
[100, 400, 900, 1600]
You could also use conditions. For example, you might want the square of each element only if the element is greater than 20.
>>> a
[10, 20, 30, 40]
>>> d = [square(x) for x in a if x > 20]
>>> d
[900, 1600]
Only 30 and 40 are greater than 20 and so, you only have 2 elements on the new array.
The elements of the new array need not be a function of the elements of the old array at all. For example, I could do something like this:
>>> a
[10, 20, 30, 40]
>>> divisibleBy20 = [1 for x in a if x % 20 == 0]
>>> divisibleBy20
[1, 1]
The above code sets the element of the new array equal to 1 if the old array element is exactly divisible by 20 (i.e., modulo is equal to zero). Only 2 elements are exactly divisible by 20. That is why you only have 2 elements with 1 in it.
Suppose you want to know if a string has 'abc' and 'xyz' in it, what would you do?
>>> s = 'Do you know your abcd?'
>>> if 'abc' in s or 'xyz' in s:
... print 'It is there!'
...
It is there!
Now, suppose you are given a list of strings to search instead of just 2? Maybe you would do it this way:
It is there!
>>> s
'Do you know your abcd?'
>>> searchThis = ['abc','xyz','uuu']
>>> found = False
>>> for x in searchThis:
... if x in s:
... found = True
... break
...
>>> if found:
... print 'found it!'
... else:
... print 'It is not there'
found it!
Well, there is a better way - use for-else:
>>> s
'Do you know your abcd?'
>>> for x in searchThis:
... if x in s:
... print 'found it!'
... break
... else:
... print 'It is not there'
...
found it!
Please note that the 'else' clause is NOT for the 'if' statement. It is for the 'for' statement and is executed if you exhausted the list without issuing a break statement
I think I have found a much better way using list comprehension:
>>> if sum([1 for x in searchThis if x in s]) > 0:
... print 'found it!'
... else:
... print 'It is not there'
...
found it!
To understand what is going on, let us see what we have inside the sum function.
>>> [1 for x in searchThis if x in s]
[1]
It assign a value from searchThis into x and whenever it can find that substring from 's', it places '1' in the new array. So, if none of the elements of searchThis matches, there will be nothing there. Now the sum function adds all elements of a list. If it is an empty list, sum will be zero and greater than zero (there is a match) otherwise.
I just found out that there is still another better way in doing the above using "any" . It is not really list comprehension but the syntax is very similar.
>>> s = 'Do you know your abcd?'
>>> searchThis = ['abc','xyz','uuu']
>>> any(x in s for x in searchThis)
True
"any" actually test if at least an element of a list satisfies a condition and bails out (i.e., do not continue testing the rest of the list) once it found one. Here is
another example of "any". It checks if any element of the list has a square greater than 78.
>>> lx = [3, 4, 5]
>>> any(x*x > 78 for x in lx)
False
>>> lx = [3, 4, 10]
>>> any(x*x > 78 for x in lx)
True
Another useful construct is "all" - it checks whether all elements of a list satisfies the condition. The example below checks if all elements has a square larger than 78.
>>> lx = [10, 11, 12]
>>> all(x*x > 78 for x in lx)
True
>>> lx = [3, 4, 10]
>>> all(x*x > 78 for x in lx)
False
Another common use that I find with list comprehension is trimming the elements of a list of spaces. If you use the Areva ODBC driver to extract data from the database, the returned values will be padded with spaces to fill up the exact size allocation for the data like this.
>>> substn = ['74S ','102S ','286S ']
>>> mySubstn = [x.strip() for x in substn]
>>> mySubstn
['74S', '102S', '286S']
In my later posts, I will discuss how to use python to extract data from within habitat
Friday, December 31, 2010
Counting the Number of Occurances of Words in a File
Suppose you want to count the number of times a word appeared in a text file, how would you do it? The best approach would be to use dictionaries. You use the word to be counted as the key and the dict content would the the word count. For example, d['and'] = 4 means the word 'and' appeared 4 times.
If you are quick, you might now be asking: I don't even know what words exist in the file, so how could I even know what keys to use? Well all you need to do is to check if a word is already a key in the dict and if not, create a dict entry with that key; otherwise simply increment that dict entry. Below is an example:
We use humpy.txt whose contents are:
Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
The script that will count the number of occurrences of a word is count.py with listing below:
fin = open('humpy.txt','r')
d = {}
for line in fin:
# break down the line into individual words
# strip() would remove leading/trailing spaces and newlines
# words below will be an array whose elements are the
# individual words
words = line.strip().split(' ')
for word in words:
if word in d: # does this word has an entry in d?
d[word] += 1 # then increment it
else: # no entry yet. So, make an entry
d[word] = 1
fin.close()
# ok, all words have been processed. Print the statistics
for key, value in d.iteritems():
print '%s appeared %d times' % (key,value)
If you run the command below in the command line,
python count.py
You will get this result:
a appeared 2 times
on appeared 1 times
great appeared 1 times
again. appeared 1 times
Humpty appeared 3 times
all appeared 1 times
Dumpty appeared 2 times
men appeared 1 times
had appeared 1 times
wall, appeared 1 times
together appeared 1 times
king's appeared 2 times
horses appeared 1 times
All appeared 1 times
fall. appeared 1 times
Couldn't appeared 1 times
put appeared 1 times
and appeared 1 times
the appeared 2 times
sat appeared 1 times
You will notice that 'all' is different from 'All'. If you want them to be considered the same, you could convert all words into lower case first by changing this
words = line.strip().split(' ')
into this
words = line.strip().lower().split(' ')
The other thing that may be unusual is the print syntax. It is actually similar to that of C where you specify the formatting string: %s for string, %d for integer, and %f for float, and then you place the corresponding variables to be printed inside the parentheses in the same order as the formatting string.
Another alternative for the print statement would be:
print key + ' appeared ' + str(value) + ' times'
Or better yet, the more efficient version
print ' '.join([key,'appeared',str(value),'times'])
This might look like a trivial example but I used this idea to create a report of how many alarms were generated for a particular alarm category and even the alarm message itself. I parsed the alarm log files and created a dict d[cat] to count the number of alarms for a particular category. I also created a dict for the alarm message and printed the top 20 alarm messages. This helps us easily identify nuisance alarms or perhaps legitimate alarms that needs immediate attention.
If you are quick, you might now be asking: I don't even know what words exist in the file, so how could I even know what keys to use? Well all you need to do is to check if a word is already a key in the dict and if not, create a dict entry with that key; otherwise simply increment that dict entry. Below is an example:
We use humpy.txt whose contents are:
Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
The script that will count the number of occurrences of a word is count.py with listing below:
fin = open('humpy.txt','r')
d = {}
for line in fin:
# break down the line into individual words
# strip() would remove leading/trailing spaces and newlines
# words below will be an array whose elements are the
# individual words
words = line.strip().split(' ')
for word in words:
if word in d: # does this word has an entry in d?
d[word] += 1 # then increment it
else: # no entry yet. So, make an entry
d[word] = 1
fin.close()
# ok, all words have been processed. Print the statistics
for key, value in d.iteritems():
print '%s appeared %d times' % (key,value)
If you run the command below in the command line,
python count.py
You will get this result:
a appeared 2 times
on appeared 1 times
great appeared 1 times
again. appeared 1 times
Humpty appeared 3 times
all appeared 1 times
Dumpty appeared 2 times
men appeared 1 times
had appeared 1 times
wall, appeared 1 times
together appeared 1 times
king's appeared 2 times
horses appeared 1 times
All appeared 1 times
fall. appeared 1 times
Couldn't appeared 1 times
put appeared 1 times
and appeared 1 times
the appeared 2 times
sat appeared 1 times
You will notice that 'all' is different from 'All'. If you want them to be considered the same, you could convert all words into lower case first by changing this
words = line.strip().split(' ')
into this
words = line.strip().lower().split(' ')
The other thing that may be unusual is the print syntax. It is actually similar to that of C where you specify the formatting string: %s for string, %d for integer, and %f for float, and then you place the corresponding variables to be printed inside the parentheses in the same order as the formatting string.
Another alternative for the print statement would be:
print key + ' appeared ' + str(value) + ' times'
Or better yet, the more efficient version
print ' '.join([key,'appeared',str(value),'times'])
This might look like a trivial example but I used this idea to create a report of how many alarms were generated for a particular alarm category and even the alarm message itself. I parsed the alarm log files and created a dict d[cat] to count the number of alarms for a particular category. I also created a dict for the alarm message and printed the top 20 alarm messages. This helps us easily identify nuisance alarms or perhaps legitimate alarms that needs immediate attention.
Monday, December 27, 2010
Python Tutorial
This tutorial is intended for those who already have previous programming experience, most notably C and VB, to get them to transition easily to python. At the start, a VB or C code will be provided and then, translate them to python for easier comparison. I have minimal VB experience and hence, the VB version might be lame and have lots of room for improvement.
Please don't skip and cherry pick as you might miss many important points.
-------------------------------------------------------------------------------------------------------
INSTALLATION
As a start, download the python installer here:
http://www.activestate.com/activepython/downloads
If you are using windows, download the windows installer (msi) for version 2.6. and then, accept all default settings during installation.
-------------------------------------------------------------------------------------------------------
RUNNING THE PYTHON EDITOR/INTERPRETER
You should now see "ActiveState ActivePython 2.6" under All Programs and select the "PythonWin Editor". You should then see the >>> prompt, in the "Interactive Window". This is where you test your code.
The normal workflow in writing python script is that first, you have an idea but not sure if it will work or not. So, you try it first in the >>> prompt and see the results immediately. You then just copy and paste the working snippet to your editor, fully confident that that portion of the code is already correct.
For the earlier part of this tutorial, everything was just copied and pasted from the "Interactive Window". I would recommend that you try it out yourself and do more experimentation to gain more confidence in the language.
-------------------------------------------------------------------------------------------------------
DECLARATION OF VARIABLES
In python, you normally don't declare variables. Those who do are black-belt pythonista but for beginners like us, we would unlikely be in a situation where we need to. Variable names are CASE SENSITIVE! Below are samples:
>>> a = 2
>>> print a
2
>>> a
2
As could be seen above, typing the variable name actually prints the contents. From now on, instead of typing 'print a', I will just be typing "a" to print its contents.
>>> a = 1.25
>>> a
1.25
>>> a = 'This is good'
>>> a
'This is good'
As you could see, I assigned "a" to an int and then float, and then string and it did not complain. I did not even bother declaring what the contents would be beforehand.
The usual convention in python is to use lower case for the first letter of variable or function; use upper case for first letter of classes; use an underscore as the first character of a variable in classes if you want others to treat it as private (this is just a convention because you could still access that variable outside of the class if you really want to. It is just a way to tell the users of the class not to mess with it)
-------------------------------------------------------------------------------------------------------
OPERATIONS
Operations are almost similar to that in C
>>> a = 3
>>> b = 4
>>> c = 6
>>> 2*a
6
>>> a**2
9
>>> c/a
2
>>> b % a
1
Exponentation is ** and modulo is %. It also allows string operations where it makes sense. In the example below, subtraction does not make sense and throws an exception
>>> a = 'noel'
>>> b = 'quintos'
>>> a + b
'noelquintos'
>>> 3 * a
'noelnoelnoel'
>>> '-' * 50
'--------------------------------------------------'
>>> a - b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for -: 'str' and 'str'
Bit-wise operation is similar to C.
>>> a = 4
>>> b = 2
>>> a | b
6
>>> c = 6
>>> a & c
4
-------------------------------------------------------------------------------------------------------
CONDITIONS
The comparison operators are similar to C (==, !=, >, <, >=, <=) but the logical operators are similar to VB (or, and)
>>> a = 4
>>> b = 10
>>> a > 6
False
>>> b >= 10
True
>>> b =>10
File "<stdin>", line 1
b =>10
^
SyntaxError: invalid syntax
>>> a < 5 and 6 < b
True
>>> a != 1
True
>>> a == 4 and not b > 200
True
>>> a < 8 < b
True
Take note of the last example!
In python, anything that evaluates to True or False may be used in the condition clause.
>>> a = 1
>>> b = 30
>>> a < 4 < b
True
>>> if a < 4 < b:
... print '4 is between a and b'
... else:
... print '4 is not between a and b'
...
4 is between a and b
Just like in C, anything that evaluates to 0 or none is False; otherwise, it is true. More of this in the last section under "Conditions Part 2"
>>> a = 0
>>> if a:
... print 'true'
... else:
... print 'false'
...
false
>>> a = 2
>>> if a:
... print 'true'
... else:
... print 'false'
...
true
>>> a = -3
>>> if a:
... print 'true'
... else:
... print 'false'
...
true
-------------------------------------------------------------------------------------------------------
VARIABLE SCOPE
Variables in the "main" function are global in scope and variables in functions and classes are known only inside those functions or classes. In case you happen to use the same variable names inside the function, the global value is used if it is on the right hand side of assignment but do not change the global value if it is on the left hand side. In the example below, I created a function called 'demo'. You use the 'def' keyword to indicate that it is a function.
>>> def demo():
... print a
...
>>> a = 4
>>> demo()
4
>>> def demo2():
... c = a
... print c
...
>>>
>>> demo2()
4
>>> def demo3():
... a = 5
... print a
...
>>> a = 1
>>> demo3()
5
>>> a
1
Notice that 'a' was assigned before the function call and it was not passed and yet the function knows about it. In demo2, 'a' was on the left hand side of the assignment and the global value was used. In demo3, 'a' was on the left hand side and was assigned a value which is retained inside the function but after the function call, the previous value of 'a' is still there.
In python, the functions and variables must be defined first before it is used or it throws an exception.
>>> def demo3():
... print notYetDefinedVariable
...
>>> demo3()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in demo3
NameError: global name 'notYetDefinedVariable' is not defined
>>> notYetDefinedVariable = 'it works now'
>>> demo3()
it works now
-------------------------------------------------------------------------------------------------------
CODE BLOCKS
Python uses indentation to designate a block. This sounds naive but when you think about it, you will indent anyway for readability of your code, hence braces or other form of delimiters are superflous. Yeah, it is waste of typing! One word of caution: Tabs are not the same as white spaces, so, do not use both! Stick to white spaces and indent consistently like 4 white spaces for every indent.
Indentation to designate a block of code will give you a feeling that you missed something but you will get over it fast!
-------------------------------------------------------------------------------------------------------
COMMENTS
Comments are denoted by #. Everything on it's right are
treated as comments like this:
# this is a comment
x = 9999 # assign x a very large value to denote EOF
=======================================
CONTROL STRUCTURES
-------------------------------------------------------------------------------------------------------
IF STATEMENT
VB:
If average>50 Then
text = "Pass"
Else
text = "Fail"
End If
python:
if average > 50:
text = 'Pass'
else:
text = 'fail
Take note of the colon at the 'if' and 'else'.
You might be wondering if there is a equivalent to C's statement below
x = a?u:v
Yes, there is and below is how you translate this statement to python
x = u if a else v
-------------------------------------------------------------------------------------------------------
NESTED IF - THEN
VB:
If average > 75 Then
text = "A"
ElseIf average > 65 Then
text = "B"
ElseIf average > 55 Then
text = "C"
ElseIf average > 45 Then
text = "S"
Else
text = "F"
End If
python:
if average > 75:
text = "A"
elif average > 65:
text = "B"
elif average > 55:
text = "C"
elif average > 45:
text = "S"
else:
text = "F"
There is no 'case' statement in python like in C. The nested if-else is normally how you translate a 'case' statement.
-------------------------------------------------------------------------------------------------------
DO - WHILE STATEMENT
There is no do-while statement in python
-------------------------------------------------------------------------------------------------------
WHILE STATEMENT
VB:
number = 1
While number <=100
number = number + 1
Wend
python:
number = 1
while number <= 100:
number += 1
Note: the last statement may also be written as: number = number + 1
Just like in other languages, you could use 'break' to break out of the loop and 'continue' to skip the rest of the loop statement and start over again from the top part.
-------------------------------------------------------------------------------------------------------
FOR - NEXT LOOP:
VB:
For x = 1 To 50
Print x
Next
python:
for x in xrange(1, 51):
print x
VB:
For x = 1 To 50 Step 2
Print x
Next
python:
for x in xrange(1, 51, 2):
print x
Just like in other languages, you could use 'break' to break out of the loop and 'continue' to skip the rest of the loop statement and start over again from the top part.
-------------------------------------------------------------------------------------------------------
WHILE - ELSE LOOP:
This exists in python but not in VB or C. The else statement is executed if the while loop completes without executing a 'break'.
>>> count = 10
>>> while count > 1:
... if count == 4:
... break
... count -= 1
... else:
... print 'you did not passed a break'
...
In this example, count passed by '4' as it went from 10 to 1, and therefore, 'break' was issued. Because of this, the 'else' clause was not issued.
>>> count = 10
>>> while count > 1:
... if count == 100:
... break
... count -= 1
... else:
... print 'you did not passed a break'
...
you did not passed a break
In this example you are counting down from 10 to 1 and you will have a break only if count is 100 (will not happen). Hence, break was not issued and 'else' clause was executed.
When is this useful? When you are comparing the elements of an array and issues a 'break' to get out of the loop once you found a match, you can have a special handler when nothing matches.
You can also have an 'else' in a 'for' loop.
=======================================
DATA TYPES:
There is nothing special on integer or int and float; behavior is similar with other languages, hence, I will skip these.
-------------------------------------------------------------------------------------------------------
STRING:
The following are all valid string assignments:
>>> s = 'abc'
>>> s
'abc'
>>> s = "abc"
>>> s
'abc'
>>> s = "abc's"
>>> s
"abc's"
>>> s = '''this could span
... multiple lines and
... will be printed
... as is '''
>>> s
'this could span\nmultiple lines and\nwill be printed\nas is '
Just like in C, the backslash character '\' has a special meaning and has to be 'escaped' using another backslash for it to be taken literally by the interpreter. This often times happen when specifying the file location in windows. So, if your file abc.txt is in C:\documents\abc.txt and you want to assign it to a string or passing it to a function as an argument, you have to replace every quote with a double quote like this.
>>> filename = 'c:\\documents\\abc.txt'
>>> filename
'c:\\documents\\abc.txt'
If you want to avoid this, you could prefix the string with "r" like this and have same effect.
>>> s = r'C:\documents\abc.txt'
>>> s
'C:\\documents\\abc.txt'
Remotely shared files would then look like this for windows machine since it uses a double back slashes:
>>> s1 = '\\\\server1\\docs\\abc.txt'
>>> s1
'\\\\server1\\docs\\abc.txt'
Or, with an "r" prefix,
>>> s2 = r'\\server1\docs\abc.txt'
>>> s2
'\\\\server1\\docs\\abc.txt'
Just like in C, the elements of a string is indexed:
>>> s
'this could span\nmultiple lines and\nwill be printed\nas is '
>>> s[0]
't'
>>> s[1]
'h'
>>> s[2]
'i'
However, strings are immutable - i.e. - you cannot change them.
>>> s
'this could span\nmultiple lines and\nwill be printed\nas is '
>>> s[0] = 'x'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment
string comparison is also allowed:
>>> s1 = 'abcd'
>>> s2 = 'abcd'
>>> s3 = 'abc'
>>> s1 == s2
True
>>> s1 == s3
False
-------------------------------------------------------------------------------------------------------
ARRAYS / LISTS
Arrays in python are called 'lists'. Tuples are also arrays but I never came across a situation where I actually used them. So, I will skip tuples and discuss lists only.
This is how you assign a list. The first one is how you assign an empty list.
a= []
u = [2, 'abc', 4.5, [23, 4], 3]
>>> u
[2, 'abc', 4.5, [23, 4], 3]
As you could see, you could assign anything in a list. even another list!
The index is referenced to '0'
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> u[0]
2
>>> u[1]
'abc'
>>> u[3]
[23, 4]
>>> u[3][0]
23
>>> u[3][1]
4
>>> u[1][0]
'a'
If you want to reference an array from the end instead of from the start, you start with -1 and then -2, and so on.
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> u[-1]
3
>>> u[-2]
[23, 4]
>>>
This makes it so easy to access, say, the last element of an array. The index is always -1. Of course, if you want the hard way, the last index is len(u)-1 as you would do in C.
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> u[len(u)-1]
3
As you might have guessed, 'len' would give you the number of elements in the array.
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> len(u)
5
In the above example, the 4th element which is another list is counted as one even though it has 2 elements inside.
Although you don't need to declare variables in python, you should somehow tell it that you are operating on a list or else, you get an exception.
>>> uu[9] = 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'uu' is not defined
You could make an index assignment only if that element exists before.
>>> uu = []
>>> uu[0] = 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range
You have an error here because you have an empty list and yet you want to replace the first element which doesn't exists.
>>> uu.append(5)
>>> uu
[5]
>>> uu[0] = 4
>>> uu
[4]
>>> uu.append(7)
>>> uu
[4, 7]
>>> uu[1] = 100
>>> uu
[4, 100]
-------------------------------------------------------------------------------------------------------
LIST CONDITIONS:
List may be compared and are considered equal if each element matches in value for each position.
>>> l1 = [5,4,3,2,1]
>>> l2 = [5,4,3,2,1]
>>> l3 = [1,2,3,4,5]
>>> l1 == l2
True
>>> l1 == l3
False
You could also easily check if an element exists in a list:
>>> lx = [10, 20, 30, 40]
>>> 30 in lx
True
>>> 100 in lx
False
-------------------------------------------------------------------------------------------------------
EXTRACTING ELEMENTS FROM A LIST:
Of course this is obvious:
>>> a = ['abc', 200, 4.5]
>>> x = a[0]
>>> x
'abc'
>>> b = a[1]
>>> b
200
>>> c = a[2]
>>> c
4.5
However, python has an idiom to make this simpler:
>>> a
['abc', 200, 4.5]
>>> x, b, c = a
>>> x
'abc'
>>> b
200
>>> c
4.5
Perhaps, due to this idiom, swapping of variable contents could be made as follows:
>>> a = 10
>>> b = 20
>>> a, b = b, a
>>> a
20
>>> b
10
What happened is that the first variable on the right is assigned to the first variable on the left, the second variable at the right is assigned to the second variable on the left. This allows you to avoid the usual use of temporary variable shown below:
>>> a = 10
>>> b = 20
>>> T = a
>>> a = b
>>> b = T
>>> a
20
>>> b
10
Shifting of variable contents is also made more simple:
>>> first = 11
>>> second = 22
>>> third = 33
>>> new = 44
>>> first, second, third = second, third, new
>>> first
22
>>> second
33
>>> third
44
-------------------------------------------------------------------------------------------------------
ITERATING THROUGH A LIST:
You could iterate a list in this manner:
>>> a = [10, 20, 30, 40, 50]
>>> N = len(a)
>>> N
5
>>> for i in xrange(N):
... print a[i]
...
10
20
30
40
50
It is also possible to replace N with len(a) as follows:
>>> for i in xrange(len(a)):
... print a[i]
...
10
20
30
40
50
The examples above in not the best way. The preferred way is:
>>> for x in a:
... print x
...
10
20
30
40
50
Not only will you type less; it is also clearer and is guaranteed not to have those "out of index" exceptions.
What if you have more than 1 list to be processed simultaneously as below?
>>> a
[10, 20, 30, 40, 50]
>>> b = [1000, 2000, 3000, 4000, 5000]
>>> for i in xrange(len(a)):
... print a[i] + b[i]
...
1010
2020
3030
4040
5050
You use zip() which you could think of some kind of a zipper which binds things together. This is how zip() works:
>>> zip(a,b)
[(10, 1000), (20, 2000), (30, 3000), (40, 4000), (50, 5000)]
As you could see, it pairs the ith elements of list a and list b together and it becomes the ith element of the new list. So, the best way to process more than 1 list together is as follows:
>>> for x, y in zip(a,b):
... print x + y
...
1010
2020
3030
4040
5050
What happened is that in the first pass, x, y is assigned the value of (10, 1000), or x gets 10 and y gets 1000 (if you don't get it, you probably skipped one of my topic above entitled: EXTRACTING ELEMENTS FROM A LIST. Read that again). On the second pass, x, y gets (20, 2000), and so on. As a result, you typed less code and have avoided indices.
-------------------------------------------------------------------------------------------------------SLICING
Suppose you want just a portion of an array or list what would you do? Resist the temptation of writing a loop to accomplish this because python offers a far easier way - slicing!
The syntax is: listname[startIndex:endIndex:increment]. If you leave the startIndex blank, it implies '0' index. If you leave the endIndex blank, that means all the way to the end. The increment is optional. If not provided, it means increment is 1. Take not that the endIndex element being pointed to is not included. Examples below:
>>> a
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
>>> a[2:7]
[20, 30, 40, 50, 60]
>>> a[:7]
[0, 10, 20, 30, 40, 50, 60]
>>> a[7:]
[70, 80, 90, 100]
>>> a[2:7:2]
[20, 40, 60]
To understand the last example, go back to a[2:7] and then get every other element because increment is 2.
Below is how you copy a list:
>>> a
[1, 2, 3, 4]
>>> b = a[:]
>>> b
[1, 2, 3, 4]
>>>
Now, you might ask, how is b = a[:] different from b = a ? In the former, b is just a copy whereas in the latter, b and a are the same list where changes done in one instance is carried over to the other. See below
>>> a
[1, 2, 3, 4]
>>> b = a[:]
>>> b
[1, 2, 3, 4]
>>> b[0] = 10
>>> b
[10, 2, 3, 4]
>>> a
[1, 2, 3, 4]
Notice that 'a' was unaffected by the changes in 'b'
>>> c = a
>>> c
[1, 2, 3, 4]
>>> c[0] = 0
>>> c
[0, 2, 3, 4]
>>> a
[0, 2, 3, 4]
Notice that when you changed 'c', 'a' was changed as well! So, be very careful especially when passing list to a function, what you want most probably, is just to pass a copy.
-------------------------------------------------------------------------------------------------------
BUILT-IN LIST FUNCTIONS
Suppose you want to get the sum of the elements in a list. How would you do it? Perhaps, you write this code:
>>> a = [1, 2, 3, 4]
>>> sum = 0
>>> for x in a:
... sum += x
...
>>> sum
10
Well, python is no different from other languages if you still need to do it this way. But it is different! You are re-inventing the wheel! The right way is:
>>> a = [1, 2, 3, 4]
>>> sum(a)
10
There is also maximum and minimum:
>>> max(a)
4
>>> min(a)
1
---------------------------------------------------------------------------------------------------------
STRINGS - SPLIT AND JOIN
Often times, you have a string that is comma delimited, most likely, a csv file, and you want to process the contents. The split function is your friend. You supply it with the delimiter character and it returns a list containing the individual elements.
s = '402S, cb, 701, opcl'
>>> s
'402S, cb, 701, opcl'
>>> v = s.split(',')
>>> v
['402S', ' cb', ' 701', ' opcl']
Notice that since there was a space between comma and next element, the space was retained.
Example below uses '$' as the delimiter intead of a comma
>>> ss = '402S$CB$701$OPCL'
>>> ss
'402S$CB$701$OPCL'
>>> vv = ss.split('$')
>>> vv
['402S', 'CB', '701', 'OPCL']
You could also use a word as a delimiter!
>>> r = 'redandblueandgray'
>>> r.split('and')
['red', 'blue', 'gray']
The opposite of split is join. You supply join with an array of strings plus the join character and the result is a single string.
>>> vv
['402S', 'CB', '701', 'OPCL']
>>> '_'.join(vv)
'402S_CB_701_OPCL'
I want to stress these 2 points:
1. join will process only a list. So if you don't have a list, create one
2. The list should have only string elements.
>>> ''.join('one','two','three','four')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: join() takes exactly one argument (4 given)
Item 1 above was violated and so, you got an error. To correct:
>>> ''.join(['one','two','three','four'])
'onetwothreefour'
>>> ' '.join(['one','two','three','four'])
'one two three four'
>>> ' '.join(['I','have',2,'hands'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: sequence item 2: expected string, int found
Item 2 above was violated. To correct:
>>> ' '.join(['I','have',str(2),'hands'])
'I have 2 hands'
The str() function converted the non-string value into a string.
-------------------------------------------------------------------------------------------------------
BUILT-IN HELP - dir()
Suppose you want to know what operations are readily available for a string, or any object, use the dir() function
>>> s = 'abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
As you could see, most of them are self-explanatory, and a lot could be done without having to write additional code. When you write in python, keep in mind that most of the routine things that you need to do were already been done for you. So, when you find yourself writing trivial code that you wished the language handles by itself, you are more likely been re-inventing the wheel!
>>> s = 'Abc'
>>> s.upper()
'ABC'
>>> s
'Abc'
>>> s.swapcase()
'aBC'
>>> s
'Abc'
>>> s.endswith('bc')
True
>>> s.isdigit()
False
What if you want more help for these built-in functions? You import help from pydoc and proceed as follows:
>>> s
'Abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>>> from pydoc import help
>>> help(s.rsplit)
The last command would give you:
Help on built-in function rsplit:
rsplit(...)
S.rsplit([sep [,maxsplit]]) -> list of strings
Return a list of the words in the string S, using sep as the
delimiter string, starting at the end of the string and working
to the front. If maxsplit is given, at most maxsplit splits are
done. If sep is not specified or is None, any whitespace string
is a separator.
(END)
If you don't want to import pydoc, you could proceed as follows: It gives you the same information but uglier, though.
>>> s.rsplit.__doc__
'S.rsplit([sep [,maxsplit]]) -> list of strings\n\nReturn a list of the words in the string S, using sep as the\ndelimiter string, starting at the end of the string and working\nto the front. If maxsplit is given, at most maxsplit splits are\ndone. If sep is not specified or is None, any whitespace string\nis a separator.'
As another example for lists:
>>> a = []
>>> dir(a)
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
>>> a.pop.__doc__
'L.pop([index]) -> item -- remove and return item at index (default last).\nRaises IndexError if list is empty or index is out of range.'
-------------------------------------------------------------------------------------------------------
SETS
If you have 2 list and you want to know which elements are in the first list but not on the second, and vice-versa, then sets would be more appropriate. To illustrate:
>>> l1 = ['abc','def','ghi','klm']
>>> l2 = ['xxx','yyy','abc','klm','uuu']
>>> s1 = set(l1)
>>> s2 = set(l2)
>>> inList1ButNotInList2 = s1 - s2
>>> inList1ButNotInList2
set(['ghi', 'def'])
>>> inList2ButNotInList1 = s2 - s1
>>> inList2ButNotInList1
set(['uuu', 'xxx', 'yyy'])
>>> for x in inList1ButNotInList2:
... print x
...
ghi
def
Sets have no index as shown below. You have to use the "for" loop to extract each element.
>>> inList1ButNotInList2[0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'set' object does not support indexing
With resplect to Areva, this is the best approach to determine which points were added and which ones deleted in the latest database release.
-------------------------------------------------------------------------------------------------------
DICTIONARIES
Dictionaries are lists where strings are used instead of integers for index. The strings which are used as indices are known as 'keys'. Below are examples on how you create one and access values:
>>> d = {} # initialize a dictionary
>>> d['fiftyfive'] = 55
>>> d['sixtyeight'] = 68
>>> d['iv'] = 4
>>> d['x'] = 10
>>> d
{'x': 10, 'fiftyfive': 55, 'sixtyeight': 68, 'iv': 4}
>>> d['x']
10
>>> d['sixtyeight']
68
If you forgot to initialize a variable first before making an assignment, you would receive and exception
>>> unInitializedDict['Richard'] = 'Nixon'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'unInitializedDict' is not defined
To correct this,
>>> unInitializedDict = {}
>>> unInitializedDict['Richard'] = 'Nixon'
I use dictionaries a lot especially in mapping one string into another or storing information for convenient retrieval later on - you don't have to remember an integer as the keys are now strings which are more meaningful.
>>> opposites = {'black':'white', 'big':'small', 'tall':'short'}
>>> opposites['tall']
'short'
Dictionaries could be used to store any information. In Areva, I normally would extract information from netmom for later use in conjunction with scadamom. Below is an example where I stored the 'from' and 'to' substations for line '704L', as well as voltage level.
>>> tl = {}
>>> tl['704L'] = ['102S','74S','138']
So, I could easily check whether a line terminates on a given substation in this manner.
>>> '100S' in tl['704L'][:2]
False
>>> '74S' in tl['704L'][:2]
True
You could also use composite key as index and store the voltage level information. In netmom, you could easily parse the composite key to reconstruct the corresponding scadamom key
>>> cb = {}
>>> cb['102S.CB.701'] = 138
Now, how do you iterate through a dictionary (dict)?
>>> d = {'aaa':'first','bbb':'second'}
>>> for key, value in d.iteritems():
... print key, value
...
aaa first
bbb second
-------------------------------------------------------------------------------------------------------
DICTIONARY CONDITIONS
To check if a key exists in a dictionary:
>>> d = {'aaa':'first','bbb':'second'}
>>> 'bbb' in d
True
>>> 'ccc' in d
False
To check if a certain value exists in a dictionary
>>> 'second' in d.values()
True
>>> 'xxx' in d.values()
False
To extract the key given a value, I use list comprehension (to be covered in detail later).
>>> [key for key in d.keys() if d[key] == 'second']
['bbb']
One of the most irritating behaviour of dictionaries is that if there is no such key, it will throw an exception. There are times when you wish it will just return a default value rather than throw an exception. Good news! Dictionaries have a method "get" that will do this. General usage is:
dictName.get(key, defaultValue)
To illustrate, say you have a dict which stores the dollars contributed by members of an organization shown below: John has contributed $32, Ann $35, and Mario $55.
>>> contribution = {'john':32, 'ann':35, 'mario':55}
Now, Pete has not made any contribution yet; hence, his name is not in the dict. Obviously, his contribution is zero. However, if you try to find his amount of contribution from this dict, it gives you an error like below:
>>> contribution['pete']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'pete'
This is reasonable because python do not attempt to guess what it should return if the key does not exist. One work around is to check if key exists first and return 0 if not there.
>>> if 'pete' in contribution:
... contribution['pete']
... else:
... 0
...
0
However, using "get" method is more straightforward and elegant.
>>> contribution.get('ann',0)
35
>>> contribution.get('pete',0)
0
It returned the value given the key if it exists and return the default (zero in this case) if it does not.
-------------------------------------------------------------------------------------------------------
MIXING AND MATCHING THE DIFFERENT DATA TYPES
In python, you could have a list that contains another list, a dictionary and what have you like this:
>>> u = [11, 22, [300, 400, 500], {'VI':'six', 'X':'ten', 'L':'fifty'}]
>>> u[2]
[300, 400, 500]
>>> u[2][2]
500
>>> u[3]
{'VI': 'six', 'L': 'fifty', 'X': 'ten'}
>>> u[3]['L']
'fifty'
>>> u[0]
11
Similarly, a dictionary can have any contents. Be careful though when creating them. It should be like this:
>>> pupils = {}
>>> pupils['boys'] = {}
>>> pupils['boys']['grade1'] = ['Bryan','James','Rick','Michael']
>>> pupils['boys']['grade1'].append('Mario')
>>> pupils['boys']['grade1']
['Bryan', 'James', 'Rick', 'Michael', 'Mario']
>>> pupils['girls'] = {}
>>> pupils['girls']['grade1'] = []
>>> pupils['girls']['grade1'].append('Laura')
>>> pupils['girls']['grade1'].append('Susan')
>>> pupils['girls']['grade1']
['Laura', 'Susan']
-------------------------------------------------------------------------------------------------------
CONDITIONS PART 2
The following evaluates to True: non-zero value, non-empty string, non-empty list, non-empty dictionary, anything that is non-empty. The snippet below illustrates this point. I defined a function which determines whether a value evaluates to true or false.
>>> def trueorFalse(x):
... if x:
... return 'True'
... else:
... return 'False'
...
>>> trueorFalse(25)
'True'
>>> trueorFalse(-12)
'True'
>>> trueorFalse(2.5)
'True'
>>> trueorFalse('whatever')
'True'
>>> trueorFalse([3, 4])
'True'
>>> trueorFalse({'name':'noel'})
'True'
>>> trueorFalse(trueorFalse)
'True'
The last example above passes the function itself as an argument. Passing a function to another function is valid (and normal) in python.
The reverse, of course evaluates to false: zero value and anything empty.
>>> trueorFalse(0)
'False'
>>> trueorFalse(0.0)
'False'
>>> trueorFalse('')
'False'
>>> trueorFalse([])
'False'
>>> trueorFalse({})
'False'
Be forewarned, though that passing an empty function (one which contains 'pass' only) would still evaluate to true.
-------------------------------------------------------------------------------------------------------
FILE OPERATIONS
To open a textfile for reading
fileInputHandle = open(/path/to/file.txt, 'r')
To open a textfile for writing
fileOutputHandle = open(/path/to/file.txt,'w')
Note: In windows, the path will have a backslash '\' and should be escaped by placing another backslash. For example, if you have
C:\My Documents\work\humpy.txt,
fileInputHandle = open('C:\\My Documents\\work\\humpy.txt','r')
A workaround would be to precede the string with r like this:
fileInputHandle = open(r'C:\My Documents\work\humpy.txt','r')
Here is an example. We have humpy.txt at the current working directory with this content:
Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
>>> fin = open('humpy.txt','r')
>>> r = fin.readlines() # read everything and store in list
>>> r
['Humpty Dumpty sat on a wall,\n', 'Humpty Dumpty had a great fall.\n', "All the king's horses and all the king's men\n", "Couldn't put Humpty together again.\n"]
>>> r.sort()
>>> r
["All the king's horses and all the king's men\n", "Couldn't put Humpty together again.\n", 'Humpty Dumpty had a great fall.\n', 'Humpty Dumpty sat on a wall,\n']
>>> fin.close()
In above example, the whole file is stored in list 'r', one line per element, with the first line being the first element. Having done that, all list built in functions may be applied, like sort(), which rearranges the lines in alphabetical order. You could also iterate through the list using the idiom discussed before. However, unless you need to do sorting, this is not the best approach as this will load everything in the memory. The better approach would be just to process one line at a time.
>>> fin = open('humpy.txt','r')
>>> for r in fin:
... u = r.replace('Humpty Dumpty','Mr Egg') # do something with line
... print u
...
Mr Egg sat on a wall,
Mr Egg had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
>>> fin.close()
Now you are wondering why you could iterate through a file handle as if it were a list. In C you would have an error if you do this and I know it does not make sense! The reason is that 'fin' is an instance of a File object which implements the iterator interface. This allows you to magically iterate through the object as if it were a list. We will discuss iterators later on. In the meantime, just use this idiom. It is easy to remember and less code!
Now, if you want to write the modified version of Humpty Dumpty (where Humpty Dumpty was replaced with Mr Egg) to egg.txt, here is how you do it.
>>> fin = open('humpy.txt','r')
>>> fout = open('egg.txt','w')
>>> for r in fin:
... u = r.replace('Humpty Dumpty','Mr Egg') # do something with line
... fout.write(u)
...
>>> fout.close()
>>> fin.close()
Here is the content of egg.txt
Mr Egg sat on a wall,
Mr Egg had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
Please don't skip and cherry pick as you might miss many important points.
-------------------------------------------------------------------------------------------------------
INSTALLATION
As a start, download the python installer here:
http://www.activestate.com/activepython/downloads
If you are using windows, download the windows installer (msi) for version 2.6. and then, accept all default settings during installation.
-------------------------------------------------------------------------------------------------------
RUNNING THE PYTHON EDITOR/INTERPRETER
You should now see "ActiveState ActivePython 2.6" under All Programs and select the "PythonWin Editor". You should then see the >>> prompt, in the "Interactive Window". This is where you test your code.
The normal workflow in writing python script is that first, you have an idea but not sure if it will work or not. So, you try it first in the >>> prompt and see the results immediately. You then just copy and paste the working snippet to your editor, fully confident that that portion of the code is already correct.
For the earlier part of this tutorial, everything was just copied and pasted from the "Interactive Window". I would recommend that you try it out yourself and do more experimentation to gain more confidence in the language.
-------------------------------------------------------------------------------------------------------
DECLARATION OF VARIABLES
In python, you normally don't declare variables. Those who do are black-belt pythonista but for beginners like us, we would unlikely be in a situation where we need to. Variable names are CASE SENSITIVE! Below are samples:
>>> a = 2
>>> print a
2
>>> a
2
As could be seen above, typing the variable name actually prints the contents. From now on, instead of typing 'print a', I will just be typing "a" to print its contents.
>>> a = 1.25
>>> a
1.25
>>> a = 'This is good'
>>> a
'This is good'
As you could see, I assigned "a" to an int and then float, and then string and it did not complain. I did not even bother declaring what the contents would be beforehand.
The usual convention in python is to use lower case for the first letter of variable or function; use upper case for first letter of classes; use an underscore as the first character of a variable in classes if you want others to treat it as private (this is just a convention because you could still access that variable outside of the class if you really want to. It is just a way to tell the users of the class not to mess with it)
-------------------------------------------------------------------------------------------------------
OPERATIONS
Operations are almost similar to that in C
>>> a = 3
>>> b = 4
>>> c = 6
>>> 2*a
6
>>> a**2
9
>>> c/a
2
>>> b % a
1
Exponentation is ** and modulo is %. It also allows string operations where it makes sense. In the example below, subtraction does not make sense and throws an exception
>>> a = 'noel'
>>> b = 'quintos'
>>> a + b
'noelquintos'
>>> 3 * a
'noelnoelnoel'
>>> '-' * 50
'--------------------------------------------------'
>>> a - b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for -: 'str' and 'str'
Bit-wise operation is similar to C.
>>> a = 4
>>> b = 2
>>> a | b
6
>>> c = 6
>>> a & c
4
-------------------------------------------------------------------------------------------------------
CONDITIONS
The comparison operators are similar to C (==, !=, >, <, >=, <=) but the logical operators are similar to VB (or, and)
>>> a = 4
>>> b = 10
>>> a > 6
False
>>> b >= 10
True
>>> b =>10
File "<stdin>", line 1
b =>10
^
SyntaxError: invalid syntax
>>> a < 5 and 6 < b
True
>>> a != 1
True
>>> a == 4 and not b > 200
True
>>> a < 8 < b
True
Take note of the last example!
In python, anything that evaluates to True or False may be used in the condition clause.
>>> a = 1
>>> b = 30
>>> a < 4 < b
True
>>> if a < 4 < b:
... print '4 is between a and b'
... else:
... print '4 is not between a and b'
...
4 is between a and b
Just like in C, anything that evaluates to 0 or none is False; otherwise, it is true. More of this in the last section under "Conditions Part 2"
>>> a = 0
>>> if a:
... print 'true'
... else:
... print 'false'
...
false
>>> a = 2
>>> if a:
... print 'true'
... else:
... print 'false'
...
true
>>> a = -3
>>> if a:
... print 'true'
... else:
... print 'false'
...
true
-------------------------------------------------------------------------------------------------------
VARIABLE SCOPE
Variables in the "main" function are global in scope and variables in functions and classes are known only inside those functions or classes. In case you happen to use the same variable names inside the function, the global value is used if it is on the right hand side of assignment but do not change the global value if it is on the left hand side. In the example below, I created a function called 'demo'. You use the 'def' keyword to indicate that it is a function.
>>> def demo():
... print a
...
>>> a = 4
>>> demo()
4
>>> def demo2():
... c = a
... print c
...
>>>
>>> demo2()
4
>>> def demo3():
... a = 5
... print a
...
>>> a = 1
>>> demo3()
5
>>> a
1
Notice that 'a' was assigned before the function call and it was not passed and yet the function knows about it. In demo2, 'a' was on the left hand side of the assignment and the global value was used. In demo3, 'a' was on the left hand side and was assigned a value which is retained inside the function but after the function call, the previous value of 'a' is still there.
In python, the functions and variables must be defined first before it is used or it throws an exception.
>>> def demo3():
... print notYetDefinedVariable
...
>>> demo3()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in demo3
NameError: global name 'notYetDefinedVariable' is not defined
>>> notYetDefinedVariable = 'it works now'
>>> demo3()
it works now
-------------------------------------------------------------------------------------------------------
CODE BLOCKS
Python uses indentation to designate a block. This sounds naive but when you think about it, you will indent anyway for readability of your code, hence braces or other form of delimiters are superflous. Yeah, it is waste of typing! One word of caution: Tabs are not the same as white spaces, so, do not use both! Stick to white spaces and indent consistently like 4 white spaces for every indent.
Indentation to designate a block of code will give you a feeling that you missed something but you will get over it fast!
-------------------------------------------------------------------------------------------------------
COMMENTS
Comments are denoted by #. Everything on it's right are
treated as comments like this:
# this is a comment
x = 9999 # assign x a very large value to denote EOF
=======================================
CONTROL STRUCTURES
-------------------------------------------------------------------------------------------------------
IF STATEMENT
VB:
If average>50 Then
text = "Pass"
Else
text = "Fail"
End If
python:
if average > 50:
text = 'Pass'
else:
text = 'fail
Take note of the colon at the 'if' and 'else'.
You might be wondering if there is a equivalent to C's statement below
x = a?u:v
Yes, there is and below is how you translate this statement to python
x = u if a else v
-------------------------------------------------------------------------------------------------------
NESTED IF - THEN
VB:
If average > 75 Then
text = "A"
ElseIf average > 65 Then
text = "B"
ElseIf average > 55 Then
text = "C"
ElseIf average > 45 Then
text = "S"
Else
text = "F"
End If
python:
if average > 75:
text = "A"
elif average > 65:
text = "B"
elif average > 55:
text = "C"
elif average > 45:
text = "S"
else:
text = "F"
There is no 'case' statement in python like in C. The nested if-else is normally how you translate a 'case' statement.
-------------------------------------------------------------------------------------------------------
DO - WHILE STATEMENT
There is no do-while statement in python
-------------------------------------------------------------------------------------------------------
WHILE STATEMENT
VB:
number = 1
While number <=100
number = number + 1
Wend
python:
number = 1
while number <= 100:
number += 1
Note: the last statement may also be written as: number = number + 1
Just like in other languages, you could use 'break' to break out of the loop and 'continue' to skip the rest of the loop statement and start over again from the top part.
-------------------------------------------------------------------------------------------------------
FOR - NEXT LOOP:
VB:
For x = 1 To 50
Print x
Next
python:
for x in xrange(1, 51):
print x
VB:
For x = 1 To 50 Step 2
Print x
Next
python:
for x in xrange(1, 51, 2):
print x
Just like in other languages, you could use 'break' to break out of the loop and 'continue' to skip the rest of the loop statement and start over again from the top part.
-------------------------------------------------------------------------------------------------------
WHILE - ELSE LOOP:
This exists in python but not in VB or C. The else statement is executed if the while loop completes without executing a 'break'.
>>> count = 10
>>> while count > 1:
... if count == 4:
... break
... count -= 1
... else:
... print 'you did not passed a break'
...
In this example, count passed by '4' as it went from 10 to 1, and therefore, 'break' was issued. Because of this, the 'else' clause was not issued.
>>> count = 10
>>> while count > 1:
... if count == 100:
... break
... count -= 1
... else:
... print 'you did not passed a break'
...
you did not passed a break
In this example you are counting down from 10 to 1 and you will have a break only if count is 100 (will not happen). Hence, break was not issued and 'else' clause was executed.
When is this useful? When you are comparing the elements of an array and issues a 'break' to get out of the loop once you found a match, you can have a special handler when nothing matches.
You can also have an 'else' in a 'for' loop.
=======================================
DATA TYPES:
There is nothing special on integer or int and float; behavior is similar with other languages, hence, I will skip these.
-------------------------------------------------------------------------------------------------------
STRING:
The following are all valid string assignments:
>>> s = 'abc'
>>> s
'abc'
>>> s = "abc"
>>> s
'abc'
>>> s = "abc's"
>>> s
"abc's"
>>> s = '''this could span
... multiple lines and
... will be printed
... as is '''
>>> s
'this could span\nmultiple lines and\nwill be printed\nas is '
Just like in C, the backslash character '\' has a special meaning and has to be 'escaped' using another backslash for it to be taken literally by the interpreter. This often times happen when specifying the file location in windows. So, if your file abc.txt is in C:\documents\abc.txt and you want to assign it to a string or passing it to a function as an argument, you have to replace every quote with a double quote like this.
>>> filename = 'c:\\documents\\abc.txt'
>>> filename
'c:\\documents\\abc.txt'
If you want to avoid this, you could prefix the string with "r" like this and have same effect.
>>> s = r'C:\documents\abc.txt'
>>> s
'C:\\documents\\abc.txt'
Remotely shared files would then look like this for windows machine since it uses a double back slashes:
>>> s1 = '\\\\server1\\docs\\abc.txt'
>>> s1
'\\\\server1\\docs\\abc.txt'
Or, with an "r" prefix,
>>> s2 = r'\\server1\docs\abc.txt'
>>> s2
'\\\\server1\\docs\\abc.txt'
Just like in C, the elements of a string is indexed:
>>> s
'this could span\nmultiple lines and\nwill be printed\nas is '
>>> s[0]
't'
>>> s[1]
'h'
>>> s[2]
'i'
However, strings are immutable - i.e. - you cannot change them.
>>> s
'this could span\nmultiple lines and\nwill be printed\nas is '
>>> s[0] = 'x'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment
string comparison is also allowed:
>>> s1 = 'abcd'
>>> s2 = 'abcd'
>>> s3 = 'abc'
>>> s1 == s2
True
>>> s1 == s3
False
-------------------------------------------------------------------------------------------------------
ARRAYS / LISTS
Arrays in python are called 'lists'. Tuples are also arrays but I never came across a situation where I actually used them. So, I will skip tuples and discuss lists only.
This is how you assign a list. The first one is how you assign an empty list.
a= []
u = [2, 'abc', 4.5, [23, 4], 3]
>>> u
[2, 'abc', 4.5, [23, 4], 3]
As you could see, you could assign anything in a list. even another list!
The index is referenced to '0'
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> u[0]
2
>>> u[1]
'abc'
>>> u[3]
[23, 4]
>>> u[3][0]
23
>>> u[3][1]
4
>>> u[1][0]
'a'
If you want to reference an array from the end instead of from the start, you start with -1 and then -2, and so on.
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> u[-1]
3
>>> u[-2]
[23, 4]
>>>
This makes it so easy to access, say, the last element of an array. The index is always -1. Of course, if you want the hard way, the last index is len(u)-1 as you would do in C.
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> u[len(u)-1]
3
As you might have guessed, 'len' would give you the number of elements in the array.
>>> u
[2, 'abc', 4.5, [23, 4], 3]
>>> len(u)
5
In the above example, the 4th element which is another list is counted as one even though it has 2 elements inside.
Although you don't need to declare variables in python, you should somehow tell it that you are operating on a list or else, you get an exception.
>>> uu[9] = 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'uu' is not defined
You could make an index assignment only if that element exists before.
>>> uu = []
>>> uu[0] = 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range
You have an error here because you have an empty list and yet you want to replace the first element which doesn't exists.
>>> uu.append(5)
>>> uu
[5]
>>> uu[0] = 4
>>> uu
[4]
>>> uu.append(7)
>>> uu
[4, 7]
>>> uu[1] = 100
>>> uu
[4, 100]
-------------------------------------------------------------------------------------------------------
LIST CONDITIONS:
List may be compared and are considered equal if each element matches in value for each position.
>>> l1 = [5,4,3,2,1]
>>> l2 = [5,4,3,2,1]
>>> l3 = [1,2,3,4,5]
>>> l1 == l2
True
>>> l1 == l3
False
You could also easily check if an element exists in a list:
>>> lx = [10, 20, 30, 40]
>>> 30 in lx
True
>>> 100 in lx
False
-------------------------------------------------------------------------------------------------------
EXTRACTING ELEMENTS FROM A LIST:
Of course this is obvious:
>>> a = ['abc', 200, 4.5]
>>> x = a[0]
>>> x
'abc'
>>> b = a[1]
>>> b
200
>>> c = a[2]
>>> c
4.5
However, python has an idiom to make this simpler:
>>> a
['abc', 200, 4.5]
>>> x, b, c = a
>>> x
'abc'
>>> b
200
>>> c
4.5
Perhaps, due to this idiom, swapping of variable contents could be made as follows:
>>> a = 10
>>> b = 20
>>> a, b = b, a
>>> a
20
>>> b
10
What happened is that the first variable on the right is assigned to the first variable on the left, the second variable at the right is assigned to the second variable on the left. This allows you to avoid the usual use of temporary variable shown below:
>>> a = 10
>>> b = 20
>>> T = a
>>> a = b
>>> b = T
>>> a
20
>>> b
10
Shifting of variable contents is also made more simple:
>>> first = 11
>>> second = 22
>>> third = 33
>>> new = 44
>>> first, second, third = second, third, new
>>> first
22
>>> second
33
>>> third
44
-------------------------------------------------------------------------------------------------------
ITERATING THROUGH A LIST:
You could iterate a list in this manner:
>>> a = [10, 20, 30, 40, 50]
>>> N = len(a)
>>> N
5
>>> for i in xrange(N):
... print a[i]
...
10
20
30
40
50
It is also possible to replace N with len(a) as follows:
>>> for i in xrange(len(a)):
... print a[i]
...
10
20
30
40
50
The examples above in not the best way. The preferred way is:
>>> for x in a:
... print x
...
10
20
30
40
50
Not only will you type less; it is also clearer and is guaranteed not to have those "out of index" exceptions.
What if you have more than 1 list to be processed simultaneously as below?
>>> a
[10, 20, 30, 40, 50]
>>> b = [1000, 2000, 3000, 4000, 5000]
>>> for i in xrange(len(a)):
... print a[i] + b[i]
...
1010
2020
3030
4040
5050
You use zip() which you could think of some kind of a zipper which binds things together. This is how zip() works:
>>> zip(a,b)
[(10, 1000), (20, 2000), (30, 3000), (40, 4000), (50, 5000)]
As you could see, it pairs the ith elements of list a and list b together and it becomes the ith element of the new list. So, the best way to process more than 1 list together is as follows:
>>> for x, y in zip(a,b):
... print x + y
...
1010
2020
3030
4040
5050
What happened is that in the first pass, x, y is assigned the value of (10, 1000), or x gets 10 and y gets 1000 (if you don't get it, you probably skipped one of my topic above entitled: EXTRACTING ELEMENTS FROM A LIST. Read that again). On the second pass, x, y gets (20, 2000), and so on. As a result, you typed less code and have avoided indices.
-------------------------------------------------------------------------------------------------------SLICING
Suppose you want just a portion of an array or list what would you do? Resist the temptation of writing a loop to accomplish this because python offers a far easier way - slicing!
The syntax is: listname[startIndex:endIndex:increment]. If you leave the startIndex blank, it implies '0' index. If you leave the endIndex blank, that means all the way to the end. The increment is optional. If not provided, it means increment is 1. Take not that the endIndex element being pointed to is not included. Examples below:
>>> a
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
>>> a[2:7]
[20, 30, 40, 50, 60]
>>> a[:7]
[0, 10, 20, 30, 40, 50, 60]
>>> a[7:]
[70, 80, 90, 100]
>>> a[2:7:2]
[20, 40, 60]
To understand the last example, go back to a[2:7] and then get every other element because increment is 2.
Below is how you copy a list:
>>> a
[1, 2, 3, 4]
>>> b = a[:]
>>> b
[1, 2, 3, 4]
>>>
Now, you might ask, how is b = a[:] different from b = a ? In the former, b is just a copy whereas in the latter, b and a are the same list where changes done in one instance is carried over to the other. See below
>>> a
[1, 2, 3, 4]
>>> b = a[:]
>>> b
[1, 2, 3, 4]
>>> b[0] = 10
>>> b
[10, 2, 3, 4]
>>> a
[1, 2, 3, 4]
Notice that 'a' was unaffected by the changes in 'b'
>>> c = a
>>> c
[1, 2, 3, 4]
>>> c[0] = 0
>>> c
[0, 2, 3, 4]
>>> a
[0, 2, 3, 4]
Notice that when you changed 'c', 'a' was changed as well! So, be very careful especially when passing list to a function, what you want most probably, is just to pass a copy.
-------------------------------------------------------------------------------------------------------
BUILT-IN LIST FUNCTIONS
Suppose you want to get the sum of the elements in a list. How would you do it? Perhaps, you write this code:
>>> a = [1, 2, 3, 4]
>>> sum = 0
>>> for x in a:
... sum += x
...
>>> sum
10
Well, python is no different from other languages if you still need to do it this way. But it is different! You are re-inventing the wheel! The right way is:
>>> a = [1, 2, 3, 4]
>>> sum(a)
10
There is also maximum and minimum:
>>> max(a)
4
>>> min(a)
1
---------------------------------------------------------------------------------------------------------
STRINGS - SPLIT AND JOIN
Often times, you have a string that is comma delimited, most likely, a csv file, and you want to process the contents. The split function is your friend. You supply it with the delimiter character and it returns a list containing the individual elements.
s = '402S, cb, 701, opcl'
>>> s
'402S, cb, 701, opcl'
>>> v = s.split(',')
>>> v
['402S', ' cb', ' 701', ' opcl']
Notice that since there was a space between comma and next element, the space was retained.
Example below uses '$' as the delimiter intead of a comma
>>> ss = '402S$CB$701$OPCL'
>>> ss
'402S$CB$701$OPCL'
>>> vv = ss.split('$')
>>> vv
['402S', 'CB', '701', 'OPCL']
You could also use a word as a delimiter!
>>> r = 'redandblueandgray'
>>> r.split('and')
['red', 'blue', 'gray']
The opposite of split is join. You supply join with an array of strings plus the join character and the result is a single string.
>>> vv
['402S', 'CB', '701', 'OPCL']
>>> '_'.join(vv)
'402S_CB_701_OPCL'
I want to stress these 2 points:
1. join will process only a list. So if you don't have a list, create one
2. The list should have only string elements.
>>> ''.join('one','two','three','four')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: join() takes exactly one argument (4 given)
Item 1 above was violated and so, you got an error. To correct:
>>> ''.join(['one','two','three','four'])
'onetwothreefour'
>>> ' '.join(['one','two','three','four'])
'one two three four'
>>> ' '.join(['I','have',2,'hands'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: sequence item 2: expected string, int found
Item 2 above was violated. To correct:
>>> ' '.join(['I','have',str(2),'hands'])
'I have 2 hands'
The str() function converted the non-string value into a string.
-------------------------------------------------------------------------------------------------------
BUILT-IN HELP - dir()
Suppose you want to know what operations are readily available for a string, or any object, use the dir() function
>>> s = 'abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
As you could see, most of them are self-explanatory, and a lot could be done without having to write additional code. When you write in python, keep in mind that most of the routine things that you need to do were already been done for you. So, when you find yourself writing trivial code that you wished the language handles by itself, you are more likely been re-inventing the wheel!
>>> s = 'Abc'
>>> s.upper()
'ABC'
>>> s
'Abc'
>>> s.swapcase()
'aBC'
>>> s
'Abc'
>>> s.endswith('bc')
True
>>> s.isdigit()
False
What if you want more help for these built-in functions? You import help from pydoc and proceed as follows:
>>> s
'Abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>>> from pydoc import help
>>> help(s.rsplit)
The last command would give you:
Help on built-in function rsplit:
rsplit(...)
S.rsplit([sep [,maxsplit]]) -> list of strings
Return a list of the words in the string S, using sep as the
delimiter string, starting at the end of the string and working
to the front. If maxsplit is given, at most maxsplit splits are
done. If sep is not specified or is None, any whitespace string
is a separator.
(END)
If you don't want to import pydoc, you could proceed as follows: It gives you the same information but uglier, though.
>>> s.rsplit.__doc__
'S.rsplit([sep [,maxsplit]]) -> list of strings\n\nReturn a list of the words in the string S, using sep as the\ndelimiter string, starting at the end of the string and working\nto the front. If maxsplit is given, at most maxsplit splits are\ndone. If sep is not specified or is None, any whitespace string\nis a separator.'
As another example for lists:
>>> a = []
>>> dir(a)
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
>>> a.pop.__doc__
'L.pop([index]) -> item -- remove and return item at index (default last).\nRaises IndexError if list is empty or index is out of range.'
-------------------------------------------------------------------------------------------------------
SETS
If you have 2 list and you want to know which elements are in the first list but not on the second, and vice-versa, then sets would be more appropriate. To illustrate:
>>> l1 = ['abc','def','ghi','klm']
>>> l2 = ['xxx','yyy','abc','klm','uuu']
>>> s1 = set(l1)
>>> s2 = set(l2)
>>> inList1ButNotInList2 = s1 - s2
>>> inList1ButNotInList2
set(['ghi', 'def'])
>>> inList2ButNotInList1 = s2 - s1
>>> inList2ButNotInList1
set(['uuu', 'xxx', 'yyy'])
>>> for x in inList1ButNotInList2:
... print x
...
ghi
def
Sets have no index as shown below. You have to use the "for" loop to extract each element.
>>> inList1ButNotInList2[0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'set' object does not support indexing
With resplect to Areva, this is the best approach to determine which points were added and which ones deleted in the latest database release.
-------------------------------------------------------------------------------------------------------
DICTIONARIES
Dictionaries are lists where strings are used instead of integers for index. The strings which are used as indices are known as 'keys'. Below are examples on how you create one and access values:
>>> d = {} # initialize a dictionary
>>> d['fiftyfive'] = 55
>>> d['sixtyeight'] = 68
>>> d['iv'] = 4
>>> d['x'] = 10
>>> d
{'x': 10, 'fiftyfive': 55, 'sixtyeight': 68, 'iv': 4}
>>> d['x']
10
>>> d['sixtyeight']
68
If you forgot to initialize a variable first before making an assignment, you would receive and exception
>>> unInitializedDict['Richard'] = 'Nixon'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'unInitializedDict' is not defined
To correct this,
>>> unInitializedDict = {}
>>> unInitializedDict['Richard'] = 'Nixon'
I use dictionaries a lot especially in mapping one string into another or storing information for convenient retrieval later on - you don't have to remember an integer as the keys are now strings which are more meaningful.
>>> opposites = {'black':'white', 'big':'small', 'tall':'short'}
>>> opposites['tall']
'short'
Dictionaries could be used to store any information. In Areva, I normally would extract information from netmom for later use in conjunction with scadamom. Below is an example where I stored the 'from' and 'to' substations for line '704L', as well as voltage level.
>>> tl = {}
>>> tl['704L'] = ['102S','74S','138']
So, I could easily check whether a line terminates on a given substation in this manner.
>>> '100S' in tl['704L'][:2]
False
>>> '74S' in tl['704L'][:2]
True
You could also use composite key as index and store the voltage level information. In netmom, you could easily parse the composite key to reconstruct the corresponding scadamom key
>>> cb = {}
>>> cb['102S.CB.701'] = 138
Now, how do you iterate through a dictionary (dict)?
>>> d = {'aaa':'first','bbb':'second'}
>>> for key, value in d.iteritems():
... print key, value
...
aaa first
bbb second
-------------------------------------------------------------------------------------------------------
DICTIONARY CONDITIONS
To check if a key exists in a dictionary:
>>> d = {'aaa':'first','bbb':'second'}
>>> 'bbb' in d
True
>>> 'ccc' in d
False
To check if a certain value exists in a dictionary
>>> 'second' in d.values()
True
>>> 'xxx' in d.values()
False
To extract the key given a value, I use list comprehension (to be covered in detail later).
>>> [key for key in d.keys() if d[key] == 'second']
['bbb']
One of the most irritating behaviour of dictionaries is that if there is no such key, it will throw an exception. There are times when you wish it will just return a default value rather than throw an exception. Good news! Dictionaries have a method "get" that will do this. General usage is:
dictName.get(key, defaultValue)
To illustrate, say you have a dict which stores the dollars contributed by members of an organization shown below: John has contributed $32, Ann $35, and Mario $55.
>>> contribution = {'john':32, 'ann':35, 'mario':55}
Now, Pete has not made any contribution yet; hence, his name is not in the dict. Obviously, his contribution is zero. However, if you try to find his amount of contribution from this dict, it gives you an error like below:
>>> contribution['pete']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'pete'
This is reasonable because python do not attempt to guess what it should return if the key does not exist. One work around is to check if key exists first and return 0 if not there.
>>> if 'pete' in contribution:
... contribution['pete']
... else:
... 0
...
0
However, using "get" method is more straightforward and elegant.
>>> contribution.get('ann',0)
35
>>> contribution.get('pete',0)
0
It returned the value given the key if it exists and return the default (zero in this case) if it does not.
-------------------------------------------------------------------------------------------------------
MIXING AND MATCHING THE DIFFERENT DATA TYPES
In python, you could have a list that contains another list, a dictionary and what have you like this:
>>> u = [11, 22, [300, 400, 500], {'VI':'six', 'X':'ten', 'L':'fifty'}]
>>> u[2]
[300, 400, 500]
>>> u[2][2]
500
>>> u[3]
{'VI': 'six', 'L': 'fifty', 'X': 'ten'}
>>> u[3]['L']
'fifty'
>>> u[0]
11
Similarly, a dictionary can have any contents. Be careful though when creating them. It should be like this:
>>> pupils = {}
>>> pupils['boys'] = {}
>>> pupils['boys']['grade1'] = ['Bryan','James','Rick','Michael']
>>> pupils['boys']['grade1'].append('Mario')
>>> pupils['boys']['grade1']
['Bryan', 'James', 'Rick', 'Michael', 'Mario']
>>> pupils['girls'] = {}
>>> pupils['girls']['grade1'] = []
>>> pupils['girls']['grade1'].append('Laura')
>>> pupils['girls']['grade1'].append('Susan')
>>> pupils['girls']['grade1']
['Laura', 'Susan']
-------------------------------------------------------------------------------------------------------
CONDITIONS PART 2
The following evaluates to True: non-zero value, non-empty string, non-empty list, non-empty dictionary, anything that is non-empty. The snippet below illustrates this point. I defined a function which determines whether a value evaluates to true or false.
>>> def trueorFalse(x):
... if x:
... return 'True'
... else:
... return 'False'
...
>>> trueorFalse(25)
'True'
>>> trueorFalse(-12)
'True'
>>> trueorFalse(2.5)
'True'
>>> trueorFalse('whatever')
'True'
>>> trueorFalse([3, 4])
'True'
>>> trueorFalse({'name':'noel'})
'True'
>>> trueorFalse(trueorFalse)
'True'
The last example above passes the function itself as an argument. Passing a function to another function is valid (and normal) in python.
The reverse, of course evaluates to false: zero value and anything empty.
>>> trueorFalse(0)
'False'
>>> trueorFalse(0.0)
'False'
>>> trueorFalse('')
'False'
>>> trueorFalse([])
'False'
>>> trueorFalse({})
'False'
Be forewarned, though that passing an empty function (one which contains 'pass' only) would still evaluate to true.
-------------------------------------------------------------------------------------------------------
FILE OPERATIONS
To open a textfile for reading
fileInputHandle = open(/path/to/file.txt, 'r')
To open a textfile for writing
fileOutputHandle = open(/path/to/file.txt,'w')
Note: In windows, the path will have a backslash '\' and should be escaped by placing another backslash. For example, if you have
C:\My Documents\work\humpy.txt,
fileInputHandle = open('C:\\My Documents\\work\\humpy.txt','r')
A workaround would be to precede the string with r like this:
fileInputHandle = open(r'C:\My Documents\work\humpy.txt','r')
Here is an example. We have humpy.txt at the current working directory with this content:
Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
>>> fin = open('humpy.txt','r')
>>> r = fin.readlines() # read everything and store in list
>>> r
['Humpty Dumpty sat on a wall,\n', 'Humpty Dumpty had a great fall.\n', "All the king's horses and all the king's men\n", "Couldn't put Humpty together again.\n"]
>>> r.sort()
>>> r
["All the king's horses and all the king's men\n", "Couldn't put Humpty together again.\n", 'Humpty Dumpty had a great fall.\n', 'Humpty Dumpty sat on a wall,\n']
>>> fin.close()
In above example, the whole file is stored in list 'r', one line per element, with the first line being the first element. Having done that, all list built in functions may be applied, like sort(), which rearranges the lines in alphabetical order. You could also iterate through the list using the idiom discussed before. However, unless you need to do sorting, this is not the best approach as this will load everything in the memory. The better approach would be just to process one line at a time.
>>> fin = open('humpy.txt','r')
>>> for r in fin:
... u = r.replace('Humpty Dumpty','Mr Egg') # do something with line
... print u
...
Mr Egg sat on a wall,
Mr Egg had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
>>> fin.close()
Now you are wondering why you could iterate through a file handle as if it were a list. In C you would have an error if you do this and I know it does not make sense! The reason is that 'fin' is an instance of a File object which implements the iterator interface. This allows you to magically iterate through the object as if it were a list. We will discuss iterators later on. In the meantime, just use this idiom. It is easy to remember and less code!
Now, if you want to write the modified version of Humpty Dumpty (where Humpty Dumpty was replaced with Mr Egg) to egg.txt, here is how you do it.
>>> fin = open('humpy.txt','r')
>>> fout = open('egg.txt','w')
>>> for r in fin:
... u = r.replace('Humpty Dumpty','Mr Egg') # do something with line
... fout.write(u)
...
>>> fout.close()
>>> fin.close()
Here is the content of egg.txt
Mr Egg sat on a wall,
Mr Egg had a great fall.
All the king's horses and all the king's men
Couldn't put Humpty together again.
Subscribe to:
Posts (Atom)