Install MySQL for Python (MySQLdb) on Windows
It took me quite a while to figure out how to build and install MySQL for Python (MySQLdb) on Windows. I’d better write it down.
There is no binary distribution of MySQLdb for Python 2.6 on Windows. I have to build it from the source. My environment is Windows XP. MySQL 5.1. Python 2.6 (windows version, not cygwin), and MySQL-python-1.2.3c1. Also, I have Microsoft Visual C++ 2008 Express Edition (Microsoft Visual Studio 9.0) installed, which is required to compile the C code in MySQL-python.
First of all, install Python setuptools, if you haven’t installed it. It is required in MySQL-python setup.py. I also added C:\Python26\Scripts into environment PATH, where easy_install is installed.
Then, make sure you have MySQL Developer Components installed. Download MySQL msi installer version, select “Developer Components” in Custom Setup. It will install C:\Program Files\MySQL\MySQL Server 5.1\include, lib\debug and lib\opt for you. They are not installed by default.
Uncompress MySQL-python-1.2.3c1.tar.gz into a directory. Open a command window (cmd), change to the directory.
Try to run,
setup.py build
I got this error in setup_windows.py:
in get_config
serverKey = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, options['registry_key'])
WindowsError: [Error 2] The system cannot find the file specified
So I edited site.cfg, changed the MySQL version from 5.0 to 5.1 (since I am using 5.1)
registry_key = SOFTWARE\MySQL AB\MySQL Server 5.1
You can use regedit to check which version you are using. It is specified at: HKEY_LOCAL_MACHINE/SOFTWARE/MySQL AB/MySQL Server 5.1.
Now try to build it again. I got this error:
build\temp.win32-2.6\Release\_mysql.pyd.manifest : general error c1010070: Failed to load and parse the manifest. The system cannot find the file specified.
error: command ‘mt.exe’ failed with exit status 31
To fix this problem, go to C:\Python26\Lib\distutils, edit msvc9compiler.py, search for ‘MANIFESTFILE’, you will find the following line
ld_args.append(‘/MANIFESTFILE:’ + temp_manifest)
Then append the following line after the above line,
ld_args.append(‘/MANIFEST’)
Then go back to run “setup.py build”, it will succeed. Finally, run
setup.py install
Test it in python
>>> import MySQLdb
>>>
Cannot open libstdc++.so.5 on fedora 11
I just upgraded to fedora 11. When I installed java EE SDK, I got the following error
> ./java_ee_sdk-5_07-linux-nojdk.bin
./java_ee_sdk-5_07-linux-nojdk.bin: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory
The default libstdc++ on fedora is libstdc++.so.6 installed from libstdc++.i586. To solve the problem, install compat-libstdc++-33. For fedora 11, the package can be installed by
> yum install compat-libstdc++-33-3.2.3-66.i586
How to read input files in maven junit
Sometimes we need to put unit test data into plain text files. For example, assume we want to test a parser using a json string as the test data. If we put the json string as a constant string in the java code, we end up with a lot of error-prone escaping characters. In that case, we may want to put the test string into a file as a resource and read the string from the file in junit.
In maven, we need to put the resource file in src/test/resources. Let me create a demo from scratch.
> mvn archetype:create -DgroupId=org.fuyun -DartifactId=junitresdemo > find . . ./junitresdemo ./junitresdemo/pom.xml ./junitresdemo/src ./junitresdemo/src/test ./junitresdemo/src/test/java ./junitresdemo/src/test/java/org ./junitresdemo/src/test/java/org/fuyun ./junitresdemo/src/test/java/org/fuyun/AppTest.java ./junitresdemo/src/main ./junitresdemo/src/main/java ./junitresdemo/src/main/java/org ./junitresdemo/src/main/java/org/fuyun ./junitresdemo/src/main/java/org/fuyun/App.java > cd junitresdemo > mkdir -p src/test/resources > vi src/test/resources/myres.txt > cat src/test/resources/myres.txt test1=testdata
As you can see, I put my test data as a key-value pair in a property file. If your test data contains special characters such as escape char, you’d better handle file reading by yourself instead of using Properties as I am going to show.
Then I modify the automatically generated test code src/test/java/org/fuyun/AppTest.java as follows.
package org.fuyun;
import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;
import java.io.InputStream;
import java.util.Properties;
public class AppTest extends TestCase {
public AppTest( String testName ) {
super( testName );
}
public static Test suite() {
return new TestSuite( AppTest.class );
}
public void testApp() throws java.io.IOException {
InputStream in =
getClass().getClassLoader().getResourceAsStream("myres.txt");
Properties p = new Properties();
p.load(in);
String mystr = p.getProperty("test1");
assertEquals("testdata", mystr);
}
}
To read the property file, we need to use getResourceAsStream. Actually this is why I want to write this blog. If you search on web, you may find that people talk about you can load the file using Class.getResourceAsStream(). So, I am supposed to write the line as
InputStream in = getClass().getResourceAsStream("myres.txt");
It can compile. But the test will fail. The InputStream variable in will be null, i.e., it cannot find myres.txt. Why? Why do we have to use the method defined in ClassLoader?
The difference between Class.getResourceAsStream and ClassLoader.getResourceAsStream is that Class.getResourceAsStream attempts to first resolve the file name by appending the package prefix (org/fuyun/) if the file name is not an absolute path, otherwise removes the leading “/” if the path is absolute. Then, it calls the ClassLoader’s getResourceAsStream to load the resolved file name. This is documented here.
For example, if I do the following hack, the test will pass temporarily.
> mv target/test-classes/myres.txt target/test-classes/org/fuyun/. > mvn test
But to really fix it, we should revise the line by adding a leading “/” in the file name as follows.
InputStream in = getClass().getResourceAsStream("/myres.txt");
On the other hand, if you use ClassLoader.getResourceAsStream, the leading “/” will make it unable to find the file.
How to convert an integer to base64 in Python
I found a bunch of solutions on web and noticed it may lead to entirely different results if you ignore a few assumptions.
First of all, the input of base64 algorithm is an array of bytes, which is a string in Python. Thus, before converting an integer to base64, you need to convert the integer to an array of bytes. To do so, you need to first decide the byte order: big endian or little endian. The best solution is using Python module struct. It can convert an integer to binary format in either byte order. Also, it can handle signed integers, etc.
I am using unsigned long long, which has 64 bits, and little endian, the simplest solution could be:
>>>import struct
>>> n = 12345
>>> struct.pack(‘<Q’, n).encode(“base64″).strip()
‘OTAAAAAAAAA=’
However, the reason I need to use base64 here is that base64 is more compact than the decimal format of integers. But the above example has a lot of A’s, which are caused by zeros padded in the unsigned long long. Also, I am using encode method provided by string to generate base64 in the above example. A better base64 support is module base64, which supports url safe version of base64.
The following example can solve these problems. Also, I remove the padding =’s here.
import base64
import struct
def encode(n):
data = struct.pack('<Q', n).rstrip('\x00')
if len(data)==0:
data = '\x00'
s = base64.urlsafe_b64encode(data).rstrip('=')
return s
def decode(s):
data = base64.urlsafe_b64decode(s + '==')
n = struct.unpack('<Q', data + '\x00'* (8-len(data)) )
return n[0]
Notice that when stripping zeros, how to handle the integer zero itself? If we simply remove all tailing zeros, we will end up with an empty string here. Thus, I keep a byte zero if the integer is zero. In decode, we have to pad zeros to variable data to 8 bytes for unpacking it as an unsigned long long. Notice the padding is in little endian order too.
Run some tests:
>>> print encode(0)
AA
>>> print decode('AA')
0
>>> print encode(12345)
OTA
>>> print decode('OTA')
12345
The code is for Python 2.5.
Finally, an interesting question. Is that true any string constructed by base64 characters is a valid base64 string? See the following example,
>>>print decode('100')
19927
>>>print decode('101')
19927
>>>print decode('102')
19927
>>>print decode('103')
19927
The results are the same. Why? I would say only 100 is a valid base64 string. But python base64 decoder can tolerate the other 3. We know that 4 base64 chars (24 bits) represent 3 bytes. If we have only 3 base64 chars (18 bits) here, it can only represent 2 bytes (16 bits). Thus, the least significant 2 bits of the 3 chars (“101″-”102″) are ignored by base64 decoder.
glassfish errors in netbeans
I don’t use IDE since I am used to editing in vim. I tried to adopt eclipse for java work several times. But I just could not keep using it. However, I recently started to see if I can adopt netbeans for java ee development. My environment is netbeans + glassfish + maven.
I experienced several problems in the beginning
(1) NetBeans: No suitable Deployment Server is defined for the project or globally.
I found the solution here: http://wiki.netbeans.org/JAXWSNB6Maven2GlassFishV2
In Netbeans:
- Right-click the project and select Properties. Navigate to the Run tab.
- In the Server field select GlassFish V2
(2) netbeans SEC5046: Audit: Authentication refused for [admin].
Why do I see this error while I can still successfully deploy my application to glassfish from netbeans?
I found the solution here http://forums.java.net/jive/thread.jspa?threadID=35551
Just simply remove ~/.asadminpass
