Python Pet Peeves

As of this posting, Python has been my main programming language for over three years. Although I definitely feel that Python is not a good fit for all programming projects, the speed and efficiency with which I can code in it has made it my go-to language whenever possible.

As such, I’ve seen a lot of Python code, and have had ample time to think about some of the more nuanced issues regarding coding standards. Here’s a few of my pet peeves, and opinions about them:

from module import *

When I first started python, I used this particular import for a lot of things. I’m using so many methods from this module, why not just import the whole thing? It was definitely a pain in the neck fixing those include issues.

Well, time in the industry has made me realize the error of my ways. This isn’t just python related, this is related to any programming language. Includes/Import should always be as obvious as possible. The correct import methodology, is to do as such:

from module import w,x,y,z

or, if you want to be even nicer:

import module
module.x()

But what if we’re using ten methods from that module? still gotta do it.

What about 20 methods? still gotta do it

What about 100 methods? don’t know how there’s 100 methods in a single module, but you still gotta do it.

The reasoning is simple: you’re providing a very helpful hint that future coders can use to debug your code years from now. That hint is : where the method is actually found.

While you yourself don’t save any time off of doing this, you’re saving hours of development time for future coders, giving them a roadmap to exactly what your function’s stack actually is. Although this can be given by any IDE that has an understanding of the language and it’s dependencies, one shouldn’t assume that this is so. In my experience, when debugging, I have spent anywhere between a good ten to twenty minutes looking for methods, especially in python files with twenty lines of imports. To know exactly where a particular method or module comes from goes a long way to making one’s code maintainable.

For example, suppose I was a programmer who had to debug, and was able to pinpoint the bug to a method that had been previously written, called a_func. The file calling it looks like:

from foo import *
from bar import *

def b_func():
    ...
    a_func()
    ...
    return

Now if I had no knowledge of the modules foo and bar, I would have to look through BOTH foo and bar, and see if either of those had the function a_func. This is only a minor inconvenience if your code only has two of these imports, but the larger a script gets, and the more includes it brings in over the years, could result in one having to look through several files in various locations, to debug one call. Precious time that could have been saved, had the original code just written:

from bar import a_func

Use ternary’s, but only where it makes sense

If you’re not familiar with tenary operators, I’d suggest acquainting yourself now. After all, ternary operators only exist because the problem they solve is so prevalent in coding everywhere. Specifically, the strict point where you want a variable to be one of two things. In Python, ternary operators are represented differently than other programming languages (the typical ( condition ? do_this_if_true : do_this_if_false ) operation). Python has:

do_this_if_true if condition else do_this_if_false

Ternary’s in general have several uses. The big one is providing a default value:

var = (value if value else default_value)

Basically, in any situation where you have:

if this:
	just_one_procedure()
else:
	just_one_other_procedure()

One should consider using a ternary. You can also nested ternarys, although I wouldn’t suggest doing so for more than one level deep. This is especially useful when you have a variable assignment with four different possible outcomes:

x = ( (1 if a else 0) if b
else (2 if c else 3))

To do so with regular if else statements, one would need ten lines of logic. Ternarys are a lesser known function within Python, and it belongs in any programmer’s set of tools.

Search and replace multi-line expressions with SED

Now here’s an interesting problem:
I wanted to do a recursive search and replace in unix, AND I wanted to do an expression that spans multiple lines. Here’s what I came up with:

find ./ -type f | xargs sed -E -i -n
'1h;1!H;${;g;s/</fileSet>.*<fileSet>.*RevisionVersion.*
</fileSet>.*</fileSets>/</fileSet>n</fileSets>/g;p}'

There a lot of examples showing you how to do this.
The first argument lists all files recursively. These are the piped to sed, which uses an inline search and replace (-i or –in-line), then using the expression ‘{}’ which is then modified for multi-line expressions (1h;1!H;).

WebPageTest and IE9

Recently, I tried updating the browser for a WebPageTest instance to IE9. This proved to have some issues, specifically due to the pop-up dialogues that IE9 has now to tell you when something suspicious occurs.

Logging into WPT, I was greeted with an error on an IE9 browser opened by URLblast. Something along the lines of:
“Are you sure you want to use this Non-Verified plugin?”

Of course, the non-verified plugin was the WebPageTest hook. In order to get that working, I modified the security settings on my browser to not care about non-verified plugins:

Internet Options (clicking on that gear icon in IE9) -> Security -> Custom Level.

I modified two settings:

  • “Download unsigned ActiveX controls” to Enable (not secure)
  • “Initialize and script ActiveX controls not marked as safe for scripting” to Enable (not secure)

This then brought me to another error, with IE9 complaining about not using secure settings. Something like:

“Your current settings are insecure”

Well, after some searching, there’s apparently a policy that you can set that disables this specific message:

http://windowsconnected.com/forums/p/959/3087.aspx#3087

Basically it says:

Run gpedit.msc (if you type ‘gpedit.msc’ in the search bar it comes up)

Then Navigate to Computer Configuration -> Administrative Templates -> Windows Components -> Internet Exporer, and right click and enable the “Turn off the Security Settings Check feature” policy.

This gets rid of the error, but then WebPageTest just seems to freeze on a run. After some more searching, there was one final step in the solution. It seems that urlblast has to open the browser using the user’s account. By default, urlblast creates and uses a specific account on which it opens a browser, not necessarily the user that is running urlblast. Having the account opening the browser be an administrator did the trick, and in my situation, I just had it be the same account running urlblast. This can be done with a change in urlblast.ini:

Use Current Account=1

And that did it for me!

Getting Python2.5 to Build with sqlite3 and zlib on Ubuntu Natty 2.5

I had a really hard time finding this, so I’m posting it here:

First one must install all the proper packages on Natty (these are the packages needed for zlib and sqlite in general, not just specifically for Python):

sudo apt-get install zlibc zlib1g zlib1g-dev
sudo apt-get install sqlite3-dev

Then one must add an LDFlag to the new lib directories (apparently Natty has a new directory for X86_64 lib files):

after the ./configure open your Makefile and find the line with
LDFLAGS =

edit to LDFLAGS = -L/usr/lib//x86_64-linux-gnu

and make

Credit for the above snippet goes to Awin Abi and source is below:

http://groups.google.com/group/google-appengine/browse_thread/thread/a8bd0a71270a3ce6

Basically, setting up Python2.5 ( and presumably any version of Python) properly involves downloading the proper package libraries , then building Python2.5 with those packages. In order to do this, the LDFlags variable must have the new library location (the /usr/lib/x86-64-linux-gnu) for Natty and 64-bit processors added.

I have not tried this on a 32-bit machine. This may not be required then, or you may need to point the flag to load the proper directory.

My IDE in Emacs (mainly for Python)

I’m writing this article up to mainly keep track of the current state of my IDE in Emacs, how to set one up, and to keep my to-do list.

Implemented Features

Default Emacs Library Includes

I use the following from the library that comes with Emacs (as of version 23)

  • Viper-mode (viper-mode 3, though I’m sure 5 would be good too)
  • Windmove (through keybindings, for moving around windows easier)
  • hideshow (for code folding)
  • ibuffer (for listing on buffers when buffer switching)
  • ido (for listing of file in a directory in the minibuffer

Code to instantiate:

(setq viper-mode t)
(require 'viper)
(load-library "hideshow")
(add-hook 'python-mode-hook 'hs-minor-mode)
(require 'ido)
(ido-mode 'both)
Keybindings
(global-set-key (kbd "C-x C-l") 'windmove-right)
(global-set-key (kbd "C-x C-h") 'windmove-left)
(global-set-key (kbd "C-x C-k") 'windmove-up)
(global-set-key (kbd "C-x C-j") 'windmove-down)
(global-set-key (kbd "C-x C-;") 'hippie-expand)
(global-set-key (kbd "C-x C-g") 'find-name-dired)
(global-set-key (kbd "C-c C-t") 'ansi-term)
Viper Keybindings (in .viper)
(setq viper-expert-level '3)
(setq viper-inhibit-startup-message 't)
(setq-default indent-tabs-mode nil) ; I think this makes tabs into spaces
(setq viper-shift-width 4) ; don't touch or else...

;; Makes searching w/ regex default
(setq viper-re-search t) ; don't touch or else...

;; The following is for hideshow to work ALMOST similar to vi folding
;; (there were keybindings I didn't like)
(define-key viper-vi-global-user-map "zt" 'hs-toggle-hiding)
(define-key viper-vi-global-user-map "zM" 'hs-hide-all)
(define-key viper-vi-global-user-map "zm" 'hs-hide-block)
(define-key viper-vi-global-user-map "zR" 'hs-show-all)
(define-key viper-vi-global-user-map "zr" 'hs-show-block)

Features implemented using external files

Yasnippet (for bundling and snippets)

Yasnippet provides me features along the lives of textmates bundling, which I think definitely makes things faster in the long run. After all, who wants to write boilerplate code?
http://manual.macromates.com/en/bundles
Yasnippet site:
http://code.google.com/p/yasnippet/

lusty-explorer.el (for a great tab completion file navigator)

Followed this emacs-fu guide:
http://emacs-fu.blogspot.com/2010/07/navigating-through-files-and-buffers.html

And downloaded the .el here:

http://www.emacswiki.org/emacs/LustyExplorer

Specifically I have the following in my .emacs:

(when (require 'lusty-explorer nil 'noerror)

  ;; overrride the normal file-opening, buffer switching
  (global-set-key (kbd "C-x C-f") 'lusty-file-explorer)
  (global-set-key (kbd "C-x b")   'lusty-buffer-explorer))

Desired features

I have yet to implement this, but I would like:

  • Better file search (the ones I could find don’t do what I’m looking for)
    • Specifically, looking for a smart find that allow autocompletion
    • Looking for something along the lines of eclipse

ax_check_mysql introduction and example

I previously mentioned ax_check_mysql.m4 in one of my posts, an m4 macro written for autoconf. So here’s a bit more information about it, and some examples on how to use it.

Introduction

So ax_check_mysql is essentially an m4 macro for autoconf that was written with MySQL plugin developers in mind. When one runs the macro, a detected MySQL installation will give you the following information:

  • The path to the directory containing the MySQL executables
  • The path to the directory containing MySQL includes (if they exist)
  • The path to the directory where MySQL plugins go
  • The version of MySQL
  • Whether MySQL is 32 or 64 bit
Basically providing most of the information, MySQL-wise, needed to install the plugin.

In the situation where an installation can not be detected or an incomplete one is found, arguments can also be entered manually with:

 --with-mysql

(where the root directory of the MySQL installation is passed (such as /usr/local/mysql or some other custom directory)  and

 --with-mysql-command, --with-mysql-plugin, --with-mysql-include

Which would just passing all the directories directly.

Examples

One can include the macro in the same fashion as any other macro in the configure.ac file:

AC_INIT(ax_check_mysql_example,version-1.0)

m4_include([m4_ax_check_mysql.m4])
AX_CHECK_MYSQL([no],[yes],[5.0],[no])
AC_MSG_NOTICE($MYSQL)
AC_MSG_NOTICE($MYSQL_COMMANDS)

Now if I run this script on a computer with MySQL installed, you should something along the lines of:

$ autoconf && ./configure
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
Testing if MySQL was installed to common source/binary directory
checking for mysql... no
Testing if MySQL was installed to common package manager directory
checking for mysql... yes
checking /usr/include/mysql/mysql_version.h/mysql_version.h usability...
 no
checking /usr/include/mysql/mysql_version.h/mysql_version.h presence...
no
checking for /usr/include/mysql/mysql_version.h/mysql_version.h... no
checking /usr/include/mysql_version.h/mysql_version.h usability... no
checking /usr/include/mysql_version.h/mysql_version.h presence... no
checking for /usr/include/mysql_version.h/mysql_version.h... no
checking if /usr/lib/mysql/plugin/ exists...... yes
checking for mysql... /usr/bin/
configure: WARNING: A package install was detected, but the include dire
ctory could not be found! MySQL development library may not be installed
. If development library is installed please use --with-mysql-include --
with-mysql-plugin --with-mysql-command to manually assign directory loca
tions
checking MySQL Architecture... 32
checking MySQL Version... 5.1.41
checking if MySQL install supports Plugins... yes
checking if MySQL version is equal or greater than 5.0... yes
configure: yes
configure: /usr/bin/

Note that the last two lines of output were echoing the MYSQL and MYSQL_COMMAND variables respectively, and that I do not have the development library installed. A full list of variables available are listed in the documentation.

One can pass four arguments when running the macro:

MYSQL-PLUGIN-NEEDED: if the MySQL version doesn’t support plugins (< 5.1), this will cause failure.

MYSQL-REQUIRED: say if MySQL is required or not.

MINIMUM-VERSION: minimum version required for MySQL (i.e. 5.0 or 5.5)

INCLUDES-REQUIRED: whether the MySQL includes are required (will fail if includes are not found)

For example, If I wanted MySQL 5.5 or higher, I could enter:

AC_INIT(ax_check_mysql_example,version-1.0)

m4_include([m4_ax_check_mysql.m4])

AX_CHECK_MYSQL([no],[yes],[5.5],[no])

And as my MySQL installation is 5.1.41, ./configure will fail:

checking MySQL Architecture... 32
checking MySQL Version... 5.1.41
checking if MySQL install supports Plugins... yes
checking if MySQL version is equal or greater than 5.5... no
configure: error: installed MySQL version is not above 5.5. 
Please upgrade your version of MySQL

Entering nothing in the version field will allow any version.

Warnings will be outputted instead of errors if components aren’t required (such as includes or MySQL itself).

And there’s a brief example! Feel free to comment or contact me (tsutsumi.yusuke@gmail.com) if there are any questions/ comments.

The script is maintained by myself on github:

https://github.com/Toumorokoshi/ax_check_mysql

Code folding in Emacs Viper-Mode

Code folding is a feature I’ve never really used, and for the most part seem to have done find without. I generally use search to navigate from place to place in my code, but I realize this isn’t always the most efficient way to go, and code folding is very useful in a couple aspects:

  • It helps focus me on what particular method or class I’m working on (way harder to tell when you’ve got several bunches of code in front of you at once)
  • Getting a good idea of the structure of the code (with everything folded, it’s much easier to see)

So I decided to play around with folding with my current development environment. I use Emacs as my base, but viper-mode for the actual text editing.

Emacs has some pretty good folding tools built-in. Namely, these are FoldingMode and HideShow. I admit I didn’t play around with FoldingMode a lot, as using it seems to involve manually adding the folding points, something which I think is unnecessary 90% of the time. Ideally, I’m looking for a folding extension that automatically determines folding points, and leaves things as hands-off for me as possible. One should be able to open a file, fold it up, and then open and fold as necessary. I’m not looking to waste time adding commented blocks of folding everywhere.

Thats where HideShow comes in. Armed with rules for an array of programming languages, HideShow automatically looks for these patterns and sets folding points appropriately. Exactly what I’m looking for. Simply loading hideshow using .emacs:

(load-library "hideshow")

And activate the hideshow minor mode whenever you load the major mode of your choice (for me it’s Python):

(add-hook 'python-mode-hook 'hs-minor-mode)

Now you have all the access to the wonderful world of dynamic folding! Unfortunately, I didn’t really like the cumbersome keystrokes:

  • C-c @ M-C-s to unfold all
  • C-c @ C-h to fold
  • C-c @ C-s to unfold
  • C-c @ M-C-h to fold all
  • C-c @ C-c to toggle folding

Yeah, a six key-stroke succession is too much for me. So I assigned these bindings to almost the same folding commands as VIM:

  • zm to unfold
  • zr to fold
  • zM to unfold all
  • zR to fold all
  • zt to toggle

To do this, I added configuration into the .viper file:

(define-key viper-vi-global-user-map "zt" 'hs-toggle-hiding)
(define-key viper-vi-global-user-map "zM" 'hs-hide-all)
(define-key viper-vi-global-user-map "zm" 'hs-hide-block)
(define-key viper-vi-global-user-map "zR" 'hs-show-all)
(define-key viper-vi-global-user-map "zr" 'hs-show-block)

(viper-vi-global-user-map tell viper it’s for any buffer in any state with viper as a major mode). So far, this is working like a charm for me. Here’s a screenshot with it at work:

Feel free to comment if you have ideas/improvements!

Autoconf: ax_check_mysql

Just mentioning a little m4 script I wrote a little while ago. ax_check_mysql is used to find a valid MySQL installation, and gives you the binary, include, and plugin directories.

If you have an autoconf project, and you need MySQL, definitely consider it!

http://www.gnu.org/software/autoconf-archive/ax_check_mysql.html#ax_check_mysql

Configuring HttpArchive + Webpagetest (Part 3: HttpArchive)

It’s time to install HttpArchive! So just as with Webpagetest, there’s some requirements for HttpArchive as well.

HttpArchive must run on a unix-based machine, as HttpArchive uses pcntl, a threading function in PHP currently available only on unix-based machines. For this guide I will be using Ubuntu.

The following will be needed on your machine:

  • Apache2+
  • PHP5 or above
  • MySQL
  • Subversion
  • pcntl (PHP)

Most of these can be installed with a package manager. However with pnctl, one must manually download the source, and either configure PHP with the pcntl argument, or compile and install the pcntl extension manually. I found an Ubuntu forum post from skout23 that explains a very easy way to install pcntl for Ubuntu users. However I’m sure aside from the package manager, BSD based Linux users can do the exact same thing: http://ubuntuforums.org/showthread.php?t=549953

Here’s the relevant code for Ubuntu users:

mkdir php
cd php
apt-get source php5
cd php5-(WHATEVER_RELEASE)/ext/pcntl
phpize
./configure
make

And don’t forget to restart Apache afterward!

Once everything is configured properly, you can checkout the HttpArchive source from the googlecode repository:

http://httparchive.googlecode.com/svn/trunk/

In addition, unless you want to download the downloads folder (which contains over 1GB of data from the sites that HttpArchive tracks), it would be best to checkout non-recursively, then check out all other folders:

$svn co -N http://httparchive.googlecode.com/svn/trunk/ .
$cd trunk
$svn up images
$svn up bulktest

Next, we will modify the settings.inc folder with the following information:

  • $gMysqlServer = "YOUR_SERVER"
  • $gMysqlDb = "YOUR_DATABASE"
  • $gMysqlUsername = "ACCOUNT_USERNAME"
  • $gMysqlPassword = "ACCOUNT_PASSWORD"

Finally, Apache needs to interpret the .js files with PHP before being served to the user. There exists a directive inside the .htaccess file in the root of the repository that already accounts for this. However, I had issues with this particular part, so I had to add the directive into the php.conf file under /etc/apache2/mods-enabled (you will need root permissions to modify this file):

<Filesmatch "(filmstrip|harviewer|interesting|interesting-images).js">
SetHandler application/x-httpd-php
</FilesMatch>

And you've configured HttpArchive! Unfortunately it won't really work without any data, but we'll talk about the final steps in part 4: Configuring the two to work with each other!

Configuring HttpArchive + Webpagetest (Part 2: Webpagetest)

Welcome to part 2! This post discusses installing Webpagetest.org. In order to do so, we will need:

  • A windows machine (XP or Windows 7 have been tested with this method)
  • Apache2.2 or higher
  • PHP5 or higher
  • IE of some sort (IE8 or 9 would be best)
  • ffmpeg

It is possible to split up the web server and the testing server, but I put them both on the same machine for ease of use. In addition I found installing and configuring Apache and PHP together on windows was surprisingly difficult, so I suggest installing Xampp. It’s a single-install program that includes many of the tools used for serving web pages and web development, such as Apache, PHP5, MySQL, and Filezilla. Windows 7 machines have IE 8 installed by default, but upgrading is straightforward for XP machines.

Once you have your machine set up properly, it’s time to install Webpagetest! You can download the source here: Webpagetest.org source.

There are also installation instructions on the webpagetest.org google site. I would recommend following these fora a complete guide, but what I have written is a shorter version and will attain the same result.

https://sites.google.com/a/webpagetest.org/docs/private-instances

Configure Apache to point to the www directory of your source, or move the contents of the folder to the “htdocs” folder under Xampp (I found that the Virtualhost directive in Apache was having issues, so I just threw everything into the htdocs folder, where Xampp is initially configured to point to). I found that on the windows machine, giving read/write permissions to the directories needed was not an issue.

You will then have to configure everything properly. This involves basically copying and pasting everything in the settings folder to it’s non-sample equivalent. I was able to do this because I wanted a basic instance, but keep in mind you may need to do more if you want more complex options.

Configuring the system to run the tests is best explained, verbatim from the private-instance setup site linked above:

  1. Configure the test system to automatically log-on to an administrator account. Running “control userpasswords2″ from the start menu is one way to configure it.
  2. Disable any screen savers (the desktop needs to remain visible for the video capture to work)
  3. Disable UAC (Vista or later – slide to “never notify”)
  4. Uninstall IE Enhanced-Security Mode (Windows Server)
  5. Copy the test software from the agent folder to the system (to “c:webpagetest” for this example)
  6. Install the DUMMYNET ipfw driver
    • Pull up the properties for the Network Adapter that is used to access the Internet
    • Click “Install”
    • Select “Service” and click “Add”
    • Click “Have Disk” and navigate to c:webpagetestdummynet
    • Select the ipfw+dummynet service (and click through any warnings about the driver being unsigned)
  7. Create a shortcut to c:webpagetestdummynetipfw.cmd in the startup folder
  8. Create a shortcut to c:webpagetesturlblast.exe in the startup folder
  9. Make a copy of the settings file (urlblast.ini) based on the sample
    • Give it the path to the server (default configuration points to a server on the local machine)
    • Configure the location to match the location defined on the server in locations.ini (if modified)
    • Configure the location key to match the server in locations.ini (if modified)
  10. Reboot to make sure everything starts up correctly

Note:On windows 7, ipfw will not properly install (it will not show up under installable services). If you want to use windows 7, you must add a “Location = LAN” directive under test in settings.inc in settings:

[Test]
Location = LAN

After that, your instance should be set up! Now that wasn’t so bad, was it?

Next time we’ll talk about installing HttpArchive!