Automatic Python virtual environments Python virtual environments are great for seperating your development environments for each project. You can start with a fresh install and dependencies for each project, which helps keep your project dependency list short and your Python path clean. I use virtualenvwrapper and this setup to make new environment creation easy, but I find myself constantly running a project only to realize that I haven’t activated the proper environment.

Recently, I have been working with the Python API for Spark to use distrbuted computing techniques to perform analytics at scale. When you write Spark code in Scala or Java, you can bundle your dependencies in the jar file that you submit to Spark. However, when writing Spark code in Python, dependency management becomes more difficult because each of the Spark executor nodes performing computations needs to have all of the Python dependencies installed locally.

To help facilitate my blogging workflow, I wanted to go from written to published post quickly. My general workflow for writing a post for this blog looks like this: Create a post in _posts Write the post Run fab sync Here is the repo fab sync is a custom command that uses the magic of Fabric to stage, commit and push changes in my blog repo to Github. Next, Fabric uses an ssh session in the Python process to connect to the server on which my blog is hosted, pull down the newest changes from the blog repo and finally, build the Jekyll blog so that the changes are immediately reflected on this site.

If you spend most of your time in the command line, you don’t want to leave to do math. Qc is a script that does in-line command line math without forcing you to exit the main bash prompt as you might with a program like bc or a language interpreter. #!/bin/bash python -c "print $1" Make the script executable with the command: $ chmod +x qc.sh Alias it to qc by editing the .