The Lab @ CarlSverre.com

Static Hosting

2010-02-09T00:00:00-08:00

Carlsverre.com has just shifted from Hostingrails.com to a combination of Github and Google App Engine. The transfer was prompted because my Hosting Rails account expires in 4 days. At first I was just going to renew, but the more I thought about the joyous union of Git commits and appengine pulls, the more my mind was made up. So now that I got all the kinks worked out, here is a short tutorial for those of you who want to do it yourselves.

$ cd ~/dev

DryDrop

DryDrop is the key to the whole process. Developed by Antonin Hildebrand, DryDrop allows anyone to quickly deploy to Google App Engine from a Github repo. DryDrop is a Python application which grabs all the files in your repo, stores them in the Google cloud and serves them on your App Engine URL. Grab DryDrop from its Github repo like so:

$ git clone git://github.com/darwin/drydrop.git

You will also need the Google App Engine SDK which you can get here.

Getting the right info

To follow with this tutorial you will need a Github repo full of static files. Something similar to my Carlsverre.com repo would be perfect. Once you have your repo setup you will need to figure out what your raw pulling url is. A generic raw pulling url looks like this:

http://github.com/USERNAME/REPONAME/raw/BRANCHNAME

USERNAME and REPONAME should be obvious. Use master for BRANCHNAME unless you have another branch full of static files. My raw pulling url looks like this:

http://github.com/carlsverre/carlsverre.github.com/raw/master

To test it out, try navigating to the url followed by the name of a file in the repo. If you have the url right, the current version of the file in the specified branch should be returned.

Take a note of your raw pulling url now - we will need it later.

Setting up App Engine

The next step is to get your appengine project registered. Login using your Google account to the App Engine management console and click Create an Application. Once you have created an application, head back to the terminal to upload DryDrop.

Make sure you replace PROJECTNAME and other obvious variables with your own words.

$ cd ~/dev/drydrop
$ rake upload project=$PROJECTNAME

Initializing DryDrop

Now that you have uploaded DryDrop, it’s configuration time. Head to your shiny new project url (PROJECTNAME.appspot.com) and click admin section. The only setting we will change right now is the Content Source which you can find in the Settings tab on the left.

Click on the pulling from url and change it to the raw pulling url you created earlier. Woo we are 99% there!

The last step

All you have left to do is setup a post-commit hook from your Github repo to your shiny new DryDrop app. Follow these steps and your set:

Go to the Admin section of your Github repo
Click the Service Hooks tab
Click Post-Receive URLs on the left
Add the following url (with keywords replaced) to your Post-Receive URLS:

http://YOURGOOGLEAPPNAME.appspot.com/hook/github

Can I use Jekyll with DryDrop?

A growing number of Github users are using Jekyll to create static sites for their Github projects. I use it to power this blog and am extremely happy with the result. As for using Jekyll with Drydrop I suggest one of two options:

Create a _site.yaml file in the root of your Github repo which wrangles Drydrop into pointing requests in the right direction
Commit Jekyll’s output directory into a different branch and point Drydrop to that branch rather than your master

I am eventually going to switch to option #2 as I see that as being a cleaner solution. For reference on how to set up your _site.yaml file you can check out mine here.

Comments? Suggestions?

This is the first tutorial posted on my shiny new Jekyll blog, so I would love any feedback that you’d care to share. I currently haven’t setup a main feedback system, but it is in the workings. For now you can get in touch with me via Twitter.

The New Lab

2010-01-06T00:00:00-08:00

Welcome to the latest incarnation of The Lab@carlsverre.com. For those of you who looked at the last site, you will understand why its not archived. As for everyone else, lets just say that it grew to be a bit too busy. I did not stick to the plan (which was to keep things minimal) and I suffered because of it. The Lab was supposed to be a place where I could expose the process of writing code. It was supposed to encourage me to write more code with my lofty one project a month goal. Unfortunatly it failed in many ways. So here I am, once again with a new site and a new dream.

The Lab is my place to write about programming. Any new and interesting projects which I am working on will hopefully make their way here. Any research or class projects which I am involved in (and can disclose) will make its way here. This will be my hub of code-freedom. This will be my face in the open source world. And hopefully somewhere along the line, this will be recognized for something. That is my goal.

Anyways, the rest of this page is the test-bed for most of the css styles used on this site. I prototyped this entire site in three phases using only two files (the index and this post):

Wrote HTML test-beds
Applied CSS styles and the 960 grid system
Added JS code for Cufón and jQuery

The Lab uses Cufón for font-replacement, 960 for a grid layout and Jekyll for static site generation. See the footer for the complete credits as well as a link to this site’s source.

Enjoy!

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim.

Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit.

This is a block pulled to the left… I hope it looks REALLY nice!

def foo
  puts "hello world"
end

Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus.

What the flickr?!

2009-09-01T00:00:00-07:00

I have once again delved into the world of imagery, and conjured up a interesting project combining Google App Engine, Django, Python, FlickrPy and Flickr. The project’s idea is that with Flickr’s huge resource of images one should be able to create a game similar to pictionary, with the player guessing and Flickr “drawing”. My attempt at making this game is called What the Flickr?!

That sounds easy!

Well your wrong. Unfortunately, playing What the Flickr?! is difficult due to the Flickr search engine being based on crowd-sourcing. Since all of their terms are user-generated, some search terms come up with completely unrelated results. Thus my first order of business was to figure out a way of getting around this problem.

Unfortunately I haven’t found it yet. So I thought about other methods of making the game easier, and the one I came up with was to make a extremely targeted word list. I wanted words which were commonly used. After scouring the interwebs I found this site. I grabbed all the words except for the “operations” column and plugged it into the Google datastore.

So what happened next?

A bunch of coding, then no coding for awhile followed by a bunch more coding. Essentially I completed the whole site in about 3 hours broken up into three sessions. Not bad for someone who has only done the Django tutorial.

The reason why I am failing at doing a nice code breakdown is for a couple reasons. First, I am keeping the code internal for now because I have some different plans for it as well as it being pretty unfinished. Second, I am completely exhausted and sick. If you have a problem, bite me.

So, can I play?

Of course! That is the reason I made this post (and to fulfill my 1 project a month goal). You can play the game here.

Have fun, and hopefully next month I will have something a bit more interesting!

XP + Linux

2009-08-01T00:00:00-07:00

Hello there! The following post regarding Windows XP, dual booting and Linux has been contributed by my friend Kris Koiner. It is quite a heavy article, but I believe its a good read. It is posted here solely in the hope that it will benefit someone somehow sometime… somewhere.

Disclaimer: If you try anything described in the following article and ruin your computer, its not our problem. If you don’t agree with this, then don’t read the article. Thank you.

Let me just say that I am really starting to hate the arrogance of Windows XP. I’ve just spent the last week and a bit trying to get XP to dual-boot with Ubuntu on my main machine. I know what you’re thinking: “That’s not that hard, I’ve done that many times.” Normally, you’d be right, the process should be relatively simple, however there were a multitude of reasons why this one didn’t.

Firstly, the current setup:

Intel Core 2 Duo E6550 @ 2.33GHz
Asus P5E-VM HDMI with 6GB OCZ DDR2-800
200GB, 320GB, 1000GB SATA drives
Ubuntu 8.04 LTS 64-bit

The main drives are the 200GB (sda) and the 320GB (sdc); why the 320GB is not sdb I don’t know, but it works, so I’m not fixing it. The 1000GB is irrelevant, although crucial in the entire process, but for all intensive purposes, it could have been a flash drive.

The 200GB and 320GB are split into 3 partitions and raided together:

md1: 100MB: raid1: sda1 & sdc1: /boot
md2: 140GB: raid0: sda2 + sdc2: /
md3: 120GB: raid1: sda3 & sdc3: /home

As you can see, there’s ~120GB of unused space at the end of the 320GB. The idea is to install Windows XP on the fourth primary partition formatted as NTFS. Seems simple, right? Let’s begin.

Definitions

BIOS: Basic Input-Output System: the config file for your computer.
MBR: Master Boot Record: the first 512 bytes of a hard drive, contains the partition entries, and the boot loader.
PBR: Partition Boot Record: the first 512 bytes of a partition, contains boot loader code for the partition.
NTLDR: NT Loader: the windows boot loader.
LILO: The Linux boot loader, but of course has more features.
GRUB: Another Linux boot loader, newer than LILO and more popular among Ubuntu distros.
FAT: File Allocation Table: an old and outdated file system that works, so everybody uses it.
NTFS: New-Technology File System: the Windows preferred file system.
PATA: Parallel ATA: a drive interface used in the olden days.
SATA: Serial ATA: PATA’s replacement.
RAID: Redundant Array of Inexpensive Disks: a way of joining multiple disks for an added benefit, usually speed or redundancy.
LBA: Logical Block Addressing: a new way of addressing hard drives that allow up to 2.0TB on a single disk.
NIC: Network Interface Card.

Forward

Before you begin reading this lengthy rant about Windows XP, let me apologize for the spelling, grammar, and inconsistencies in this article. I have written it over the course of many days, and each day brought a different perspective to the writing.

This article requires a certain level of computer knowledge. I use quite a few nix terms, make reference to particular programs, and don’t go in to a lot of depth explaining things. If you have installed a dual-boot before and have used thenix shell, you’ll probably be fine. If not, there are hopefully enough analogies and comedic material in here to at least provide some entertainment.

One last thing, I flip between “Windows” and “XP” as if they were the same thing. Also, not everything I say is completely accurate, most things are simplified to keep them short and on-track, if you think it’s really bad… too bad.

For those of you in a hurry, or hoping for a solution to the problem, just skip to the last section.

Hour 1

For this task, I’ll be using a Windows XP Professional SP3 disc. I decided to use this version as it would avoid installing the service packs on top of older versions. So I put the disc in, say goodbye to Linux for a while, reboot, and set the CD to boot in the BIOS.

Problem 1: Windows disc won’t get past the “Setup is inspecting your hardware.”

Windows shows the message, then the message goes away, and nothing else comes up; the screen just stays black. Ctrl+Alt+Del to try it again and the same error, so we’re already off to a great start.

After some google searching, because after all isn’t that always the case: complaining to google to see if anybody else in the world has the same problems as you, I came across something that said that the Windows XP disc boot loader doesn’t like SATA drives, so you have to disable them in the BIOS. Okay, we can try that.

Fix 1: Disable all SATA drives in the BIOS.

This works! We now get into the Windows “loading drivers…” screen, all in those beautiful white-on-blue colors. One thought occurs: we disabled the drives, so how will Windows install? We’ll deal with that when it shows up.

After about 8 minutes of “loading drivers” and “getting ready to install Windows”, the system partitions are listed, and XP asks where you’d like it to deploy itself. So, apparently, while it was loading drivers, it discovered a bunch of drives, so we wonder why we had to disable the drives in the first place: confusion++.

After being presented with a list of 8 locations to install XP, I selected the unformatted space at the end of my 320GB. Windows asks me if I would like to create a partition, I say yes, and it tells me it can’t.

Problem 2: Windows cannot create a fourth primary partition.

This problem makes little sense, but upon reconsideration, it is fairly logical: the MBR only has space for 4 entries (thus, 4 primary partitions), yet, it is possible to have more than 4 partitions by using extended (or logical) partitions. This takes up an entry in the MBR and allows a second partition table somewhere else on the disk to deal with extended partitions. I assume that Windows reserves the fourth entry in the primary partition table for an extended partition, thereby not allowing a fourth primary partition. Makes sense (still stupid). Aside: it might be possible to install XP on an extended partition, but I had my mind set on the fourth primary, so I needed a fix. Maybe Linux can help me out…

Fix 2: Boot into Linux, install NTFS drivers, create fourth primary partition as NTFS.

This process went relatively smoothly, so no real complaints. Don’t forget to re-enable the drives or you won’t be able to get into Linux. Also, after creating the partition and rebooting, don’t forget to disable the drives before trying to load Windows.

Hour 2

Back into Windows install, waiting… Okay, choose your partition. Now we get to choose the Linux-made NTFS partition. Windows likes it, but isn’t satisfied and would like the first partition on the 200GB as well.

Problem 3: Windows wants first partition on first drive.

You can imagine how I feel, here I am giving over 100GB to XP and taking time from my Linux and it wants more. I guess this is XP’s way of bargaining: to install it on the second drive, you must also give it space on the first. This brings up an issue however, my 200GB is full with raided partitions, so I can’t give XP what it wants without breaking something. Luckily for me though, the first partition is mirrored and only 100MB, so I’d be able to restore it somewhat easily.

After careful consideration, I decided that XP could have the first partition on the first disk, as it would only be temporary, and once XP was set up, I would take it back.

Fix 3: Give it what it wants, take it back later.

So XP happily removes the partition, breaks my /boot mirror and puts a nice FAT16 partition in it’s place. Good news is, that with XP happy, it starts to copy files. Time to take a nap.

Hour 2.5

After waking up, imagine my surprise to find Windows install back at square one asking me where I would like to install XP. Once again, it becomes clear: the drives were disabled, so when the computer rebooted, all the BIOS found was the CD and we repeated the process.

No big deal, just F3 to quit the install, reboot the computer, re-enable the drives, and let Windows carry on. Everything but the last step ended up working: just before getting into Windows, however, we are told that XP cannot find “Hal.dll” in it’s system root and we come to a stand-still.

Problem 4: XP cannot boot: Hal.dll not found or corrupt.

Let’s ask google, but first, some background: Hal.dll is for the Hardware Abstraction Layer. It let’s Windows and your applications pretend that all computers are the same, in simplistic terms. After some good ol’ internet prodding, I find out that it might be because XP is installed on the second drive, and NTLDR doesn’t know this. Therefore, we must edit the boot.ini configuration file in the first partition of the first disk.

Fix 4: Edit boot.ini to reflect non-standard setup.

Since it’s a FAT16 partition, Linux (which is already installed) will be able to modify it. Let’s just boot that back up. Oh wait, XP wrote over the MBR and /boot partition.

Problem 5: XP was naughty, and didn’t make a backup: wrote over MBR.

So before we can modify our corrupted boot.ini file, we must now fix Ubuntu. Pop in the Ubuntu install disk, boot up the live CD, open a terminal, and rewrite GRUB to the MBR. That works. Shutdown the live CD, and reboot into our own version of Ubuntu. It informs us that our /boot partition is degraded, and we tell it to forget about it and just deal with it.

Fix 5: Use a Linux live-CD to reinstall GRUB.

After logging in, we mount the first partition and open the boot.ini file in our favorite editor (mine is vim). Just as I suspected, the XP entries point to the fourth partition on the first disk. Each boot.ini entry contains three fields: the controller, the disk, and the partition. Now since SATA only hosts one drive per controller (okay, not quite true, but roll with it), we know that the disk will be ‘0’, as opposed to in olden days when you would differentiate between the master and the slave drive. Thus, we could bet that the correct path to the XP partition would be 1-0-4; the controller and disk numbers start at zero whereas the partitions start at one, why, I don’t know, but that’s a different rant. Just to be safe, I’ll make a bunch more entries with the different combinations so that I don’t have to repeat this process over and over again.

Upon rebooting, all boot.ini possibilities resulted in the same “Hal.dll might be missing or corrupt” error. This lead me to believe that despite all its efforts, XP really didn’t want to run from a second hard drive. Since the situation wasn’t about to change, and knowing that an alternate boot loader, LILO, can boot Windows from a second hard drive, I figured I’d try LILO instead of GRUB.

Hour 4

After some pondering as to the easiest way to install LILO, I decided to install it from my Backtrack live-CD. An aside: I really like Backtrack, it is probably one of the most advanced and complete live-CDs out there.

Long-story-short, between the man pages of LILO and google, I was fighting my way through the LILO install. Maybe I was doing it wrong, or maybe it was because I was trying to install it off a live-CD, but it wasn’t working very well, and in the end, I decided that I could probably make it work just fine with GRUB.

Knowing that GRUB can chain-load boot records, I would need a Windows XP boot record on my XP partition. But, XP installs it on the first partition of the first drive, remember?

Problem 6: Need XP boot record on fourth partition.

So here’s the idea: we’re going to take the boot record from the first drive, first partition and move it to the second hard drive, fourth partition. At which point GRUB can go in the MBR, load its stage2 from the first drive, and then chain-load to the second drive.

But, we have another problem: the XP install creates the first partition on the first drive as FAT16, whereas our fourth partition is NTFS, so we can’t directly copy the PBR. So, we’re going to repeat the entire install process, but this time on a FAT32 partition so that we can transfer the PBR.

Fix 6: Reinstall XP on a FAT32 partition and transfer the PBR.

Okay, here we go.

Hour 5

Reboot the computer, disable the drives, wait for the installer to load everything it needs, delete the old partition, create the new… oops, forgot, Windows can’t create a fourth partition. Reboot, enable the drives, boot into Linux, open parted, delete the NTFS partition, create a FAT32 partition, save, reboot. Disable drives, wait for the installer, XP needs first partition on first drive? Yes, take a nap.

Wake up to the start screen again, reboot, enable drives, this time, however, we’re not expecting XP to load, and it doesn’t, so no surprises.

Hour 6

Now that that’s done, we need to get our PBR and move it to our XP partition.

Problem 7: GRUB is once again broken, but we can’t fix it, because we need what has replaced it, which is of course, also broken.

So we turn to our invaluable Backtrack once again. Using it, I made a copy of all the PBRs and MBRs around for further investigation. I also had a copy of my GRUB MBR and partition (that I made after re-installing GRUB the first time) so I copied that back on to the drive, rebooted, and got back into Linux a little easier.

Fix 7: Use Backtrack. Make copies.

Now back in my own Linux, I start analysing all the boot records I’ve collected and start piecing together a new one for the XP partition.

Hour 8

Okay, I lied that last step took more than two hours, but when it was done, I had my own boot record for the XP partition. Copied it over to the now FAT32 partition, adjusted GRUB’s menu.lst and prepared to reboot.

As I’m sure you probably could have guessed, it didn’t work (I had gotten my hopes up unfortunately), but I got a different error: referring to a “hardware detection error”. Unlike the “hal.dll” error, this one seemed more upstream, but like the “hal.dll” error, it too resided in the NTLDR.

Problem 8: Tried own boot loader, new error, same problem.

By this time, I was just getting frustrated with all the problems that XP was throwing at me and eventually decided that it might be more efficient to give in a little more. After all, I was already okay with XP breaking my /boot partition every time.

Fix 8: Give up, comply with XP and give it space on the first drive.

Hour 9

I decided that I would flip the 200GB and 320GB drives so that the 320GB was the first drive and the 200GB was the second. This would make the XP partition the fourth partition on the first drive, where XP would hopefully be a little happier. But first, I have to restore everything the way it was before I started, or my Linux might stop working.

Loading Linux again, I deleted the Windows partitions, re-created my raid partitions, re-added them, and rebuilt the arrays. Turned off the computer, opened it up, swapped the SATA cables, and powered it back on. Load into Linux, make sure all the drives are still in the arrays, adjust any mount points that have been broken. Overall, system looks good, time to try Windows again.

Open parted, create NTFS partition, now on the first hard drive, reboot, disable drives, insert Windows disc, wait for install. This is getting kind of repetitive isn’t it? This time however, XP install did not ask for the first partition on the first disk. The install worked fine, rebooted, enabled the drives, get to the boot loader… nope, still doesn’t work: back to a “hal.dll” error.

Problem 9: XP still doesn’t work, wants more than the first drive?

So XP won’t even work when it’s given space on the first drive, what could it possibly want? The first partition? It can’t have that because it’s too big to fit in the space of my /boot partition, and I’m not going to spend the next day shuffling my raid partitions to give it more space. At this point, it’s getting really frustrating to deal with XP and all of its “me” attitude, so I decided to grab an old laptop hard drive, an 80GB SATA, and just see if XP would install on that. I know the drive has some bad sectors on it, hence why I was avoiding it before, but if XP finds a way to get at least to the second part of the install, I’d be happy.

Fix 9: Completely abandon all plans and just give XP its own drive.

Hour 10

Open the computer again, install the now-fourth hard drive, disable the drives, wait for the installer, point it to the fourth drive this time, and for some reason, it doesn’t want the first partition on the first drive. That’s weird, so it only must have it when it’s not the first partition or on the first drive. Moving on, it installs for a while, reboots, I enable the drives again, and… no way, it actually boots in to the second part of the install. Victory!

So now we get to sit back and laugh in disgust at all the propaganda that is shown to us during the install: “..faster and more reliable” “..easiest Windows yet”, “..latest hardware and software”, “most dependable Windows…”, “..safety, security, and privacy”, the list just keeps going and going.

Finally, it reboots again and we get our first glimpse of the Windows’ user interface. I open IE in order to download Firefox, but for some reasons, XP didn’t install the drivers for my network card.

Problem 10: No NIC drivers.

Hahaha, we go through all this trouble, and XP can’t even get us a little bit of out-of-the-box Internet? Wasn’t there a quote in the install about supporting the latest hardware?

Rather than switch to Linux right away, I just used my other computer to download the NIC driver, store it to a flash drive, wait for XP to detect the drive, install the drivers for the flash drive… Okay, now I can install my network card. Ten minutes later, I’m downloading Firefox, then my graphics card drivers, then my sound drivers…

Fix 10: Use another computer.

Now that XP is finally getting settled in, completely unaware and ungrateful of all the things that I’ve done for it, it’s time to get Linux back and integrated.

Hour 12

A simple boot into Backtrack, replace the MBR from my GRUB backup, reboot, and we’re back into Linux. Make sure all the raids are intact, and edit the GRUB menu.lst to chain-load Windows on the fourth drive. Reboot again to confirm that everything works, and it does, unbelievable.

Now, something I noticed in XP: it detected the old broken install of its predecessor on fourth partition of the first hard drive and decided that it would be a good idea to call that one “C:” and the working copy “L:”. It choose “L” because it noticed that there were a lot of other partitions that one day would surely be visible to XP (since they’re empty now) and reserved letters for them, how nice.

Unfortunately, because Windows had commandeered and adopted itself a “C:” drive, I couldn’t tell it to let it go and call the working version of Windows “C:”. So now, because I’m not going through the install process again, I get the added luxury of changing the install paths of all my programs. Either way, it works, so I’m not touching it. It can figure out the hard drive is corrupt on its own.

Final Thoughts

Although it seems like I was able to complete the entire process in about 12 hours, it actually took more like 48 hours spread out over a little more than a week; a couple of things contributed.

Firstly, I needed the Linux portion working throughout the day, every day. So, when it got around to midnight, I stopped the “progress” and began restoring the computer. This included rebuilding the /boot partition, rewriting the MBR, and lots of rebooting.

Secondly, with the amount of reboots, the mount-count for each partition increased quite rapidly. Once it hits a certain number, Linux checks the entire partition for errors, which is good, but takes a couple of hours where you can’t do anything else but watch it.

I described attempting to put a PBR on the FAT32 partition, well, I also tried it on an NTFS partition after that. I pretty much went through every single possibility of the above process and then just chose the best one to write about. What I didn’t like about the NTFS partition was that it has reserved files, as opposed to FAT’s reserved sectors. This makes it harder to edit the drive in a hex editor and maintain consistency.

Speaking of possibilities, at one point I figured my XP-CD was corrupt, so I tried some other discs, including an older XP disc and a Windows 2000 disc. I did not know this, but apparently Microsoft took a while to get on the large-hard-drive bandwagon as neither discs used LBA and had a 128GiB size limit. Waiting each time for the installer to load files, rebooting, enabling and disabling drives, these all took many hours of headache.

And finally, the countless hours spent searching through google and online forums every time I hit a brick wall.

Start reading here if you skipped the article.

So in the end, I actually never got XP to do what I wanted. I tinkered a lot with it, got frustrated at it, learned a lot about it, but at some point in the process, deemed it a waste of time, and just gave in to XP’s forceful demands. I feel like I was just bullied into installing an unsafe, unappreciative, and unimpressive OS just to play a couple of games. But after all, that was the whole point: to play games, so I can’t complain too much.

Moral of the story: XP doesn’t play well with others, but we knew that, I mean, just at look at his parents…

Time.txt

2009-07-01T00:00:00-07:00

This post is out of date. It is only here for posterity.

The web is plagued with programs and websites pledging that they can solve all your time-management and todo-list problems. People such as David Allen have come up with systems such as GTD to help get things done. For some people these various methods work fine, but for others we just need to be able to create our own system. About a month ago I finally got fed up with all the various todo lists and time tracking interfaces I had to deal with, so I set out to create my own evolving masterpiece. I call it time.txt inspired by todo.txt (a great command-line-interface (CLI) todo list).

What does it solve?

The idea is that Time.txt will evolve to become a complete CLI project management suite which can be updated through twitter (and via twitter, sms) and provide all the statistics you will ever need. The next release will integrate the following:

A backend stats sqlite3 database. This will be synced from the text database in order to efficiently store long term stats and to provide an universal interface to the stats for your own hacking purposes.
Export-to-excel functionality

Once those parts have been added, time.txt will hit v1. After that, a twitter bot will be added and more features will be planned.

Contribute!

Time.txt is open source and licensed under the GPL so feel free to fork it on github and have at it! If I like what you change ill merge it back into the main branch.

Get Time.txt

FlickrFortune

2009-06-01T00:00:00-07:00

This post is out of date. It is only here for posterity.

Ah the sweet smell of Python in the morning. It’s true, I have fallen for the sweet embrace which Python offers; the easy to read coding style, the amazing documentation system, django, all of it. It speaks to me like no other language has. I would like to start this post by quoting directly from the Python Interactive Shell:

  >>> import this
  The Zen of Python, by Tim Peters

  Beautiful is better than ugly.
  Explicit is better than implicit.
  Simple is better than complex.
  Complex is better than complicated.
  Flat is better than nested.
  Sparse is better than dense.
  Readability counts.
  Special cases aren't special enough to break the rules.
  Although practicality beats purity.
  Errors should never pass silently.
  Unless explicitly silenced.
  In the face of ambiguity, refuse the temptation to guess.
  There should be one-- and preferably only one --obvious way to do it.
  Although that way may not be obvious at first unless you're Dutch.
  Now is better than never.
  Although never is often better than *right* now.
  If the implementation is hard to explain, it's a bad idea.
  If the implementation is easy to explain, it may be a good idea.
  Namespaces are one honking great idea -- let's do more of those!

Moving on.

FlickrWhat?

FlickrFortune is the product of my imagination. I wanted to create an innovative way of twisting together the massive power of the FlickrAPI and the often overlooked insight of fortune. Fortune is a program included on most Unix operating systems that retrieves a random fortune cookie or quote from a database included with the program. You can read more here: Fortune. FlickrFortune is implemented as a Python script which accesses the FlickrAPI using Beej's Python module after extracting the largest word in a random fortune (from the fortune program). That’s the gist of it.

First some “live-coding”

If you want to follow along with the following code you will need the FlickrAPI for python referenced above, a flickr API key, and the fortune program included on most linux/unix systems. Also Python 2.5.2 would be helpful.

Disclaimer: I will try to explain some of the flickr/fortune related concepts, but you’re on your own when it comes to the Python. Also the next couple code snippets are all in a single Python Interactive Shell, thus imports are at the top.

>>> import flickrapi
>>> import os
>>> pipe = os.popen("fortune -s")
>>> fortune = pipe.read()
>>> pipe.close()
>>> fortune
'Among the lucky, you are the chosen one.'
>>>

So what’s happened. In line 3 we create a pipe connected to “fortune -s”; the flag specifies “short fortunes”. In line 4 we grab the output of the fortune pipe and store it for later. So far so good.

>>> apikey = "YOUR API KEY HERE"
>>> flickr = flickrapi.FlickrAPI(apikey)
>>> tag = "chosen"    # Put the biggest word in the fortune here
>>> photos = flickr.photos_search(tags=tag,sort="relevance",per_page="1")
>>> photos.attrib['stat']    # Ensure photo was found
'ok'
>>> photo = photos.find('photos').findall('photo')[0]
>>> photoURLFormat = "http://farm%s.static.flickr.com/%s/%s_%s_b.jpg"
>>> photoURL = photoURLFormat % (photo.attrib['farm'],
             photo.attrib['server'],photo.attrib['id'],photo.attrib['secret'])
>>> photoURL
'http://farm1.static.flickr.com/9/12379604_20464c804d_b.jpg'
>>>

What does the above code do? Well, first we initialize a flickrapi object (line: 2), which is followed by using said object to search flickr (line: 4). If line 5 doesn’t return “ok” then search Flickr using a different tag. On line 6 we get a list of photos returned by the search and choose the first one (there should only be one because we specified “per_page=”1”“). Finally after running the various attributes of the photo through the string formatter we have the url.

So there you have it. We are practically done. Just need to wrap all of the above into a script and then… Oh, right, image manipulation. The essence of image manipulation is that it’s easier to do when you are not working in the interpreter, so let’s start laying out the script!

Round 1

In the design process of round one I just had a basic program in mind.

Get the fortune
Get the tag
Get the flickrImage
Make the wallpaper
Update the desktop background

As this was satisfactory, I started programming.

A couple hours later

Now don’t get me wrong: this should not take a couple hours. However, this is the first program I have written in Python and thus I spent awhile understanding how to use the FlickrAPI as well as debugging my python code. Unfortunately, since I wrote this program before I decided to create this website, I am unable to show you code revisions at each step, so instead I’ll go over some things I ran into.

The PIL

The PIL is the Python Imaging Library which adds a ton of image related operations to the already huge Python feature set. Here is how I used it:

from PIL import Image
import ImageDraw
import ImageFont

def createWallpaper(flickrImage, fortune, prefix):
    wallpaper = Image.new("RGB", wallpaperSize, "black")
    draw = ImageDraw.Draw(wallpaper)

    flickrImage = Image.open(localDir + flickrImage)
    x,y = flickrImage.size

    x0 = (wallpaperSize[0]-x) / 2
    y0 = (wallpaperSize[1]-y) / 2

    x1 = x0+x
    y1 = y0+y

    wallpaper.paste(flickrImage, (x0,y0,x1,y1))

    boxMargin = 10

    font = ImageFont.truetype(localDir + "Sugo.ttf", fontsize)
    textSize = draw.textsize(fortune, font=font)
    textTopLeft = (wallpaperSize[0]/2 - textSize[0]/2,
                 wallpaperSize[1]-textSize[1]*4)

    boxTopLeft = (textTopLeft[0]-boxMargin, textTopLeft[1])
    boxBottomRight = (wallpaperSize[0]/2 + textSize[0]/2 + boxMargin,
                    wallpaperSize[1]-textSize[1]*3+boxMargin)

    draw.rectangle([boxTopLeft, boxBottomRight], fill="white")

    draw.text(textTopLeft, fortune, font=font, fill="black")

    del draw

    filename = wallpaperDir + prefix+"_"+wallpaperPrefix+".jpg"
    wallpaper.save(filename, "JPEG")
    return filename

It should be quite straight forward but for those who want a bit of a breakdown here it is:

Make a new blank image
Create a draw object which provides various image-manipulation operations
Open the previously downloaded Flickr image
“Paste” the Flickr image into the center of the wallpaper
The next 11 lines create a font instance using the Sugo font, calculate its position, draw a white box and finally draw the fortune.
Finally we save and return the file name of the new wallpaper

Simple as pie. I realize that the above code is heavily unoptimized, however this will be fixed in version two (discussed later).

Downloading and saving the Flickr image

import urllib2

def getImageData(url):
    try:
        return urllib2.urlopen(url).read()
    except urllib2.URLError:
        print urlib2_URLError
        exit(1)

def saveFlickrImage(imageData, prefix):
    fileName = prefix+"_"+flickrPrefix+".jpg"

    try:
        f = open(localDir + fileName, 'w')
    except IOError:
        print IOError
        exit(1)

    f.write(imageData)
    f.close()

    return fileName

I love urllib2. Oh and Python’s file handling. Anyways, this code is extremely self explanatory. The reason this process was divided into two functions is that I wasn’t sure if I needed to use the save function multiple times and so I seperated it. In the end I only used it once, and thus it might be merged with getImageData in V2.

Dealing with arguments

At this point you have seen most of the code I used in the final script; the fortune/Flickr accessing part from our Python Interactive Shell experience, and the above two code snippets. However after I wrote the script I added a couple other simple features which required the usage of program arguments. These new features were:

Looping n times (bulk generate wallpapers)
Add color to the output (using bash shell color codes)
Only change the desktop background if the -s flag is provided

This of course brings me to one of my favorite modules in Python: getopt.

try:
    opts, args = getopt.getopt(argv, optsString, optsList)
except getopt.GetoptError:
    usage()

for opt, arg in opts:
    if opt in ("-h", "--help"):
        usage()
    elif opt in ('-n', "--number"):
        numWallpapers = int(arg)
    elif opt == '-s':
        updateBackground = True
    elif opt == '-c':
        colors = True

Wow. That is way too easy. I recently implemented my own version of getopt for a class assignment and its a significant amount of code that is a pain to write. I assume that the getopt module is implemented in c which is another benefit of using python’s implementation.

So whats left?

Next Steps

As I said above in a number of places I am planning on continuing the development of FlickrFortune. Here are the following changes which will be implemented:

Make the code smarter, lighter and more optimized. Specifically the create wallpaper function.
Create an interface for setting the desktop background so that it can be easily implemented on other systems.
A smarter searching algorithm which will choose a different tag from the fortune if the first one did not return a usable image (or any image).
Allow the user to specify a file of fortunes and create a wallpaper for each one
Give the option to generate wallpapers in multiple sizes