Author Archives: carson

Parsing the SXSW Twitter Stream for Fun

Over the weekend I decided to toss together a simple twitter stream monitoring app that would capture SXSW tweets. I wanted to build on some of what I learned while hacking together the stuff for code2009 and it was also an excuse to play with node.js and a few other things. I figured I would put together a quick post with all the different parts and as I have time I'll pull together some of the more important sections into larger posts. Be warned that the site may stop functioning since it was only a few hours work and mostly put together with bailing wire and gum.

Server side:

  • node.js – Used to process the long polling requests
  • ngnix – Front for multiple nodejs instances and serving some static files
  • underscore.js
  • rabbitmq – Used to monitor the realtime feed
  • Redis – Used as a datastore for everything
  • ruby – Used to glue a bunch of things together, tons of gems used
  • Resque – Used to handle the screen captures

Browser side:

Hosting:

Using Daemon-Kit and RobustThread to Build Ruby Daemons

On a number of occasions I have found myself needing to assemble a daemon process for some type or processing done using Ruby. Each time I roll things a little different and I finally started to wonder if someone had already put together tools for doing the daemon parts. After some quick digging I ran into Daemon-Kit and after adding it together with a couple other tools it seems like what I've needed. I've put together a few recipes here to help guide others who might be looking for something similar.

Continue reading

Building HipHop PHP for Fedora 12 on 64 bit and 32 bit Systems

Now that Facebook has finally released the source for HipHop PHP it is time to give it a spin. Of course it is still a little rough around the edges so I figured I would toss together a quick howto on getting it to build.

The first thing to note is that they are only supporting 64 bit systems officially. Having said that it isn't too hard to modify the code to make it work on a 32 bit system although it may turn out that such early modifications are missing some fundamental bits on why they were only support 64 bit systems. I'm going to assume at first that you are using a 64 bit system and then end with what you need if you are still using a 32 bit system.

Continue reading

Fedora 12 Bootable Root EBS on EC2

I recently needed to create a clean EC2 AMI using a fairly new linux distro. It has been a while since I've needed to create a new AMI so I also wanted to move away from the older pre-packaged AMI and boot using EBS. After taking a look at what was currently available publicly I decided I would just create my own EBS bootable AMI using Fedora 12. It wasn't all that complicated but there are a decent number of steps so I figured I would document them for anyone else who might want to give it a try.

I'm going to assume you already understand how to do things like create instances, create EBS volumes and ssh into your running instance using key based authentication. I use the AWS management console for a lot of what follows with the exception of needing to register the AMI and for that you will need the Amazon EC2 API Tools and Amazon EC2 AMI Tools

Continue reading

PHP MySQLi and Multiple Prepared Statements

While sprucing up the PHP code I use to provide my own Stack Overflow API for GeeQe I ran into an error caused by trying to use multiple prepared statements with MySQLi. It turned up when I tried to execute one prepared statement while looping over the result set from another prepared statement that were both created on the same connection. What came out was the following error:

"Commands out of sync; you can't run this command now"

Details about this error can be found in the mysql docs. Reading those details makes it clear that the result sets of a prepared statement execution need to be fetched completely before executing another prepared statement on the same connection.

Continue reading

Developing Adobe Air Apps with Linux

I finally found a little project I wanted to do using Adobe Air and after some searching I found out you can use Linux to develop Air applications. At first I thought I would have to use Flex Builder which is still in alpha for Linux but it turns out there is a better option from Aptana.

The Aptana Air plugin supports developing Adobe Air applications using HTML and Javascript. It even support the 2.0 release of Air that is currently in beta. Aptana uses the Eclipse framework as an editor so if you are familure with Eclipse it will be even easier to use.

I started by downloading and installing the latest version of the Air runtime. Next I grabbed the Air SDK, the SDK doesn't come with the plugin so it is something you have to get directly from the Air developers site. After getting the SDK unpacked I installed the latest Aptana core release. Once the core is installed there is a big plugin button on the startup screen that currently has Air listed.

The install went smoothly except for a few issues. The first one I ran into was very noticeable since it kept any dialog buttons from working when they were clicked although they did work when I clicked them and then hit enter or navigated to them with the keyboard. Luckily someone has already figured out that there is an issue with Eclipse and GTK+ that is the cause (even though the post is for Ubuntu the same problem and solution worked for me on Fedora). The fix is to set the GDK_NATIVE_WINDOWS variable before running the Aptana binary:

GDK_NATIVE_WINDOWS=true; AptanaStudio

The next thing I noticed was the application.xml descriptor that Aptana created didn't generate correctly. It needs to start with the correct xmlns or the following error will be thrown on run: "invalid application descriptor: descriptor version does not match runtime version". To fix this check the version of the Air SDK by running the following command:

./adt -version
adt version "1.5.3.9120"

For the version of the Air SDK I downloaded the correct xmlns was http://ns.adobe.com/air/application/1.5 so I needed the following application tag:

<application xmlns="http://ns.adobe.com/air/application/1.5">

Once I had that working I was able to compile and execute a demo application. I was also able to create an Air application package from within Aptana using File > Export > Adobe AIR > Adobe AIR Package. Before creating the Air package I had to create a signing certificate. Creating the certificate can be done within Aptana too but because I had not yet fixed the above button issue I created a cert on the command line with the Air SDK and then imported it. To create the Air signing certificate from the command line I used the adt command from the SDK:

adt -certificate -cn SelfSigned 1024-RSA sampleCert.pfx samplePassword

Remember the password that gets used to generate the certificate because it will have to be used before a package is signed.

Finally Adobe has a lot of information on developing Air applications on their Air devnet site. The Air ajax section is especially important.

Heroku Tips for the Cheap

I've been playing around with the Ruby/Rails cloud provider Heroku a little bit lately just to try it out. It is somewhat like Google App Engine or Microsoft Azure in the way it works since you bundle your application and push it out to the Heroku cloud for deployment. It is very easy to get things going but I ran into a few interesting items that I figured I would share.

Continue reading

Building Chromium and Chrome OS with EC2

When the initial cut of the Chromium OS source was released last week I decided to use the opportunity to see if it would run on my EEE PC 900 netbook (check out EEE PC 900 running Chrome OS on Youtube to see the final result). The first roadblock I hit with the build instructions was the Ubuntu requirement (I did give a little effort to getting it working on Fedora first). I don't have an Ubuntu box so I started out trying to use VirtualBox but that was going to take forever so I decided to move things to EC2 and what follows is the result. This isn't meant to be a replacement for the build docs since they are surely going to change, it is more of a cookbook to build Chromium (the browser) and Chromium OS using EC2 (EBS is used as well if you want to cache the source over time).

When I first started down the path of using EC2 I thought I would grab the source each time I wanted to build. I quickly ran into a snag however because it took forever to sync the source and download the Ubuntu repo. Once I had the initial sync of the source I decided I would copy it all to an EBS volume and keep that volume up to date. Using EBS to store the source feels better too since I assume Google expects people to be syncing changes only as opposed to pulling the entire source tree down every time they want to build.

I started out by finding this Ubuntu AMI for a base to work from. For the most efficient compile times I ended up using the High CPU (c1.medium) instance. I started with the default small instance but it was just too slow. With the high cpu instance you are looking at about 45 minutes to build the OS after you have the source synced for the first time and if you add building Chromium in there you are looking at around 55 additional minutes. All told you can have a complete build in less than 2 hours even if there are some source updates needed. For EBS you need a 3G volume for the Chrome OS source plus Ubuntu package repo and a 4G volume for the Chromium source.

Continue reading

Upgrade to Fedora 12 from Fedora 11

Fedora 12 was just released and it is time to upgrade again of course. I almost thought this was going to be a version to yawn at but then I saw that there was going to be a new version of Fedora based on Moblin and it seemed exciting again. Of course that isn't the only thing being upgraded in the latest version of Fedora. Some of the more notable changes in this version:

You can find the complete list of Fedora 12 enhancements as well if you want more details.

Continue reading

Full Text Search with Sphinx

While developing my GeeQE iPhone application I decided I needed a way to let users search posts so I started looking around for a simple search engine that I could use with PHP. I took a look at a number of different options like MySQL Full Text search, Sphinx, Solr and others based on Lucene. After looking at what it would take to get started with each I decided to go with Sphinx. Sphinx looked like it would be the easiest and quickest to set up, didn't require a lot of resources to run in an idle state and would integrate with PHP easily.

This post goes over how I went about configuring Sphinx and gives an example of how to integrate it with PHP. I'm using MySQL as the data store filled with the Stack Overflow CC data dump although it should be easy to adapt the instructions to other data sources. To follow along just download a copy of the data dump and use my schema and loader to get the same MySQL database.

Continue reading