I knew that it had some sort of encoder, but I was surprised how basic it was. The encoder consisted of a single photodiode and a toothed wheel... I presume that the engineers must have used additional information from the motor driver (current direction?) to resolve the direction of the rotation. I connected the diode to a 220 ohm resistor, 5 V power source, and reverse biased the photodiode with a 560 KOhm (also 5 V source). I then ran this through a comparator (LM311) against a 2.5V reference (I just used two 120 KOhm resistors in a voltage divider) to digitize the output.
With an Arduino, I wrote a simple sketch (posted after the break) to take the input from the comparator and increment a counter. I tested that turning the wheel increments the encoder. I ran the signal to an interrupt pin as well as a digital input, so that I didn't have to keep polling the pin. Should allow me to add extra inputs from my other wheel to the system easily I hope.
I don't want to drive the motor from my Arduino so I'm going to have to figure out a second power source and motor driver (perhaps a shield might be easiest) before going too much further with this.
Also below the break I posted a few more photos of the Roomba disassembly.
