MS SenseCam

Back in December 05, I mentioned that I was going to get one of Microsoft Research's SenseCams for an experience capture project. Microsoft delayed the release of the hardware, but we got hold of them in late June. I've been wearing it lately to see what it does.

Basic gist. The camera has sensors onboard that trigger image collection. Motion, temperature, light, and objects coming into view are the core sensors. Makes sense...somebody steps into the view of the IR sensor, take a picture of 'em. Big temperature change might mean leaving my air conditioned van ("Big Pimp'n") and entering the heat of State College, PA. If I happen to run like hell in the heat to get to the next air conditioner...well, record that too. In theory, this should lead to the camera catching interesting life events.

The goal of our project is to see whether the camera could do what we've done in past research automatically. We used to have people take pictures of food and exercise, coordinate these with blood glucose readings. Now we want to see if the imaging can be done on the fly. The problem is that the camera can take an enormous amount of images. For instance, the video below was created from 104 pictures taken in 17 minutes (990 pictures were collected during the full three hours of...well, I'll explain that in bit). And, unfortunately, there's no way to know if food is in these images. That is, last I checked there were no ways to extract such information from images.

Luckily, the camera gives access to its sensor data. And with some connections to other sources (like a calendar...those are good places for inferring things like "lunch"), we might be able to point to rough locations of relevant events. The other thing we're investigating is combining the camera with a continuous glucose monitoring system. Cause we really want to pay attention to big sweeps in glucose levels, be able to understand what behaviors may have led to 'em. More soon on that...we hope to have a CGMS next week.

Meanwhile, the video below is me eating dinner at Shannon's 20 year high school reunion. When participants in past studies captured images manually, we weren't always able to see how much of a meal was consumed (unless we asked them to take pics at beginning and end of eating, as we did in several studies). But the SenseCam shows it all, step by step...including the consumption of a certain "Lite" beverage. Most of these seem to have been collected by triggering the IR sensor...hands moving all over the place as I cut and move food to my mouth. Whether this is useful for nutritionists remains to be seen.

The other funny note is that I wore the SenseCam during the reunion without mentioning what it was or what it was doing. Luckily, Shannon is used to my strange experiments, so she went along with it. By end of night someone asked her if it was a camera, and she told 'em it was. But even she didn't say it was recording most of the evening automatically. Like making my own Candid Camera show...with a slight dork factor.