| The science of everyday
living requires data on human activities collected over extended periods
of time. This is especially true for the research conducted in the Perception
and Awareness Technology (PAT) thrust area; its learning methods rely on
the availability of rich and often ground-truthed training data. The Grand
Challenge Data Collection project addresses this need. In the initial phase
of the project, we collected data in a controlled environment allowing
for complete ground-truthing of the data and integration of multiple sensor
modalities. We collected data by observing a human subject in a kitchen
conducting common activities, such as preparing meals. This scenario has
the advantage of being well-defined and confined to a relatively small
space, and still being matched with the QoLT objectives of helping in instrumental
activities of daily living, e.g., Active Home, PerMMA and Virtual Coach.
We built a mock-up kitchen
in the lab, and data from several sensors and cameras were recorded while
a subject was going over four different "recipes" scenarios. A motion capture
system was used to record the subject's motion by using 32 infrared cameras;
five motion sensors (each with accelerometers and gyroscopes) were also
attached to the subject’s body. A visual record was also obtained from
six high-resolution cameras covering the entire workspace. In addition,
a wearable camera was added to collect vision data from the user's perspective,
simulating the inside-out vision system. A significant technical challenge
was to properly synchronize and time-tag the outputs of all the sensors
to integrate them in the final data format. Data from the eWatch sensors
was simultaneously collected in order to compare accuracy between different
sensor modalities.
The resulting combined data
over four different cooking scenarios amount to a total of over 1 terabyte.
Our plan is to make the data available, not just within the QoLT ERC, but
also to the worldwide research community. Within QoLT, we plan to use the
data initially to develop more general versions of the learning and prediction
techniques.
The strong impact and benefits
of such data sharing for advancement of science and engineering have been
proven. In the robotics and computer vision areas, CMU has experience of
having developed such databases, including the Motioncap Database of human
actions and the PIE database for face recognition. Furthermore, we plan
to publish the data collection protocol, so that other research groups
can collect similar data sets that are sharable and usable in the same
manner as our Grand Challenge data. The Digital Human Research Center in
Japan will be the first user of the protocol to contribute to the data
collection.
|
