Kura's Mask-bot information

Welcome to Dr. Takaaki Kuratate's (Kura's) TUM webpage!

(Still Under Construction...)

Kura's Mask-bot information (Japeanese Version)

CFP: Special Issue on Face-to-Face Communication, IJHR

Please check details at IJHR Special Issue Information

Team Members (Ver.1): Dr. Takaaki Kuratate, Brennand Pierce and Prof. Dr. Gordon Cheng

"Mask-bot" is our new generation of talking head. Unlike usual computer graphics facial animation on a flat computer screen, Mask-bot will give you more realistic 3D existence of synthetic face. Even though, it's basic concept is very classic - like "singing busts" at Haunted Mansion, Disneyland, or "talking head projection" from MIT in 1980. Also there are many similar rear-projected robotic-face have been conducted by various researchers. So, what is new about this "Mask-bot"?

Mask-bot: what is so unique?
Some researchers already published some systems using a rear-projected screen on robotic platform. But from following points, our system is very unique.
- Calibrated realistic facial animation on realistic 3D face surface:
  
  If you project video or film image of actor's face onto 3D screen like a mask or a statue, it may looks fine at certain view (usually exact font of the face). However, if you look such face from side, you may notice image distortion because of the mismatch between projected video image and 3D surface. This kind of distortion become quite visible and affects output face image quality when you use rear-projection system via wide/fisheye converter with macro adapter to to project wider angle within short projection distance. However, all related research (as I know) before us, did not properly challenge this problem - or just tried a simple cartoon-like face and didn't go further.
  So, we did it! - After our first prototype calibration was done in April, 2011 - and demonstrated at our internal workshop, then immediately submitted AVSP2011 - and accepted.
- Carrying "visual speech" information, based on Auditory-Visual speech research
  Our talking head animation system was originally came from Auditory-Visual speech processing (AVSP) research, started in 1997 at ATR, Kyoto, Japan (directed by Eric Vatikiotis-Bateson, now he is a professor at UBC, Vancouver, Canada), then continued at MARCS Auditory laboratories in Australia from June 2006 (directed by Prof. Denis Burnham), and now at ICS/TUM since January 2010.
- Various face models available, based on 3D face analysis and synthesis research
  
  I'm working on 3D face models and built 3D face databases with multiple postures (~500 subjects at ATR: 1998-2006, ~200 subjects at MARCS: 2006-2009), we can access various type of faces - original subjects faces (who agreed to be shown on publications and media) or synthesized 3D faces from photographs using these 3D face databases. These 3D faces can be easily transferred to Mask-bot system.
- Using rear-projector paint on a transparent 3D mask
  
  This method works extremely well. We tested various materials like semi-transparent plastics, thin papers, see-thorough cloths, etc... Most Materials are either too dark with rear-projection, or too thin and cannot see the image clearly, or too difficult to make a 3D face shape. Then I found out about 3M's rear projector film which can transform any glass window to a rear-projection screen. So, we though about using this film to put transparent 3D mask somehow, or put on a transparent plastic sheet and vacuum-form to 3D mask.
  
  But finally, I found about a special paint which can spray to a transparent surface to use as a rear-projection screen. So, we tried it and it works great! Very simple solution - took a while to get this answer...
Mask-bot: how do you make it?
Well, hardware part is very simple - but configuration is important.
- LED projector
  
  We use a 200 ANSI lumens model, which we could obtain easily when we started building it. As a start, Our selection criterias were followings:
  - not so big, not so heavy (< 1kg)
  - not so dark (150-300 ANSI lumens)
  - enough contrast (1000:1 - 2000:1)
  We wanted to try 300 ANSI lumens model first, just in case, but from our bureaucracy procedure reasons, we decided Acer K11 (200 ANSI lumens, 2000:1 contrast, 0.61kg).
- Fisheye lens and macro adapter
- 3D mask / mannequin head - but get transparent one!
- paint for rear projection home theater (and spray-gun or air brush, and good skill to spray)
- metal frame (or any solid frame) to support projector, fisheye lens and 3D mask
- If you control head motion, pan-tilt (and yaw or more) control unit
- PC or Mac with MATLAB or similar software, C compiler
- and software: to build a distortion model, calibrate 3D face models, animation program, 3D face data, etc...

Mask-bot news

Please don't forget to check our YouTube channel!!

29-04-2011: Internal demo: Mask-bot was demonstrated at CoTeSys spring workshop
11-06-2011: First paper describes Mask-bot display system was accepted at AVSP2011!
28-06-2011: Mask-bot system was invited to a demo session at AVSP2011!
22-07-2011: Mask-bot - first public appearance at CoTeSys Open House
18-07-2011: Our paper about the demo system using OpenHRI-base speech interface and Mask-bot system was accepted at Humanoids 2011 poster session!!
02-09-2011: First academic presentation / live-demo of Mask-bot at AVSP2011, Volterra, Italy.
27-10-2011: Poster session and live-demo (no pan-tilt unit) of Mask-bot at Humanoids 2011, Bled, Solovenia.
07-11-2011: Press release from TUM
22,23-02-2012: Poster session and live-demo (no pan-tilt unit) of Mask-bot at CogSys 2012, Vienna, Austria.

Mask-bot web links

Mask-bot publications

Brennand Pierce, Takaaki Kuratate, Christian Vogl, and Gordon Cheng, "mask-bot 2i": An active customisable robotic head with interchangeable face, Proceedings of the 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012), Osaka, Japan, Nov.29-Dec.1, 2012.
Akinobu Maejima, Takaaki Kuratate, Brennand Pierce, Shigeo Morishima, and Gordon Cheng, Automatic face replacement for a humanoid robot with 3D face shape display, Proceedings of the 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012), Osaka, Japan, Nov.29-Dec.1, 2012.
Takaaki Kuratate, Marcia Riley, Brennand Pierce, and Gordon Cheng, Gender identification bias induced with texture images on a life size retro-projected face screen, Proceedings of the 21st IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2012), Paris, France, Sep 9-13, 2012.
Akinobu Maejima, Takaaki Kuratate, Brennand Pierce, Shigeo Morishima, and Gordon Cheng, Automatic texture generation for face-shaped display from a frontal face image (in Japanese), will be published at Visual Computing / Graphics & CAD Joint Symposium, Tokyo, Japan, June 23-24, 2012.
Takaaki Kuratate, Brennand Pierce, Christian Vogl, Akinobu Maejima, Yosuke Matsusaka, and Gordon Cheng, Maskbot as a communication interface, Proceedings of the 5th International Conference on Cognitive Systems (CogSys2012), poster No.50, 2012.
T. Kuratate, B. Pierce and G. Cheng, ``Mask-bot'' - a life-size talking head animated robot for AV speech and human-robot communication research, Proc. of the international conference on Auditory Visual Speech Processing (AVSP2011), pp.111-116, Volterra, Italy, 2011.
T. Kuratate, Y. Matsusaka, B. Pierce and G. Cheng, ``Mask-bot'': a life-size robot head using talking head animation for human-robot communication, Proceedings of the 11th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2011), pp.99-104, Bled, Slovenia, 2011.