Multimodal User Interfaces (eBook)

From Signals to Interaction

Dimitros Tzovaras (Herausgeber)

eBook Download: PDF

2008 | 2008
X, 315 Seiten
Springer Berlin (Verlag)
978-3-540-78345-9 (ISBN)

Infos

tionship indicates how multimodal medical image processing can be unified to a large extent, e. g. multi-channel segmentation and image registration, and extend information theoretic registration to other features than image intensities. The framework is not at all restricted to medical images though and this is illustrated by applying it to multimedia sequences as well. In Chapter 4, the main results from the developments in plastic UIs and mul- modal UIs are brought together using a theoretic and conceptual perspective as a unifying approach. It is aimed at defining models useful to support UI plasticity by relying on multimodality, at introducing and discussing basic principles that can drive the development of such UIs, and at describing some techniques as proof-of-concept of the aforementioned models and principles. In Chapter 4, the authors introduce running examples that serve as illustration throughout the d- cussion of the use of multimodality to support plasticity.

Inhalt

Content 6
Introduction 12
Multimodality Theory 16
2.1 Introduction 16
2.2 What is a Multimodal System? 16
2.2.1 An Enigma 16
2.2.2 A Solution 17
2.3 Which Modalities are There? 21
2.3.1 Deriving a Taxonomy of Input/Output Modalities 21
2.3.2 Basic Concepts 23
2.3.3 Modality Taxonomy 26
2.3.4 Information Channels 30
2.3.5 Interaction Devices 31
2.3.6 Practical Uses of the Theory 31
2.4 Multimodal Information Representation 35
2.4.1 Advantages of Multimodality 35
2.4.2 Constructing Multimodality from Unimodal Modalities 36
2.4.3 Linear Modality Addition and Replacement 36
2.4.4 Non-linear Effects, Users, Design Detail, Purpose 39
Information-Theoretic Framework for Multimodal Signal Processing 42
3.1 Introduction 42
3.2 Some Information Theoretic Concepts 43
3.2.1 Stochastic Process and Error Probability 43
3.2.2 Fano’s Inequality and the Data Processing Inequality 44
3.2.3 Information Theoretic Feature Extraction. 46
3.3 From Error Probability to Multimodal Signal Processing 47
3.3.1 Multimodal Stochastic Processes 47
3.3.2 Objective Functions for Multimodal Signal Processing 50
3.4 Optimization 59
3.5 Results 61
3.5.1 Multimodal Medical Images 61
3.5.2 Speech-Video Sequences 66
Multimodality for Plastic User Interfaces: Models, Methods, and Principles 72
4.1 Introduction 72
4.2 Running Examples 73
4.2.1 The Assisted Neuro-surgery System 73
4.2.2 The Sedan-Bouillon Web Site 75
4.3 Modality and Multimodality 77
4.3.1 Definitions 77
4.3.2 The CARE Properties 78
4.4 The Problem Space of Plastic Multimodal UIs 81
4.4.1 Two Adaptation Means: UI Re-molding and UI Re- distribution 82
4.4.2 UI Components Granularity 83
4.4.3 State Recovery Granularity 83
4.4.4 UI Deployment 84
4.4.5 Coverage of the Context of Use 84
4.4.6 Coverage of Technological Spaces 85
4.4.7 Existence of a Meta-UI 85
4.4.8 UI Re-molding and Modalities 86
4.4.9 UI Re-molding and Levels of Abstraction 87
4.4.10 Summary 87
4.5 Domain of Plasticity of a User Interface 88
4.6 Three Principles for the Development of Plastic Multimodal UIs 92
4.6.1 Blurring the Distinction between Design-time and Run-time 92
4.6.2 Mixing Close and Open Adaptiveness 93
4.6.3 Keeping Humans in the Loop 93
Face and Speech Interaction 96
5.1 Face and Facial Feature Detection 96
5.1.1 Face Detection 97
5.1.2 Facial Feature Detection 108
5.2 Interaction 109
5.2.1 Multimodal Speaker Localization 110
5.2.2 Audio-Visual Speech Recognition 115
Recognition of Emotional States in Natural Human- Computer Interaction 130
6.1 Introduction 130
6.2 Fundamentals 132
6.2.1 Emotion Representation 132
6.2.2 Methodology Outline 134
6.2.3 Running Example 135
6.3 Feature Extraction 136
6.3.1 Visual Modality 136
6.3.2 Auditory Modality 149
6.4 Multimodal Expression Classification 151
6.4.1 The Elman Net 151
6.4.2 Classification 154
6.5 Experimental Results 157
6.5.1 The Case for Naturalistic Data 157
6.5.2 Statistical Results 158
6.5.3 Quantitative Comparative Study 160
6.5.4 Qualitative Comparative Study 162
Two SIMILAR Different Speech and Gestures Multimodal Interfaces 166
7.1 Introduction and State-of-the-art 166
7.2 ICANDO Multimodal Interface 171
7.2.1 Objectives 171
7.2.2 System’s Description 172
7.3 Experimental Results 179
7.4 MOWGLI Multimodal Interface 180
7.4.1 Objectives 181
7.4.2 System’s Description 181
Multimodal User Interfaces in Ubiquitous Environments 196
8.1 Introduction 196
8.2 Related Work 197
8.3 Migratory User Interfaces 199
8.4 The Dimensions of Migration 199
8.5 An Architecture for a Migration Platform 203
8.6 Example Applications 207
8.6.1 Domotic Scenario ( mobile vocal+ graphical-> graphical desktop)
8.6.2 Museum Scenario (pda-> digital tv)
Software Engineering for Multimodal Interactive Systems 212
9.1 Introduction 212
9.2 PAC-Amodeus: a Conceptual Architectural Solution 214
9.2.1 Concurrent Processing of Data 215
9.2.2 Data Fusion 215
9.3 Software Tools for Multimodality 218
9.3.1 Existing Tools 218
9.3.2 ICARE Platform 220
9.3.3 ICARE Conceptual Model 221
9.3.4 ICARE Graphical Editor 222
9.3.5 OpenInterface Platform 224
Gestural Interfaces for Hearing- Impaired Communication 230
10.1 Introduction 230
10.2 Modality Processing and Analysis 232
10.2.1 Preprocessing 233
10.2.2 Hand Shape 235
10.2.3 Hand Location 240
10.2.4 Hand Motion 241
10.2.5 Facial Movements 244
10.2.6 Lip Reading 245
10.2.7 Facial Expressions 246
10.3 Temporal Analysis 247
10.3.1 Sign Language 247
10.3.2 Cued Speech 248
10.4 Multimodal Fusion 248
10.4.1 Temporal Modelling 249
10.4.2 Heterogenic Multiplexing 251
10.5 Applications 257
10.5.1 Sign Language Tutoring Tool 257
10.5.2 Cued Speech Manual Gesture Interpreter 259
Modality Replacement Framework for Applications for the Disabled 262
11.1 Introduction 262
11.2 The Modality Replacement Concept 265
11.3 Cued Speech 266
11.4 Feature Extraction and Representation for the Cued Speech Language 268
11.4.1 Audio Feature Extraction 268
11.4.2 Lip Shape Feature Extraction 268
11.4.3 Gesture Feature Extraction 269
11.5 Coupled Hidden Markov Models 270
11.6 Modality Reliability 271
11.7 Modified Coupled Hidden Markov Model 272
11.7.1 Training 272
11.8 Evaluation of the Cued Speech Recognition System 273
11.9.1 Multimodal Collaborative Game 276
A medical Component-based Framework for Image Guided Surgery 282
12.1 Introduction 282
12.2 MedicalStudio Framework 283
12.2.1 Architecture 283
12.2.2 Framework Implementation 284
12.3 General Purpose Components 285
12.3.1 Multimodal Registration 286
12.3.2 Segmentation and 3D Reconstruction 287
12.4 Applications 288
12.4.1 3D Medical Assistant for Orthognatic Computer Surgery 288
12.4.2 ACROGuide 289
Multimodal Interfaces for Laparoscopic Training 292
13.1 Introduction 292
13.2 Functionality 293
13.3 Technical Issues 293
13.3.1 Simulator Architecture 294
13.3.2 Collision Detection and Handling 296
13.3.3 Design of Multimodal Interface Scenarios and Surgical Simulation Tasks 298
13.4 Research Issues 299
13.4.1 Study of Laparoscopic Sensory Interaction 299
13.4.2 Laparoscopic Simulation Conceptual Framework 300
References 302

Produkt-Details

• Erscheint lt. Verlag: 27.2.2008
• Reihe/Serie: Signals and Communication Technology
• Reihe/Serie: Signals and Communication Technology
• Zusatzinfo: X, 315 p. 238 illus., 72 illus. in color.
• Verlagsort: Berlin
• Sprache: englisch
• Themenwelt: Mathematik / Informatik ► Informatik ► Grafik / Design
• Themenwelt: Technik ► Elektrotechnik / Energietechnik
• Themenwelt: Technik ► Nachrichtentechnik
• Schlagworte: digital signal processing • Direct Brain Connection • Haptics • Haptik • Human-Computer interaction • Human-Computer Interaction (HCI) • Image Processing • Image Registration • Information • interaction • Interface • Multimedia • Segment • Segmentation • Signal Processing • Speech • user interface • User Interfaces • Vision
• ISBN-10: 3-540-78345-8 / 3540783458
• ISBN-13: 978-3-540-78345-9 / 9783540783459