ImageCLEF (eBook)

Experimental Evaluation in Visual Information Retrieval

Henning Müller, Paul Clough, Thomas Deselaers, Barbara Caputo (Herausgeber)

eBook Download: PDF

2010 | 2010
XXVIII, 544 Seiten
Springer Berlin (Verlag)
978-3-642-15181-1 (ISBN)

Infos

The pervasive creation and consumption of content, especially visual content, is ingrained into our modern world. We're constantly consuming visual media content, in printed form and in digital form, in work and in leisure pursuits. Like our cave- man forefathers, we use pictures to record things which are of importance to us as memory cues for the future, but nowadays we also use pictures and images to document processes; we use them in engineering, in art, in science, in medicine, in entertainment and we also use images in advertising. Moreover, when images are in digital format, either scanned from an analogue format or more often than not born digital, we can use the power of our computing and networking to exploit images to great effect. Most of the technical problems associated with creating, compressing, storing, transmitting, rendering and protecting image data are already solved. We use - cepted standards and have tremendous infrastructure and the only outstanding ch- lenges, apart from managing the scale issues associated with growth, are to do with locating images. That involves analysing them to determine their content, clas- fying them into related groupings, and searching for images. To overcome these challenges we currently rely on image metadata, the description of the images, - ther captured automatically at creation time or manually added afterwards.

Inhalt

Foreword 6
Preface 8
Acknowledgements 10
Contents 12
List of Contributors 24
Introduction 28
Seven Years of Image Retrieval Evaluation 30
Introduction 30
Evaluation of IR Systems 32
IR Test Collections 33
Cross--Language Evaluation Forum (CLEF) 36
ImageCLEF 36
Aim and Objectives 36
Tasks and Participants 38
Data sets 39
Contributions 39
Organisational Challenges 41
Conclusions 42
References 43
Data Sets Created in ImageCLEF 46
Introduction 46
Collection Creation 47
Requirements and Specification 48
Collection Overview 50
Image Collections for Photographic Retrieval 51
The St. Andrews Collection of Historic Photographs 51
The IAPR TC--12 Database 53
The Belga News Agency Photographic Collection 55
Image Collections for Medical Retrieval 56
The ImageCLEFmed Teaching Files 57
The RSNA Database 61
Automatic Image Annotation and Object Recognition 62
The IRMA Database 62
The LookThatUp (LTU) Data set 63
The PASCAL Object Recognition Database 64
The MIR Flickr Image Data Set 65
Image Collections in Other Tasks 65
The INEX MM Wikipedia Collection 66
The KTH--IDOL2 Database 67
Conclusions 68
References 69
Creating Realistic Topics for Image Retrieval Evaluation 71
Introduction 71
User Models and Information Sources 74
Machine--Oriented Evaluation 74
User Models 75
Information Sources for Topic Creation 76
Concrete Examples for Generated Visual Topics in Several Domains 79
Photographic Retrieval 79
Medical Retrieval 80
The Influence of Topics on the Results of Evaluation 81
Classifying Topics Into Categories 82
Links Between Topics and the Relevance Judgments 83
What Can Be Evaluated and What Can Not? 83
Conclusions 84
References 85
Relevance Judgments for Image Retrieval Evaluation 88
Introduction 88
Overview of Relevance Judgments in Information Retrieval 89
Test Collections 89
Relevance Judgments 90
Relevance Judging for the ImageCLEF Medical Retrieval Task 97
Topics and Collection 97
Judges 98
Relevance Judgment Systems and the Process of Judging 99
Conclusions and Future Work 103
References 104
Performance Measures Used in Image Information Retrieval 106
Evaluation Measures Used in ImageCLEF 106
Measures for Retrieval 107
Measuring at Fixed Recall 108
Measuring at Fixed Rank 110
Measures for Diversity 112
Collating Two Measures Into One 113
Miscellaneous Measures 113
Considering Multiple Measures 114
Measures for Image Annotation and Concept Detection 115
Use of Measures in ImageCLEF 116
Conclusions 117
References 117
Fusion Techniques for Combining Textual and Visual Information Retrieval 120
Introduction 120
Information Fusion and Orthogonality 122
Methods 123
Results 123
Early Fusion Approaches 123
Late Fusion Approaches 124
Inter--media Feedback with Query Expansion 129
Other Approaches 130
Overview of the Methods from 2004--2009 130
Justification for the Approaches and Generally Known Problems 130
Conclusions 133
References 133
Track Reports 140
Interactive Image Retrieval 142
Interactive Studies in Information Retrieval 142
iCLEF Experiments on Interactive Image Retrieval 144
iCLEF Image Retrieval Experiments: The Latin Square Phase 145
iCLEF Experiments with Flickr 148
The Target Collection: Flickr 149
Annotations 149
The Task 150
Experiments 152
Task Space, Technology and Research Questions 159
Use Cases for Interactive Image Retrieval 159
Challenges: Technology and Interaction 160
References 162
Photographic Image Retrieval 165
Introduction 165
Ad hoc Retrieval of Historic Photographs: ImageCLEF 2003--2005 166
Test Collection and Distribution 167
Query Topics 168
Relevance Judgments and Performance Measures 171
Results and Analysis 171
Ad hoc Retrieval of Generic Photographs: ImageCLEFphoto 2006-2007 173
Test Collection and Distribution 174
Query Topics 175
Relevance Judgments and Performance Measures 176
Results and Analysis 177
Visual Sub--task 178
Ad hoc Retrieval and Result Diversity: ImageCLEFphoto 2008--2009 179
Test Collection and Distribution 179
Query Topics 180
Relevance Judgments and Performance Measures 182
Results and Analysis 182
Conclusion and Future Prospects 184
References 185
The Wikipedia Image Retrieval Task 187
Introduction 187
Task Overview 188
Evaluation Objectives 188
Wikipedia Image Collection 189
Additional Resources 189
Topics 190
Relevance Assessments 191
Evaluation 193
Participants 193
Approaches 194
Results 199
Discussion 203
Best Practices 203
Open Issues 204
Conclusions and the Future of the Task 205
References 205
The Robot Vision Task 208
Introduction 208
The Robot Vision Task at ImageCLEF 2009: Objectives and Overview 210
The Robot Vision Task 2009 211
Robot Vision 2009: The Database 211
Robot Vision 2009: Performance Evaluation 212
Robot Vision 2009: Approaches and Results 215
Moving Forward: Robot Vision in 2010 217
The Robot Vision Task at ICPR2010 217
The Robot Vision Task at ImageCLEF2010 219
Conclusions 220
References 220
Object and Concept Recognition for Image Retrieval 222
Introduction 222
History of the ImageCLEF Object and Concept Recognition Tasks 223
2006: Object Annotation Task 224
2007: Object Retrieval Task 225
2008: Visual Concept Detection Task 226
2009: Visual Concept Detection Task 227
Approaches to Object Recognition 227
Descriptors 229
Feature Post--processing and Codebook Generation 230
Classifier 230
Post--Processing 231
Results 231
2006: Object Annotation Task 232
2007: Object Retrieval Task 232
2008: Visual Concept Detection Task 233
2009: Visual Concept Detection Task 234
Evolution of Concept Detection Performance 236
Discussion 237
Combinations with the Photo Retrieval Task 238
Conclusion 238
References 239
The Medical Image Classification Task 243
Introduction 243
History of ImageCLEF Medical Annotation 244
The Aim of the Challenge 244
The Database 245
Error Evaluation 249
Approaches to Medical Image Annotation 251
Image Representation 252
Classification Methods 252
Hierarchy 253
Unbalanced Class Distribution 253
Results 253
Conclusion 257
References 259
The Medical Image Retrieval Task 261
Introduction 261
Participation in the Medical Retrieval Task 262
Development of Databases and Tasks over the Years 264
2004 264
2005--2007 265
2008--2009 269
Evolution of Techniques Used by the Participants 271
Visual Retrieval 272
Textual Retrieval 272
Combining Visual and Textual Retrieval 273
Case--Based Retrieval Topics 273
Results 273
Visual Retrieval 274
Textual Retrieval 274
Mixed Retrieval 275
Relevance Feedback and Manual Query Reformulation 275
Main Lessons Learned 275
Conclusions 277
References 277
Participant reports 280
Expansion and Re--ranking Approaches for Multimodal Image Retrieval using Text--based Methods 282
Introduction 283
Integrated Retrieval Model 284
Handling Multi--modality in the Vector Space Model 285
Document and Query Expansion 286
Re--ranking 288
Level 1: Narrowing-down and Re-indexing 290
Level 2: Cover Coefficient Based Re--ranking 290
Results 292
Conclusions 294
References 295
Revisiting Sub--topic Retrieval in the ImageCLEF 2009 Photo Retrieval Task 297
Introduction 298
Background and Related Work 300
Sub--topic Retrieval 300
The Probability Ranking Principle 302
Beyond Independent Relevance 302
Document Clustering and Inter--Cluster Document Selection 304
Re--examining Document Clustering Techniques 304
Clustering for Sub--topic Retrieval 305
Empirical Study 307
Results 310
Conclusions 311
References 313
Knowledge Integration using Textual Information for Improving ImageCLEF Collections 315
Introduction 315
System Description 317
Photo Retrieval System 317
Medical Retrieval System 318
Photo Task 318
Using Several IR and a Voting System 321
Filtering 322
Clustering 325
The Medical Task 326
Metadata Selection using Information Gain 326
Expanding with Ontologies 328
Fusion of Visual and Textual Lists 331
Conclusion and Further Work 331
References 333
Leveraging Image, Text and Cross--media Similarities for Diversity--focused Multimedia Retrieval 334
Introduction 334
Content--Based Image Retrieval 336
Fisher Vector Representation of Images 337
Image Retrieval at ImageCLEF Photo 339
Text Representation and Retrieval 340
Language Models 340
Text Enrichment at ImageCLEF Photo 341
Text--Image Information Fusion 345
Cross--Media Similarities 346
Cross--Media Retrieval at ImageCLEF Photo 348
Diversity--focused Multimedia Retrieval 351
Re--ranking Top--Listed Documents to Promote Diversity 352
Diversity--focused Retrieval at ImageCLEF Photo 355
Conclusion 358
References 359
University of Amsterdam at the Visual Concept Detection and Annotation Tasks 362
Introduction 362
Concept Detection Pipeline 363
Point Sampling Strategy 364
Color Descriptor Extraction 365
Bag--of--Words model 366
Machine Learning 367
Experiments 368
Spatial Pyramid Levels 368
Point Sampling Strategies and Color Descriptors 369
Combinations of Sampling Strategies and Descriptors 370
Discussion 372
ImageCLEF 2009 372
Evaluation Per Image 374
Conclusion 374
ImageCLEF@ICPR 2010 375
Conclusion 375
References 376
Intermedia Conceptual Indexing 378
Introduction 378
Conceptual Indexing 380
Concept Usage and Definition in IR 380
Concept Mapping to Text 381
Mapping Steps 382
IR Models Using Concepts 385
Experiments using the ImageCLEF Collection 386
Image Indexing using a Visual Ontology 388
Image Indexing Based on VisMed Terms 389
FlexiTile Matching 392
Medical Image Retrieval Using VisMed Terms 393
Spatial Visual Queries 394
Multimedia and Intermedia Indexing 395
Conclusions 397
References 398
Conceptual Indexing Contribution to ImageCLEF Medical Retrieval Tasks 400
Introduction 401
Semantic Indexing Using Ontologies 401
Conceptual Indexing 402
Language Models for Concepts 402
Concept Detection 403
Concept Evaluation Using ImageCLEFmed 2005--07 404
From Concepts to Graphs 405
A Language Model for Graphs 405
Graph Detection 406
Graph Results on ImageCLEFmed 2005--07 407
Mixing Concept Sources 407
Query Fusion 408
Document Model Fusion 408
Joint Decomposition 409
Results on ImageCLEFmed 2005--07 411
Adding Pseudo--Feedback 412
Pseudo--Relevance Feedback Model 412
Results 413
Conclusions 414
References 414
Improving Early Precision in the ImageCLEF Medical Retrieval Task 416
Introduction 416
What is Early Precision? 417
Why Improve Early Precision? 418
ImageCLEF 418
Our System 419
User Interface 419
Image Database 420
Query Parsing and Indexing 421
Improving Precision 422
Modality Filtration 422
Using Modality Information for Retrieval 425
Using Interactive Retrieval 427
Conclusions 430
References 431
Lung Nodule Detection 433
Introduction 433
Lung Cancer --- Clinical Motivation 434
Computer--Aided Detection of Lung Nodules 436
Ground Truth for Lesions 437
Review of Existing Techniques 438
Gray--Level Threshold 439
Template Matching 439
Spherical Enhancing Filters 440
Description of Siemens LungCAD System 441
Lung Segmentation 441
Candidate Generation 441
Feature Extraction 442
Classification 443
Multiple Instance Learning 443
Exploiting Domain Knowledge in Data--Driven Training--Gated Classifiers 444
Ground Truth Creation: Learning from Multiple Experts 445
ImageCLEF Challenge 446
Materials and Methods 446
Results 447
Discussion and Conclusions 448
Clinical Impact 448
Future Extensions of CAD 450
References 451
Medical Image Classification at Tel Aviv and Bar Ilan Universities 453
Introduction 453
Visual Words in Medical Archives 454
The Proposed TAU--BIU Classification System Based on a Dictionary of Visual--Words 455
Patch Extraction 456
Feature Space Description 456
Quantization 457
From an Input Image to a Representative Histogram 458
Classification 459
Experiments and Results 460
Sensitivity Analysis 462
Optimizing the Classifier 464
Classification Results 467
Discussion 468
References 469
Idiap on Medical Image Classification 470
Introduction 470
Multiple Cues for Image Annotation 471
High--Level Integration 472
Mid--Level Integration 473
Low--Level Integration 473
Exploiting the Hierarchical Structure of Data: Confidence Based Opinion Fusion 474
Facing the Class Imbalance Problem: Virtual Examples 475
Experiments 475
Features 475
Classifier 478
Experimental Set--up and Results 479
Conclusions 480
References 481
External views 483
Press Association Images --- Image Retrieval Challenges 485
Press Association Images --- A Brief History 485
The Press Association 485
Images at the Press Association 487
User Search Behaviour 488
Types of Users 488
Types of Search 489
Challenges 490
Semantic Web for Multimedia Applications 491
Introduction to the Semantic Web 491
Success Stories and Research Areas 491
The Semantic Web Project at Press Association Images 493
Utilizing Semantic Web Technologies for Improving User Experience in Image Browsing 494
PA Data set: Linking to the Linked Data Cloud 494
Information Extraction and Semantic Annotation 496
Conclusions and Future Work 497
References 497
Image Retrieval in a Commercial Setting 499
Introduction 499
Evaluating Large Scale Image Search Systems 502
Query Logs and Click Data 503
Background Information on Image Search 506
Multilayer Perceptron 507
Click Data 509
Data Representation 511
Textual Features 511
Visual Features 513
Evaluation and Results 514
Analysis of Features 516
Discussion of Results 518
Looking Ahead 519
References 520
An Overview of Evaluation Campaigns in Multimedia Retrieval 522
Introduction 522
ImageCLEF in Multimedia IR (MIR) 524
INEX XML Multimedia Track 525
MIREX 526
GeoCLEF 526
TRECVid 527
VideOlympics 529
PASCAL Visual Object Classes (VOC) Challenge 529
MediaEval and VideoCLEF 530
Past Benchmarking Evaluation Campaigns 531
Comparison with ImageCLEF 532
Utility of Evaluation Conferences 533
Impact and Evolution of Metrics 534
Conclusions 536
References 537
Glossary 541
Index 546

Produkt-Details

• Erscheint lt. Verlag: 20.8.2010
• Reihe/Serie: The Information Retrieval Series
• Reihe/Serie: The Information Retrieval Series
• Zusatzinfo: XXVIII, 544 p. 138 illus., 16 illus. in color.
• Verlagsort: Berlin
• Sprache: englisch
• Themenwelt: Mathematik / Informatik ► Informatik ► Grafik / Design
• Schlagworte: Annotation • classification • Image Retrieval • Information Retrieval • media retrieval • Medical Image Processing • Multimedia Retrieval • Performance • Performance Evaluation • Robot vision • text retrieval
• ISBN-10: 3-642-15181-7 / 3642151817
• ISBN-13: 978-3-642-15181-1 / 9783642151811