Handbook of Big Data Technologies (eBook)
XIII, 895 Seiten
Springer International Publishing (Verlag)
978-3-319-49340-4 (ISBN)
Albert Zomaya is the Chair Professor of High Performance Computing & Networking in the School of Information Technologies, University of Sydney. Dr. Zomaya published more than 500 scientific papers and articles and is author, co-author or editor of more than 20 books. He served as the Editor in Chief of the IEEE Transactions on Computers (2011-2014) and was elected recently as a Founding Editor in Chief for the newly established IEEE Transactions on Sustainable Computing. Dr. Zomaya also serves as an associate editor for more than 20 leading journals. He is Fellow of AAAS, IEEE, and IET.
Albert Zomaya is the Chair Professor of High Performance Computing & Networking in the School of Information Technologies, University of Sydney. Dr. Zomaya published more than 500 scientific papers and articles and is author, co-author or editor of more than 20 books. He served as the Editor in Chief of the IEEE Transactions on Computers (2011-2014) and was elected recently as a Founding Editor in Chief for the newly established IEEE Transactions on Sustainable Computing. Dr. Zomaya also serves as an associate editor for more than 20 leading journals. He is Fellow of AAAS, IEEE, and IET. Sherif Sakr is currently a professor of computer and information science in the Health Informatics department at King Saud bin Abdulaziz University for Health Sciences. He is also affiliated with the University of New South Wales and DATA61/CSIRO. He received his PhD degree in Computer and Information Science from Konstanz University, Germany in 2007. Dr. Sakr held visiting appointments in several academic and research institutes including Microsoft Research (2011), Alcatel-Lucent Bell Labs (2012), University of Zurich (2016) and TU Dresden (2016). His current research is revolved around advanced big data management and processing technologies. In addition to his dozens of peer-reviewed articles in reputable conferences and journals, he is the author and editor of several valuable books in this domain.
Foreword 6
Preface 7
Contents 9
Part I Fundamentals of Big Data Processing 12
Big Data Storage and Data Models 13
1 Storage Models 13
1.1 Block-Based Storage 14
1.2 File-Based Storage 17
1.3 Object-Based Storage 20
1.4 Comparison of Storage Models 23
2 Data Models 25
2.1 NoSQL (Not only SQL) 26
2.2 Relational-Based 32
2.3 Summary of Data Models 37
References 37
Big Data Programming Models 40
1 MapReduce 40
1.1 Features 41
1.2 Examples 42
2 Functional Programming 43
2.1 Features 43
2.2 Example Frameworks 44
3 SQL-Like 47
3.1 Features 47
3.2 Examples 47
4 Actor Model 52
4.1 Features 52
4.2 Examples 52
5 Statistical and Analytical 56
5.1 Features 56
5.2 Examples 56
6 Dataflow-Based 57
6.1 Features 58
6.2 Examples 58
7 Bulk Synchronous Parallel 60
7.1 Features 60
7.2 Examples 60
8 High Level DSL 62
8.1 Pig Latin 62
8.2 Crunch/FlumeJava 63
8.3 Cascading 64
8.4 Dryad LINQ 65
8.5 Trident 66
8.6 Green Marl 66
8.7 Asterix Query Language (AQL) 67
8.8 IBM Jaql 68
9 Discussion and Conclusion 68
References 70
Programming Platforms for Big Data Analysis 73
1 Introduction 73
2 Requirements of Big Data Programming Support 75
3 Classification of Programming Platforms 76
3.1 Data Source 76
3.2 Processing Technique 77
4 Major Existing Programming Platforms 78
4.1 Data Parallel Programming Platforms 79
4.2 Graph Parallel Programming Platforms 85
4.3 Task Parallel Platforms 92
4.4 Stream Processing Programming Platforms 93
5 A Unifying Framework 100
5.1 Comparison of Existing Programming Platforms 101
5.2 Need for Unifying Framework 102
5.3 MatrixMap Framework 103
6 Conclusion and Future Directions 105
References 105
Big Data Analysis on Clouds 108
1 Introduction 109
2 Introducing Cloud Computing 110
2.1 Basic Concepts 111
2.2 Cloud Service Distribution and Deployment Models 111
3 Cloud Solutions for Big Data 113
3.1 Microsoft Azure 114
3.2 Amazon Web Services 114
3.3 OpenNebula 115
3.4 OpenStack 115
4 Systems for Big Data Analytics in the Cloud 116
4.1 MapReduce 117
4.2 Spark 118
4.3 Mahout 119
4.4 Hunk 120
4.5 Sector/Sphere 120
4.6 BigML 121
4.7 Kognitio Analytical Platform 123
4.8 Data Analysis Workflows 123
4.9 NoSQL Models for Data Analytics 132
4.10 Visual Analytics 136
4.11 Big Data Funding Projects 138
4.12 Historical Review 138
4.13 Summary 139
5 Research Trends 143
6 Conclusions 145
References 146
Data Organization and Curation in Big Data 150
1 Big Data Indexing Techniques 151
1.1 Overview 151
1.2 Record-Level Non-adaptive Indexing 154
1.3 Record-Level Adaptive Indexing 157
1.4 Split-Level Indexing 158
1.5 Hadoop-RDBMS Hybrid Indexing 160
2 Data Organization and Layout Techniques 161
2.1 Overview 161
2.2 Result Materialization and Caching Techniques 162
2.3 Pre-processing and Colocation Techniques 164
2.4 None Row-Oriented Storage Layouts 166
3 Non-traditional Workloads in Big Data 168
3.1 Overview 168
3.2 Techniques for Recurring Workloads 171
3.3 Techniques for Fast Online Analytics 172
4 Curation and Metadata Management in Big Data 175
4.1 Overview 175
4.2 Execution-Centric Metadata Approach 177
4.3 Provenance-Centric Metadata Approach 177
4.4 Data-Centric Metadata Approach 179
5 Conclusion 181
References 182
Big Data Query Engines 186
1 Introduction 187
1.1 MPP Query Engines 187
1.2 Hadoop Query Engines 188
1.3 Chapter Organization 189
2 Massively Parallel Query Engines 189
2.1 Teradata 189
2.2 Greenplum 191
2.3 Vertica 192
3 Hadoop Query Engines 194
3.1 MapReduce 194
3.2 Hive 195
3.3 Spark 196
4 SQL on Hadoop 197
4.1 HAWQ 198
4.2 Impala 200
4.3 Presto 202
5 Query Optimization 203
5.1 Research Problems 204
5.2 Orca 206
5.3 Catalyst 209
5.4 V2Opt 212
5.5 Impala Query Optimizer 212
6 Query Execution 214
6.1 Research Problems 215
6.2 Hadoop-Based Execution Engines 216
6.3 Parallel Databases Execution Engines 218
6.4 Code Generation 221
7 Summary 223
References 223
Large-Scale Data Stream Processing Systems 225
1 Introduction 226
1.1 Stream Processing and Its Precursors 227
1.2 Large-Scale Data Stream Processing on Commodity Clusters 228
1.3 Distinctive Features of Data Stream Processing Systems 229
1.4 Chapter Overview 231
2 Programming Models 231
2.1 Programming with Streams 232
2.2 Lower-Level Dataflow Programming 233
2.3 Functional APIs 237
2.4 Stream Windows 239
3 System Support for Distributed Data Streaming 241
3.1 An Analysis of Large-Scale Stream Processing Systems 242
3.2 Execution Models 243
3.3 Processing Guarantees Upon Failure 245
3.4 Flow Control 251
3.5 Execution Plan Optimisations 252
4 Case Study: Stream Processing with Apache Flink 253
4.1 The Apache Flink Stack 254
4.2 The Apache Flink System Architecture 255
4.3 Lightweight Asynchronous Snapshots 255
5 Applications, Trends and Open Challenges 258
5.1 Graph Stream Processing 259
5.2 Online Learning 260
5.3 Complex Event Processing 260
6 Conclusions and Outlook 261
References 262
Part II Semantic Big Data Management 267
Semantic Data Integration 268
1 An Important Challenge 268
1.1 Linked Data 271
1.2 Ontologies 273
1.3 Ontology and Data Alignment 275
2 Current State-of-the-Art 281
2.1 Interactive and Collaborative Approaches 281
2.2 Visualizing the Data Integration Process 286
2.3 Integrating Geospatial Data 291
2.4 Integrating Biomedical Data 295
3 The Path Forward 300
3.1 Moving Beyond 1-to-1 Equivalence Mappings 300
3.2 Advancing Alignment Evaluation 301
3.3 Contextualizing Alignments 303
References 305
Linked Data Management 311
1 Introduction 311
2 Background Information 313
3 Native Linked Data Stores 317
3.1 Quadruple Systems 318
3.2 Index Permuted Stores 321
3.3 Graph-Based Systems 325
4 Provenance for Linked Data 333
4.1 Provenance Representations 333
4.2 Provenance in Data Management Systems 334
References 339
Non-native RDF Storage Engines 343
1 Introduction 343
2 Storing Linked Data Using Relational Databases 344
2.1 Statement Table 344
2.2 Optimizing Data Storage 346
2.3 Property Tables 348
2.4 Query Execution 350
3 No-SQL Stores 352
4 Massively Parallel Processing for Linked Data 357
4.1 Data Storage and Partitioning 357
4.2 Query Execution 362
References 365
Exploratory Ad-Hoc Analytics for Big Data 369
1 Exploratory Analytics for Big Data 369
1.1 Requirements 372
1.2 Architecture Overview 374
2 A Top-K Entity Augmentation System 375
2.1 Motivation and Challenges 376
2.2 Requirements 379
2.3 Top-k Consistent Entity Augmentation 380
2.4 Related Work 389
3 DrillBeyond -- Processing Open World SQL 391
3.1 Motivation and Challenges 391
3.2 Requirements 393
3.3 The DrillBeyond System 395
3.4 Processing Multi-result Queries 404
3.5 Related Work 405
4 Summary and Future Work 407
4.1 Future Work 408
References 409
Pattern Matching Over Linked Data Streams 412
1 Overview 412
2 Linked Data Dissemination System 414
2.1 System Overview 415
2.2 TP-Automata for Single Triple Pattern Query Matching 416
2.3 CTP-Automata for Conjunctive Triple Pattern Query Matching 417
3 Experimental Evaluation 421
3.1 Experimental Setup 421
3.2 Evaluation of TP-Automata 422
3.3 Evaluation of CTP-Automata 424
3.4 Limitations 427
4 Related Work 427
5 Summary 428
References 429
Searching the Big Data: Practices and Experiences in Efficiently Querying Knowledge Bases 431
1 Introduction 431
2 Background 434
2.1 Knowledge Base Preliminary 434
3 The Framework of Cache-Based Knowledge Base Querying 436
4 Similar Queries Suggestion 436
4.1 Query Distance Calculation 437
4.2 Feature Modeling 440
5 Cache Replacement 442
5.1 Modified Simple Exponential Smoothing 442
5.2 Replacement Algorithms 443
6 Implementation and Experimental Evaluation 445
6.1 Setup 445
6.2 Performance of Cache Replacement Algorithm 446
6.3 Comparison of Feature Modeling Approaches 447
6.4 Performance Comparison with the State-of-the-Art Work 449
6.5 Experimental Conclusion 450
7 Related Work 450
7.1 Semantic Caching 451
7.2 Query Suggestion 452
8 Discussion and Conclusion 452
References 453
Part III Big Graph Analytics 456
Management and Analysis of Big Graph Data: Current Systems and Open Challenges 457
1 Introduction 457
2 Graph Databases 460
2.1 Recent Graph Database Systems 460
2.2 Graph Data Models 463
2.3 Query Language Support 466
3 Graph Processing 467
3.1 General Architecture 468
3.2 Think Like a Vertex 468
3.3 Think Like a Graph 475
4 Graph Dataflow Systems 477
4.1 Apache Flink 478
4.2 Apache Flink Gelly 480
4.3 Comparison to Other Graph Dataflow Frameworks 484
5 Gradoop 485
5.1 Architecture 485
5.2 Extended Property Graph Model 486
6 Comparison 493
7 Current Research and Open Challenges 495
7.1 Graph Data Allocation and Partitioning 495
7.2 Benchmarking and Evaluation of Graph Data Systems 496
7.3 Analysis of Dynamic Graphs 497
7.4 Graph-Based Data Integration and Knowledge Graphs 498
7.5 Interactive Graph Analytics 499
8 Conclusions and Outlook 500
References 501
Similarity Search in Large-Scale Graph Databases 506
1 Introduction 506
2 Preliminaries 508
3 The Pruning-Verification Framework 511
4 State-of-the-Art Approaches 512
4.1 A Tree-Based Approach: K-Adjacent Tree 513
4.2 A Star-Based Approach: SEGOS 514
4.3 A Path-Based Approach: GSimJoin 518
4.4 A Partition-Based Approach: Pars 519
5 Future Research Directions 522
5.1 New GED Bounds and Search Algorithms 522
5.2 Rich Semantics of Similarity Search 523
5.3 Graph Query Formulation and Understanding 524
6 Summary 525
References 525
Big-Graphs: Querying, Mining, and Beyond 529
1 Introduction 529
2 Graph Data Models 531
2.1 RDF 531
2.2 Property Graph 533
3 Pattern Matching Techniques Over Big-Graphs 534
3.1 SQL and NoSQL Approaches 534
3.2 Keyword Search 537
3.3 Graph Matching Query 545
3.4 Graph Query by Example 551
4 Mining Techniques Over Big-Graphs 553
4.1 Frequent Subgraph Mining 554
4.2 Mining Discriminative Subgraphs 560
4.3 Mining Statistically Significant Subgraphs 562
4.4 Mining Representative Subgraphs 570
5 Open Problems 571
5.1 Large-Scale Graph Processing Systems 571
5.2 Graph Databases, Languages, and Query Interfaces 572
5.3 Datasets and Benchmarks 573
6 Conclusions 573
7 About Authors 574
References 574
Link and Graph Mining in the Big Data Era 581
1 Introduction 581
2 Definitions 583
3 Temporal Evolution 586
4 Link Prediction 588
5 Community Detection 592
5.1 Modularity Maximization 592
5.2 The Louvain Method for Community Detection 593
6 Graphs in Big Data 594
6.1 Graphs in the Big Data Era 595
6.2 Knowledge Graphs 596
6.3 Graph Sampling 598
6.4 Graph Analytics Tools 598
7 Weighted Networks 600
8 Extending Graph Models: Multilayer Networks 601
8.1 The Layered Point of View: Multilayer Networks 601
8.2 Models, Methodologies and Other Tools 602
8.3 Theoretical Models, Empirical Applications and Other Examples 604
9 Open Challenges 605
10 Conclusions 607
References 607
Granular Social Network: Model and Applications 615
1 Introduction 615
2 Preliminaries 617
2.1 Social Network Analysis 617
2.2 Fuzzy Sets 621
2.3 Rough Sets 621
2.4 Granular Computing 622
3 Literature Review 623
3.1 Modeling Social Networks 623
3.2 Target Set Selection 625
3.3 Community Detection 626
4 Fuzzy Granular Social Networks (FGSN) 627
4.1 The Model 628
4.2 Network Measures of FGSN 629
4.3 Uncertainties in FGSN 631
4.4 Granular Degree Heuristic for Target Set Selection in FGSN 632
4.5 Fuzzy-Rough Community (FRC) Detection 634
4.6 Scalability of FGSN 641
5 Discussions and Conclusions 643
References 645
Part IV Big Data Applications 650
Big Data, IoT and Semantics 651
1 Introduction 651
2 Semantics for Big Data 654
2.1 Semantic Representation of Things, People and Web 655
2.2 Semantic Based Classification and Learning 656
2.3 Linked Data and Open Data 658
2.4 Reasoning over Big Data 660
3 Big Data and Semantics in the Internet of Things 661
3.1 Impact of IoT on Big Data 662
3.2 Ongoing Research Efforts 664
4 Social Mining 667
4.1 Text Mining 668
4.2 Sentiment Analysis 671
4.3 Social and Political Trends 673
5 Graph Mining 674
5.1 Link Mining 678
6 Big Stream Data Mining 679
6.1 Data Sampling 681
6.2 Data Filtering 681
7 Geo-Referenced Data Mining 682
8 Conclusion 684
References 684
SCADA Systems in the Cloud 687
1 Introduction 688
2 Related Work 689
3 An Overview of SCADA 691
3.1 Generalized SCADA Architecture 691
3.2 SCADA Characteristics 693
3.3 SCADA and Big Data 695
4 Moving SCADA to the Cloud 695
4.1 Benefits of Cloud-Based SCADA Systems 695
4.2 SCADA Requirements Versus Cloud Solutions 696
4.3 Overview of Cloud Migration 698
4.4 Cloud Migration Technologies 699
5 Conceptual SCADA Cloud Orchestration Framework 701
5.1 Migration Recommendations 701
5.2 SCADA as a Service 703
5.3 Cloud Service Deployment Environment 704
5.4 SCADA Service Design 705
5.5 SCADA Cloud Orchestration Framework 706
6 Results 710
7 Conclusion 711
References 713
Quantitative Data Analysis in Finance 715
1 Introduction 716
1.1 History of Quantitative Finance 716
1.2 Compendium of Terminology and Abbreviations 718
2 The Three V's of Big Data in High Frequency Data 719
2.1 Velocity 719
2.2 Variety 720
2.3 Volume 720
2.4 Challenges for High Frequency Data 721
3 Data Cleaning, Aggregating and Management 721
3.1 Data Cleaning 723
3.2 Data Aggregating 724
3.3 Scalable Database and Distributed Processing 726
4 Modeling High Frequency Data in Finance 727
4.1 Volatility Curve 728
4.2 Stochastic Volatility 730
4.3 Multivariate Volatility 731
4.4 Expected Return 732
4.5 Duration 734
4.6 Scalable Parallel Algorithms on Supercomputers 735
5 Portfolio Selection and Evaluation 737
5.1 Markowitz Portfolio Optimization with Transaction Costs 737
5.2 On-Line Portfolio Selection 739
6 The Future 743
6.1 Advanced Statistics and Information Theory 744
6.2 Combination of Machine Learning, Game Theory and Statistics 745
6.3 Efficient Algorithms in Linear Algebra and Convex Optimization 745
7 Conclusion 746
References 747
Emerging Cost Effective Big Data Architectures 750
1 Introduction 750
2 Emerging Solutions for Big Data 754
2.1 Workload-Aware Solutions 755
2.2 Scaling-Down Big Data Systems 760
2.3 Approximate Computing 764
3 Future Directions 765
3.1 Hybrid Big Data Architectures 765
3.2 Multi-tenancy in Cloud Infrastructures 766
3.3 Virtualized Environments 767
4 Conclusion 767
References 768
Bringing High Performance Computing to Big Data Algorithms 772
1 Introduction 772
1.1 High Performance Computing Meets Big Data 772
1.2 Application Areas 774
1.3 Tricks of the Trade 776
2 GPU Acceleration of Alternating Least Squares 777
2.1 Explicit Feedback 778
2.2 Implicit Feedback 779
2.3 CPU Implementation 781
2.4 GPU Implementation 782
2.5 Setup and Datasets 784
2.6 Auto Tuning 785
2.7 Performance Evaluation 786
3 GPU Acceleration of Singular Value Decomposition 788
3.1 Introduction 788
3.2 Randomized Algorithms to Compute SVD 789
3.3 Hybrid CPU/GPU Implementation 790
3.4 Randomized Algorithms to Update SVD 793
4 Conclusions 796
References 798
Cognitive Computing: Where Big Data Is Driving Us 802
1 Cognitive Computing: An Alternative Approach for Clear Understanding 802
2 Big Data Impulsing Cognitive System 807
3 Traditional Systems versus Cognitive Systems? 809
4 Data Mining in the Era of Cognitive Systems 810
5 Design Methods for Cognitive Systems 814
5.1 Quantitative and Qualitative Methods 814
5.2 Data Gathering Methods 816
5.3 Design Methods 822
5.4 Evaluation Methods 824
5.5 Data Analysis Methods 826
5.6 Using Information Visualization to Understanding Users 829
6 Cognitive Systems 831
6.1 IBM Watson 831
6.2 Other Cognitive Systems 834
7 The Future of Cognitive Systems 837
8 Final Remarks 839
References 840
Privacy-Preserving Record Linkage for Big Data: Current Approaches and Research Challenges 846
1 Introduction 847
2 Background 849
2.1 Overview and Challenges of PPRL 849
2.2 The PPRL Process and Techniques Used 851
3 Privacy Aspects and Techniques for PPRL 854
3.1 PPRL Scenarios 854
3.2 Adversary Models 855
3.3 Attacks 856
3.4 Data Masking or Encoding 857
3.5 Bloom Filters 860
4 Scalability Techniques for PPRL 861
4.1 Blocking Techniques 862
4.2 Filtering Techniques 865
4.3 Parallel PPRL 869
5 Multi-party PPRL 872
5.1 Multi-party Private Blocking Techniques 872
5.2 Multi-party Private Comparison and Classification Techniques 874
6 Open Challenges 877
6.1 Improving Scalability 877
6.2 Improving Linkage Quality 878
6.3 Dynamic Data and Real-Time Matching 878
6.4 Improving Security and Privacy 879
6.5 Evaluation, Frameworks, and Benchmarks 880
6.6 Discussion 881
7 Conclusions 882
References 883
| Erscheint lt. Verlag | 25.2.2017 |
|---|---|
| Zusatzinfo | XIII, 895 p. 307 illus. |
| Verlagsort | Cham |
| Sprache | englisch |
| Themenwelt | Mathematik / Informatik ► Informatik ► Programmiersprachen / -werkzeuge |
| Informatik ► Weitere Themen ► Hardware | |
| Technik ► Nachrichtentechnik | |
| Wirtschaft | |
| Schlagworte | Big Data • Big data applications • Big Data integration • Big Data programming models • Big Data query engines • Big data storage • Big graph analytics • Big SQL • data analytics • flink • Giraph • Hadoop • MapReduce • Spark |
| ISBN-10 | 3-319-49340-X / 331949340X |
| ISBN-13 | 978-3-319-49340-4 / 9783319493404 |
| Haben Sie eine Frage zum Produkt? |
PDF (Wasserzeichen)Größe: 32,1 MB
DRM: Digitales Wasserzeichen
Dieses eBook enthält ein digitales Wasserzeichen und ist damit für Sie personalisiert. Bei einer missbräuchlichen Weitergabe des eBooks an Dritte ist eine Rückverfolgung an die Quelle möglich.
Dateiformat: PDF (Portable Document Format)
Mit einem festen Seitenlayout eignet sich die PDF besonders für Fachbücher mit Spalten, Tabellen und Abbildungen. Eine PDF kann auf fast allen Geräten angezeigt werden, ist aber für kleine Displays (Smartphone, eReader) nur eingeschränkt geeignet.
Systemvoraussetzungen:
PC/Mac: Mit einem PC oder Mac können Sie dieses eBook lesen. Sie benötigen dafür einen PDF-Viewer - z.B. den Adobe Reader oder Adobe Digital Editions.
eReader: Dieses eBook kann mit (fast) allen eBook-Readern gelesen werden. Mit dem amazon-Kindle ist es aber nicht kompatibel.
Smartphone/Tablet: Egal ob Apple oder Android, dieses eBook können Sie lesen. Sie benötigen dafür einen PDF-Viewer - z.B. die kostenlose Adobe Digital Editions-App.
Zusätzliches Feature: Online Lesen
Dieses eBook können Sie zusätzlich zum Download auch online im Webbrowser lesen.
Buying eBooks from abroad
For tax law reasons we can sell eBooks just within Germany and Switzerland. Regrettably we cannot fulfill eBook-orders from other countries.
aus dem Bereich
