Building upon his earlier book that detailed agile data warehousing programming techniques for the Scrum master, Ralph's latest work illustrates the agile interpretations of the remaining software engineering disciplines:
- Requirements managements benefits from streamlined templates that not only define projects quickly, but ensure nothing essential is overlooked.
- Data engineering receives two new 'hyper modeling' techniques, yielding data warehouses that can be easily adapted when requirements change without having to invest in ruinously expensive data-conversion programs.
- Quality assurance advances with not only a stereoscopic top-down and bottom-up planning method, but also the incorporation of the latest in automated test engines.
Use this step-by-step guide to deepen your own application development skills through self-study, show your teammates the world's fastest and most reliable techniques for creating business intelligence systems, or ensure that the IT department working for you is building your next decision support system the right way.
- Learn how to quickly define scope and architecture before programming starts
- Includes techniques of process and data engineering that enable iterative and incremental delivery
- Demonstrates how to plan and execute quality assurance plans and includes a guide to continuous integration and automated regression testing
- Presents program management strategies for coordinating multiple agile data mart projects so that over time an enterprise data warehouse emerges
- Use the provided 120-day road map to establish a robust, agile data warehousing program
Ralph Hughes, former DW/BI practice manager for a leading global systems integrator, has led numerous BI programs and projects for Fortune 500 companies in aerospace, government, telecom, and pharmaceuticals. A certified Scrum Master and a PMI Project Management Professional, he began developing an agile method for data warehouse 15 years ago, and was the first to publish books on the iterative solutions for business intelligence projects. He is a veteran trainer with the world's leading data warehouse institute and has instructed or coached over 1,000 BI professionals worldwide in the discipline of incremental delivery of large data management systems.
A frequent keynote speaker at business intelligence and data management events, he serves as a judge on emerging technologies award panels and program advisory committees of advanced technology conferences. He holds BA and MA degrees from Stanford University where he studied computer modeling and econometric forecasting. A co-inventor of Zuzena, the automated testing engine for data warehouses, he serves as Chief Systems Architect for Ceregenics and consults on agile projects internationally.
Building upon his earlier book that detailed agile data warehousing programming techniques for the Scrum master, Ralph's latest work illustrates the agile interpretations of the remaining software engineering disciplines: Requirements management benefits from streamlined templates that not only define projects quickly, but ensure nothing essential is overlooked. Data engineering receives two new "e;hyper modeling"e; techniques, yielding data warehouses that can be easily adapted when requirements change without having to invest in ruinously expensive data-conversion programs. Quality assurance advances with not only a stereoscopic top-down and bottom-up planning method, but also the incorporation of the latest in automated test engines. Use this step-by-step guide to deepen your own application development skills through self-study, show your teammates the world's fastest and most reliable techniques for creating business intelligence systems, or ensure that the IT department working for you is building your next decision support system the right way. Learn how to quickly define scope and architecture before programming starts Includes techniques of process and data engineering that enable iterative and incremental delivery Demonstrates how to plan and execute quality assurance plans and includes a guide to continuous integration and automated regression testing Presents program management strategies for coordinating multiple agile data mart projects so that over time an enterprise data warehouse emerges Use the provided 120-day road map to establish a robust, agile data warehousing program
Front Cover 1
Agile Data Warehousing for the Enterprise 4
Copyright Page 5
Advance Praise for Agile Data Warehousing for the Enterprise 6
Short Contents 8
Full Contents 10
List of Figures 18
List of Tables 24
Abbreviations 26
Development Team Roles 26
Foreword 28
Acknowledgments 30
1 Solving Enterprise Data Warehousing’s “Fundamental Problem” 32
The Agile Solution in a Nutshell 32
Five Legs to Stand Upon 34
The Agile EDW Alternative is Ready to Deploy 36
Defining a Baseline Method for Agile EDW 36
Plenty of Motivation to “Go Agile” 38
Structure of the Presentation Ahead 38
Summary 40
I. Summaries of Generic Agile Development Methods 42
2 Primer on Agile Development Methods 44
Defining “Agile” 44
Agile Manifesto Values and Principles 50
Scrum in a Nutshell 51
User Stories 52
Scrum’s Five-Step Delivery Iteration 54
Story Conference 54
Task Planning 54
Development 55
User Demo 56
Iteration Retrospective 57
Contributions from Extreme Programming 57
XP Values and Principles 58
XP’s Values 59
Principles 59
Summary 60
3 Introduction to Alternative Iterative Methods 62
Lean Software Development 62
Lean Origins 62
Lean Methods as a Long-Term Destination 63
Lean Principles and Tools 64
Principle 1: Eliminate Waste 64
Tool 1: Seeing Waste 65
Tool 2: Value Stream Mapping 65
Principle 2: Amplify Learning 66
Tool 3: Feedback 66
Tool 4: Iterations 66
Tool 5: Synchronization 66
Tool 6: Set-Based Development 67
Principle 3: Decide as Late as Possible 67
Tool 7: Options Thinking 67
Tool 8: The Last Responsible Moment 67
Tool 9: Making Decisions 68
Principle 4: Deliver as Fast as Possible 68
Tool 10: Pull-Based Systems 68
Tool 11: Queuing Theory 69
Tool 12: Cost of Delay 69
Principle 5: Empower the Team 69
Tool 13: Self-Determination 70
Tool 14: Motivation 70
Tool 15: Leadership 70
Tool 16: Expertise 70
Principle 6: Build Integrity in 70
Tool 17: Perceived Integrity 70
Tool 18: Conceptual Integrity 71
Tool 19: Refactoring 71
Tool 20: Testing 71
Principle 7: See the Whole 71
Tool 21: Measurements 71
Tool 22: Contracts 72
Kanban 72
Quick Sketch of the Kanban Method 72
Visualizing and Maintaining Continuous Flow 74
Evidence-Based Service Levels 75
Comparing Kanban to Scrum 76
The Hybrid “Scrumban” Approach 78
Rational Unified Process 80
RUP Overview 80
Why Not RUP for DW/BI? 83
Summary 84
Part I References 86
II. Review of Fast EDW Coding and Risk Mitigation 88
4 Essential DW/BI Background and Definitions 90
Primary Source for DW/BI Standards 91
Defining Enterprise Data Warehousing 92
Basic Business Terms 94
Enterprise 94
Business Unit 94
Business Department 95
Executive, Director, and Manager 95
Business Rules 96
Data and Information Terms 96
Information Services Terms 97
Information Technology (IT) 97
Software and Applications 97
End Users 97
Operational and Analytical Systems 97
IT Service Groups 97
Shadow IT 98
Competency Centers 98
Software Engineering Terms 98
Software Development Life Cycle (SDLC) 98
Developers and Programmers 99
Software Engineering 99
Units and Components 100
Configuration Management and Application Builds 101
Environments 101
Basic Architectural Concepts 101
System Architecture 101
Data Architecture 102
Business Conceptual Model 102
Logical Data Model 103
Physical Data Model 103
Reference Architecture 105
Enterprise Architecture 106
Architectural Frameworks 107
Zachman Enterprise Architectural Framework 107
Dama Functional Framework 107
Hammergren DW Planning Matrix 108
Additional Data Warehousing Concepts 110
Database Management System 110
ETL/ELT 110
Data Loads 111
Primary and Foreign Keys 111
Natural and Surrogate Keys 111
Indexes 111
Constraints and Referential Integrity 112
Views and Data Virtualization 112
Data Schema 112
Subject and Topic Areas 112
Data Dictionary 112
Normalized Data Model 112
Dimensional Data Model 113
Data Marts 113
Data Warehouse Appliance 113
Corporate Information Factory 113
Enterprise Data Bus 113
Traditional Project Management Terms 113
Project 114
Stakeholder 114
Programs 114
Portfolios 114
Program Management Office 115
Project Charter 115
Project and Program Manager 115
Summary 115
5 Recap of Agile DW/BI Coding Practices 116
Iterative Coding Alone Significantly Improves BI Projects 116
Yet Data Integration Remains a Challenge 116
New Roles for DW/BI Projects 117
Project Architect 118
Data Modeler 119
Systems Analyst 119
System Tester 120
Proxy Product Owner 120
Scrum Master 121
Including the New Roles on the Team’s Whale Chart 121
80/20 Specifications 121
Developer Stories 123
DW/BI User Stories Hide Much of the Data Integration Work 123
Developer Stories Make DW/BI Work More Manageable 124
Developer Stories Require a Deeper Understanding of Value 125
Current Estimates 126
Adding Techniques from Kanban 128
Pipelined Delivery 129
Work-in-Progress Limits for Developers 131
Iteration –1 and 0 131
Two-Pass Testing 132
Evidence-Based Service Level Agreements 133
Proof that Agile DW/BI Works 135
Investigating Project Cost Impacts in More Detail 137
Some Myths Prove True 138
Summary 138
6 Eliminating Risk Through Nested Iterations 140
EDW Programs Slip into “231 Swamps” 140
231 Swamps Derive from a Command and Control Strategy 141
Agile’s Fundamental Risk Mitigation Technique 142
Agile’s General Risk Mitigation Strategy 142
Eliminating Miscommunication with Multiplexed Engineering Phases 144
Agile Edw’s Extended Risk Mitigation Techniques 145
Three Types of Risk Threaten EDW Programs 145
Examples of the Three Levels 146
Mitigating the Risk of Application Coding Concept Errors 147
Mitigating the Risk of Solution Concept Errors 147
Mitigating the Risk of Business Concept Errors 150
Summary 151
Part II References 152
III. Agile EDW Requirements Management 154
7 Balancing between Two Extremes 156
Building the Case for Effective Requirements Management 157
Developers Often Neglect Requirements Work 159
Motivating Teams to Take Requirements Seriously 159
The Boehm Multiplier 159
The Blivit Factor 160
The Curse of Working Nights and Weekends 160
Easy to Overinvest in Requirements Management 161
“Requirements Management” Formally Defined 161
Traditional Projects Employ a Big Spec Up Front 161
Requirements are Inherently Diverse 163
Business Process Reengineering Can Add to the Complexity 166
Reasons Not to Overinvest in Requirement Work 167
Precision at the Expense of Accuracy 168
Business Partners are Adverse to Traditional Requirements Gathering Efforts 169
Traditional Requirements Management Fails More than it Succeeds 170
The Greatest Failure is Losing Business Opportunity 170
Agile’s Approach Centers on Balance 172
Agile Objectives for Requirements Management 172
Provide Enough Context to Make User Stories Easy to Author 173
Engage the Close Stakeholders 173
Must Address All Types of Requirements 173
Empower IT to Judge Requirements Completeness and Accuracy 174
Provide a Whole Project Sketch to Avoid the Big Mistakes 174
Knowing when a Backlog is “Good Enough” 174
Enable Regular “Current Estimates” 175
Keeping the Requirements Management Process Agile 175
Two Intersecting Requirements Management Value Chains 175
Salient Differences between GRM and ERM 178
Business Analysts Implicit in Two Project Lead Roles 180
Summary 181
8 Redefining the Epic Stack to Enable Value Accounting 182
Toward a Robust Epic Decomposition Framework 182
Defining the Backlog Hierarchy’s Structure 182
Aligning the Epic Stack to the Company’s Hierarchy 183
Clearly Defining Each Level within the Epic Stack 185
Testing Whether Stories are Good Enough 187
Demonstrable 187
Independent 189
Layered 189
Business-valued 190
Estimateable 190
Refinable 190
Testable 190
Small 190
Clarifying Everything with Value Accounting 190
The Basics of Value Accounting 191
Value Accounting Makes Developers More Effective 192
Value Accounting Mitigates Project Risk 193
Allocating Value Throughout an Epic Tree 194
Identifying the Value of a Project 194
Allocating Value to Epics 195
Allocating Value to Themes and User Stories 195
Value Buildups by Environment Provide Motivation and Clarity 196
Summary 198
9 Artifacts for the Generic Requirements Value Chain 200
Beware of Requirements Churn 200
User Modeling/Personas 201
End Users’ Hierarchy of Needs 202
Data Access 203
Reporting 204
Research 204
Analysis 204
Prediction 204
Benefits Offered by the BI Hierarchy of Needs 204
Mind Maps and Fishbone Diagrams 205
Vision Boxes 207
Vision Statements 207
Product Roadmaps 209
Summary 211
10 Artifacts for the Enterprise Requirements Value Chain 212
The Generic Value Chain Can Overlook Crucial Requirements 212
ERM as a Flexible RM Approach 214
Focusing on Enterprise Aspects of Project Requirements 215
Functionality Dimension 215
Polarity Dimension 216
Orientation Dimension 216
Streamlined ERM Templates 217
Uncovering Project Goals with Sponsor’s Concept Briefing 217
Justification Type 218
Customer Experience Impacts 219
Functional Area Impacts Assessments 219
Value of the Program 219
Program Success Metrics 220
Identifying Project Objectives with Stakeholder’s Requests 220
Business System Challenges 220
Current Manual Solution 220
Desired Business Solution 221
Volume Requirements and End-User Census 221
Dependent Systems 221
Sketching the Solution with a Vision Document 222
Solutions Statements 222
Features and Benefits List 222
Context Diagram 225
Target Business Model 227
High-Level Architectural Diagram 228
Nonfunctional Requirements 228
Segmenting the Project with Subrelease Overview 229
Subrelease Identifier 231
Subrelease Scope 231
Expressed as Data Services 231
Expressed as a Target Business Model 233
Expressed Using the Fact Qualifier Matrix 233
Business Process Supported 233
Use Case Model 233
Analysis Venn Diagram 234
Business-Level Data Validation Steps 236
Sample Business Queries 238
Technical Description 238
Target Fact Tables Details 238
Reusable Target Dimensions Details 238
Non-Reusable Target Dimensions Details 239
Data Sourcing Details 239
Nonfunctional Requirements 239
Providing Developer Guidance with Module Use Cases 240
Goal 240
Standard Flow of Events 240
Alternative Flow of Events 241
Special Requirements 243
Source-to-Target Mappings as Supplemental Specifications 243
Nonfunctional Requirements as Supplemental Specifications 243
Summary 244
11 Intersecting Value Chains for a Stereoscopic Project Definition 246
Intersecting the Two Value Chains 246
Agile EDW’s Version of Requirements Traceability 246
Addressing Nonfunctional Requirements 248
The Proper Problem Domain for Agile EDW 248
Agile EDW Supports Broader Architectural Activities 250
Supporting the Organization’s Software Release Cycle 252
Phases Borrowed from Rational Unified Process 252
Iterations –1 and 0 Fit into the Inception Phase 252
Arriving at a Predevelopment Project Estimate 254
Managing the Predevelopment Estimate 256
Completing the Release Cycle 257
Techniques for the Elaboration Phase 257
Choosing Developer Stories for the Elaboration Phase 257
Proving Out Architectures Using a “Steel Thread” 258
Prioritizing Project Backlogs 259
Priority 1: Business Value 259
Priority 2: Predecessor/Successor Dependencies 259
Priority 3: Architectural Uncertainties 260
Priority 4: Meeting Interproject Milestones 260
Priority 5: Smoothing Out Iterations 260
Priority 6: “Funding Waypoints” 260
Priority 7: Resource Scheduling 260
Managing Incremental Precision 260
A Framework for Visualizing Progressive Requirements 261
The Freezer, Fridge, Counter Metaphor 261
Effort Levels by Team Roles 263
Visualizing Requirements Management Demands with Effort Curves 263
Allocating Time for Nonfunctional Requirements 265
Conquering Complex Business Rules with an Embedded Method 266
Add the Data Cowboy Role 266
Special Skills and Tools for the Data Cowboy 267
Modified Data Mining Method Can Help 267
Placing Business Rules Discovery and Analysis into the Effort Curves 269
Interfacing with Project Governance 270
Not Returning to a Waterfall Approach 273
Summary 273
Part III References 276
IV. Agile EDW Data Engineering 278
12 Traditional Data Modeling Paradigms and Their Discontents 280
EDW at a Crossroads 280
Reviewing the Reference Architecture 280
Standard Normal Forms Lead to Complex Integration Layers 282
Conformed Dimensions Lead to Complex Presentation Layers 284
A Peek at the Agile Alternatives 286
Models, Architectures, and Paradigms 288
Data Architecture 288
Data Model 289
Data Modeling Paradigm 290
Normalization Basics 291
Designing Databases to Eliminate Update Anomalies 291
Example: One Table from First to Fifth Normal Form 293
1NF Correction 295
2NF Correction 295
3NF Correction 296
4NF Correction 297
5NF Correction 301
Generalization Basics 302
Advantages and Disadvantages of Generalization 302
Example: Generalizing a Sales Table for the Party Entity 305
Level 1 Generalization for Party 307
Level 2 Generalization 308
Two Patterns for Role Hierarchies 309
Level 3 Generalization 309
Further Generalization Concepts 310
The Standard Approach and its Data Modeling Paradigms 310
The Traditional Integration Layer as a Challenged Concept 312
Involves an Expensive Hidden Layer 312
Results are Difficult to Understand 313
Entails High Maintenance Conversion Costs 314
“Straight-To-Star” as a Controversial Alternative 317
Four Change Cases for Appraising a Data Modeling Paradigm 317
Change Case 1: Correcting Fourth Normal Form Errors 318
Change Case 2: Generalizing to the Party Model 318
Change Case 3: New Trigger Attribute for a Slowly Changing Dimension 320
Change Case 4: Changing a Fact Table’s Grain 321
Summary 322
13 Surface Solutions Using Data Virtualization and Big Data 324
Leveraging Shadow it 325
Example of a Five-Step Collaborative Effort 325
Lessons from the Case History 327
Faster Value Delivery with Data Virtualization 327
Defining Data Virtualization 328
The Basic Use Case 328
DVS Performance Features 330
The Economics of Virtual Solutions 331
DVS Surface Solutions and Progressive Deployment 333
Comparing DVS Surface Solutions to the Previous Example 335
Data Virtualization’s Value Proposition 336
EDW’s Reference Architecture Becomes Dynamic 337
An Agile Role for Big Data 339
Introducing Big Data Technologies 339
The Need for Big Data Technology 340
The Promise of Schema-On-Read 341
An Introduction to Hadoop 342
Hadoop Distributed File System 343
Hadoop MapReduce 343
Notable Contrasts between SQL and MapReduce 345
Making MapReduce Look Like SQL with Hive 348
A Tempered View of Hive 351
Big Data Is Not Just Hive 355
Using Big Data to Enhance EDW Agility 356
Summary 358
14 Agile Integration Layers with Hyper Normalization 360
Hyper Normalization Hinges on “Ensemble Modeling” 360
Several Varieties of Hyper Normalization Exist 361
Hyper Normalized Data Modeling Concepts 362
Business Key Entities 364
Linking Entities 365
Attribute Entities 366
Lightly Integrated, Persistent Staging Area 368
Ensemble Modeling Components Allow Light Integration and Agility 370
Business Key Tables Abet Agility 370
Linking Tables Abet Agility 370
Attribute Tables Abet Agility 371
An Insert-Only Paradigm 373
Swedish Variation: Anchor Modeling 374
Reusable ETL Modules Accelerate New Development 375
One ETL Pattern Needed Per Hyper Normalized Table Type 376
Parameter-Driven ETL Module Prototypes 377
Calling the Reusable ETL Modules 379
Self-Validating Reusable ETL Modules 381
Estimate of Comparative Development Efforts 383
Common Data Retrieval Challenges and Their Solutions 383
HNF Aids the Leading Edge of the Integration Layer Only 384
Retrieving Data from an HNF Repository Doubly Difficult 385
Solution 0: Focus on Presentation Layer Objects 387
Solution 1: Dummy Attribute Records 387
Solution 2: Current Record Indicators 387
Solution 3: Point-in-Time Tables 387
Solution 4: Table Pruning 389
Solution 5: Bridging Tables 390
Solution 6: Retrieval Query Writers 391
Clearing an Architectural Review 392
Re-Architecting the EDW for Hyper Normalization 392
The Simple Vault Style 393
The Enhanced Vault Style 394
The Source Vault Style 395
The Raw Vault Style 395
Blending Styles to Achieve Agility 396
Enabling Evolution of Existing EDW Components 397
Change Case 1: Splitting Out Entities 397
Change Case 2: Upgrading to a Party Model 398
HNF-Powered Agile Solutions 399
Step 1: Surface Solution with Raw Data Vault 401
Step 2: Audit Sublayer 401
Step 3: Value-Added Sublayer 402
Step 4: Fully Managed Data Delivery Chain 402
Step 5: Performance Sublayer 402
Evidence of Success 402
Online Financial Services 403
The Free University 403
Summary 404
15 Fully Agile EDW with Hyper Generalization 406
Hyper Generalization Involves a Mix of Modeling Strategies 406
Extreme Generalization 408
Adding Time-Oriented Object Classification 411
Managing Things and Links with an Associative Data Model 412
Storing Attributes as Name-Value Pairs 415
Storing Transaction Data in a Lightly Dimensionalized Format 416
Managing Hyper Generalized Data in HGF Requires an Automation Tool 417
HGF Enables Model-Driven Development and Fast Deliveries 418
Eliminating Most Logical and Physical Data Modeling 418
Controlling the EDW Design from a Business Model Diagram 418
Driving Design Changes Using a Business Model 420
Loading Data into the Hyper Generalized Integration Layer 421
Loading the Dimensional Objects 421
Loading the Transactional Objects 422
Retrieving Information from a Hyper Generalized EDW 423
HGF Systems Maintain a Performance Sublayer 423
Performance Layer Objects Enable Business-Intelligible Data Retrieval 424
Model-Driven Evolution and Fast Adaptation 426
Impact of Model Changes on Existing Data 426
Hyper Generalization Tools Facilitate Data Conversions 427
Supporting Derived Elements 428
Value-Added Loops 428
Model-Driven Master Data Components 429
Addressing Performance Concerns 433
Demonstrating Agility Through Four Change Cases 434
Change Case 1: Upgrading Attributes to Entities 434
Change Case 2: Consolidating Entities into the Party Model 437
Change Case 3: New Trigger for a Slowly Changing Dimension 440
Change Case 4: Increasing the Grain of a Fact Table 441
Recap of Change Case Findings 444
HGF-Powered Agile Solutions 445
Easier Backfills for Surface Solutions 446
Evidence of Success 447
Case History 1: Model-Driven Development in Pharmaceuticals 447
Case History 2: Hyper Generalized Data Warehousing in Specialty Retail 448
Hyper Generalized Tools Offer a Better Way to Work 449
Hyper Generalized Tools Mitigate Risk 449
Hyper Generalized Tools Allow Better Use of Resources 450
Hyper Generalized Tools Deliver a Better Warehouse 450
Barriers to Wider Adoption 450
Summary 451
Part IV References 452
V. Agile EDW Quality Management Planning 454
16 Why We Test and What Tests to Run 456
Why Test? 457
Testing Keeps Agile Teams from Cutting Corners 457
Testing Keeps Root Cause Analysis Manageable 458
Testing Integrates Teamwork Across the Pipeline 459
Testing Leads to Better Requirements 459
Testing Makes Real Progress Visible to Everyone 459
An Agile Approach to Quality Assurance 460
Striving for Balance 460
Keeping Quality Assurance “Agile” 461
It is Collaborative, with High Customer Involvement 462
It is Iterative and Incremental 462
It Embodies the 80/20 Rule 462
It Relies on Self-Organization 462
It is Highly Transparent 462
Its Artifacts are Lightweight 462
It is Stereoscopic 462
It Retains QA Practices Already Included in the Base Agile Method 463
Extending Test-Led Development Far Above Unit Testing 463
“What to Test?” Answered with Top-Down Planning 464
The Six Dimensions of DW/BI Testing 464
Preliminary Definitions 466
Dimension 1: Planning 467
Dimension 2: System 468
Dimension 3: Functional 470
Dimension 4: Polarity 470
Dimension 5: Time Frame 471
Dimension 6: Point-of-View 471
A 2×2 Planning Matrix for Top-Down Test Selection 472
A Framework for Assessing a QA Plan’s Coverage 472
Linking Test Planning to Requirements and Risk Management 474
“What to Test?” Answered Bottom-Up 475
Data Warehousing Testing Techniques 475
Referential Integrity Test 476
Data Corners 476
Reconciliation 477
Examples and Expected Results 477
Traditional Application Testing Techniques 477
Equivalence Class Partitioning 478
Boundary Value Analysis (“Edge Testing”) 478
Combinatorial Reduction 479
Agile-Specific Test Techniques 480
Epic Stack Testing 480
Exploratory Testing 481
An Easy-to-Follow Test Technique Matrix for Low-Level Validations 482
Reusable Test Widgets 483
Test Cases Roll Forward Along the System Dimension 484
Testing for Convergence 484
Summary 487
17 Designating Who, When, and Where 488
Who Shall Write the Tests? 488
A Framework for Understanding Who Must Do What 489
Using the Classic V-Model for Analyzing QA Responsibilities 489
Adapted V-Model for Agile DW/BI Test Cases 490
Communicating the QA Assignments 491
Using the 2×2 Top-Down Matrix 492
Using a More Detailed Grid 492
One-Up, One-Down Validation Can Save Time 492
When Should Teammates Perform Their QA Duties? 494
Quality Activities Within an Iteration Cycle 495
Quality Duties at the End of a Release Cycle 497
Where Should Teammates Perform Their qa Duties? 499
Distributing Test Activities Across Environments 499
Distributing Test Techniques Across Environments 500
Key Quality Responsibilities by Team Role 501
Guiding the Team to Self-Organized Quality Planning 501
Suggested Quality Duties by Role 502
Product Owner 503
Project Architect 503
Data Modeler 503
Systems Analyst 503
Programmers and Programming Leads 504
Scrum Master 504
The Overarching Duties of the System Tester 504
Certifying the User Demo’s Data 505
How Many Testers are Needed? 506
Summary 507
18 Deciding How to Execute the Test Cases 508
Good Agile Quality Plans Involve Numerous Test Executions 508
Alternatives to Sufficient Testing Unattractive 511
Test Fewer Items 511
Test Only a Few Times 511
Build a Smaller Scope 511
Facing Up to Test Automation 512
Step 1: Update the Top-Down Plan 513
Step 2: Start Building the Parameter-Driven Widgets 513
Step 3: Plan Out the Test Data Sets 513
Identifying How Many Data Sets are Required 515
Planning to Create Dozens of Data Sets 516
Must Subset Production Data 516
Data Will Need to be Re-Created 517
The Refresh Must Leave the Data Set Unchanged as Much as Possible 517
Must Use Repeatable Masking 517
Planning Storage for Dozens of Data Sets 518
Planning also for Expected Results 518
Step 4: Implement the Engine, Whether Manual or Automated 518
Defining Test Scenarios 520
Step 5: Define the Project’s Set of Testing Aspects 520
Step 6: Build and Populate the Test Data Repository 521
Step 7: Quantify the Testing Objectives 522
Step 8: Begin Creating Test Cases 524
Step 9: Start Up the Engine 524
Step 10: Visualize Project Progress with Quality Assurance 525
Tests Implemented by Environment 525
Connect Top-Down and Bottom-Up Quality Planning 527
Defects Over Time 527
Current Iteration Burndown Chart 527
Step 11: Document the Team’s Success 528
Summary 529
Part V References 530
VI. Integrating the Pieces of the Agile EDW Method 532
19 The Agile EDW Subrelease Cycle 534
Making the Release Cycle a Repeatable Process 534
Traditional Notions of Data Governance 535
A Life Cycle for Data Governance 536
Data Governance Actions for the EDW Team 539
Machine-Assisted Data Governance for the Subrelease Cycle 540
The Agile EDW Subrelease Value Cycle 541
The Fast Requirements Portion of the Subrelease Cycle 542
Step 1: Workflow-Driven Data Governance and Prototyping 542
Step 2: Associative Data Discovery 542
Step 3: Collaborative Source-to-Target Mapping 543
Step 4: Live Data Prototyping 543
The Fast Delivery Portion of the Subrelease Cycle 543
Step 5: Hyper-Modeled Key Integration Points 543
Step 6: Enriched Hyper-Modeled Solution 544
Step 7: Collaborative Analytics 544
Step 8: Model-Driven Solutions 544
Centering the Value Cycle on Data Governance and Quality 545
Deepening the Support for Data Governance 545
Achieving World-Class Quality Assurance 546
Guiding the Agile EDW Transition 546
The DW/BI customer’s Bill of Rights 547
The Need for a Business-Side Project Architect 548
Toward an Agile EDW Manifesto 549
Prompt, Sponsor-Appreciated Results Over Technical Perfection 549
Evolving Data Over Iterating on Application Code 550
Managing Risk Over Eliminating Uncertainty 550
Appropriate Technology Over Maintaining an Infrastructural Monoculture 551
Summary 551
Part VI References 552
Index 554
Back Cover 563
List of Figures
| Figure 1.1 | The negative feedback loop present in most traditionally managed projects. | 2 |
| Figure 1.2 | The five major components to agile enterprise data warehousing. | 4 |
| Figure 1.3 | Agile EDW practices switch projects to a positive feedback loop. | 4 |
| Figure 1.4 | How a team might acquire agile EDW techniques working from the inside out. | 8 |
| Figure 2.1 | Mind map of generic iterative methods summarized in Chapters 2 and 3. | 14 |
| Figure 2.2 | A family tree of methods and influences leading to the agile EDW method. | 15 |
| Figure 2.3 | The traditional waterfall method. | 17 |
| Figure 2.4 | The Agile manifesto cover page. | 17 |
| Figure 2.5 | Values and principles of the agile manifesto and Extreme Programming. | 18 |
| Figure 2.6 | The essence of the Scrum method. | 21 |
| Figure 2.7 | Typical user story. | 22 |
| Figure 2.8 | A sample Scrum task board as it would appear in mid-iteration. | 24 |
| Figure 2.9 | A Scrum burndown chart as it would appear in mid-iteration. | 25 |
| Figure 3.1 | Lean values, principles, and tools. | 33 |
| Figure 3.2 | Value-stream analysis of development work for a challenged waterfall project. | 34 |
| Figure 3.3 | Typical Kanban work board. | 42 |
| Figure 3.4 | Kanban-style cumulative flow diagram. | 43 |
| Figure 3.5 | Sample cycle time distribution analysis for a Kanban team. | 44 |
| Figure 3.6 | Typical stages of “Scrumban”—the transition from Scrum to Kanban. | 48 |
| Figure 3.7 | Two-tiered Scrumban task board. | 48 |
| Figure 3.8 | Values and principles of the Rational Unified Process. | 50 |
| Figure 3.9 | RUP Whale Chart. | 51 |
| Figure 3.10 | Google Ngram of “Scrum” and “RUP” through 2008. | 53 |
| Figure 4.1 | Business organizational terms used in this book. | 64 |
| Figure 4.2 | Business conceptual model. | 72 |
| Figure 4.3 | Logical data model. | 72 |
| Figure 4.4 | Physical data model. | 73 |
| Figure 4.5 | Sample enterprise data warehouse “reference architecture”. | 75 |
| Figure 4.6 | Zachman framework adapted for an enterprise data warehousing program. | 77 |
| Figure 4.7 | DAMA’s framework for data management functions. | 78 |
| Figure 4.8 | Hammergren’s matrix for sequencing DW/BI development work. | 79 |
| Figure 5.1 | Typical RUP-style whale chart for an agile EDW project. | 91 |
| Figure 5.2 | Agile EDW user stories result in too many developer stories for one, short Iteration. | 92 |
| Figure 5.3 | Deriving developer stories from user stories. | 94 |
| Figure 5.4 | A “current estimate” for an agile data warehousing project. | 96 |
| Figure 5.5 | Agile data warehousing requires pipelined work specialties. | 99 |
| Figure 5.6 | Work packages tend to flow diagonally across technical specialties and iterations. | 101 |
| Figure 5.7 | Cycle time distribution analysis for an agile data warehousing project. | 103 |
| Figure 5.8 | A current estimate adjusted for observed delivery cycle times. | 104 |
| Figure 5.9 | Success rates for agile data warehousing teams, by number of agile projects completed, compared to traditional methods. | 105 |
| Figure 5.10 | Agile’s impact upon key performance indicators for data warehousing development projects. | 105 |
| Figure 5.11 | Agile data warehousing surveys indicate that practitioners have overcome some challenge areas. | 107 |
| Figure 6.1 | Relative cost of correcting defects grows by 100 between requirements and promotion into production. | 112 |
| Figure 6.2 | Incremental delivery mitigates risk by increasing the number of product check points. | 113 |
| Figure 6.3 | The sources of EDW project risk mitigated with three types of iterations. | 115 |
| Figure 6.4 | Relative timing for the three types of iterations that Agile EDW employs. | 118 |
| Figure 7.1 | Mind map of topics addressed in Part III. | 126 |
| Figure 7.2 | Sample EDW requirements expressed at three levels. | 127 |
| Figure 7.3 | Waterfall-style requirements management. | 131 |
| Figure 7.4 | Typical requirements work breakdown for a traditional project. | 133 |
| Figure 7.5 | As-is business process diagram showing a sample work flow requiring re-engineering. | 135 |
| Figure 7.6 | To-be business process re-engineered to use EDW to communicate between agents. | 136 |
| Figure 7.7 | Accuracy vs. precision. | 137 |
| Figure 7.8 | Standard risk analysis. | 140 |
| Figure 7.9 | Standard analysis adjusted for dollar value of each type of risk. | 141 |
| Figure 7.10 | Agile EDW’s requirements management benefits greatly from intersecting value chains. | 146 |
| Figure 7.11 | Overall agile EDW requirements management plan. | 148 |
| Figure 7.12 | Enterprise requirements management roles. | 150 |
| Figure 8.1 | Big picture – decomposing epics into a backlog of stories. | 152 |
| Figure 8.2 | Immediate business stakeholder formalizing all levels of stories by linking them to the hierarchy among business stakeholders. | 153 |
| Figure 8.3 | Primary technique for decomposing user stories into developers stories. Note the 25-to-1 multiplier for this project’s user story. | 158 |
| Figure 8.4 | INVEST and DILBERT’S test. | 158 |
| Figure 8.5 | Big picture – recompling modules for perceived value. | 160 |
| Figure 8.6 | Value build-up charts distingishing between delivery environments. | 166 |
| Figure 9.1 | User modeling example. | 171 |
| Figure 9.2 | Business... |
| Erscheint lt. Verlag | 22.9.2015 |
|---|---|
| Sprache | englisch |
| Themenwelt | Informatik ► Datenbanken ► Data Warehouse / Data Mining |
| Informatik ► Office Programme ► Outlook | |
| Mathematik / Informatik ► Informatik ► Software Entwicklung | |
| Sozialwissenschaften ► Kommunikation / Medien ► Buchhandel / Bibliothekswesen | |
| ISBN-10 | 0-12-396518-7 / 0123965187 |
| ISBN-13 | 978-0-12-396518-9 / 9780123965189 |
| Haben Sie eine Frage zum Produkt? |
PDF (Adobe DRM)Größe: 30,1 MB
Kopierschutz: Adobe-DRM
Adobe-DRM ist ein Kopierschutz, der das eBook vor Mißbrauch schützen soll. Dabei wird das eBook bereits beim Download auf Ihre persönliche Adobe-ID autorisiert. Lesen können Sie das eBook dann nur auf den Geräten, welche ebenfalls auf Ihre Adobe-ID registriert sind.
Details zum Adobe-DRM
Dateiformat: PDF (Portable Document Format)
Mit einem festen Seitenlayout eignet sich die PDF besonders für Fachbücher mit Spalten, Tabellen und Abbildungen. Eine PDF kann auf fast allen Geräten angezeigt werden, ist aber für kleine Displays (Smartphone, eReader) nur eingeschränkt geeignet.
Systemvoraussetzungen:
PC/Mac: Mit einem PC oder Mac können Sie dieses eBook lesen. Sie benötigen eine
eReader: Dieses eBook kann mit (fast) allen eBook-Readern gelesen werden. Mit dem amazon-Kindle ist es aber nicht kompatibel.
Smartphone/Tablet: Egal ob Apple oder Android, dieses eBook können Sie lesen. Sie benötigen eine
Geräteliste und zusätzliche Hinweise
Zusätzliches Feature: Online Lesen
Dieses eBook können Sie zusätzlich zum Download auch online im Webbrowser lesen.
Buying eBooks from abroad
For tax law reasons we can sell eBooks just within Germany and Switzerland. Regrettably we cannot fulfill eBook-orders from other countries.
EPUB (Adobe DRM)Größe: 20,6 MB
Kopierschutz: Adobe-DRM
Adobe-DRM ist ein Kopierschutz, der das eBook vor Mißbrauch schützen soll. Dabei wird das eBook bereits beim Download auf Ihre persönliche Adobe-ID autorisiert. Lesen können Sie das eBook dann nur auf den Geräten, welche ebenfalls auf Ihre Adobe-ID registriert sind.
Details zum Adobe-DRM
Dateiformat: EPUB (Electronic Publication)
EPUB ist ein offener Standard für eBooks und eignet sich besonders zur Darstellung von Belletristik und Sachbüchern. Der Fließtext wird dynamisch an die Display- und Schriftgröße angepasst. Auch für mobile Lesegeräte ist EPUB daher gut geeignet.
Systemvoraussetzungen:
PC/Mac: Mit einem PC oder Mac können Sie dieses eBook lesen. Sie benötigen eine
eReader: Dieses eBook kann mit (fast) allen eBook-Readern gelesen werden. Mit dem amazon-Kindle ist es aber nicht kompatibel.
Smartphone/Tablet: Egal ob Apple oder Android, dieses eBook können Sie lesen. Sie benötigen eine
Geräteliste und zusätzliche Hinweise
Zusätzliches Feature: Online Lesen
Dieses eBook können Sie zusätzlich zum Download auch online im Webbrowser lesen.
Buying eBooks from abroad
For tax law reasons we can sell eBooks just within Germany and Switzerland. Regrettably we cannot fulfill eBook-orders from other countries.
aus dem Bereich
