Bookstore

Covers experiment planning, execution, analysis, and reporting
 
This single-source resource guides readers in planning and conducting credible experiments for engineering, science, industrial processes, agriculture, and business. The text takes experimenters all the way through conducting a high-impact experiment, from initial conception, through execution of the experiment, to a defensible final report. It prepares the reader to anticipate the choices faced during each stage.
 
Filled with real-world examples from engineering science and industry, Planning and Executing Credible Experiments: A Guidebook for Engineering, Science, Industrial Processes, Agriculture, and Business offers chapters that challenge experimenters at each stage of planning and execution and emphasizes uncertainty analysis as a design tool in addition to its role for reporting results. Tested over decades at Stanford University and internationally, the text employs two powerful, free, open-source software tools: GOSSET to optimize experiment design, and R for statistical computing and graphics. A website accompanies the text, providing additional resources and software downloads.
* A comprehensive guide to experiment planning, execution, and analysis
* Leads from initial conception, through the experiment's launch, to final report
* Prepares the reader to anticipate the choices faced throughout an experiment
* Hones the motivating question
* Employs principles and techniques from Design of Experiments (DoE)
* Selects experiment designs to obtain the most information from fewer experimental runs
* Offers chapters that propose questions that an experimenter will need to ask and answer during each stage of planning and execution
* Demonstrates how uncertainty analysis guides and strengthens each stage
* Includes examples from real-life industrial experiments
* Accompanied by a website hosting open-source software
 
Planning and Executing Credible Experiments is an excellent resource for graduates and senior undergraduates--as well as professionals--across a wide variety of engineering disciplines.

Planning and Executing Credible Experiments

About the Authors xxi





Preface xxiii





Acknowledgments xxvii





About the Companion Website xxix





1 Choosing Credibility 1





1.1 The Responsibility of an Experimentalist 2





1.2 Losses of Credibility 2





1.3 Recovering Credibility 3





1.4 Starting with a Sharp Axe 3





1.5 A Systems View of Experimental Work 4





1.6 In Defense of Being a Generalist 5





Panel 1.1 The Bundt Cake Story 6





References 6





Homework 6





2 The Nature of Experimental Work 7





2.1 Tested Guide of Strategy and Tactics 7





2.2 What Can Be Measured and What Cannot? 8





2.2.1 Examples Not Measurable 8





2.2.2 Shapes 9





2.2.3 Measurable by the Human Sensory System 10





2.2.4 Identifying and Selecting Measurable Factors 11





2.2.5 Intrusive Measurements 11





2.3 Beware Measuring Without Understanding: Warnings from History 12





2.4 How Does Experimental Work Differ from Theory and Analysis? 13





2.4.1 Logical Mode 13





2.4.2 Persistence 13





2.4.3 Resolution 13





2.4.4 Dimensionality 15





2.4.5 Similarity and Dimensional Analysis 15





2.4.6 Listening to Our Theoretician Compatriots 16





Panel 2.1 Positive Consequences of the Reproducibility Crisis 17





Panel 2.2 Selected Invitations to Experimental Research, Insights from Theoreticians 18





Panel 2.3 Prepublishing Your Experiment Plan 21





2.4.7 Surveys and Polls 22





2.5 Uncertainty 23





2.6 Uncertainty Analysis 23





References 24





Homework 25





3 An Overview of Experiment Planning 27





3.1 Steps in an Experimental Plan 27





3.2 Iteration and Refinement 28





3.3 Risk Assessment/Risk Abatement 28





3.4 Questions to Guide Planning of an Experiment 29





Homework 30





4 Identifying the Motivating Question 31





4.1 The Prime Need 31





Panel 4.1 There's a Hole in My Bucket 32





4.2 An Anchor and a Sieve 33





4.3 Identifying the Motivating Question Clarifies Thinking 33





4.3.1 Getting Started 33





4.3.2 Probe and Focus 34





4.4 Three Levels of Questions 35





4.5 Strong Inference 36





4.6 Agree on the Form of an Acceptable Answer 36





4.7 Specify the Allowable Uncertainty 37





4.8 Final Closure 37





Reference 38





Homework 38





5 Choosing the Approach 39





5.1 Laying Groundwork 39





5.2 Experiment Classifications 40





5.2.1 Exploratory 40





5.2.2 Identifying the Important Variables 40





5.2.3 Demonstration of System Performance 41





5.2.4 Testing a Hypothesis 41





5.2.5 Developing Constants for Predetermined Models 41





5.2.6 Custody Transfer and System Performance Certification Tests 42





5.2.7 Quality-Assurance Tests 42





5.2.8 Summary 43





5.3 Real or Simplified Conditions? 43





5.4 Single-Sample or Multiple-Sample? 43





Panel 5.1 A Brief Summary of "Dissertation upon Roast Pig" 44





Panel 5.2 Consider a Spherical Cow 44





5.5 Statistical or Parametric Experiment Design? 45





5.6 Supportive or Refutative? 47





5.7 The Bottom Line 47





References 48





Homework 48





6 Mapping for Safety, Operation, and Results 51





6.1 Construct Multiple Maps to Illustrate and Guide Experiment Plan 51





6.2 Mapping Prior Work and Proposed Work 51





6.3 Mapping the Operable Domain of an Apparatus 53





6.4 Mapping in Operator's Coordinates 57





6.5 Mapping the Response Surface 59





6.5.1 Options for Organizing a Table 59





6.5.2 Options for Presenting the Response on a Scatter-Plot-Type Graph 61





Homework 64





7 Refreshing Statistics 65





7.1 Reviving Key Terms to Quantify Uncertainty 65





7.1.1 Population 65





7.1.2 Sample 66





7.1.3 Central Value 67





7.1.4 Mean, or ? 67





7.1.5 Residual 67





7.1.6 Variance, 2 or S2 68





7.1.7 Degrees of Freedom, Df 68





7.1.8 Standard Deviation, Y or SY 68





7.1.9 Uncertainty of the Mean, 69





7.1.10 Chi?Squared, 2 69





7.1.11 p?Value 70





7.1.12 Null Hypothesis 70





7.1.13 F?value of Fisher Statistic 71





7.2 The Data Distribution Most Commonly Encountered The Normal Distribution for Samples of Infinite Size 71





7.3 Account for Small Samples: The t?Distribution 72





7.4 Construct Simple Models by Computer to Explain the Data 73





7.4.1 Basic Statistical Analysis of Quantitative Data 73





7.4.2 Model Data Containing Categorical and Quantitative Factors 75





7.4.3 Display Data Fit to One Categorical Factor and One Quantitative Factor 76





7.4.4 Quantify How Each Factor Accounts for Variation in the Data 76





7.5 Gain Confidence and Skill at Statistical Modeling Via the R Language 77





7.5.1 Model and Plot Results of a Single Variable Using the Example Data diceshoe.csv 77





7.5.2 Evaluate Alternative Models of the Example Data hiloy.csv 78





7.5.2.1 Inspect the Data 78





7.5.3 Grand Mean 78





7.5.4 Model by Groups: Group?Wise Mean 78





7.5.5 Model by a Quantitative Factor 78





7.5.6 Model by Multiple Quantitative Factors 78





7.5.7 Allow Factors to Interact (So Each Group Gets Its Own Slope) 79





7.5.8 Include Polynomial Factors (a Statistical Linear Model Can Be Curved) 80





7.6 Report Uncertainty 80





7.7 Decrease Uncertainty (Improve Credibility) by Isolating Distinct Groups 81





7.8 Original Data, Summary, and R 82





References 83





Homework 83





8 Exploring Statistical Design of Experiments 87





8.1 Always Seeking Wiser Strategies 87





8.2 Evolving from Novice Experiment Design 87





8.3 Two?Level and Three?Level Factorial Experiment Plans 88





8.4 A Three?Level, Three?Factor Design 89





8.5 The Plackett-Burman 12?Run Screening Design 93





8.6 Details About Analysis of Statistically Designed Experiments 95





8.6.1 Model Main Factors to Original Raw Data 95





8.6.2 Model Main Factors to Original Data Around Center of Each Factor 96





8.6.3 Model Including All Interaction Terms 97





8.6.4 Model Including Only Dominant Interaction Terms 97





8.6.5 Model Including Dominant Interaction Term Plus Quadratic Term 98





8.6.6 Model All Factors of Example 2, Centering Each Quantitative Factor 99





8.6.7 Refine Model of Example 2 Including Only Dominant Terms 100





8.7 Retrospect of Statistical Design Examples 101





8.8 Philosophy of Statistical Design 101





8.9 Statistical Design for Conditions That Challenge Factorial Designs 102





8.10 A Highly Recommended Tool for Statistical Design of Experiments 103





8.11 More Tools for Statistical Design of Experiments 103





8.12 Conclusion 103





Further Reading 104





Homework 104





9 Selecting the Data Points 107





9.1 The Three Categories of Data 107





9.1.1 The Output Data 107





9.1.2 Peripheral Data 108





9.1.3 Backup Data 108





9.1.4 Other Data You May Wish to Acquire 108





9.2 Populating the Operating Volume 109





9.2.1 Locating the Data Points Within the Operating Volume 109





9.2.2 Estimating the Topography of the Response Surface 109





9.3 Example from Velocimetry 109





9.3.1 Sharpen Our Approach 110





9.3.2 Lessons Learned from Velocimetry Example 111





9.4 Organize the Data 112





9.4.1 Keep a Laboratory Notebook 112





9.4.2 Plan for Data Security 112





9.4.3 Decide Data Format 112





9.4.4 Overview Data Guidelines 112





9.4.5 Reasoning Through Data Guidelines 113





9.5 Strategies to Select Next Data Points 114





9.5.1 Overview of Option 1: Default Strategy with Intensive Experimenter Involvement 115





9.5.1.1 Choosing the Data Trajectory 115





9.5.1.2 The Default Strategy: Be Bold 115





9.5.1.3 Anticipate, Check, Course Correct 116





9.5.1.4 Other Aspects to Keep in Mind 116





9.5.1.5 Endpoints 117





9.5.2 Reintroducing Gosset 118





9.5.3 Practice Gosset Examples (from Gosset User Manual) 119





9.6 Demonstrate Gosset for Selecting Data 120





9.6.1 Status Quo of Experiment Planning and Execution (Prior to Selecting More Samples) 120





9.6.1.1 Specified Motivating Question 120





9.6.1.2 Identified Pertinent Candidate Factors 121





9.6.1.3 Selected Initial Sample Points Using Plackett-Burman 121





9.6.1.4 Executed the First 12 Runs at the PB Sample Conditions 122





9.6.1.5 Analyzed Results. Identified Dominant First-Order Factors. Estimated First-Order Uncertainties of Factors 123





9.6.1.6 Generated Draft Predictive Equation 124





9.6.2 Use Gosset to Select Additional Data Samples 125





9.6.2.1 Example Gosset Session: User Input to Select Next Points 125





9.6.2.2 Example Gosset Session: How We Chose User Input 126





9.6.2.3 Example Gosset Session: User Input Along with Gosset Output 128





9.6.2.4 Example Gosset Session: Convert the Gosset Design to Operator Values 131





9.6.2.5 Results of Example Gosset Session: Operator Plots of Total Experiment Plan 132





9.6.2.6 Execute Stage Two of the Experiment Plan: User Plus Gosset Sample Points 132





9.7 Use Gosset to Analyze Results 133





9.8 Other Options and Features of Gosset 133





9.9 Using Gosset to Find Local Extrema in a Function of Several Variables 134





9.10 Summary 137





Further Reading 137





Homework 137





10 Analyzing Measurement Uncertainty 143





10.1 Clarifying Uncertainty Analysis 143





10.1.1 Distinguish Error and Uncertainty 144





10.1.1.1 Single-Sample vs. Multiple-Sample 145





10.1.2 Uncertainty as a Diagnostic Tool 146





10.1.2.1 What Can Uncertainty Analysis Tell You? 146





10.1.2.2 What is Uncertainty Analysis Good For? 148





10.1.2.3 Uncertainty Analysis Can Redirect a Poorly Conceived Experiment 148





10.1.2.4 Uncertainty Analysis Improves the Quality of Your Work 148





10.1.2.5 Slow Sampling and "Randomness" 149





10.1.2.6 Uncertainty Analysis Makes Results Believable 150





10.1.3 Uncertainty Analysis Aids Management Decision-Making 150





10.1.3.1 Management's Task: Dealing with Warranty Issues 150





10.1.4 The Design Group's Task: Setting Tolerances on Performance Test Repeatability 152





10.1.5 The Performance Test Group's Task: Setting the Tolerances on Measurements 152





10.2 Definitions 153





10.2.1 True Value 153





10.2.2 Corrected Value 153





10.2.3 Data Reduction Program 153





10.2.4 Accuracy 153





10.2.5 Error 154





10.2.6 XXXX Error 154





10.2.7 Fixed Error 154





10.2.8 Residual Fixed Error 154





10.2.9 Random Error 154





10.2.10 Variable (but Deterministic) Error 155





10.2.11 Uncertainty 155





10.2.12 Odds 155





10.2.13 Absolute Uncertainty 155





10.2.14 Relative Uncertainty 155





10.3 The Sources and Types of Errors 156





10.3.1 Types of Errors: Fixed, Random, and Variable 156





10.3.2 Sources of Errors: The Measurement Chain 156





10.3.2.1 The Undisturbed Value 158





10.3.2.2 The Available Value 158





10.3.2.3 The Achieved Value 158





10.3.2.4 The Observed Value 159





10.3.2.5 The Corrected Value 159





10.3.3 Specifying the True Value 160





10.3.3.1 If the Achieved Value is Taken as the True Value 160





10.3.3.2 If the Available Value is Taken as the True Value 163





10.3.3.3 If the Undisturbed Value is Taken as the True Value 166





10.3.3.4 If the Mixed Mean Gas Temperature is Taken as the True Value 167





10.3.4 The Role of the End User 167





10.3.4.1 The End-Use Equations Implicitly Define the True Value 167





10.3.5 Calibration 168





10.4 The Basic Mathematics 170





10.4.1 The Root-Sum-Squared (RSS) Combination 170





10.4.2 The Fixed Error in a Measurement 171





10.4.3 The Random Error in a Measurement 172





10.4.4 The Uncertainty in a Measurement 173





10.4.5 The Uncertainty in a Calculated Result 174





10.4.5.1 The Relative Uncertainty in a Result 176





10.5 Automating the Uncertainty Analysis 178





10.5.1 The Mathematical Basis 178





10.5.2 Example of Uncertainty Analysis by Spreadsheet 179





10.6 Single-Sample Uncertainty Analysis 181





10.6.1 Assembling the Necessary Inputs 184





10.6.2 Calculating the Uncertainty in the Result 185





10.6.3 The Three Levels of Uncertainty: Zeroth-, First-, and Nth-Order 185





10.6.3.1 Zeroth-Order Replication 186





10.6.3.2 First-Order Replication 187





10.6.3.3 Nth-Order Replication 188





10.6.4 Fractional-Order Replication for Special Cases 188





10.6.5 Summary of Single-Sample Uncertainty Levels 189





10.6.5.1 Zeroth-Order 189





10.6.5.2 First-Order 190





10.6.5.3 Nth-Order 190





References 190





Further Reading 191





Homework 191





11 Using Uncertainty Analysis in Planning and Execution 197





11.1 Using Uncertainty Analysis in Planning 197





11.1.1 The Physical Situation and Energy Analysis 198





11.1.2 The Steady?State Method 199





11.1.3 The Transient Method 200





11.1.4 Reflecting on Assumptions Made During DRE Derivations 201





11.2 Perform Uncertainty Analysis on the DREs 202





11.2.1 Uncertainty Analysis: General Form 202





11.2.2 Uncertainty Analysis of the Steady?State Method 203





11.2.3 Uncertainty Analysis - Transient Method 204





11.2.4 Compare the Results of Uncertainty Analysis of the Methods 205





11.2.5 What Does the Calculated Uncertainty Interval Mean? 206





11.2.6 Cross?Checking the Experiment 207





11.2.7 Conclusions 207





11.3 Using Uncertainty Analysis in Selecting Instruments 208





11.4 Using Uncertainty Analysis in Debugging an Experiment 209





11.4.1 Handling Overall Scatter 209





11.4.2 Sources of Scatter 210





11.4.3 Advancing Toward Calibration 211





11.4.4 Selecting Thresholds 212





11.4.5 Iterating Analysis 212





11.4.6 Rechecking Situational Uncertainty 212





11.5 Reporting the Uncertainties in an Experiment 213





11.5.1 Progress in Uncertainty Reporting 214





11.6 Multiple?Sample Uncertainty Analysis 214





11.6.1 Revisiting Single?Sample and Multiple?Sample Uncertainty Analysis 214





11.6.2 Examples of Multiple?Sample Uncertainty Analysis 215





11.6.3 Fixed Error and Random Error 216





11.7 Coordinate with Uncertainty Analysis Standards 216





11.7.1 Describing Fixed and Random Errors in a Measurement 217





11.7.2 The Bias Limit 217





11.7.2.1 Fossilization 218





11.7.2.2 Bias Limits 218





11.7.3 The Precision Index 219





11.7.4 The Number of Degrees of Freedom 220





11.8 Describing the Overall Uncertainty in a Single Measurement 220





11.8.1 Adjusting for a Single Measurement 220





11.8.2 Describing the Overall Uncertainty in a Result 221





11.8.3 Adding the Overall Uncertainty to Predictive Models 222





11.9 Additional Statistical Tools and Elements 222





11.9.1 Pooled Variance 222





11.9.1.1 Student's t?Distribution - Pooled Examples 223





11.9.2 Estimating the Standard Deviation of a Population from the Standard Deviation of a Small Sample: The Chi?Squared 2 Distribution 223





References 225





Homework 226





12 Debugging an Experiment, Shakedown, and Validation 231





12.1 Introduction 231





12.2 Classes of Error 231





12.3 Using Time-Series Analysis in Debugging 232





12.4 Examples 232





12.4.1 Gas Temperature Measurement 232





12.4.2 Calibration of a Strain Gauge 233





12.4.3 Lessons Learned from Examples 234





12.5 Process Unsteadiness 234





12.6 The Effect of Time-Constant Mismatching 235





12.7 Using Uncertainty Analysis in Debugging an Experiment 236





12.7.1 Calibration and Repeatability 236





12.7.2 Stability and Baselining 238





12.8 Debugging the Experiment via the Data Interpretation Program 239





12.8.1 Debug the Experiment via the DIP 239





12.8.2 Debug the Interface of the DIP 239





12.8.3 Debug Routines in the DIP 240





12.9 Situational Uncertainty 241





13 Trimming Uncertainty 243





13.1 Focusing on the Goal 243





13.2 A Motivating Question for Industrial Production 243





13.2.1 Agreed Motivating Questions for Industrial Example 244





13.2.2 Quick Answers to Motivating Questions 244





13.2.3 Challenge: Precheck Analysis and Answers 245





13.3 Plackett-Burman 12-Run Results and Motivating Question #3 245





13.4 PB 12-Run Results and Motivating Question #1 247





13.4.1 Building a Predictive Model Equation from R-Language Linear Model 248





13.4.2 Parsing the Dual Predictive Model Equation 249





13.4.3 Uncertainty of the Intercept in the Dual Predictive Model Equation 250





13.4.4 Mapping an Answer to Motivating Question #1 251





13.4.5 Tentative Answers to Motivating Question #1 252





13.5 Uncertainty Analysis of Dual Predictive Model and Motivating Question #2 252





13.5.1 Uncertainty of the Constant in the Dual Predictive Model Equation 252





13.5.2 Uncertainty of Other Factors in the Dual Predictive Model Equation 253





13.5.3 Include All Coefficient Uncertainties in the Dual Predictive Model Equation 254





13.5.4 Overall Uncertainty from All Factors in the Predictive Model Equation 254





13.5.5 Improved Tentative Answers to Motivating Questions, Including Uncertainties 256





13.5.6 Search for Improved Predictive Models 256





13.6 The PB 12-Run Results and Individual Machine Models 256





13.6.1 Individual Machine Predictive Model Equations 258





13.6.2 Uncertainty of the Intercept in the Individual Predictive Model Equations 258





13.6.3 Uncertainty of the Constant in the Individual Predictive Model Equations 259





13.6.4 Uncertainty of Other Factors in the Individual Predictive Model Equation 259





13.6.4.1 Uncertainties of Machine 1 259





13.6.4.2 Uncertainties of Machine 2 260





13.6.4.3 Including Instrument and Measurement Uncertainties 260





13.6.5 Include All Coefficient Uncertainties in the Individual Predictive Model Equations 260





13.6.6 Overall Uncertainty from All Factors in the Individual Predictive Model Equations 261





13.6.7 Quick Overview of Individual Machine Performance Over the Operating Map 262





13.7 Final Answers to All Motivating Questions for the PB Example Experiment 263





13.7.1 Answers to Motivating Question #1 263





13.7.2 Answers to Motivating Question #2 263





13.7.3 Answers to Motivating Question #3 (Expanded from Section 13.3) 263





13.7.4 Answers to Motivating Question #4 264





13.7.5 Other Recommendations (to Our Client) 264





13.8 Conclusions 265





Homework 266





14 Documenting the Experiment: Report Writing 269





14.1 The Logbook 269





14.2 Report Writing 269





14.2.1 Organization of the Reports 270





14.2.2 Who Reads What? 270





14.2.3 Picking a Viewpoint 271





14.2.4 What Goes Where? 271





14.2.4.1 What Goes in the Abstract? 272





14.2.4.2 What Goes in the Foreword? 272





14.2.4.3 What Goes in the Objective? 273





14.2.4.4 What Goes in the Results and Conclusions? 273





14.2.4.5 What Goes in the Discussion? 274





14.2.4.6 References 274





14.2.4.7 Figures 275





14.2.4.8 Tables 276





14.2.4.9 Appendices 276





14.2.5 The Mechanics of Report Writing 276





14.2.6 Clear Language Versus "JARGON" 277





Panel 14.1 The Turbo-Encabulator 278





14.2.7 "Gobbledygook": Structural Jargon 279





Panel 14.2 U.S. Code, Title 18, No. 793 279





14.2.8 Quantitative Writing 281





14.2.8.1 Substantive Versus Descriptive Writing 281





Panel 14.3 The Descriptive Bank Statement 281





14.2.8.2 Zero-Information Statements 281





14.2.8.3 Change 282





14.3 International Organization for Standardization, ISO 9000 and other Standards 282





14.4 Never Forget. Always Remember 282





Appendix A: Distributing Variation and Pooled Variance 283





A.1 Inescapable Distributions 283





A.1.1 The Normal Distribution for Samples of Infinite Size 283





A.1.2 Adjust Normal Distributions with Few Data: The Student's t-Distribution 283





A.2 Other Common Distributions 286





A.3 Pooled Variance (Advanced Topic) 286





Appendix B: Illustrative Tables for Statistical Design 289





B.1 Useful Tables for Statistical Design of Experiments 289





B.1.1 Ready-made Ordering for Randomized Trials 289





B.1.2 Exhausting Sets of Two-Level Factorial Designs ( Five Factors) 289





B.2 The Plackett-Burman (PB) Screening Designs 289





Appendix C: Hand Analysis of a Two-Level Factorial Design 293





C.1 The General Two-Level Factorial Design 293





C.2 Estimating the Significance of the Apparent Factor Effects 298





C.3 Hand Analysis of a Plackett-Burman (PB) 12-Run Design 299





C.4 Illustrative Practice Example for the PB 12-Run Pattern 302





C.4.1 Assignment: Find Factor Effects and the Linear Coefficients Absent Noise 302





C.4.2 Assignment: Find Factor Effects and the Linear Coefficients with Noise 303





C.5 Answer Key: Compare Your Hand Calculations 303





C.5.1 Expected Results Absent Noise (compare C.4.1) 303





C.5.2 Expected Results with Random Gaussian Noise (cf. C.4.2) 304





C.6 Equations for Hand Calculations 305





Appendix D: Free Recommended Software 307





D.1 Instructions to Obtain the R Language for Statistics 307





D.2 Instructions to Obtain LibreOffice 308





D.3 Instructions to Obtain Gosset 308





D.4 Possible Use of RStudio 309





Index 311