10.4 Some model experimental  10. Experimental Design  10.2 Components in experimental

10.3 Exploratory vs confirmatory experiments

Most experimental work in atmospheric science is exploratory in nature, and formal confirmatory experiments are seldom conducted. (An exception is in weather modification, where some confirmatory experiments have been attempted.)  Confirmatory experiments are those that replicate an earlier experiment with specific statements of hypotheses, tests, measurement and analysis strategies, and expected outcome. Such experiments are especially important when the results of an exploratory experiment provide marginal statistical support for a hypothesis that arose during the analysis of the data. A confirmatory experiment usually tests a single hypothesis (e.g., that cloud seeding increases precipitation) vs the null hypothesis (e.g., that no increase in precipitation results from seeding) under fully specified experimental conditions.

An important reason for confirmatory experiments is that exploratory analyses risk the danger of multiplicity, the increased likelihood that some effects will appear to meet tests of statistical significance simply because many possibilities are considered. If 20 classifications of data are considered, on the average one will appear to meet a test of being expected only 5% of the time, so there is a danger that apparently significant results can result by chance when the possibilities are not limited in advance. A value of confirmatory experiments is that they minimize this danger by testing a specific hypothesis stated in advance.

Another argument for confirmatory experiments is that it is often difficult to know the underlying probability function governing meteorological events, which are often highly correlated in time and hence difficult to use with some approaches to statistical inference. Confirmatory experiments do not eliminate this problem but can reduce it when used with different experimental populations. In truly exploratory experiments, where a large number of possible relationships are considered in the data, there is often value in reserving a portion of the data for confirmatory use before undertaking the analysis. Once trends are identified in a portion of the data, the remainder can be used to test for these same trends in an independent dataset.

   
 10.4 Some model experimental  10. Experimental Design  10.2 Components in experimental