The Functional Neuroimaging Data Pipeline. Activation patterns obtained from functional neuroimaging studies reflect interactions among the components of a complicated data pipeline of experimental design parameters, and a series of methodological choices including data acquisition, post-acquisition processing and data-analysis model selection.

Measuring prediction error requires estimating model parameters in a training set and using those model parameters to predict the "design matrix" values (e.g., scan state labels such as "activation/baseline") of an independent test set.

Why Optimize The Data Pipeline? As the field of functional neuroimaging matures and becomes more widely used, standardization and optimization of data-analytic approaches, and automated quality control procedures will become increasingly important. For many researchers the generation of a "plausible result" that can be linked to the neuroscientific literature, perhaps through a hypothesis, is often taken as justification of the pipeline choices made providing a systematic bias in the field towards prevailing neuroscientific expectations ( Strother et al., 1995; Skudlarski et al., 1999). We do not advocate ignoring the existing neuroscientific knowledge base, but both its implicit and explicit use needs to be balanced against a concerted effort to independently define and test the validity of the rapidly increasing range of experimental and methodological techniques used in functional neuroimaging. The NPAIRS approach is guided by the rapidly developing field of predictive learning in statistics (Friedman, 1994; Larsen and Hansen, 1997; Ripley, 1998).

NPAIRS Performance Metrics. NPAIRS defines the "validity or quality" of functional neuroimaging results, and the experimental and pipeline choices made in obtaining them, by quantitatively measuring and optimizing their predictive performance in a crossvalidation resampling framework. This is defined as the ability of the pipeline to produce data-analytic model(s) parameters from a training dataset that can accurately predict the values of experimental design parameters (e.g., brain-state labels, performance measures) in an independent test dataset, and also reproduce the activation image parameters between the training and test datasets.

Activation pattern values from both split-half groups are plotted against each other and normalized by their standard deviation (SD) producing the scatter plot above. Projecting onto the PCA major and minor axes and scaling both by the SD of the noise axis provides the z-score scale signal and noise histograms,respectively.

Such validity, defined as prediction accuracy and activation pattern reproducibility in an independent test dataset is not guaranteed by inferential statistical procedures, even when the underlying model assumptions are true, unless we have asymptotically large datasets. The use of prediction and reproducibility metrics for model optimization has been proposed in the functional neuroimaging community ( Lautrup et al., 1995; Strother et al., 1995, 97, 98, 01; Mørch et al., 1996, 97; Mørch, 1998; Hansen et al., 1999; Tegeler et al., 1999; Kustra, 2000; Liow et al., 2000; McKeown, 2000; Ngan et al., 2000; Goutte et al., 2001; Kjems et al., 2001; Kustra and Strother, 2001 ) but has been largely ignored, typically in favor of inferential tests of unknown generality (e.g., Friston, 1998; Petersson et al., 1999a, 1999b ).