How you get 244 different ways to measure performance on the same test of decision making. And what it means for the reliability of behavioural science
In complex, high dimensional systems like the human mind, the data required to infer underlying mechanisms from observation alone quickly exceeds what can realistically be collected. The math from complex systems and information theory show that the number of observations needed grows exponentially with dimensions and entropy.
The issue may not be only methodology of empirical testing. Empirical testing by itself may simply be wrong tool for the job of uncovering the mechanics of human mind.
Quite a doctrine of despair. There are whole swathes of psychologists who think they know things about the mechanisms of the human mind because of empirical methods. Maybe not empiricism alone, but that is definitely part of the recipe
A big part of the problem is that classical hypothesis testing promises much more than it can deliver. "Rejecting the null with 99 per cent confidence", even with good practice, means something like "there's more than random noise here, and we should look into it". If you start with a broadly Bayesian view of the world, you can restate this as "I've adjusted my probabilities to give a bit a more support to the hypothesis I wanted to test".
Yes. Accurate scholarship might unearth the whole offence, but certainly inappropriate and inadequate statistical approaches are part the reproducibility crisis
In complex, high dimensional systems like the human mind, the data required to infer underlying mechanisms from observation alone quickly exceeds what can realistically be collected. The math from complex systems and information theory show that the number of observations needed grows exponentially with dimensions and entropy.
The issue may not be only methodology of empirical testing. Empirical testing by itself may simply be wrong tool for the job of uncovering the mechanics of human mind.
Quite a doctrine of despair. There are whole swathes of psychologists who think they know things about the mechanisms of the human mind because of empirical methods. Maybe not empiricism alone, but that is definitely part of the recipe
A big part of the problem is that classical hypothesis testing promises much more than it can deliver. "Rejecting the null with 99 per cent confidence", even with good practice, means something like "there's more than random noise here, and we should look into it". If you start with a broadly Bayesian view of the world, you can restate this as "I've adjusted my probabilities to give a bit a more support to the hypothesis I wanted to test".
Yes. Accurate scholarship might unearth the whole offence, but certainly inappropriate and inadequate statistical approaches are part the reproducibility crisis