From time to time, the term “ecological validity” is thrown around in connection with linguistic research. And you’d think I’d be calloused by now, but no: I’m astounded anew every time someone treats this as something that’s self-evidently desirable (and not, say, as anathema to how most science works).
The term “ecological validity”, which I think has its origins in experimental psychology and sociology, is used in linguistic research as an informal assessment of how well the experimental conditions in a given study reflect the conditions and factors involved in real-world, day-to-day language use. (And before we get too far in: acceptability judgments, including introspective ones, are very much an instance of robust, reliable experimentation.)
Now, if your scientific question is something about how language is used in real-life situations, then by all means, “ecological validity” might be something you should think about.
But suppose what you’re after is the structure of human language. That is, suppose you’re treating human language as a naturally-occurring phenomenon, and you’re interested in uncovering its inner workings. Reason dictates that you should probably steer as far away from “ecological validity” as you possibly can! When some naturally-occurring phenomenon is thought to be a massive interaction effect of many, many independent and interdependent factors, the way sciences typically approach things is by creating highly artificial experimental setups – sometimes strictly thought-experimental, other times carried out – in the hopes of isolating one (or at least a relatively small number) of these many factors. Ask yourself: could you imagine a critique of the Large Hadron Collider on the grounds that the conditions inside it are not “ecologically valid”?
And here’s the thing: linguistic behavior is self-evidently a massive interaction effect, involving working memory, attention, motivation, fatigue, etc. etc. This makes physical phenomena like Brownian motion (wherein one can’t predict the motion of an individual particle) – or, to cite one of Chomsky’s favorite examples, the paths of individual leaves blowing in the wind – look positively simple by comparison. It’s beyond me why anyone would seek to confront this undifferentiated mess head-on.
More concretely: we have every reason to suspect that humans throw all their cognitive resources (or at least those that they can spare in the moment) at whatever task they’re currently faced with. The task of using language is no exception. E.g. do we have a capacity for rote memorization? We sure do! (Once upon a time we used it to memorize phone numbers. Remember that??) Why not make use of it, in those circumstances where rote-memorization can be fruitfully applied to language?1This is why, as I never tire of telling my students, “One rote-learned construction does not a head-final language make.” But since rote-learning is not a linguistic capacity per se, it follows that research into the structure of language itself needs to abstract away from it. So there you go: in real language-use situations, you can probably lean on rote-learned information to some extent. Therefore, research into the structure of language needs to be “ecologically invalid” in at least this sense. (E.g. by using jabberwocky items, or unlikely-to-be-encountered-before combinations of more familiar items.) And rote-memorization is of course but one example of the many ways that “ecological validity” would undermine research into the structure of human language.
And so, the next time someone tells you something like, “That sentence is not the kind of thing anyone would ever say in regular speech!”, you should proudly respond, “Thank you! I too think this is well-designed stimuli for testing what I’m after.”