Show what you know: musings on the reporting of negative results in speech recognition research Hynek Hermansky 1 and Nelson Morgan 2 1 Dalle Molle Institute for Perceptual Artificial Intelligence (IDIAP), Martigny, Switzerland 2 International Computer Science Institute (ICSI), Berkeley, USA WHAT IS A “NEGATIVE RESULT”? In a sense, well-designed experiments never have a completely negative result, since there is always the opportunity to learn something. In fact, unexpected results by definition provide the most information. Conventionally, negative results refer to those that do not support the hypothesis that an experiment has been designed to test; that is, results that are unable to disprove the null hypothesis (e.g., that the distinction between results from novel and baseline approaches can be explained by chance variability). Such a result can certainly be due to many causes, including bugs, and does not by itself confirm any hypothesis. However, learning about negative as well as positive results can be instrumental in providing the context for the development of new hypotheses to be tested. Hearing only about the successes is equivalent to throwing away half of the information. Personally, we have often been more intrigued with reports of significant unexpected failures than with the usual reports of method A being 5% better than baseline method B. Such reports often provide little surprise at all. We hope that the new journal will provide a forum for experimenters who have unexpected results from well-designed experiments. WHITHER SPEECH RECOGNITION More than thirty years ago, John Pierce wrote a short but very strongly worded letter to editor of the Journal of the Acoustical Society of America (JASA) (Pierce 1969). In the letter he questioned research in automatic speech recognition (ASR), which he viewed as more of an art than a science. At that time, the letter did not make him many friends in the field, given its critical perspective. The fury of ASR researchers may have been justified. Given Pierce’s standing in the scientific community and his high-level position in the Bell Labs management hierarchy, the letter had an extremely negative effect. The letter was stark in its criticism, and its tone might have been more appropriate for a private communication rather than for a public forum such as JASA. While the letter and the attitude of its influential author apparently had some significant effects, research and development in ASR eventually recovered. So today we may have the luxury of taking some distance and perhaps even appreciate that many issues that were raised in the letter were valid (Jelinek 1996). Particularly critical was Pierce’s accusation that ASR researchers do not behave like scientists but rather like mad inventors or untrustworthy engineers (sic). To avoid that, his advice was ... If there was no clear experimental evidence, ... (one It seems that Pierce’s advice is still sometimes forgotten. SCIENTIFIC METHOD What is involved in designing a good experiment? Briefly, the scientific method requires: 1. Observe the phenomenon. 2. Form the hypothesis. 3. Make the prediction. 4. Test the hypothesis (run experiment to see whether you prediction is valid).