Thick 2D Relations for Document Understanding Marco Aiello *,1 , Arnold M.W. Smeulders Intelligent Sensory Information Systems, University of Amsterdam, Kruislaan 403, 1098 SJ Amsterdam, The Netherlands Abstract We use a propositional language of qualitative rectangle relations to detect the read- ing order from document images. To this end, we define the notion of a document encoding rule and we analyze possible formalisms to express document encoding rules such as L A T E X and SGML. Document encoding rules expressed in the proposi- tional language of rectangles are used to build a reading order detector for document images. In order to achieve robustness and avoid brittleness when applying the sys- tem to real life document images, the notion of a thick boundary interpretation for a qualitative relation is introduced. The framework is tested on a collection of heterogeneous document images showing recall rates up to 89%. Key words: document image analysis, document understanding, spatial reasoning, bidimensional Allen relations, constraint satisfaction: applications 1 Introduction When Dave placed his own drawing in front of the ‘eye’ of HAL—in 2001: A Space Odyssey—HAL showed to have correctly comprehended and interpreted the sketch. “That’s Dr. Hunter, isn’t it?” (Rosenfeld, 1997). But what would have happened if Dave used the first page of a newspaper in front of the eye * Corresponding author. Email addresses: aiellom@ieee.org (Marco Aiello), smeulder@science.uva.nl (Arnold M.W. Smeulders). URLs: http://www.aiellom.it (Marco Aiello), http://www.science.uva.nl/~smeulders (Arnold M.W. Smeulders). 1 Also, Institute for Logic, Language and Computation, University of Amsterdam, Plantage Muidergracht 24, 1018 TV Amsterdam, The Netherlands Preprint submitted to Elsevier Science 26 March 2002