>1569347543< CogSIMA 2011 Conference Presentation Submission. 1 Abstract — This paper describes a methodology for incorporating human observations into a hard+soft information fusion process for counterinsurgency intelligence analysis. The goal of incorporating human observations into the information fusion process is important as it extends the ability of the fusion algorithms to associate and merge disparate pieces of information by allowing for information collected from soft data sources (e.g., human observations) to be included in the process along with information collected from hard data sources (e.g., radar sensors). This goal is accomplished through the employment of fuzzy membership functions used in similarity scoring, for data association and situation assessment. These membership functions are based on situationally qualified error characteristics. Error characteristics represent the key to this process by allowing for accurate uncertainty alignment based on the known and/or unknown state of context dependent variables that have been empirically determined to influence the accuracy of human estimation for a given category- in this case human age estimation. Index Terms— Age Estimation; Data Fusion; Fuzzy Membership Functions; Human Observations; Intelligence Analysis; Uncertainty. Manuscript received January 5, 2011. This work was supported Army Research Office MURI grant W911NF-09-1-0392 for “Unified Research on Network-based Hard/Soft Information Fusion”. Michael P. Jenkins is a research assistant at the State University of New York (SUNY) at Buffalo. He received his M.S. in Human Factors in Information Design and MBA from Bentley University in 2008 and is currently pursuing his Ph.D. in human factors engineering at SUNY Buffalo. (phone: 716-645-4704; e-mail: mpj6@buffalo.edu). Geoff Gross is a research assistant at the State University of New York (SUNY) at Buffalo. He is currently pursuing his Ph.D. in operations research at SUNY Buffalo (e-mail: gagross@buffalo.edu). Ann M. Bisantz is an associate professor of industrial and systems engineering at the State University of New York (SUNY) at Buffalo. She received her M.S. degree in industrial engineering from SUNY Buffalo in 1991 and a Ph.D. in industrial and systems engineering at the Georgia Institute of Technology in 1997 (e-mail: bisantz@buffalo.edu). Rakesh Nagi is chair and professor of industrial and systems engineering at the State University of New York (SUNY) at Buffalo. He received his Ph.D. (1991) and M.S. (1989) degrees in Mechanical Engineering from the University of Maryland at College Park, while he worked at the Institute for Systems Research and INRIA, France, and B.E. (1987) degree in Mechanical Engineering from University of Roorkee (now IIT-R), Roorkee, India. I. INTRODUCTION ntelligence analysis (IA) is a context dependent, time sensitive, dynamic and complex series of tasks requiring human analysts to perceive and often predict the future of situations based on multiple sources of information characterized by varied levels of uncertainty. Due to the ever increasing amounts of raw data available for consideration and the often ambiguous requests for information from clients of the IA process, the use of computer systems and intelligent agents to aid in the intelligence analysis process has become more and more common in practice. However, even with the assistance of these systems, analysts often face various problems in trying to make sense of multiple pieces of disparate information coming from the same or different sources, under varying conditions — for example: (i) attributing and assessing uncertainties associated with different information under different conditions [4] [17] [30], (ii) identifying information as relevant to the problem at hand for review or retrieval [1] [2] [3] [25] [29], (iii) inferring relations within and between source data in order to integrate often disparate information artifacts [12] [15] [17] [32], and (iv) identifying states of the world in order to make and valuate predictions of potential future states based on various potential courses of action [1] [18] [32]. Data fusion systems may provide one solution to help alleviate the resulting cognitive load. Data fusion systems combine data from multiple sources, through various statistical and mathematical techniques, to obtain meaningful information not available from one source alone [34]. This automated situation assessment process relies on algorithms which match different pieces of data based on the likelihood that they represent the same or similar events, people, objects, etc. Source data for this situation assessment process can be either soft data (e.g., observations made by humans in the field) or hard data (e.g., satellite imaging data). While hard data sources often have predetermined uncertainty levels known to the system, soft data sources rarely, if ever, come with this meta-information that is required for the situation assessment process. The process of attributing accurate error characteristics to incoming data sources is further complicated because the uncertainty associated with both hard and soft data sources is Towards context-aware hard/soft information fusion: Incorporating situationally qualified human observations into a fusion process for intelligence analysis Michael P. Jenkins, Geoff Gross, Ann M. Bisantz, and Rakesh Nagi I