Evaluation Metrics for Nuclear Forensics Search Fredric C Gey Electra Sutton University of California, Berkeley Inst. for Study of Societal Issues Berkeley, CA 94720-5670 +1-510-292-8421 {gey, electra}@berkeley.edu Charles Wang Ray R Larson University of California, Berkeley Information School Berkeley, CA 94720-4600 +1-510-642-6046 {charleswang, ray}@ischool.berkeley.edu Chloe J Reynolds University of California, Los Angeles IT Services | 3327 Murphy Hall Los Angeles, CA 90095-1434 +1-310-206-1621 creynolds@it.ucla.edu ABSTRACT Nuclear forensics search is an emerging sub-field of scientific search: Nuclear forensics plays an important technical role in international security. Nuclear forensic search is grounded in the science of nuclear isotope decay and the rigor of nuclear engineering. However two aspects are far from determined: Firstly, what matching formulae should be used to match between unknown (e.g. smuggled) nuclear samples and libraries of analyzed nuclear samples of known origin? Secondly, what is the appropriate evaluation measure to be applied to assess the effectiveness of search? Using a database of spent nuclear fuel samples we formulated a search experiment to try to identify the particular nuclear reactor from which an unknown sample might have came. This paper describes the experiment and also compares alternative evaluation metrics (precision at 1, 5 and 10 and mean reciprocal rank) used to judge search success Categories and Subject Descriptors H.3.3 [Information Systems]: Information Search and Retrieval—retrieval models, search process. General Terms Experimentation, Performance, Measurement Keywords Nuclear Forensics, Evaluation Metrics, Scientific Search. 1. INTRODUCTION According to [Mayer, Wallenius and Fanghänel 2007] “Since the beginning of the 1990s, when the first seizures of nuclear material were reported, the IAEA (International Atomic Energy Agency) recorded more than 800 cases of illicit trafficking of nuclear or other radioactive materials.” Security agencies worldwide continue to work to prevent nuclear terrorist incidents from happening. The two aspects of prevention are detection and forensics. Millions of dollars are being spent on improvement of devices to detect contraband radioactive material which might be hidden in shipping containers. The flip side of detection is forensics – if a significant amount of smuggled nuclear material is seized, can its origin be traced to both track down the would-be terrorists and to prevent further smuggling activities [IAEA 2002, APS/AAAS 2008, GAO 2009]. To do this, a seized sample can be analyzed to ascertain its “nuclear signature” which can be compared to an archived digital library of nuclear signatures which have been abstracted by radio-chemical analysis of a large number (tens of thousands) of nuclear samples from uranium mines or nuclear reactors worldwide. 2. NUCLEAR FORENSICS SEARCH Given a nuclear sample obtained from whatever process (interdiction, for example), the problem is to identify its source. Such identification requires clues to match against a dataset of samples for which sources and compositions have been identified. The process, abstractly, is not that different from matching fingerprints or DNA samples from a crime scene – both require a library against which the match will be made, and both require specialized matching technologies which execute the search. In the case of nuclear forensics, the library will consist of radioactive samples and their digital signatures obtained by radiochemical analyses. For the example of nuclear weapons grade material, the commonly found isotopes are highly enriched uranium (>90% 235 U) and plutonium (~93% 239 Pu). 1 The signatures of both isotopes can be characterized by their daughter isotope production from the nuclear decay process. [Gey et al 2012] describes the general search process as a temporal directed graph matching problem. In that paper and our experiments so far, temporal effects have been ignored. This is not unreasonable considering the half life of 235 U is 704 million years and of 239 Pu is 24,100 years. 3. NUCLEAR FORENSICS DATA 3.1 Spent Fuel Rod Measurements SFCOMPO is a database of spent nuclear fuel (fuel rods from a nuclear reactor after the energy has been extracted by the nuclear fission process) measurements. The data has been carefully vetted and deemed reliable by nuclear engineering experts and has been released to the public via the Organization for Economic Cooperation Nuclear Energy Agency (OECD-NEA) web site. 2 The process by which the samples are measured (the geometry of where the sample has been drilled and extracted from the fuel rod) is described on the web site. The data consists of 274 samples from 14 nuclear reactors (some no longer in operation) in four countries (Germany, Italy, Japan and the USA). There are a variable number of samples from each reactor, ranging from two for the Genkai-1 reactor in Japan to 39 for the Trino Vercellese reactor in Italy. Each sample has a variable number of isotope, isotope ratio and burn-up measurements, ranging from one measurement for Europium 155 ( 155 Eu) to 261 measurements (a measurement is found in almost all samples) for two Uranium ratios ( 235 U/ 238 U and 236 U/ 238 U). The total number of measurements is 10,339. 1 http://en.wikipedia.org/wiki/Weapons-grade#Weapons- grade_plutonium 2 http://www.oecd-nea.org/sfcompo/ 7KH)LIWK,QWHUQDWLRQDO:RUNVKRSRQ(YDOXDWLQJ,QIRUPDWLRQ$FFHVV(9,$-XQH7RN\R-DSDQ 