By Calvin Biesecker
A second round of testing that was completed on next-generation radiation portal monitors falls short in terms of the amount of testing needed to generate a statistically reliable sample and should not be used to support a production decision by the Secretary of Homeland Security, the Government Accountability Office (GAO) says in a new report released last week.
The second round of tests, called Phase III, were originally intended to improve the performance the software algorithms in the Advanced Spectroscopic Portals (ASP), the GAO says, citing statements by the Domestic Nuclear Detection Office (DNDO). DNDO is the branch of the Department of Homeland Security charged with research, development, test and evaluation of radiological and nuclear detection equipment.
However, according to the GAO report, a year ago DNDO switched course and said that the Phase III tests, combined with other test results, could be used in making a determination about whether the ASP systems represent a significant increase in operational effectiveness over current deployed systems. If so, then would allow DHS to move forward in awarding production contracts for ASPs.
In July 2006 DHS awarded a potential $1.2 billion combined in ASP development awards to Canberra Industries, a division of France’s AREVA Group, Raytheon [RTN], and Thermo Fisher Scientific [TMO]. Whereas current radiation portal monitors cannot discriminate between naturally occurring and potentially threatening sources of radiation in a cargo shipment, the ASPs are supposed to be able to do that as well as identify the source material. That greater capability of the ASPs is supposed to then lead to lower false alarm rates.
According to the GAO report, Combating Nuclear Smuggling: DHS’s Phase III Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results (GAO-08-979), DNDO didn’t run enough tests in Phase III to gain the precision necessary to judge ASP performance.
“For example, the [Phase III] report narrative states in one instance that an ASP could identify a source material during a specific test 50 percent of the time,” GAO says. “However, the narrative does not disclose that, given the limited number of test runs, DNDO can only estimate that the ASP would be able to correctly identify the source from about 15 percent to about 85 percent of the time–a result that lacks the precision implied by DNDO’s narrative.”
Given the GAO’s new report, Rep. John Dingell (D-Mich.), chairman of the House Energy and Commerce Committee, said in a statement last Thursday, “We still don’t know if this extremely expensive technology works any better than the current equipment.” Dingell’s committee requested the report.
According to GAO, DNDO says the Phase III test results show that the ASP systems are better than current radiation portal monitors at detecting low levels of radiation, that the performance of all three contractors’ systems was basically similar for different categories of radiological source material, and that they also performed much better than handheld radioisotope identifiers in identifying specific radiological or nuclear source material inside containers.
DHS, responding in August to a draft of the GAO report, says it “strongly disagrees” with the GAO. DHS acknowledges that the Phase III tests were not originally designed to support certification of the ASPs but that it ultimately realized the tests results should be considered toward that end. DHS also says it was upfront that the Phase III test program plan was limited to fewer “trials over a larger range of objects” to get more variation among the test cases.
GAO has previously stated that it believes the Phase I ASP tests done by DNDO in early 2007 used biased test methods that allowed the companies to adjust their systems to detect the source materials (Defense Daily, Sept. 19, 2007).
DHS says that the testing done so far is just part of an ongoing series leading to certification and production of the ASPs. Additional studies and testing are underway, including system qualification testing to verify compliance with system performance specifications as well as operational test and evaluation activities, DHS says.