As the House and Senate armed services committees wade deeper into their acquisition reform effort, they intend to take a close look at not just the contracting officers and program managers that oversee weapons systems development, but anyone else in a position to slow down the acquisition process. And the first target of the increased oversight: the office of the director of operational test and evaluation.
The House Armed Services Committee (HASC) wrote in its version of the fiscal year 2015 defense authorization bill, “the Director shall consider the potential for increases in program cost estimates or delays in schedule estimates in the implementation of policies, procedures, and activities related to operational test and evaluation and shall take appropriate action to ensure that operational test and evaluation activities do not unnecessarily increase program costs or impede program schedules.”
The idea behind this directive, according to a HASC staffer, is not to discredit the work DOT&E performs but rather to make clear that all parties will be held responsible for their roles in holding up acquisition.
But Director of Operational Test and Evaluation J. Michael Gilmore, asked if his work does in fact “unnecessarily increase program costs or impede program schedules,” stated “no, and demonstrably it doesn’t.”
The Language
Both the HASC and the SASC included language in their FY ’15 defense bills. The House, in addition to its directive to Gilmore to ensure no unnecessary cost and schedule impacts, calls for a Government Accountability Office investigation into the impact DOT&E has on cost and schedule.
SASC calls for a similar report, asking that “not later than March 31, 2015, the Comptroller General of the United States shall submit to the congressional defense committees a report on disputes between the Office of the Director, Operational Test and Evaluation and the acquisition community over testing requirements for major weapon systems.”
The report will look at whether those tensions have been the result of efforts to reduce testing or increase testing beyond the level “necessary to demonstrate (A) compliance with program requirements validated by the Joint Requirements Oversight Council; and (B) effectiveness and suitability for combat.”
Beyond KPPs
The notion of testing beyond the requirements is a big sticking point among those involved in program development. Gilmore, who spoke to Defense Daily Aug. 26 at the Pentagon, rattled off a long list of programs whose key performance parameters (KPPs) and key system attributes (KSAs) had little to nothing to do with its combat mission. The Navy’s P-8, for example, has KPPs dealing with the number of sonobuoys it can hold, its loiter time, its endurance and more–but nothing to do with its capability to find and destroy submarines. Boeing [BA] is the prime contractor for the P-8.
Gilmore said repeatedly that it was both the law and common sense for him to test to realistic combat conditions, regardless of the KPPs.
“We use the entire requirements document, including the key performance parameters,” he said, along with the services’ concepts of operations, war plans, system threat assessment reports and other documents outlining how a weapon would be used in a real combat environment.
In the case of the P-8, Gilmore was so dismayed with the KPPs that he wrote to the vice chairman of the Joint Chiefs of Staff, Adm. James Winnefeld, last September to raise the issue.
Winnefeld wrote back a month later to say “I share your concern regarding the provided examples of threshold KPP/KSA value selection and their inability to effective contribute to operational effectiveness assessments. These disconnects, as you accurately point out, inject unnecessary confusion into the process and are often the genesis of lengthy, costly program reviews and delays.”
Winnefeld added that clarifying language would be added to the Joint Capabilities Integration and Development System guidance documents to ensure better requirements definition.
Gilmore said he hopes this effort will help, noting “I don’t want the testing to be any more time consuming or resource intensive than it has to be, but we have to test under realistic combat conditions.”
“In this office, and I know it’s true of all my predecessors…none of us have regarded this as anything approaching a bureaucratic exercise,” Gilmore said. “This is not developing another document that’s going to go on a shelf somewhere after staffs up here in the Office of the Secretary of Defense look at it and essentially no one else, to collect dust. What we are doing is trying to assure that the people that have to depend on these systems in combat know fully well what they can and cannot do. And that’s a very serious business–it’s not a bureaucratic game, it’s not a matter of checking boxes on some list that some bureaucrat generated.
“These systems that we develop cost multiple tens of billions of dollars, and in the case of joint Strike Fighter nearly $400 billion just to buy it, let alone operate it. So that’s just one reason this is an important business. But to me, the most important reason is, the most important reason we do operational testing under realistic combat conditions, is again to assure the men and women that have to use these systems that they really understand what they can and can’t do. Because the worst circumstance I can imagine is a surprise in combat because they did not know what the systems could and couldn’t do.”
More Oversight
The HASC staffer said in an Aug. 28 interview with Defense Daily that the committee may agree with Gilmore’s sentiment and, as part of the acquisition reform process, is looking for better requirements early in a program’s development. But ultimately, the staffer said, it is not Gilmore’s job to judge the KPPs by testing beyond them.
“At the end of the day, I don’t think it’s DOT&E’s job though to tell the services what the requirements are. I think that’s probably a higher level than him,” the staffer said. “I agree there is improvement that needs to be made in the requirements side of the house, but that’s not DOT&E’s determination.”
HASC plans to look into other offices that impact program cost and schedule but is simply starting with this look at DOT&E to get the ball rolling on increasing accountability.
“Why should the [program managers] be the only ones held accountable for cost, schedule and performance?” the staffer said, adding that outside groups might need different metrics for accountability but should somehow still be monitored to ensure they are doing good and not harm for a program. “It’s like you have all these silent partners that aren’t held accountable for their decisions. The intent is not to take away from any of the work DOT&E does, it’s just that when you look at all these other agencies there’s no metrics.”
A key question for lawmakers is perhaps not whether Gilmore should test beyond the KPPs–which Gilmore insists is his legal and moral obligation–but instead how far beyond the KPPs he should go.
“Whenever they decide to do something that’s going to add a couple tests and six months to the program, okay, but go through and justify why it should be adding six months of schedule and not three months of schedule. Or instead of adding five tests why not three. So it’s going through that process, that’s all it is,” the staffer said.
A Way Forward
Gilmore, though unwilling to back down on testing to realistic combat conditions, agreed that budgetary pressures on the Pentagon mean that testing needs to be done smartly and cost-consciously. Among the steps he’s already taken is to encourage program offices to fill in the details of their Test and Evaluation Master Plans sooner, though he’s met some resistance.
“It’s not a matter at all of the services developing their own plans for testing…and then me coming in at the last minute and somehow or another upsetting the apple cart,” he said, noting that his office is involved in planning the testing as early as the program office will allow–“but that typically means approaching or after milestone C.”
Gilmore said he wants to have an operational test plan by the time a program reaches milestone B, though in many cases program managers say they cannot plan testing until they see what the contractors propose.
He also said he hopes programs will move towards more modeling and simulation instead of relying solely on live-fire tests, but that can only happen if the services invest heavily upfront to develop and accredit their models.
In the case of one Navy program, Aegis Baseline 9, program officials crafted their test and evaluation plan around the idea of saving time and money by doing simulated testing on a computer–much like what was previously done by Aegis’ counterpart, the Ship Self-Defense System.
“The Aegis modeling suite is actually not as good as the modeling suite for the Ship Self Defense System,” Gilmore said. “And I engaged with the Navy about that over a year ago, and I wrote another memo, which is classified.. which spelled out the deficiencies of the Aegis modeling suite and why it couldn’t be validated and accredited for use in operational test and evaluation.”
He said the Navy is fixing the problems with the Aegis modeling suite, and “what I anticipate is that in the next couple of years we will be able to validate portions of that modeling suite for operational test purposes.”
Doing so would involve testing the results of the model against results of corresponding live-fire tests, which would take significant time and money, but “once you’ve accredited these models it really does give you a very, very good capability to explore regions of performance that you’ll never be able to explore in live testing,” he said.
DOT&E has accepted simulated testing for the F-22 program in the past via the accredited Air Combat Simulation, and it plans to use simulation for parts of the F-35 and the Ballistic Missile Defense System operational testing.
Overall, Gilmore said he believes he can get the rigorous, realistic testing he needs to feel confident writing a positive test report for a program, and the program managers can minimize time and resources invested in the testing, if they work together early and often. He wants to ensure the right qualities are being measured with the right test plan and the right metrics–but he won’t lower his standards, even with sequestration cuts squeezing programs throughout the military.
“When it comes to sequester…the enemy gets a vote, and the enemy is not responding to sequester,” he said. “So my view is that while I don’t want the testing, the operational testing, to be more time-consuming or resource-intensive than it needs to be, I would emphasize again that we’ve done studies that show the operational tests costs themselves are a very small fraction of overall acquisition costs. It has to be realistic, and I’m always willing to engage in a good technical debate over what realistic tests should constitute.”