Completing a Doctorate—Aboard an Aircraft Carrier

Commander Donald “Bucket” Costello does not recommend working on an engineering doctorate via night school while on active duty.

A career naval aviator, engineering test pilot, and soon-to-be permanent military professor at the U.S. Naval Academy, Costello spent two of his five years during his doctoral work stationed on an aircraft carrier. With many hours a day on the bridge during its around-the-world deployment, he also endeavored to make progress on his dissertation. Besides the grueling schedule, there were other barriers, including spotty internet and the challenge of syncing schedules with those back in the U.S.

“At one point we were in the Red Sea and I had to set an alarm for the middle of the night so I could make a scheduled meeting with my advisor at the University of Maryland (UMD). Unfortunately, I was unable to get a phone line off the ship so we had to reschedule via email for the following week,” he recalls.

All these hurdles aside, Costello persisted and was able to finish his research—a groundbreaking study on the certification process for autonomous aircraft within the United States Navy. His work, which dovetailed with that of the Naval Air Warfare Center,Aircraft Division (Naval Air Station Patuxent River, Maryland) center for autonomy, could go a long way towards enabling broader use of autonomy within naval aviation, with such vehicles taking on many tasks—such as delivering supplies or even transporting soldiers—that currently require a human in the decision loop for aircraft.

Building a Framework for Certification

Whether manned or unmanned, any aircraft used by the Navy must be certified through established processes. Engineers determine its performance characteristics and draw up a set of parameters—known as an “envelope”—for safe operation of the air vehicle. Pilots are required to operate within that envelope; for example, they can’t exceed the maximum speed. Working out the specifics can be a complicated endeavor, involving much back and forth between test pilots and engineers.

In some crucial ways, it’s even more complicated for autonomous aircraft, as they will not have a pilot (or operator in the case of unmanned systems) in or on the loop to make critical decisions.

Autonomous aircraft aren’t particularly good at responding to anomalies or conditions the system was not designed to encounter, Costello explains. As long as all sensors are functioning properly and the flight conditions are relatively predictable, they’ll take off, cruise, and land in exactly the way they’re supposed to. But throw in some unforeseen condition—a dust storm clouding the sensors, for example—and things may not go as planned. As Costello puts it, “garbage in, garbage out.”

“Our approach was to define the requirements—that is, the aircraft needs to do X,Y, and Z—and identify the series of decisions that the aircraft must make: whether it should proceed with the landing, for example. We matched the requirements to the aircraft specifications and showed that algorithms could be developed that allow the aircraft to make the needed decisions and complete the mission.”

Navy Commander Donald "Bucket" Costello, Ph.D.

“Situational awareness is a critical concept in aviation,” Costello said. “That means being able to sense the environment accurately, understand what’s happening around you and predict what’s going to happen in the future. If a vehicle’s situational awareness depends on a set of sensors, then as long as the sensors are providing 100% valid information, then the vehicle is probably going to make a good decision. In the real world, though, the input data will probably not be 100%, and the resulting decision by an autonomous system may not be a sound aeronautical decision.”

An effective certification process, he said, must be able to account for such likely gaps in situational awareness, even as engineers work to mitigate them. In his research, Costello drilled down to focus on unprepared landing field mission carried out by the rotorcraft community, as there have been mishaps in civilian and military aviation during this phase of flight over the last decade.

“Our approach was to define the requirements—that is, the aircraft needs to do X,Y, and Z—and identify the series of decisions that the aircraft must make: whether it should proceed with the landing, for example,” Costello said. “We matched the requirements to the aircraft specifications and showed that algorithms could be developed that allow the aircraft to make the needed decisions and complete the mission.”

“We were then given access to data from an autonomous demonstration vehicle to evaluate actual flight test data on the performance of a vehicle completing the mission we were researching. Ultimately, when the conditions were closely controlled and scripted the autonomous air vehicle was able to complete the mission. However, when unexpected conditions were encountered the autonomous air vehicle was unable to cope,” he said.

Supplementing this work, Costello also identified specific correlations between an aircraft’s performance and the validity of data received by its sensors, thus making it possible to better predict how it will respond when the data is degraded. Having successfully defended his dissertation in October 2020, Costello plans to continue his research.

Bridging Academia and Real-World Implementation

For UMD aerospace engineering professor Mumu Xu, his Ph.D. advisor, Costello’s work represents an all-too-rare intersection between on-paper and in-the-field approaches to a topic. Academic researchers, she noted, often content themselves with so-called “toy problems”, with the findings only valid under idealized conditions. Engineers and test pilots, meanwhile, become frustrated when possibilities that exist in the abstract falter during actual implementation.

“Within the ivory tower, it’s hard to come up with applications that are grounded sufficiently in reality that they would actually be useful in the real world, without several iterations or developmental leaps to get there,” Xu said. “We like to use mathematical techniques to prove that systems are correct and to ensure that everything is safe. The tools that we use are great from an academic standpoint, but often they only work at a small scale.”

“What Bucket has done is to bring a lot of practical knowledge as to how these systems would need to be certified in real life. He knows what the requirements are, and can bridge the gap between the formal methods community and the engineers working on real systems. His work is laying the groundwork for reconciling these two perspectives,” she said.

Published May 7, 2021