Alphabet Inc’s Waymo and General Motors Co’s Cruise are leading a backlash against a California reporting requirement on self-driving vehicle test data that the companies claim is not relevant or accurate in measuring performance or progress.
Data released Wednesday by California showed Waymo and Cruise once again had the greatest number of test miles between “disengagements,” when a human driver must intervene to take control from a self-driving system during testing on public roads.
The disengagement data is widely used as a yardstick to compare companies testing self-driving vehicles on California roads, and is often cited as evidence that Waymo leads the sector.
Waymo tweeted that the disengagement metric “does not provide relevant insights” nor does it distinguish Waymo’s “performance from others in the self-driving space.”
At an investor conference on Wednesday, Dan Ammann, chief executive officer of GM’s Cruise subsidiary, sidestepped a question about the relevance of the California disengagement data, saying “there is no really great way to track” the company’s progress, other than its own internal data.
Data provided to California showed Waymo had 13,219 miles between disengagements, compared with 11,017 miles in 2018. Cruise reported an even greater improvement, with 12,221 miles between disengagements in 2019, compared with 5,205 miles the previous year.
Waymo said its self-driving test vehicles logged 1.45 million miles on California roads last year, while Cruise vehicles tallied 831,000 miles, most of them on the streets of San Francisco.
Mark Rosekind, a former head of the National Highway Traffic Safety Administration and now chief safety innovation officer for self-driving startup Zoox, addressed the discrepancy between testing on congested city streets and on freeways, saying “we have found that not all miles are created equally.”
“Our focus has been on the quality of these miles, not the quantity,” he said, noting that Zoox more than doubled the number of test miles in 2019.
Aurora co-founder Chris Urmson, who previously headed Waymo’s self-driving program, wrote last month that “these numbers mean little when there’s no clear definition of what constitutes a disengagement.”
The larger self-driving companies want to incorporate more and different data when measuring performance and progress, including miles logged in computer simulations.
Waymo also noted that much of its real-world system validation data comes from Phoenix, where the company has been testing self-driving vehicles for several years and operates a small commercial ride sharing fleet.
The company’s self-driving performance in Phoenix “is largely unrelated to our California testing,” which Waymo said is mainly for engineering development.
“We don’t think (California) disengagement data should be used to compare performance, or judge readiness or competency,” Waymo said.
(Content and photos syndicated via Reuters)
