Disengagements and the race for self-driving supremacy

For the first time in history, a Chinese company has taken the top spot among firms testing autonomous vehicles on California public roads.

In 2019, Baidu drove 108,300 miles and reported six disengagements across its four vehicles, making for the lowest disengagement rate of all the companies listed in California’s annual report: 0.055 per 1,000 self-driven miles.

Baidu had drastically improved its performance over last year’s report, In 2018, the company reported one disengagement every 205 miles. This year, that number fell to one for every 18,050 miles. In doing so, the company managed to knock Waymo out of its top-ranked position. Baidu attributed the drop to rapid expansion in testing fields over the past three years. 

But as the industry matures, disengagements are increasingly being seen as a poor measure of performance, since road and weather conditions, which play a huge role in report results, are not included in the data. Meanwhile, Baidu’s reports contain significantly less information about disengagements than its peers, causing industry insiders to raise questions about the quality of the company’s tests.

Baidu’s human intervention

According to Baidu’s report, the company’s vehicles required human intervention in certain situations: when surrounding objects were not detected or were misclassified, when a decision made by the autonomous system was not appropriate to the scenario, or when there was a problem with the hardware.

However, Baidu does not provide any additional information about the situation under which these disengagements occurred, only broad categories. Meanwhile, several of its rivals’ reports provide more detail about each incident that resulted in a disengagement.

For example, where Chinese counterpart Pony.ai said of one disengagement: “Driver precautionarily intervened for a reckless neighboring vehicle cutting into vehicle’s lane,” Baidu would simply say “perception discrepancy,” making it difficult to gauge just how well the company’s AV system functions.

To be fair, self-driving startup WeRide also lacked detailed descriptions in its reports. These companies are not required to include comprehensive accounts of every disengagement. However, many well-established players do, including Cruise, Didi, and Zoox.

Other aspects of the company’s testing regime are also absent. The company does not mention in its report where the tests took place. Most other companies’ reports indicate where they are testing and whether they have expanded their operations in California.

Baidu is predominantly running its AVs in Sunnyvale in very simple traffic scenarios, two industry insiders told TechNode, who asked not to be named due to their proximity to the matter.

By contrast, General Motors-owned Cruise conducted all of its tests on urban roads in San Francisco, the third-most congested city in the US, according to Tomtom’s 2019 Traffic Index. Cruise reported a disengagement rate of 0.082 per 1000 miles.

A Baidu spokesperson told TechNode that the company tests in “diverse conditions,” including urban roads and scenarios involving pedestrian avoidance, left and right turns, lane changes, and traffic light recognition.

Road conditions can have a profound effect on disengagements, with more complex urban roads leading to more disengagements. Conversely, highway driving is typically seen as easy for AVs.

“If I wanted to look even better, I’d do a ton of easy freeway miles in California and do my real testing anywhere else,” Bryant Walker Smith, a self-driving car expert, told The Verge.

China’s major players

While Baidu took the top spot in the tests, four Chinese AV startups also made it into the Top 10. AutoX and Pony.ai came in fourth and fifth—right behind GM’s Cruise—with one disengagement every 10,684 and 6,475 miles, respectively.

Meanwhile, Didi Chuxing took the eighth position, reporting 1,535 miles per disengagement, a good result for a relative newcomer. Didi, China’s biggest ride-hailing platform, started testing in California in June 2018.

In addition, China- and California-based WeRide recorded 151.7 miles driven per disengagement, performing much worse than its Chinese peers but ranking higher than companies such as Apple, Mercedes Benz, and Toyota.

Most Chinese companies conducting tests in California revealed no further details about their operations when contacted by TechNode. However, their individual reports reveal a blurred glimpse into their performance.

Pony.ai, AutoX, and WeRide all claimed to have covered a big pool of testing scenarios in various traffic and weather conditions—either sunny days or heavy rain. However, none of them detailed when and where exactly a driver has to disengage the system. These companies gave no indication of whether these incidents occurred in downtown traffic during commutes or on empty highways at night.

In terms of test areas, all the four companies have vehicles being tested in the South Bay, while Pony.ai further expanded to Fremont, where it launched a pilot robotaxi program providing transport services from a train station to two government offices.

However, most of the areas have modest population density, around one-quarter of that of San Francisco, where GM Cruise tested its vehicles in the city’s “very complex urban environment.”

Among the four Chinese companies, Pony.ai reported that its vehicles covered the greatest distance. Its fleet of 22 vehicles logged 174,845 miles in California, the third-largest number in the ranking, although nearly a fifth of that of Cruise.

The Toyota-backed AV startup also detailed their disengagements in more detail than its Chinese counterparts. In the 27 disengagements recorded over the 12 months ending in November 2019, Pony.ai attributed eight of them to reckless driving by other vehicles, and 11 to suboptimal routes planned by software for the car to maneuver. The situations its vehicles encountered vary from insufficient yielding to reckless driving on the part of other road users.

A flawed system?

Other AV companies reported disengagements resulting from poor detection of road objects or mapping flaws in different traffic scenarios. Although such details were presented in Didi’s reports, the ride-hailing giant revealed few reasons for disengagements, not categorizing them as planning, mapping or control issues.

Alibaba-backed AutoX referred very generally to the company’s three human intervention cases as localization and planning problems. The low number of disengagements may result from fewer miles driven than other companies. Meanwhile, Nvidia-backed WeRide reduced its miles driven by nearly two-thirds in 2019 from the year before, making little progress compared to last year.

The furor over the reports has led an increasing number of experts in the field to call into question the effectiveness of using disengagements as a metric to gauge how a vehicle is able to drive autonomously.

Disengagement reports provide an opportunity to compare AV performance between companies but discrepancies in reporting make the metric insufficient to measure performance, experts say.

In a series of tweets last month, Waymo asked whether disengagement metrics lead to meaningful insights. The company added that most of its real-world driving experience comes from outside California.

Meanwhile, Cruise Co-founder Kyle Vogt shared similar views, saying in a blog post that the reports are “woefully inadequate” to judge whether an AV is ready to be deployed commercially.

An earlier version of this article quoted a TechNode source as saying that Baidu tests AVs on Bay Area interstate highways. In fact, the company denies that its vehicles have been tested on interstate highways, and a review of interview recordings suggested that we may have misunderstood our sources’ comments.