Fully Self-Driving Future Hits the Brakes

Chris Palmer

Senior Technology Writer

In 2021, when pandemic-related supply chain delays drove up the cost of already-expensive radar and ultrasonic sensors, the automaker Tesla(Palo Alto,CA,USA)removed the technology from its vehicles,and pivoted to a cameras-only approach for providing input to its autonomous driving systems[1].The result was a dramatic increase in crashes, followed by lawsuits, federal investigations, and a recall of Tesla’s latest self-driving software from hundreds of thousands of its cars.Tesla’s woes, as well as Ford’s and Volkswagen’s October 2022 desertion of plans for fully selfdriving cars [2], highlight serious, industry-wide concerns about how long it will be before fully autonomous vehicles—from any manufacturer—are accessible to everyday drivers.

‘‘Four or five years ago it looked as though we were headed toward rapid deployment for fully autonomous vehicles on the roadway,” said Eric Jackson, associate research professor of civil engineering at the University of Connecticut (Storrs, CT, USA),and the director of the Connecticut Transportation Safety Research Center.‘‘Over the last six months, things seem to be unraveling in terms of what is possible.”

Tesla’s Autopilot software,introduced in 2014,is a suite of systems allowing drivers to hand over physical control of their electric vehicles, under the condition that they remain vigilant while behind the wheel(Fig.1).Eight cameras capture real-time footage of activity surrounding the car,allowing it to assess and maneuver accordingly around hazards such as pedestrians,bicyclists,or other vehicles [3].Autopilot directs cars to maintain speed and safe distance behind other vehicles,stay within their lane lines,and make lane changes on highways.In October 2020, Tesla introduced an expanded set of Autopilot features, called Full Self-Driving Beta.Those features included the ability to maneuver city and residential streets, halt at stop signs and traffic lights, and make turns while navigating from one point to another (Fig.2) [4].

Tesla chief executive officer Elon Musk has said that Autopilot is safer than normal driving, citing the vast number of traffic crash deaths on US roadways annually—42 915 in 2021 as estimated by the US National Highway Traffic Safety Administration (NHTSA)[4].However,Tesla vehicles running Autopilot accounted for nearly 70%of the 392 crashes—and most of the fatalities and serious injuries—involving advanced driver-assistance systems reported between July 2021 and June 2022 [4].As of June 2022, Tesla’s Autopilot had been installed on around 830 000 vehicles, giving Tesla the highest ratio of crashes to self-driving vehicles of any automaker [5].By comparison, Honda’s six million vehicles equipped with driver-assistance systems were involved in 90 crashes in the United States over the same period.Subaru racked up ten driver-assistance wrecks,and all other automakers reported five or fewer[6].

Fig.1.Tesla’s Autopilot at work controlling a 2017 Tesla Model X 75D in highway traffic.The driver-assistance system maintains a safe speed and a safe distance from the car ahead of it.Autopilot also centers the vehicle in the lane and changes lanes.Credit: Ian Maddox, English Wikipedia (CC BY-SA 4.0).

Fig.2.Tesla Autopilot’s expanded set of features, called Full Self-Driving Beta,installed in a 2022 Model S.Full Self-Driving Beta, introduced by Tesla in October 2020,includes the ability to maneuver city and residential streets,halt at stop signs and traffic lights, and make turns.Credit: Ted Drake, Flickr (CC BY-NA 2.0).

These data emerged from a 2022 US government mandate requiring automakers to report crashes involving driver assistance to assess its potential safety risks.The NHTSA order required manufacturers to disclose crashes where the software was in use within 30 s of the crash,in part to alleviate the concern that manufacturers might hide crashes by claiming the software was not in use at the time of the wreck.In many cases, NHTSA has claimed, Tesla’s vehicles shut off Autopilot about one second before impact[4].

In February 2023,following a request from NHTSA,Tesla voluntarily put a hold on new installations of its 15 000 USD Full Self-Driving Beta software [7].Owners who already had Full Self-Driving Beta installed and activated were able to use the software but had to wait until late March 2023 [8] for an over-the-air update(essentially a recall of the old software)that addressed issues identified by NHTSA, including cars breaking traffic laws and making risky maneuvers.The recall affected nearly 363 000 Model S,Model 3, Model X, and Model Y vehicles [7].‘‘Many Beta users selfreported encountering situations that either made them uncomfortable or are against their region’s traffic laws, such as performing illegal turns,” Jackson said.

In February 2023, the first lawsuit against Tesla for a crash involving Autopilot went to trial; four more suits are set to follow in the coming months[9].The claims of the lawsuits center around a common complaint: Tesla consistently overpromised on Autopilot’s capabilities and downplayed the risks.Ever since the release of Autopilot in October 2015,Musk has encouraged drivers to think of it as more advanced than it was[9].The driver’s manual,though, made clear that the Full Self-Driving software is an ‘‘Society of Automotive Engineers (SAE) Level 2 driver support feature”[7].This designation means that it can steer, brake, and accelerate automatically and that the driver must be engaged and ready to take over at any time; drivers are also completely responsible for any mistakes the system might make.However, that language received little attention compared to Musk’s claims, leading many drivers to be confused about Autopilot’s limitations.The name of the technology also implies that it can operate independently.The US Securities and Exchange Commission is currently examining the legality of Musk’s claims [10].

Experts believe the main reason Tesla vehicles are stuck in SAE Level 2 and why the company saw a dramatic uptick in the number of crashes in the past two years, was Musk’s May 2021 decision,over the objections of several now former employees, to remove expensive radar and ultrasonic sensors—short-range devices that can detect obstructions within inches of the car—in favor of multiple optical cameras[1,3].Musk claimed that relying solely on cameras made for a safer system by reducing the complications of integrating cameras and radar [11].Industry observers have speculated,however,that the change was made due to the spiking cost of radar and ultrasonic components [1,3].

The switch to the so-called Tesla Vision approach, which made vehicles susceptible to perception errors brought on by raindrops,bright sunlight,or entering and exiting tunnels,resulted in NHTSA downgrading Tesla vehicles’ safety ratings [12].The change also coincided with the expansion of the Full Self-Driving Beta testing program from thousands to tens of thousands of drivers.‘‘In tunnels,these vehicles have a hard time because the camera suddenly becomes dark; the white balancing goes completely off when you enter and exit, and this is a huge problem for Tesla,” said Aviral Shrivastava, professor of computing and augmented intelligence at Arizona State University in Tempe, AZ, USA, who researches autonomous vehicles.‘‘If you have a vision-based system, you can really fall into that trap.”

In addition to jettisoning radar and ultrasonic sensors,Tesla has chosen not to install light detection and ranging(LiDAR)sensors on its cars.Unlike cameras,which require computer vision algorithms trained on human-labeled images to interpret two-dimensional input, LiDAR can capture extremely accurate three-dimensional(3D) depth measurements [13].However, LiDAR can only build a wireframe 3D environment.It cannot read traffic signs, traffic lights, or anything devoid of depth.That is why, according to Shrivastava, most other automakers’ autonomous driving assistance systems, including those built by General Motors,Mercedes, and Waymo (Mountain View, CA, USA), a division of Google’s parent company, Alphabet, incorporate both types of sensors.‘‘It is super risky to go cameras-only,”Jackson said.‘‘There is a lot of liability that comes with not having any backup sensors.”

Shrivastava called Tesla Vision a business decision.‘‘If they were focused on building an autonomous car, then they would go with LiDAR,” he said, adding that a full suite of cameras, lidar,and radar can cost upwards of 25 000 USD.‘‘But instead,they were focused on a car configuration that allowed them to sell it at a low cost but still claim it is full self-driving.”

Problems began to emerge almost immediately following Tesla’s switch to a cameras-only system.NHTSA data,for example,revealed 107‘‘phantom braking”complaints over the three months following the removal of radar,compared to only 34 in the preceding 22 months [3,14].In a potential retreat from its cameras-only position, Tesla said in December 2022 that it may outfit new models with a high-definition radar unit that it claims is more advanced than any other on the market [11,15].

No other manufacturer appears anywhere close to building a fully autonomous vehicle for purchase by consumers,despite using more advanced sensors, including multiple radar and LiDAR units and more than ten cameras in some prototypes.Reflecting the daunting technical and economic challenges,Ford and Volkswagen shuttered their multi-billion US dollars Argo AI (Pittsburgh, PA,USA) self-driving startup in October 2022 [2].

Several companies have, however, made headway with driverless ‘‘robo-taxis,” including General Motor’s Cruise (San Francisco,CA, USA), which in June 2022 received permission from the California Public Utilities Commission to charge for driverless rides in San Francisco[16],and Alphabet’s Waymo,which has been testing its autonomous taxis in Phoenix since early 2017[17](Waymo was also granted permission,in November 2022,to offer rides in its driverless taxis in San Francisco [18]).To minimize risk, the cities have to date limited the use of the vehicles,in terms of both areas and hours of operation (e.g., at night from 10 pm to 6 am only when fewer pedestrians and cyclists are present and car traffic is light).In Phoenix,Waymo’s self-driving cars can provide paid rides in an approximately 260 km2service area,although its fully driverless taxis are restricted to an area half that size.‘‘Waymo’s taxis operate autonomously but only on highly mapped, highly coordinated routes, with designated areas where you can be picked up and dropped off.It is not like they are dropping a car in the middle of a foreign city and expecting it to navigate on its own,” Jackson said.‘‘Maybe that is the safest way to go but I do not know if that model is scalable across the entire US.”

For now, the high cost of Waymo vehicles’ suite of sensors,which can run into the tens of thousands of dollars,means the company’s fully autonomous cars will only operate as robo-taxis, with no plans—at present—for a consumer version.‘‘These vehicles run on the road 22 hours a day.They just stop to recharge.That is the only way you get a real return on your investment with these highly specialized,super-expensive vehicles,”Shrivastava said.‘‘It is going to be a very long time before your average consumer can purchase and operate a truly safe self-driving vehicle.”Jackson agreed about the long timeline for an autonomous driving future.‘‘We are on the road to full automation, but it may take a long time,” he said.‘‘We are probably still 30–50 years away from having a full self-driving vehicle that can drive across the country.”