端到端自动驾驶系统研究综述

中图法分类号：U495文献标识码：A文章编号：1006-8961（2024）11-3216-22 论文引用格式：ChenYY，TianDX，LinCMandYinHB.2024.Surveyofend-to-endautonomousdrivingsystems.JournalofImageandGraphics，29（11）：3216-3237（陈妍妍，田大新，林椿眄，殷鸿博.2024.端到端自动驾驶系统研究综述.中国图象图形学报，29（11）：3216-3237）［DOI： 10.11834/jig.230787］ 端到端自动驾驶系统研究综述 陈妍妍，田大新*，林椿眄，殷鸿博 北京航空航天大学交通科学与工程学院，北京102200 摘要：近年深度学习技术助力端到端自动驾驶框架的发展和进步，涌现出一系列创新研究议题与应用部署方案。本文首先以经典的模块化系统切入，对自动驾驶感知—预测—规划—决策4大功能模块进行简要概述，分析传统的模块化和多任务方法的局限性；其次从输入—输出模态到系统架构角度对当前新兴的端到端自动驾驶框架进行广泛地调研，详细描述弱解释性端到端与模块化联合端到端两大主流范式，深入探究现有研究工作存在的不足和弊端；之后简单介绍了端到端自动驾驶系统的开环—闭环评估方法及适用场景；最后总结了端到端自动驾驶系统的研究工作，并从数据挖掘和架构设计角度展望领域潜在挑战和亟待解决的关键问题。 关键词：人工智能（AI）；自动驾驶；模块式系统；端到端系统；数据驱动；可解释性 Surveyofend-to-endautonomousdrivingsystems ChenYanyan，TianDaxin*，LinChunmian，YinHongbo SchoolofTransportationScienceandEngineering，BeihangUniversity，Beijing102200，China Abstract：Deeplearningtechnologieshaveacceleratedthedevelopmentandadvancementofend-to-endautonomousdriv⁃ingframeworksinrecentyears，sparkingtheemergenceofnumerouscutting-edgeresearchtopicsandapplicationdeploy⁃mentsolutions.The“divideandconquer”architecturedesignconcept，whichaimstoconstructmultipleindependentbutrelatedmodulecomponents，integratethemintothedevelopedsoftwaresysteminaspecificsemanticorgeometricorder，andultimatelydeploythesecomponentstotheactualvehicle，isthefoundationforthemajorityoftheautonomousdrivingsystemscurrentlyinuse，alsoknownasmodularsystems.However，awell-developedmodulardesigntypicallycomprisesthousandsofcomponents，placingaconsiderableburdenonthegraphicsmemoryandprocessingcapacityofautomotiveCPUs.Furthermore，theintrinsicmistakesofeachstackedmoduleduringpredictionwillrisewiththenumberofstackedmodules，andupstreamflawscannotbefixedindownstreammodules，presentingamajorrisktovehiclesafety.Amulti⁃taskarchitecturebasedonthe“taskparallelism”principleaimstoefficientlyinfermultipletasksinparallelbydesigningvariousdecodedheadswithasharedbackbonenetworktoreducecomputationalconsumption.However，theoptimizationgoalsforvarioustasksmaynotbeconsistent，andsharingfeaturesmindlesslycanevendegradetheoverallperformanceofthesystem.Incontrasttotheprevioustwosystemarchitectures，theend-to-endtechnologyparadigmeliminatesinformationbottlenecksandcumulativeerrorsduetotheintegrationofnumerousintermediatecomponentsbasedonruleinterfaces，allowingthenetworktocontinuallyoptimizetowardaunifiedobjective.Alargemodelcanbeusedtogeneratelow-level controlsignalsorvehiclemotionplanningbasedoninputssuchassensordataandvehiclestatus.Withsensorsservingas 收稿日期：2023-11-07；修回日期：2024-01-06；预印本日期：2024-01-13 *通信作者：田大新dtian@buaa.edu.cn 基金项目：国家自然科学基金项目（U20A20155，62173012，52202391） Supportedby：NationalNaturalScienceFoundationofChina（U20A20155，62173012，52202391） inputs，theearlyend-to-enddesignbasedonimitationandreinforcementlearningdirectlyoutputsthefinalcontrolcom⁃mandsforsteering，braking，andacceleration.However，noexplicitrepresentationofdrivingscenariosinthiscompletely“blackbox”network，whichisalsoreferredtoasweaklyinterpretableend-to-endmethods，isavailable.Thus，understand⁃ingthereasoningbehindthedecisionorpredictionofavehicleisdifficultforhumans，makingdebugging，validation，andoptimizationchallenging.Evenworse，oncethemodelmalfunctionsorunexpectedsituationsoccur，accuratelydetecting，avoiding，andrepairingproblemsinatimelymannerbecomesdifficult，allofwhicharecrucialformaintainingthesafeoperationofintelligentvehicles.Thecomponentdecouplingapproachfacilitatesthedevelopmentandoptimizationofindi⁃vidualmodulesintheconventionalmodularsystem，therebyguaranteeingsteadyrepresentationperformanceandstronginterpretabilityforeachsubmodule.Unfortunately，thismethodfallsshortofachievingunifiedgoalsattheoptimizationlevel，thatis，integratingoptimizationandlearningtowardtheultimateplanninggoal.Amodularjointend-to-endautono⁃mousdrivingarchitecture，whichpreservesthemodulardrivingsystemwhileallowingthedifferentiabilityofeachmodule，isaworkablesolutiontoensurethateverymodulehassufficientinterpretabilityandoverallautomaticoptimizationcapabili⁃ties.Thebasicideabehindthistechnologyliesinthecreationofauniqueneuralnetworkthatconnectsallindependentmodulesandenablesthegradientsfromtheplanningmodulestobefedbackdowntotheinitialsensorinputforend-to-endexecution.Inotherwords，thiskindofapproachmerelymodifiesthesubmoduleconnectionmechanismwhilemaintainingtheclassicmodulartechnologystack；thatis，thisapproachsubstitutesanewimplicitinterfaceforthepreviousexplicitinterfaces，whichwererule-basedandrequiredmanualcreation.Modularjointend-to-endproceduresofferacertaininter⁃pretabilitybecauseofthedistinctseparationbetweenmodules.Theexplicitend-to-endsystemisarelativedecouplingbasedonoveralldesignandexhibitssomedegreeoflogicinitssequentialfunctioningfromperceptiontoprediction，andthentoplanningmodulesduringdecisioninference.Themodelcanbeintentionallyadjustedwhenitencountersunknownanduncontrollableresultsbyunderstandingtheoperationallogicunderlyingtheexplicitsolution.Furthermore，visualiza⁃tionmethods，suchasinternalfeaturesorintermediateresultsofspecifictasksormodules，canbeutilizedtoanalyzethedecision-ma