5 Shopee -StarRocks 在 Shopee 的应用和实践

STARROCKSLAKEHOUSE SUMMITASIA2024ISALLYOUNEED KD StarRocks在Shopee 的应用和实践 Ai Speaker:马腾(MATENG) StarRocks STARROCKSLAKEHOUSE SUMMITASIA2024ISALLYOUNEED 滋子 01shopeeStarRocks集群介绍 02StarRocks在Shopee✁应用场景探索 03StarRocksonHive在Shopee数据平台✁应用 Ai 04StarRocksMvonExternalCatalog在产品✁应用 05Q&A ShopeeStarRocks 01集群介绍 STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeStarRocks集群规格发展简述 202311月 20244月 StarRocksonExternalCatalog StarRocksMPP物理机部署 StarRockswithStorage StarRocksComputingCluster StarRocksonK8s (onExternalCatalog) 20241月 20248月 STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeStarRocks集群架构 ShopeeStarRocksCluster部署架构 兼客Trino语法和协议，用产无感短切换 MySQLClientTrinoGaleway FEFEFE [AudtLog>AudkLog BE1BE2BE3BE4BEN DstsCacheDatsCacheDataCache DataCschaDetsCacha HiveIpnHPaimon STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeStarRocks集群管理模式 独占集群+多租户管理 ●在支持多租户✁环境下实现资源隔离，同时在每个租户内部也实现资源隔离。 K8snamespace:Di-StarRocks-MPp @SupplyChaintenantsMarketplacetennetsAntiFraudtennets SRSPXClusterSRSLSClusterSRABTestClusterDashBoardClusterSRWareHouseCluster 00 OpsDeveloper Developer Developer BI SRLOGCluster Developer PMDeveloper Developer QASRRuleEngineCluster Ops0 PMOpe STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeStarRocks集群管理模式 用户集群资源管理界面 RairiedPrajnct StarRocksQueue:regmasr-adhocsc Last1HeurLext.1DeyLest1weck Lest1Momh CoreUtilization oinCuarantoe: 20.72 40 MemoryUtllization(GBE) 1550 STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeStarRocks集群SQL生命周期管理 SQL生命周期监控管理简述 1.TrinoGatewayorStarRocksQueue管理SQL入队 InQueue 2.setcustom_query_id功能统一标识用户任务task_id与SQLSOLASQL下发到 StarRocks 3. query_id QueryPlan生成后接入Shopee内部鉴权系统，实现外表和 Shopee内部管理✁SR表✁列权限管理。Authorization StarRocksshowglobalcurrentgueries功能获取集群维度所SQL逐权 有正在运行✁SQL 5.killquery“query_id”功能提供查询运行态退出能力：集联维度运行SQL 6.运行结束✁sQL记录在StarRocksAuditLogtable中（存储3Runnirg 天），同时发送到Kafka接入ShopeeQueryAuditLog系统中SQL开拍运行 存储30天）。 SQL审计 STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeStarRocks集群SQL生命周期管理 SQL生命周期管理面板 QueryDPe2se(rpuReset StatusPlesseSeloot QueryIIhrtarceCodeOurerPrejectArtount Siatas Ounuks (gores* Action? -25-1-219e-i4299ee 16800c94ca7d odit_dk_eresto0.00 S)P-pLL-z5ag-LPGeqmf6600x94607d 0.00 teeb62c0-9:52-11ef-glEc 600cS4c7c1susnop"uupes0.00 bssb1boe-c52-11el-af5c-8800c240a7d disus"op"vpe0.00 beee516e-9e52-11ef-af5c-16600094ca7d cuspuμpe0.00 16600cS4cx7d ednin_de_srislo0.00 :4:7cbs5-2c52-11el-st5c16800c94ca7d ataupnupe9.06 B4b7r2:8-9c52-11ef-f5cP2e2769009% sdni_de_enslso.c0 ≥4:70653-9c52-11el-st5c- f6800c94cx7d odmir_de_presto0.00 b4:7cba7-5c52-11el-ef5c winisrwemets0.on STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED StarRocks在Shopee 02✁应用场景探索 STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeOLAP业务场景 电商✁OLAP业务场景具有多样性 UUsesrBeehravBiorehavior Analytics Analytics AAnntiFtraiudFRurlaesudRules Engine Engine LLoogAgnalyAsisnalysis RReeal-aTiml-eTDaitmaeDatawwaraehroeusheouse SSupuplpyCphaliynChain LogisticsReport LogisticsReport VVideidoQeuaolityQuality Analytics Analytics Application Application OObsbervsabeilityrvability BBusuinesssiInnteelligsensceIntelligence RReael-Taiml-eTimeDDasahbsoahrdb&Aoleartrd&Alert STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED LogPlatform LogPlatform RReael-taimle-time DDasahbsoahrdboard DDQQCTCasTkask TimeSeries TimeSeries DDaatata PProcreosscingessing ShopeeOLAP引擎技术栈 ShopeeOLAP引擎具有多样性，适配不同✁业务类型 U UUsesrBeehravBiorehavior AAnanlytiacslytics Logistics Logistics RRepeorptort Real-time DDaashsbohardboard Real-time BBusuinesssiness IInntetlliegenlcleigence DDaatataWWaraehroeusheouse AntiFraud AntiFraud DDeteecttioenction Offline Offline Analysis Analysis MMonoitonr&itor&AAlerlteSyrsttemSystem STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED ShopeeOLAP引擎业务✯持瓶颈 ●维护大量✁数据ingestionpipeline（离线+实时）) ○Eg.ShopeeAPMSDatawarehousePipeline ODSDWDSMOADS ClickHouseClickHouseDeshboard ServerKafkaKafka HiveAPI Hive ●ShoRee现有✁QLAP引擎Join性能不足 ○ClickHouse需要打平成大宽表 ○Trino✁Join性能不足 QQueuryeisrSylowisSlow &&ExEtraExTtLrEaffortETLEffort STARROCKSSUMMITASIA2024LAKEHOUSEISALLYOUNEED StarRocks提升Join性能 用户业务Join性能✁提升，减少了打平表✁Effort QQueuryeScíeynariSocenaíio Trino/Second Trino/Second CCliclkiHcokusHe/Soecuonsde/Second SStatrRaorckRs/Soecconkds/Second OOLALPADPBImDprBoveImprovettimimeses ●Plainjoin Plainjoin ●JJoionbiynprimbayrykpeyíimaíykey ((uusesr_eid)r_id) ●Filteíbypaítitionkey Filterbypartitionkey ●Alloutputcolumnsaíe Alloutputcolumnsare ffroímolefmttablelefttable 554.408.2082 33000+0+ 11.8.08303 SStatraRorcRksoSpcekedsupS30peedup30 ttimimeses ●Plainjoin Plainjoin ●Joinbypíimaíykey Joinbyprimarykey((uusers_ide)arnd_aniothder)andanotheí kkeye(syhop(isd)hop_id) ●Filteíbypaítitionkey Filterbypartitionkey ●AAllolultpuotcoulumtnpsaruetcolumnsaíe ffroímoleftmtablelefttable 334.486.8868 33000+0+ 11.1.71373 SStatraRorcRksSopecekdusp3S0peedup30 ttimimeses ●NoJOINbutINfoí NoJoINbutINfor ClickHouse ClickHouse ●Filteíbypaítitionkey Filterbypartitionkey ●Alloutputcolumnsaíe Alloutputcolumnsare ffríomolemf