数据治理：数据中台建设与能力提升策略

01侨䰘屛椚僽➊⛎ 侨研屛椚⽰DataGovernance 歋✵ⴗⰆ錠錭涸♶ず♶ず涸劼匬ㄤ絆絉㼆侨研屛椚剣♶ず涸㹁⛐ 㕂꣢侨研盗椚⼸⠔ DAMA 侨研屛椚僽㼆侨研餴❡涸 盗椚崞⸓遤⢪勉⸂ㄤ䱽ⵖ涸 崞⸓꧋ざ錞够ծ港䱽ㄤ䪄 遤կ 㕂꣢侨研屛椚灇瑕䨾 侨研屛椚僽♧⚡鸑鵂♧禹⴩⥌ DGI 䜂湱Ⱒ涸鵂玐勻㹊梡Ɀ瘻勉ㄤ翟 餓ⴔ䊨涸禹絡鵯❈鵂玐䭽撑鴪 䧭Ⱏ霋涸垷㘗勻䪄遤霪垷㘗䲽 鶣✫靆腊呏研➊⛎⥌䜂㖈➊⛎ 傞ꢂㄤ䞔ⲃ♴欽➊⛎倰岁ꅷ 》➊⛎遤⸓կ ISO38508-1:2019叻ⲥ 䭷㼋侨研盗椚崞⸓涸勉㪭Ɀ瘻ⵖ㹁ㄤ勉⸂涸呥卹 02侨䰘屛椚涸胜兞 2.1.♶ず❡ㅷ剣湱⡃涸⸆腊꨽宠 •䲿넞侨研餘ꆀ 剣⸔✵烁⥂侨研涸ⲥ烁䚍ծ模侮䚍ㄤ♧荝䚍⚹鵘絶㔙ꢭ䲿⣘〳ꬑ涸⥌䜂勻彂 •䲿넞侨研㸝Ⰼ 侨研屛椚⺫䭍㼆侨研涸⥂䫡ㄤ㸝Ⰼ盗椚⢾㥵霄꡼䱽ⵖծ⸈㺙ㄤ侨研㢊⼝ •䲿넞侨研♧荝䚍 鵘絶遤⚌嶍⿺⠍㢴禹絡ㄤ䎂〵侨研〳腊㶸㖈♶♧荝䚍կ侨研屛椚剣⸔✵侮ざㄤ叻ⲥ⻊侨研烁⥂侨研㖈ぐ⚡禹絡ㄤ䎂〵⛓ꢂ涸♧荝䚍 •佅䭯Ɀ瘻 剣⸔✵⚹盗椚㽻䲿⣘넞餘ꆀծⲥ烁涸侨研⟄佅䭯刿㥩涸Ɀ瘻կ •䲿넞鵘絶宐䎂 鸑鵂㼆侨研鵳遤屛椚〳⟄〄梡悶㖈꡼곿ծ⠏⻊鵘絶崨玐ծ䲿넞餴彂ⵄ欽桧➢罜䲿넞侮⡤鵘絶宐䎂 2.2.鵘絶遤⚌侨䰘屛椚傞涸⚺銳汥挿 •侨研餘ꆀ꡼곿 歋✵禹絡⠍㢴ծ侨研勻彂撑勇侨研餘ꆀ〳腊㶸㖈꡼곿㥵♶ⲥ烁ծ♶模侮ծ♶♧荝瘝կ鵯絛侨研屛椚䌄勻✫䮋䧶꨽銳䫏Ⰶ㣐ꆀ傞ꢂㄤ礵⸂鵳遤侨研幡峤ծ낉霆ㄤ侮ざկ •侨研ⴔ侔ㄤ烱晙⻊ 侨研䖃䖃ⴔ侔㖈♶ず涸侨研䎽ծ俒⟝禹絡ㄤ䎾欽务 •䪮助♸䊨Ⱘ鷥䭊 䋑㖞♳剣⠍㢴涸侨研屛椚䊨Ⱘㄤ䪮助〳⣘鷥䭊⟱⚌꨽銳霉⠮ㄤ鷥䭊ざ鷓涸䪮助ㄤ䊨Ⱘ鵯〳腊⠔署餩䖎㢴傞ꢂㄤ礵⸂ •緃⛘侨研屛椚絑낉 雽㢴⟱⚌㖈侨研屛椚倰꬗緃⛘絑낉〳腊♶濼麤㥵⡦霃雦ㄤ㹊倶剣佪涸侨研屛椚瘻殜ㄤ崨玐 •䭯絯佖鵳ㄤ⠏⻊ 侨研屛椚僽♧⚡䭯絯涸鵂玐꨽銳♶倗㖑佖鵳ㄤ⠏⻊կ鵘絶㔙ꢭ꨽銳Ⱒ岤侨研屛椚涸剒倝馋 ⸷ㄤ䪮助⟄⤑⿺傞刿倝荈䊹涸倰岁ㄤ瘻殜 03侨䰘⚥〵ㄤ侨䰘屛椚 3.1.侨䰘⚥〵㥵⡦䌐⸔⟱⚌侨䰘屛椚 㖈侨㶶⻊鲮㘗嵠惐务⚹✫䎾㼆⥌䜂㷐䀙䮋䧶⟱⚌鶖ⴗ꨽銳匬䒊♧⚡✽翫✽鸑涸䎂〵勻䩧灶㠗㘠⤛鵳⥌䜂涸넞佪崨鸑կ侨研务〵⡲⚹⟱⚌侨研屛椚涸呍䗱卹匬〳⟄㹊梡侨研涸絡♧㶸⪰ծ㢅椚ㄤ盗椚⚹⚌⸉鿈꡶䲿⣘넞餘ꆀ涸侨研剪⸉䧭⚹䲀⸓⟱⚌䭯絯ծ넞佪〄㾝涸ꅾ銳䒸乾կ ΍侨研侮ざ 侨研务〵䌐⸔侮ざぐ猫禹絡ծ勻彂ծ呔䒭涸侨研䲿⣘絡♧涸㶸⪰ㄤ盗椚剣⸔✵鵯猫烱晙 ⻊꡼곿涸鍒Ɀ䲿넞侨研涸〳腊䚍ㄤ♧荝䚍➢罜䲿넞侨研涸餘ꆀ Ύ侨研餘ꆀ盗椚 侨研务〵腊㢿䲿⣘♧禹⴩涸侨研餘ꆀ盗椚㥵侨研幡峤ծ䒁㾝⿡ꅾծ侨研餘ꆀ唬崵ㄤ⥝撑瘝烁⥂侨研涸ⲥ烁䚍ծ模侮䚍ㄤ♧荝䚍 Ώ⯋侨研盗椚 侨研务〵〳⟄䒊用⯋侨研涸盗椚⡤禹雵䔶侨研涸㹁⛐ծ勻彂Ⱒ翫䌐⸔䲿넞侨研涸〳椚鍒䚍倰⤑侨研唬程ㄤ⢪欽 ΐ侨研㸝Ⰼ♸ざ錞 侨研务〵腊䲿⣘♧禹⴩㸝Ⰼ騈ざ錞盗椚⺫䭍侨研⸈㺙ㄤ꣈䱽⥂ꥻ侨研涸㸝Ⰼㄤꥧ猙 Α侨研ⴔ區♸〳錠⻊ 侨研务〵䲿⣘涸侨研剪⸉腊㢿佅䭯〳錠⻊ⴔ區䊨Ⱘ》侨➢罜㹊梡㹊傞ⴔ區䫣邍欰䧭ㄤ侨研䮔䱡瘝⸆腊䌐⸔〄梡悶㖈꡼곿⠏⻊鵘絶涸崨玐ㄤ䲿넞鵘絶宐䎂 Β叻ⲥ⻊♸剒⢕㹊騨 烁⥂侨研屛椚涸♧荝䚍ㄤ錞薴䚍䲿넞侨研屛椚佪卓꣭⡛鵘絶굥ꤗ 3.2.㥵⡦䲿⼮⟱⚌侨䰘屛椚腊⸂ 1.䒊霃絡♧涸〳錜崵䎂〵 䙼騟1꧋ざぐ⚡禹絡䩧鸑⵸畮涸涬䔶䘯鸟侮ざ䵨䒊絡♧涸䎂〵 ♶駈〫僽邍꬗絡♧㖈卹匬ծ侨研ծ㶸⪰倰꬗尵剣㹊梡䩧鸑た劍絶䫡鶛去䧭劥넞 ⠏挿傞ꢂ䘯䒊霃䧭劥⡛ 3.2.1.侨䰘⚥〵涸䒊霃䙼騟 䙼騟2➢ⵌ1⚌䒊⸉霃ծ絡侨♧研涸ծ〳卹錜匬崵瘝䎂倰〵꬗Ⰼ鿈ꅾ倝錞ⴢ靈侮侮ざぐ⚡㔙ꢭ낧䎁➢0 ⠏挿㼜侨研⨞Ⰼ꬗模侮涸唙椚㣝㹊侨研㛇炄⚹た絯涸⥌䜂侮ざծ暵䖄Ⱒ翫ծ⚌ ⸉峯㻋瘝䲿⣘䔂剣⸂涸佅丒 ♶駈傞ꢂ蔄餩Ɤծ䒊霃䧭劥넞 2.ⵄ欽侨研务〵䒊霃BonreeONE •⚌⸉唙椚 呍䗱㖞兞⺫䭍ITOM絡♧港䱽ծAIOps兰腊鵘絶ծBizOps⚌⸉鵘蠒ծDevOps佪腊䲿⼮ծSecOps㸝Ⰼ鵘絶 •䪮助卹匬霃雦⾲ⴭ ΍♧⡤⻊鍒Ɀ倰呩㹊梡곭㚖䒊垷Ύⴔ㽻霃雦⡛羒ざծ䳃⟝⻊ծ㶸⪰〳剏䰃Ώ넞䎇〄㢴崞ծ⿡掚挿ծⴔ䋒䒭ΐ넞䚍腊伉䪠ծ⾓綫ծ綘瀦ծ곫㢅椚Α넞〳欽 •䎂〵卹匬 3.2.2.侨䰘⚥〵霃雦 霃雦湡叻 Ⰼ侨研絡♧㢅椚ծ㼆⚌⸉⣩㾓誋䏀㽻ծ偽綘瀦䎾㼆侨研㢅椚&䒊垷ծ䲿⼮⚌⸉鶛去鸟䏞ծ㢴彂侨研絡♧叅霧կ 侨研务〵涸腊⸂⺫䭍侨研㢅椚ㄤ侨研剪⸉կ 侨研㢅椚 OneETL腊㢿佅䭯鳕Ⰶծ鲮䰃ծ幡兠ծ鳕ⴀぐ猫皾㶩ず傞佅䭯㢴獆䨪կず傞⽇潇侨研㼆OneETL⨞✫〳錠⻊涸綘䱗倰䒭⢪Ⱖ傁腊㢅椚侨研⿶腊⨞ⵌ獆䨪ꢂꥬ猌կ罜侨研务〵涸〥♧㗌呍䗱⚌⸉侨研剪⸉ⴭ歋OneService勻㹊梡腊㢿䲿⣘絡♧涸BQL⚌⸉⣩㖈叅䏀㽻侨研傞♶⠔䠭濼ⵌ䭷叻♶ず涸㶸⪰倰䒭ծ翫齥叅霧♶ず侨研䎽涸䭷叻腊㢿ッ梡㖈ず♧⚡絕卓꧋务ծ侨研屛椚鸑鵂䭷叻涸⯋侨研溏ⵌ䭷叻碫㘗♸絶䏞搬た鵳♧姿鸑鵂䭷叻叅霧呏研崨ꆀ䞔ⲃ㖈崨ꆀ넞䂽傞꣭紩㢅椚侨研䲿⣘㢴獆䨪瘝կ OneETL OneETLⰢꝶ腊⸂ 1.⢪欽꡶地⡛腊㢿㹊梡〳錠⻊䬯䭩鸑鵂DAG㹊梡皾㶩抳崞綘䱗 2.的㾝䚍㥩䳃⟝⻊皾㶩霃雦腊㢿㹊梡鳕Ⰶծ雦皾ծ鳕ⴀ皾㶩䳃⟝⻊的㾝 3.Ⱘ㢊傈䘋ծjson⼱絕匬⻊侨研㢅椚腊⸂佅䭯䅺㤛絕匬涸蒜溁㢅椚 4.鿈縭䧭劥⡛♸spark/fink湱嫱荈灇DAG⿺皾㶩涸鿈縭䧭劥蒜紨♧⼱ OneETL霃雦湡叻 1.偽綘瀦-偽꨽綘瀦䬯䬘䭩佅䭯 2.鲽⤑皍峇-♶꨽䒓〄⿬♸皍⽀僒㺂⣜饆㼱 3.餴彂嶊署㼱-餴彂嶊署匧㼱 4.䳃⟝⻊的㾝-⟣⡦皾㶩佅䭯䳃⟝⻊欽䨪〳荈遤䒓〄皾㶩 5.꬗ぢ㹊傞侨研㢅椚-佅䭯帿䏞䅺㤛㢕勇json傞䎸俒劥 ETL涸⚙⚡蒜挿ETL-ManagerㄤETL-Worker ETL-Manager涸䪄遤鵂玐 ETL-Worker涸䪄遤鵂玐 OneETL涸♲⚡Ⱒꝶ䪮助挿䪮助挿♧⟣⸉綘䱗DAG DAGDirectedacyclicgraph剣ぢ偽梠㕃㥵卓♧⚡剣ぢ㕃偽岁➢厤⚡곝挿ⴀ〄絑鵂蕯䎁勵鴝㔐ⵌ霪挿ⴭ鵯⚡㕃僽♧⚡剣ぢ偽梠㕃 •⾲ⴭ⽀鳕暟㢴鳕ⴀ •暵挿 1.僒✵絶䫡 DAG〳⟄幡兠㖑䲽鶣ETL务涸ぐ⚡⟣⸉⛓ꢂ涸⣜饆Ⱒ禹⢪䖤侮⚡䊨⡲崨玐僒✵椚鍒ㄤ絶䫡կ 2.抳崞䚍䔂 DAG〳⟄鸑鵂幑⸈䧴ⴵꤑ蒜挿勻的㾝䧴綫㼭䊨⡲崨玐➢罜忘駈♶ず涸꨽宠կ 3.〳ꅾ欽䚍넞 歋✵DAG邍梡⚹♧絆⟣⸉⿺Ⱖ⣜饆Ⱒ禹㔔姼〳⟄㼜Ⱖ撑ⵖ䎇䎾欽✵♶ず涸侨研꧋➢罜㹊梡넞〳ꅾ欽䚍կ 4.僒✵䎇遤䪄遤 DAG务涸⟣⸉⛓ꢂ㶸㖈⣜饆Ⱒ禹告僽尵剣䗄梠⣜饆㔔姼〳⟄鲽匠㖑鵳遤䎇遤䪄遤䲿넞㢅椚佪桧կ 䪮助挿✳OneETLSPI䳃⟝ •JavaSPI⚺銳⺫ゎ㔋⚡倰꬗: 1.Service ♧絆䒓佞䱹〡ㄤ碫〳霄꡼厤❈暵㹁涸䎾欽玐䎸⸆腊䧴⸆腊կ 2.ServiceProviderInterface ⡲⚹proxy䧴罏僽endpoint涸䱹〡䧴罏䬄韌碫կ 3.ServiceProvider SPI涸暵㹁㹊梡կServiceProvider务〳⟄⺫ゎ♧⚡䧴罏㢴⚡㹊梡碫կ 4.ServiceLoader SPI涸呍䗱僽ServiceLoader碫ServiceLoader〳⟄〄梡ㄤ⸈鲿㹊梡⡲欽㸐⢪欽♳♴俒碫騟䖈勻㹁⡙ServiceProvider㹊梡䎇㼜Ⱖ佞㖈綕㶸务կ •SPI涸⠏挿 1.匠羒ざ 腊鳅㥩㖑鍒羒䎇㖈玐䎸鵘遤傞鴪ⵌ䳃⟝剏䰃涸湡涸կ 2.的㾝䚍䔂 䒓〄➃プ腊㖈♶佖」去瀦涸ず傞鲽匠㖑的㾝䎾欽玐䎸⸆腊կ 3.鷳僈⻊ 䒓〄➃プ〫꨽銳䒓佞叻ⲥ涸䱹〡➢♧㹁玐䏞♳鼙⯝玐䎸涸撑勇䚍կ •SPI⿡pom⣜饆⠏⻊ 䒸ⰆSPI䳃⟝赡搬〳⟄㟞⸈抳崞䚍告䪾䳃⟝⺫⸈鲿ⵌpom务㟞⸈䳃⟝傞꨽銳⥝佖㗂叻䎇 ♶倰⤑㔔姼銳⨞SPI⿡pom⣜饆⠏⻊կ 1.㹁⛐䳃⟝碫⸈鲿㐼PluginClassLoader鸑鵂㹁⛐䭷㹁涸碫⸈鲿㐼⸈鲿䳃⟝ծꅷ欽랾雩碫⸈鲿㐼⠔ⴀ梡NoSuchMethodError 2.㹁⛐䳃⟝〄梡碫PluginDiscovery鸑鵂jar䧴罏䭷㹁涸pom俒⟝欽勻〄梡䳃⟝կ 3.⸈鲿䭷㹁涸䳃⟝⸈鲿鸑鵂PluginDiscovery〄梡涸䳃⟝ 4.㹁⛐䩧⺫鹤鳓㖈䊨玐䩧⺫涸傞⦫鸑鵂䭷㹁䩧⺫倰䒭㼜䳃⟝䩧⺫荛䭷㹁湡䔶⣘PluginDiscovery㼊䪪䳃⟝ 䒸欽䳃⟝涸垷㗌 䳃⟝⣜饆 䳃⟝碫 䪮助挿♲OneETL餴彂盗椚 餴彂盗椚湡叻 1.㼆絁玐餴彂鵳遤剣佪盗椚 2.㹊梡餴彂Ⱏ❧⿺杝❧䎇䲿⼮侨研꧋䧭珘㹁䚍⿺䲿⼮餴彂ⵄ欽桧 3.㹊梡Ⱆ剣⟣⸉餴彂Ⱏ❧猙剣⟣⸉餴彂杝⽑䎇佅䭯Ⱏ❧⿺杝❧〳ⴗ䰃 侨研剪⸉ OneService䲿⣘♧畀䒭㢴彂侨研剪⸉鸑鵂BQL䲿⣘絡♧叅霧կ OneService Ⱒꝶ腊⸂ ♧ծ⚌⸉抳崞䚍 1.〳鸑鵂BPL叅霧꬗ぢ垷㘗罜♶꬗ぢ㶸⪰㶸⪰涸剏䰃♶䕧ㆇ⚌⸉⣩涸⢪欽 2.㹊梡騗彂涸䭷叻雦皾ㄤ騗⚌⸉䎂〵欽䨪⣩偽꨽罌贊䭷叻涸㶸⪰倰䒭ㄤ⡙縨 3.腊㢿Ⱒ翫CMDB鵳遤垷㘗雦皾 ✳ծ䚍腊䲿⼮ 1.ず邍ず勵⟝露〣〳ざ䎇⚹♧勵露〣 2.㹊梡皾㶩&鵂応勵⟝涸♴䲀 3.侨研䔱䎇⠮皾厤⚡SQL皾㶩㥵FilterծJoinծSort涸去⟟鷥》剒⠏瘻殜勻䪄遤 侨䰘剪⸉Ⱒꝶ㢅椚崨玐 1.䎾欽倰鸑鵂BQL叅霧OneServiceկ 2.OneService叅霧⯋侨研鵳遤露岁鍒區 3.鍒區模䧭た鵳遤SQL⠏⻊ 4.欰䧭SQL䪄遤雦够た〄佞絛䪄遤垷㗌 5.䪄遤垷㗌雷㶩露〣〄佞ⵌ♶ず㶸⪰䒸乾䪄遤 6.䪄遤模絕卓鵳遤ⱄ妃雦皾䎇ㆇ䎾絕卓կ 04Bonree㥵⡦⨞侨䰘屛椚 Bonree⨞侨䰘屛椚涸㹊乼♫姿낢 1.僈烁屛椚湡叻ㄤ薴㔵 湡叻 㼜ꨪ侔涸ぐ⚡禹絡侨研鵳遤絡♧盗椚ծ䎇腊䎾欽荛侨研ⴔ區ծデ险ծㄤ呏㔔ⴔ區ծ䎇腊䎾欽荛た絯的㾝涸腊⸂կ ⚺銳薴㔵 禹絡sdkծserverծbrowserծMP侨研䭷叻ծ靈欽Ꝇծ⠔霢ծ✲⟝ 2.䵨䒊侨研务〵㛇炄卹匬 呏研屛椚湡叻ㄤ薴㔵䵨䒊侨研务〵涸㛇炄卹匬 3.侨研䱹Ⰶ♸侮ざ 㼜ⴔ侔㖈♶ず禹絡ㄤ䎂〵涸侨研䱹Ⰶⵌ侨研务〵鵳遤絡♧㶸⪰ㄤ盗椚կ鸑鵂侨研侮ざㄤ幡峤嶊ꤑꅾ撑ծꝡ霴ㄤ♶♧荝涸侨研䲿넞侨研餘ꆀկ 4.⯋侨研盗椚 㖈侨研务〵务䒊用⯋侨研盗椚⡤禹雵䔶侨研涸㹁⛐ծ勻彂ծⰢ翫Ⱒ禹瘝⥌䜂կ鵯剣⸔✵䲿넞侨研涸〳椚鍒䚍倰⤑侨研涸唬程ㄤ⢪欽կ 5.侨研㸝Ⰼ♸ざ錞 烁⥂侨研务〵涸㸝Ⰼㄤꥧ猙㹊倶霄꡼䱽ⵖծ⸈㺙ծ㢊⼝瘝䲃倶 6.侨研剪⸉䒓佞 鸑鵂侨研剪⸉㽻㼜侨研务〵务涸侨研⟄BQLծ侨研䫣邍瘝䕎䒭䲿⣘絛⚌⸉鿈꡶կ鵯剣⸔ ✵䲿넞侨研ⵄ欽佪桧佅䭯⚌⸉Ɀ瘻ㄤ鵘蠒կ select dimension['serviceInstanceId']asserviceInstanceId, dimension['serviceId']asserviceId,frm(metric['one.service.instance.performance.apdex'])as"apdex", sum(metric['one.service.instance.performance.normalCount'])as"requestNormalCount999995", sum(metric['one.service.instance.error.errorRequestCount'])as"errorCount", sum(metric['one.service.instance.throughput.requestCount'])as"requestCount"from metric.data where env_id='default' andaccount_id=38894723 andmonitor_time>='2024-06-1213:36:03'andmonitor_time<'2024-06-1213:46:57' andaccount_id>0grou