热门搜索：

【英文原版】StableDiffusion3技术报告

信息技术2024-03-13-Stability AI葛***

AI智能总结

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Summary

The research focuses on enhancing the capabilities of rectified flow models for high-resolution image synthesis through innovative noise sampling techniques and a novel transformer-based architecture.

Key Contributions:

Noise Scaling Improvement: The introduction of a re-weighting method for noise scales in rectified flow models, akin to noise-predictive diffusion models, significantly boosts performance over traditional approaches.
Bidirectional Architecture: A new architecture is presented that incorporates learnable streams for both image and text tokens, enabling a two-way flow of information between them, which improves text comprehension, typography, and human preference ratings.
Predictable Scaling Trends: The model demonstrates predictable scaling trends in validation loss, showing strong correlations with improved text-to-image synthesis assessments through quantitative metrics and human evaluations.

Core Findings:

The novel architecture outperforms established methods like SDXL, SDXL Turbo, Pixart-α, and DALL-E 3 in both quantitative evaluations and human preference ratings.
The research includes publicly available results, code, and model weights, contributing valuable resources to the field of text-to-image synthesis.

Methodology Highlights:

The study compares various diffusion model formulations and rectified flow approaches systematically, identifying optimal settings through a large-scale investigation.
The introduction of new noise samplers for rectified flow models enhances performance over existing samplers.
The novel architecture combines with improved rectified flow formulation to explore scalability, demonstrating predictable trends in validation loss correlation with enhanced text-to-image performance.

Impact and Significance: This work advances the state-of-the-art in high-resolution image synthesis, offering a more efficient and effective method for generating images from textual inputs, with implications for various applications requiring high-quality visual content creation.

ScalingRectifiedFlowTransformersforHigh-ResolutionImageSynthesis PatrickEsser*SumithKulalAndreasBlattmannRahimEntezariJonasMu¨llerHarrySainiYamLeviDominikLorenzAxelSauerFredericBoeselDustinPodellTimDockhornZionEnglish KyleLaceyAlexGoodwinYannikMarekRobinRombach* StabilityAI Figure1.High-resolutionsamplesfromour8Brectifiedflowmodel,showcasingitscapabilitiesintypography,precisepromptfollowingandspatialreasoning,attentiontofinedetails,andhighimagequalityacrossawidevarietyofstyles. Abstract Diffusionmodelscreatedatafromnoisebyinvert-ingtheforwardpathsofdatatowardsnoiseandhaveemergedasapowerfulgenerativemodelingtechniqueforhigh-dimensional,perceptualdatasuchasimagesandvideos.Rectifiedflowisare-centgenerativemodelformulationthatconnectsdataandnoiseinastraightline.Despiteitsbettertheoreticalpropertiesandconceptualsimplicity,itisnotyetdecisivelyestablishedasstandardprac-tice.Inthiswork,weimproveexistingnoisesam-plingtechniquesfortrainingrectifiedflowmod-elsbybiasingthemtowardsperceptuallyrelevantscales.Throughalarge-scalestudy,wedemon- *Equalcontribution.<first.last>@stability.ai. stratethesuperiorperformanceofthisapproachcomparedtoestablisheddiffusionformulationsforhigh-resolutiontext-to-imagesynthesis.Ad-ditionally,wepresentanoveltransformer-basedarchitecturefortext-to-imagegenerationthatusesseparateweightsforthetwomodalitiesanden-ablesabidirectionalflowofinformationbetweenimageandtexttokens,improvingtextcomprehen-sion,typography,andhumanpreferenceratings.Wedemonstratethatthisarchitecturefollowspre-dictablescalingtrendsandcorrelateslowervali-dationlosstoimprovedtext-to-imagesynthesisasmeasuredbyvariousmetricsandhumanevalua-tions.Ourlargestmodelsoutperformstate-of-the-artmodels,andwewillmakeourexperimentaldata,code,andmodelweightspubliclyavailable. 1.Introduction Diffusionmodelscreatedatafromnoise(Songetal.,2020).Theyaretrainedtoinvertforwardpathsofdatatowardsrandomnoiseand,thus,inconjunctionwithapproximationandgeneralizationpropertiesofneuralnetworks,canbeusedtogeneratenewdatapointsthatarenotpresentinthetrainingdatabutfollowthedistributionofthetrainingdata(Sohl-Dicksteinetal.,2015;Song&Ermon,2020).Thisgenerativemodelingtechniquehasproventobeveryeffectiveformodelinghigh-dimensional,perceptualdatasuchasimages(Hoetal.,2020).Inrecentyears,diffusionmodelshavebecomethede-factoapproachforgeneratinghigh-resolutionimagesandvideosfromnaturallanguageinputswithimpressivegeneralizationcapabilities(Sahariaetal.,2022b;Rameshetal.,2022;Rombachetal.,2022;Podelletal.,2023;Daietal.,2023;Esseretal.,2023;Blattmannetal.,2023b;Betkeretal.,2023;Blattmannetal.,2023a;Singeretal.,2022).Duetotheiriterativenatureandtheassociatedcomputationalcosts,aswellasthelongsamplingtimesduringinference,researchonformulationsformoreefficienttrainingand/orfastersamplingofthesemodelshasincreased(Karrasetal.,2023;Liuetal.,2022). Whilespecifyingaforwardpathfromdatatonoiseleadstoefficienttraining,italsoraisesthequestionofwhichpathtochoose.Thischoicecanhaveimportantimplicationsforsampling.Forexample,aforwardprocessthatfailstoremoveallnoisefromthedatacanleadtoadiscrepancyintrainingandtestdistributionandresultinartifactssuchasgrayimagesamples(Linetal.,2024).Importantly,thechoiceoftheforwardprocessalsoinfluencesthelearnedbackwardprocessand,thus,thesamplingefficiency.Whilecurvedpathsrequiremanyintegrationstepstosimulatetheprocess,astraightpathcouldbesimulatedwithasinglestepandislesspronetoerroraccumulation.Sinceeachstepcorrespondstoanevaluationoftheneuralnetwork,thishasadirectimpactonthesamplingspeed. Aparticularchoicefortheforwardpathisaso-calledRec-tifiedFlow(Liuetal.,2022;Albergo&Vanden-Eijnden,2022;Lipmanetal.,2023),whichconnectsdataandnoiseonastraightline.Althoughthismodelclasshasbettertheoreticalproperties,ithasnotyetbecomedecisivelyes-tablishedinpractice.Sofar,someadvantageshavebeenempiricallydemonstratedinsmallandmedium-sizedex-periments(Maetal.,2024),butthesearemostlylimitedtoclass-conditionalmodels.Inthiswork,wechangethisbyin-troducingare-weightingofthenoisescalesinrectifiedflowmodels,similartonoise-predictivediffusionmodels(Hoetal.,2020).Throughalarge-scalestudy,wecompareournewformulationtoexistingdiffusionformulationsanddemonstrateitsbenefits. Weshowthatthewidelyusedapproachfortext-to-imagesynthesis,whereafixedtextrepresentationisfeddirectly intothemodel(e.g.,viacross-attention(Vaswanietal.,2017;Rombachetal.,2022)),isnotideal,andpresentanewarchitecturethatincorporateslearnablestreamsforbothimageandtexttokens,whichenablesatwo-wayflowofinformationbetweenthem.Wecombinethiswithourimprovedrectifiedflowformulationandinvestigateitsscala-bility.Wedemonstrateapredictablescalingtrendintheval-idationlossandshowthatalowervalidationlosscorrelatesstronglywithimprovedautomaticandhumanevaluations. Ourlargestmodelsoutperformstate-of-theartopenmodelssuchasSDXL(Podelletal.,2023),SDXL-Turbo(Saueretal.,2023),Pixart-α(Chenetal.,2023),andclosed-sourcemodelssuchasDALL-E3(Betkeretal.,2023)bothinquantitativeevaluation(Ghoshetal.,2023)ofpromptun-derstandingandhumanpreferencera

点击免费查看完整报告

你可能感兴趣

【英文原版】StableDiffusion3技术报告

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Summary

你可能感兴趣

【中文译版】StableDiffusion3技术报告

外部报告·联合国AI报告：发展4.0（英文原版）

燃料电池在道路交通的部署现状20212021 Deployment status of fc in road transport（IEA英文原版）

2023年巴菲特股东信英文原版

2021巴菲特股东信英文原版