AI and Macroeconomic Modeling: Deep Reinforcement Learning in an RBC model

AIandMacroeconomicModeling:DeepReinforcementLearninginanRBCModel TohidAtashbarandRui(Aruhan)ShiWP/23/40 IMFWorkingPapersdescriberesearchinprogressbytheauthor(s)andarepublishedtoelicitcommentsandtoencouragedebate. TheviewsexpressedinIMFWorkingPapersarethoseoftheauthor(s)anddonotnecessarilyrepresenttheviewsoftheIMF,itsExecutiveBoard,orIMFmanagement. 2023 FEB ©2023InternationalMonetaryFundWP/23/40 IMFWorkingPaper* Strategy,PolicyandReviewDepartment AIandMacroeconomicModeling:DeepReinforcementLearninginanRBCModelPreparedbyTohidAtashbarandRui(Aruhan)Shi AuthorizedfordistributionbyStephanDanningerFebruary2023 IMFWorkingPapersdescriberesearchinprogressbytheauthor(s)andarepublishedtoelicitcommentsandtoencouragedebate.TheviewsexpressedinIMFWorkingPapersarethoseoftheauthor(s)anddonotnecessarilyrepresenttheviewsoftheIMF,itsExecutiveBoard,orIMFmanagement. ABSTRACT:Thisstudyseekstoconstructabasicreinforcementlearning-basedAI-macroeconomicsimulator.WeuseadeepRL(DRL)approach(DDPG)inanRBCmacroeconomicmodel.Wesetuptwolearningscenarios,oneofwhichisdeterministicwithoutthetechnologicalshockandtheotherisstochastic.Theobjectiveofthedeterministicenvironmentistocomparethelearningagent'sbehaviortoadeterministicsteady-statescenario.Wedemonstratethatinbothdeterministicandstochasticscenarios,theagent'schoicesareclosetotheiroptimalvalue.Wealsopresentcasesofunstablelearningbehaviours.ThisAI-macromodelmaybeenhancedinfutureresearchbyaddingadditionalvariablesorsectorstothemodelorbyincorporatingdifferentDRLalgorithms. RECOMMENDEDCITATION:Atashbar,T.andShi,R.A.2023.“AIandMacroeconomicModeling:DeepReinforcementLearninginanRBCmodel”,IMFWorkingPapers,WP/22/40. JELClassificationNumbers: C63,C54;D83;D87;E37 Keywords: Reinforcementlearning;Deepreinforcementlearning;Artificialintelligence,RL;DRL;Learningalgorithms;Macromodeling,RBC;Realbusinesscycles;DDPG;Deepdeterministicpolicygradient;Actor-criticalgorithms Author’sE-MailAddress: tatashbar@imf.org;ashi@imf.org *TheauthorswouldliketothankStephanDanningerforhishelpfulcommentsandsuggestions.WeappreciatetheviewsandsuggestionsprovidedbyMicoMrkaic,DmitryPlotnikov,SergioRodriguezandattendeesattheIMFSPRMacroPolicyDivisionBrownbagSeminar.CommentsbyAllanDizioliarealsogratefullyacknowledged.Allerrorsremainourown. WORKINGPAPERS AIandMacroeconomicModeling:DeepReinforcementLearninginanRBCModel PreparedbyTohidAtashbarandRui(Aruhan)Shi Contents GLOSSARY3 INTRODUCTION4 I.ANOVERVIEWOFTHELITERATURE5 II.AREALBUSINESSCYCLE(RBC)MODEL8 A.Households8 B.Firms9 C.Functionalformsandparameters10 D.Adeterministicsteadystate10 III.AIEXPERIMENTS11 A.ExperimentI:deterministicenvironment15 B.ExperimentII:stochasticenvironment19 C.Issuesduringlearning22 IV.CONCLUSION24 ANNEXI.DDPGALGORITHM26 REFERENCES27 FIGURES Figure1.SL,ULandRLinML7 Figure2Laborhoursduringtraining(200episodes)17 Figure3Laborhourseriesduringtrainingandtesting17 Figure4Distancethesteadystate(SS)valuesforlaborhourandconsumption18 Figure5Productivityshockserieszt19 Figure6Simulatedseriesduring100testingperiods20 Figure7Laborhourchoicebeforeandafterlearning(200episode)21 Figure8Distancetodeterministicsteadystates(SS)forlaborhourandconsumption22 Figure9Distancetodeterministicsteadystates(SS)foroutputandinvestment22 Figure10Outputperunitoflabor23 Figure11Investmentperunitoflabor24 TABLES Table1.BaselineparametersforRBCmodel10 Table2Algorithmrelatedparameters13 Table3RLsetupoftheRBCmodel15 Glossary AGIArtificialGeneralIntelligenceAIArtificialIntelligence ANNArtificialNeuralNetworks DDPGDeepDeterministicPolicyGradientDLDeeplearning DNNDeepNeuralNetwork DPGDeterministicPolicyGradientDQNDeepQ-Network DRLDeepReinforcementLearning MADDPGMulti-AgentDeepDeterministicPolicyGradientRBCRealBusinessCycle RLReinforcementLearningSACSoftActor-Critic SLSupervisedLearningTD3TwinDelayedDDPGULUnsupervisedLearning Introduction Macroeconomicmodelingistheprocessofconstructingamodelthatdescribesthebehaviorofamacroeconomicsystem.Thisprocesscanbeusedtodeveloppredictionsaboutthefuturebehaviorofthesystem,tounderstandtherelationshipsbetweendifferentvariablesinthesystem,ortosimulatebehavior. Artificialintelligence(AI)isabranchofcomputersciencethatdealswiththedesignanddevelopmentofintelligentcomputersystems.AIresearchdealswiththequestionofhowtocreateprogramsthatarecapableofintelligentbehavior,i.e.,thekindofbehaviorthatisassociatedwithhumanbeings,suchasreasoning,learning,problem-solving,andactingautonomously. Thetwofieldscouldbeconceptuallycombined,asAItechniquescouldbeusedtodevelopmoreaccuratemacroeconomicmodels,oronecouldusemacroeconomicmodelstohelpdesignartificialgeneralintelligentsystemsthatarebetterabletosimulateeconomic(ormorebroadlysocial)behaviors,amongmanyothertasks.AIcanbeusedtoautomaticallyidentifyrelationshipsbetweenvariables,ortodevelopnewwaysofrepresentingeconomicsystems.AIcanalsobeusedtodevelopmethodsforautomaticallylearningfromdata,whichcanbeusedtoimprovetheaccuracyofpredictions.AIalsocouldbeusedtodevelopmoresophisticatedmodelsthattakeintoaccountawiderrangeoffactors,includingn