【数据科学名家讲坛】On the Model-mispecification of Reinforcement Learning (Lin Yang, Assistant Professor, Electrical and Computer Engineering Department, University of California, Los Angeles)
主题:On the Model-mispecification of Reinforcement Learning
报告人:Lin Yang, Assistant Professor, Electrical and Computer Engineering Department, University of California, Los Angeles
主持人:Baoxiang Wang & Guiliang Liu, Assistant Professor, School of Data Science, CUHK-Shenzhen
日期:28 June (Wednesday), 2023
时间:11:00 AM to 12:00 PM, Beijing Time
形式:Hybrid
地点:401 Meeting Room, Daoyuan Building
SDS视频号直播:
语言:English
摘要:
The success of reinforcement learning (RL) heavily depends on the approximation of functions such as policy, value, or models. Misspecification—a mismatch between the ground-truth and the best function approximators—often occurs, particularly when the ground-truth is complex. Because the misspecification error does not disappear even with an infinite number of samples, it's crucial to design algorithms that demonstrate robustness under misspecification. In this talk, we will first present a lower bound illustrating that RL can be inefficient (e.g., possessing exponentially large complexity) if the features can only represent the optimal value functions approximately but with high precision. Subsequently, we will show that this issue can be mitigated by approximating the transition probabilities. In such a setting, we will demonstrate that both policy-based and value-based approaches can be resilient to model misspecifications. Specifically, we will show that these methods can maintain accuracy even under large, locally-bounded misspecification errors. Here, the function class might have a ξ\Omega(1)ξ approximation error in specific states and actions, but it remains small on average under a policy-induced state-distribution. Such robustness to model misspecification partially explains why practical algorithms perform so well, paving the way for new directions in understanding model misspecifications.
简介:
Dr. Lin Yang (杨林) is an assistant professor in the Electrical and Computer Engineering Department at the University of California, Los Angeles. His research focuses on developing and applying fast algorithms for machine learning and data science. He is keen on understanding the fundamental theory and computation limits of optimization for different machine learning problems. His current research focus is on reinforcement learning theory and applications, learning for control, non-convex optimization, and streaming algorithms. He was a recipient of the Simons' Research Fellowship, Dean Robert H. Roy Fellowship, and JHU MINDS best dissertation award.