强化学习_06_pytorch-PPO2实践(Humanoid-v4)

news/2024/11/15 4:57:28/

一、PPO优化

PPO的简介和实践可以看笔者之前的文章 强化学习_06_pytorch-PPO实践(Pendulum-v1)
针对之前的PPO做了主要以下优化:

-笔者-PPO笔者-PPO2ref
data collectone episodeseveral episode(one batch)
activationReLUTanh
adv-compute-compute adv as one serires
adv-normalizemini-batch normalizeservel envs-batch normalize影响PPO算法性能的10个关键技巧
Value Function Loss Clipping- L V = m a x [ ( V θ t − V t a r ) 2 , ( c l i p ( V θ t , V θ t − 1 − ϵ , V θ t − 1 + ϵ ) ) 2 ] L^{V}=max[(V_{\theta_t} - V_{tar})^2, (clip(V_{\theta_t}, V_{\theta_{t-1}}-\epsilon, V_{\theta_{t-1}}+\epsilon))^2] LV=max[(VθtVtar)2,(clip(Vθt,Vθt1ϵ,Vθt1+ϵ))2]The 37 Implementation Details of Proximal Policy Optimization
optimizeractor-opt & critic-optuse common opt
lossactor-loss-backward & critic-loss-backwardloss weight sum
paramate-init-1- hidden layer orthogonal initialization of weights 2 \sqrt{2} 2 ; 2- The policy output layer weights are initialized with the scale of 0.01; 3- The value output layer weights are initialized with the scale of 1.0The 37 Implementation Details of Proximal Policy Optimization
training envssingle gym envSyncVectorEnv

相比于PPO2_old.py 这次实现了上述的全部优化,

1.1 PPO2 代码

详细可见 Github: PPO2.py

python">
class PPO:"""PPO算法, 采用截断方式"""def __init__(self,state_dim: int,actor_hidden_layers_dim: typ.List,critic_hidden_layers_dim: typ.List,action_dim: int,actor_lr: float,critic_lr: float,gamma: float,PPO_kwargs: typ.Dict,device: torch.device,reward_func: typ.Optional[typ.Callable]=None):dist_type = PPO_kwargs.get('dist_type', 'beta')self.dist_type = dist_typeself.actor = policyNet(state_dim, actor_hidden_layers_dim, action_dim, dist_type=dist_type).to(device)self.critic = valueNet(state_dim, critic_hidden_layers_dim).to(device)self.actor_lr = actor_lrself.critic_lr = critic_lrself.actor_opt = torch.optim.Adam(self.actor.parameters(), lr=actor_lr)self.critic_opt = torch.optim.Adam(self.critic.parameters(), lr=critic_lr)self.gamma = gammaself.lmbda = PPO_kwargs['lmbda']self.k_epochs = PPO_kwargs['k_epochs'] # 一条序列的数据用来训练的轮次self.eps = PPO_kwargs['eps'] # PPO中截断范围的参数self.sgd_batch_size = PPO_kwargs.get('sgd_batch_size', 512)self.minibatch_size = PPO_kwargs.get('minibatch_size', 128)self.action_bound = PPO_kwargs.get('action_bound', 1.0)self.action_low = torch.FloatTensor([-1 * self.action_bound]).to(device)self.action_high = torch.FloatTensor([self.action_bound]).to(device)if 'action_space' in PPO_kwargs:self.action_low = torch.FloatTensor(PPO_kwargs['action_space'].low).to(device)self.action_high = torch.FloatTensor(PPO_kwargs['action_space'].high).to(device)self.count = 0 self.device = deviceself.reward_func = reward_funcself.min_batch_collate_func = partial(mini_batch, mini_batch_size=self.minibatch_size)def _action_fix(self, act):if self.dist_type == 'beta':# beta 0-1 -> low ~ highreturn act * (self.action_high - self.action_low) + self.action_lowreturn act def _action_return(self, act):if self.dist_type == 'beta':# low ~ high -> 0-1 act_out = (act - self.action_low) / (self.action_high - self.action_low)return act_out * 1 + 0return act def policy(self, state):state = torch.FloatTensor(np.array([state])).to(self.device)action_dist = self.actor.get_dist(state, self.action_bound)action = action_dist.sample()action = self._action_fix(action)return action.cpu().detach().numpy()[0]def update(self, samples: deque):state, action, reward, next_state, done = zip(*samples)state = torch.FloatTensor(np.stack(state)).to(self.device)action = torch.FloatTensor(np.stack(action)).to(self.device)reward = torch.tensor(np.stack(reward)).view(-1, 1).to(self.device)if self.reward_func is not None:reward = self.reward_func(reward)next_state = torch.FloatTensor(np.stack(next_state)).to(self.device)done = torch.FloatTensor(np.stack(done)).view(-1, 1).to(self.device)old_v = self.critic(state)td_target = reward + self.gamma * self.critic(next_state) * (1 - done)td_delta = td_target - old_vadvantage = compute_advantage(self.gamma, self.lmbda, td_delta, done).to(self.device)# recomputetd_target = advantage + old_v# trick1: batch_normalizeadvantage = (advantage - torch.mean(advantage)) / (torch.std(advantage) + 1e-5)action_dists = self.actor.get_dist(state, self.action_bound)# 动作是正态分布old_log_probs = action_dists.log_prob(self._action_return(action))if len(old_log_probs.shape) == 2:old_log_probs = old_log_probs.sum(dim=1)d_set = memDataset(state, action, old_log_probs, advantage, td_target)train_loader = DataLoader(d_set,batch_size=self.sgd_batch_size,shuffle=True,drop_last=True,collate_fn=self.min_batch_collate_func)for _ in range(self.k_epochs):for state_, action_, old_log_prob, adv, td_v in train_loader:action_dists = self.actor.get_dist(state_, self.action_bound)log_prob = action_dists.log_prob(self._action_return(action_))if len(log_prob.shape) == 2:log_prob = log_prob.sum(dim=1)# e(log(a/b))ratio = torch.exp(log_prob - old_log_prob.detach())surr1 = ratio * advsurr2 = torch.clamp(ratio, 1 - self.eps, 1 + self.eps) * advactor_loss = torch.mean(-torch.min(surr1, surr2)).float()critic_loss = torch.mean(F.mse_loss(self.critic(state_).float(), td_v.detach().float())).float()self.actor_opt.zero_grad()self.critic_opt.zero_grad()actor_loss.backward()critic_loss.backward()torch.nn.utils.clip_grad_norm_(self.actor.parameters(), 0.5) torch.nn.utils.clip_grad_norm_(self.critic.parameters(), 0.5) self.actor_opt.step()self.critic_opt.step()return Truedef save_model(self, file_path):if not os.path.exists(file_path):os.makedirs(file_path)act_f = os.path.join(file_path, 'PPO_actor.ckpt')critic_f = os.path.join(file_path, 'PPO_critic.ckpt')torch.save(self.actor.state_dict(), act_f)torch.save(self.critic.state_dict(), critic_f)def load_model(self, file_path):act_f = os.path.join(file_path, 'PPO_actor.ckpt')critic_f = os.path.join(file_path, 'PPO_critic.ckpt')self.actor.load_state_dict(torch.load(act_f, map_location='cpu'))self.critic.load_state_dict(torch.load(critic_f, map_location='cpu'))self.actor.to(self.device)self.critic.to(self.device)self.actor_opt = torch.optim.Adam(self.actor.parameters(), lr=self.actor_lr)self.critic_opt = torch.optim.Adam(self.critic.parameters(), lr=self.critic_lr)def train(self):self.training = Trueself.actor.train()self.critic.train()def eval(self):self.training = Falseself.actor.eval()self.critic.eval()

1.2 ppo2_train

其实就是向量环境多个step进行一次ppo update
详细可见 Github: ppo2_train

python">
def ppo2_train(envs, agent, cfg, wandb_flag=False, train_without_seed=False, step_lr_flag=False, step_lr_kwargs=None, test_ep_freq=100,online_collect_nums=1024,test_episode_count=3,wandb_project_name="RL-train_on_policy",add_max_step_reward_flag=False):test_env = envs.envs[0]env_id = str(test_env).split('>')[0].split('<')[-1]if wandb_flag:wandb.login()cfg_dict = cfg.__dict__if step_lr_flag:cfg_dict['step_lr_flag'] = step_lr_flagcfg_dict['step_lr_kwargs'] = step_lr_kwargsalgo = agent.__class__.__name__now_ = datetime.now().strftime('%Y%m%d__%H%M')wandb.init(project=wandb_project_name,name=f"{algo}__{env_id}__{now_}",config=cfg_dict,monitor_gym=True)mini_b = cfg.PPO_kwargs.get('minibatch_size', 12)if step_lr_flag:opt = agent.actor_opt if hasattr(agent, "actor_opt") else agent.optschedule = StepLR(opt, step_size=step_lr_kwargs['step_size'], gamma=step_lr_kwargs['gamma'])tq_bar = tqdm(range(cfg.num_episode))rewards_list = []now_reward = 0recent_best_reward = -np.infupdate_flag = Falsebest_ep_reward = -np.infbuffer_ = replayBuffer(cfg.off_buffer_size)steps = 0rand_seed = np.random.randint(0, 9999)final_seed = rand_seed if train_without_seed else cfg.seeds, _ = envs.reset(seed=final_seed)for i in tq_bar:if update_flag:buffer_ = replayBuffer(cfg.off_buffer_size)tq_bar.set_description(f'Episode [ {i+1} / {cfg.num_episode} ](minibatch={mini_b})')    step_rewards = np.zeros(envs.num_envs)step_reward_mean = 0.0for step_i in range(cfg.off_buffer_size):a = agent.policy(s)n_s, r, terminated, truncated, infos = envs.step(a)done = np.logical_or(terminated, truncated)steps += 1mem_done = done buffer_.add(s, a, r, n_s, mem_done)s = n_sstep_rewards += rif (steps % test_ep_freq == 0) and (steps > cfg.off_buffer_size):freq_ep_reward = play(test_env, agent, cfg, episode_count=test_episode_count, play_without_seed=train_without_seed, render=False, ppo_train=True)if freq_ep_reward > best_ep_reward:best_ep_reward = freq_ep_reward# 模型保存save_agent_model(agent, cfg, f"[ ep={i+1} ](freqBest) bestTestReward={best_ep_reward:.2f}")max_step_flag = (step_i == (cfg.off_buffer_size - 1)) and add_max_step_reward_flagif max_step_flag:step_reward_mean = step_rewards.mean()if (("final_info" in infos) or max_step_flag) and step_i >= 5:info_counts = 0.0001episode_rewards = 0for info in infos.get("final_info", dict()):if info and "episode" in info:# print(f"global_step={step_i}, episodic_return={info['episode']['r']}")if isinstance(info["episode"]["r"], np.ndarray):episode_rewards += info["episode"]["r"][0]else:episode_rewards += info["episode"]["r"]info_counts += 1# if(steps % cfg.max_episode_steps == 0):rewards_list.append(max(episode_rewards/info_counts, step_reward_mean))# print(rewards_list[-10:])  0: in buffer_size step not get any pointnow_reward = np.mean(rewards_list[-10:])if max_step_flag:step_reward_mean = 0.0if (now_reward > recent_best_reward):# best 时也进行测试test_ep_reward = play(test_env, agent, cfg, episode_count=test_episode_count, play_without_seed=train_without_seed, render=False, ppo_train=True)if test_ep_reward > best_ep_reward:best_ep_reward = test_ep_reward# 模型保存save_agent_model(agent, cfg, f"[ ep={i+1} ](recentBest) bestTestReward={best_ep_reward:.2f}")recent_best_reward = now_rewardtq_bar.set_postfix({'lastMeanRewards': f'{now_reward:.2f}', 'BEST': f'{recent_best_reward:.2f}',"bestTestReward": f'{best_ep_reward:.2f}'})if wandb_flag:log_dict = {'lastMeanRewards': now_reward,'BEST': recent_best_reward,"episodeRewards": episode_rewards,"bestTestReward": best_ep_reward}if step_lr_flag:log_dict['actor_lr'] = opt.param_groups[0]['lr']wandb.log(log_dict)update_flag = agent.update(buffer_.buffer, wandb=wandb if wandb_flag else None)if step_lr_flag:schedule.step()envs.close()if wandb_flag:wandb.finish()return agent

二、 Pytorch实践

2.1 智能体构建与训练

PPO2主要是收集n_envs * n_step的结果序列进行训练,针对Humanoid-v4,需要同时对多个环境进行游戏采样(num_envs = 128),同时环境的步数需要进行尝试(笔者尝试了[60, 64, 70, 75, 80, 100, 120, 159, 164)最终采用n_step=80。这里还有一个非常重要的是需要对环境进行NormalizeObservation

2.1.1 NormalizeObservation

下图是几个训练的比较好的未进行环境Normalize的BestTestReward VS 进行环境Normalize的BestTestReward。 所有1500分以上的均是环境Normalize的
在这里插入图片描述
NormalizeObservation的核心就是RunningMeanStd,每个step都对环境进行迭代

python"># update_mean_var_count_from_momentsdelta = batch_mean - meantot_count = count + batch_countnew_mean = mean + delta * batch_count / tot_countm_a = var * countm_b = batch_var * batch_countM2 = m_a + m_b + np.square(delta) * count * batch_count / tot_countnew_var = M2 / tot_countnew_count = tot_count

2.1.2 进行训练

详细可见 Github: test_ppo.Humanoid_v4_ppo2_test

python">env_name = 'Humanoid-v4'
num_envs = 128 #64
gym_env_desc(env_name)
print("gym.__version__ = ", gym.__version__ )
path_ = os.path.dirname(__file__)
norm_flag = True
reward_flag = False
envs = gym.vector.SyncVectorEnv([make_env(env_name, obs_norm_trans_flag=norm_flag, reward_norm_trans_flag=reward_flag) for _ in range(num_envs)]
)
dist_type = 'beta'
cfg = Config(envs, # 环境参数save_path=os.path.join(path_, "test_models" ,f'PPO_Humanoid-v4-{norm_flag}-1'), seed=202405,# 网络参数actor_hidden_layers_dim=[128, 128, 128],critic_hidden_layers_dim=[128, 128, 128],# agent参数actor_lr=4.5e-4, gamma=0.99,# 训练参数num_episode=3000, off_buffer_size=80, # batch_size = off_buffer_size * num_envmax_episode_steps=80,PPO_kwargs={'lmbda': 0.985, 'eps': 0.125,  'k_epochs': 3,'sgd_batch_size': 2048, # 1024, # 512,'minibatch_size': 1024,  # 512,  # 64,'action_space': envs.single_action_space,'act_type': 'tanh','dist_type': dist_type,'critic_coef': 1,'max_grad_norm': 3.5, # 45.5'clip_vloss': True,# 'min_adv_norm': True,'anneal_lr': False, # not work'num_episode': 3000}
)
cfg.test_max_episode_steps = 300
cfg.num_envs = num_envs
minibatch_size = cfg.PPO_kwargs['minibatch_size']
max_grad_norm = cfg.PPO_kwargs['max_grad_norm']
agent = PPO2(state_dim=cfg.state_dim,actor_hidden_layers_dim=cfg.actor_hidden_layers_dim,critic_hidden_layers_dim=cfg.critic_hidden_layers_dim,action_dim=cfg.action_dim,actor_lr=cfg.actor_lr,critic_lr=cfg.critic_lr,gamma=cfg.gamma,PPO_kwargs=cfg.PPO_kwargs,device=cfg.device,reward_func=None
)
agent.train()
ppo2_train(envs, agent, cfg, wandb_flag=True, wandb_project_name=f"PPO2-{env_name}",train_without_seed=False, test_ep_freq=cfg.off_buffer_size * 10, online_collect_nums=cfg.off_buffer_size,test_episode_count=10)
# save norm env
save_env(envs.envs[0], os.path.join(cfg.save_path, 'norm_env.pkl'))

2.2 训练出的智能体观测

最后将训练的最好的网络拿出来进行观察,这里需要注意:我们在训练的时候对环境进行了Normalize,所以在环境初始化的时候,需要将obs_rms (即 RunningMeanStd)中的mean, var, count进行初始化,然后再play

python">agent.load_model(cfg.save_path)
agent.eval()with open(os.path.join(cfg.save_path, 'norm_env.pkl'), 'rb') as f:env = cloudpickle.load(f)# p = '/home/scc/sccWork/myGitHub/RL/src/test/test_models/PPO_Humanoid-v4-True-2/norm_env.pkl'
# with open(p, 'rb') as f:
#     env = cloudpickle.load(f)obs_rms = env.get_wrapper_attr('env').get_wrapper_attr("obs_rms")
env = make_env(env_name, obs_norm_trans_flag=norm_flag, render_mode='human')()
env.get_wrapper_attr('env').get_wrapper_attr("obs_rms").mean = obs_rms.mean
env.get_wrapper_attr('env').get_wrapper_attr("obs_rms").var = obs_rms.var
env.get_wrapper_attr('env').get_wrapper_attr("obs_rms").count = obs_rms.count
# env = make_env(env_name, obs_norm_trans_flag=norm_flag)()
# cfg.max_episode_steps = 1020 
play(env, agent, cfg, episode_count=3, play_without_seed=False, render=True)

在这里插入图片描述


http://www.ppmy.cn/news/1463490.html

相关文章

C-数据结构-树状存储基本概念

‘’’ 树状存储基本概念 深度&#xff08;层数&#xff09; 度&#xff08;子树个数&#xff09; 叶子 孩子 兄弟 堂兄弟 二叉树&#xff1a; 满二叉树&#xff1a; 完全二叉树&#xff1a; 存储&#xff1a;顺序&#xff0c;链式 树的遍历&#xff1a;按层遍历&#xff0…

springboot相关知识集锦----1

一、springboot是什么&#xff1f; springboot是一个用于构建基于spring框架的独立应用程序的框架。它采用自动配置的原则&#xff0c;以减少开发人员在搭建应用方面的时间和精力。同时提升系统的可维护性和可扩展性。 二、springboot的优点 约定优于配置 版本锁定&#xf…

【建议收藏】30个较难Python脚本,纯干货分享

本篇较难&#xff0c;建议优先学习上篇 &#xff1b;20个硬核Python脚本-CSDN博客 接上篇文章&#xff0c;对于Pyhon的学习&#xff0c;上篇学习的结束相信大家对于Pyhon有了一定的理解和经验&#xff0c;学习完上篇文章之后再研究研究剩下的30个脚本你将会有所成就&…

《AI学习笔记》大模型-微调/训练区别以及流程

阿丹&#xff1a; 之前一直对于大模型的微调和训练这两个名词不是很清晰&#xff0c;所有找了一个时间来弄明白到底有什么区别以及到底要怎么去使用去做。并且上手实践一下。 大模型业务全流程&#xff1a; 大模型为啥要微调&#xff1f;有哪些微调方式&#xff1f; 模型参数…

C++:vector基础讲解

hello&#xff0c;各位小伙伴&#xff0c;本篇文章跟大家一起学习《C&#xff1a;vector基础讲解》&#xff0c;感谢大家对我上一篇的支持&#xff0c;如有什么问题&#xff0c;还请多多指教 &#xff01; 如果本篇文章对你有帮助&#xff0c;还请各位点点赞&#xff01;&#…

Nodejs 第七十三章(网关层)

什么是网关层(getway)&#xff1f; 技术选型fastify 速度快适合网关层 fastify教程上一章有讲 网关层是位于客户端和后端服务之间的中间层&#xff0c;用于处理和转发请求。它充当了请求的入口点&#xff0c;并负责将请求路由到适当的后端服务&#xff0c;并将后端服务的响应…

STM32无法向设备写入固件

&#x1f3c6;本文收录于「Bug调优」专栏&#xff0c;主要记录项目实战过程中的Bug之前因后果及提供真实有效的解决方案&#xff0c;希望能够助你一臂之力&#xff0c;帮你早日登顶实现财富自由&#x1f680;&#xff1b;同时&#xff0c;欢迎大家关注&&收藏&&…

校园一体式气象站功能特点与应用

TH-XQ4随着教育理念的更新和科技的发展&#xff0c;校园一体式气象站在现代校园中的应用越来越广泛。它不仅为学生提供了实时、准确的气象数据&#xff0c;还成为了推动科学教育、培养学生实践能力和环保意识的重要工具。本文将从功能、特点和应用三个方面对校园一体式气象站进…