D4RL的踩坑记录

D4RL 的env.get_normalized_score

在调用d4rl的库函数时用了env.get_normalized_score
这里遇到的问题是
env has no attribute get_normalized_score

后记

D4RL的用法指南
D4RL本质上是一个数据库，用于离线的强化学习
D4RL的env和gym的环境并不同
简单来说，gym提供的是在线的交互式env，而D4RL则是离线的env
最直观的来看名字就不同

#gym
env = gym.make('Hopper')
#D4RL
env = gym.make('hopper-medium-v0')

分析

首先的分析的问题是env写法的问题

到D4RL官网看一看
Normalizing Scores
You can use the env.get_normalized_score(returns) function to compute a normalized score for an episode, where returns is the undiscounted total sum of rewards accumulated during an episode.
我在网上找到了一个用法，似乎没有问题

分析一下是不是D4RL的安装问题

protobuf

按照官方的下载路线之后得到了以下的报错
在这里插入图片描述
可以看到问题是protobuf
尝试的方法1
pip uninstall protobuf
pip install protobuf==3.20.1
打开根目录下的/d4rl后发现，下载的文件可能有问题
计划重新下载安装
rm -rf files

d4rl 的下载方案

git clone https://github.com/rail-berkeley/d4rl.git
cd d4rl
pip install -e .

遇到的报错为

fatal: unable to access ‘https://github.com/aravindr93/mjrl/’: GnuTLS recv error (-110): The TLS connection was non-properly terminated.
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with ‘git status’
and retry with 'git restore --source=HEAD 😕
尝试的解决方案为
问题成功解决了
重新安装成功了

运行安装测试

import gym
import d4rl # Import required to register environments

# Create the environment
env = gym.make('maze2d-umaze-v1')# d4rl abides by the OpenAI gym interface
env.reset()
env.step(env.action_space.sample())# Each task is associated with a dataset
# dataset contains observations, actions, rewards, terminals, and infos
dataset = env.get_dataset()
print(dataset['observations']) # An N x dim_observation Numpy array of observations# Alternatively, use d4rl.qlearning_dataset which
# also adds next_observations.

测试的结果

运行到env = gym.make(‘maze2d-umaze-v1’)遇到的问题是

  logger.warn(
pybullet build time: May 20 2022 19:44:17env = gym.make('maze2d-umaze-v1')
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 592, in makeenv = env_creator(**_kwargs)File "/d4rl/d4rl/pointmaze/maze_model.py", line 175, in __init__mujoco_env.MujocoEnv.__init__(self, model_path=f.name, frame_skip=1)File "/opt/conda/lib/python3.8/site-packages/gym/envs/mujoco/mujoco_env.py", line 97, in __init__observation, _reward, done, _info = self.step(action)File "/d4rl/d4rl/pointmaze/maze_model.py", line 192, in stepself.clip_velocity()File "/d4rl/d4rl/pointmaze/maze_model.py", line 222, in clip_velocityqvel = np.clip(self.sim.data.qvel, -5.0, 5.0)
AttributeError: 'MazeEnv' object has no attribute 'sim'

import d4rl 的 warning


```python
>>> import d4rl
Warning: Flow failed to import. Set the environment variable D4RL_SUPPRESS_IMPORT_ERROR=1 to suppress this message.
No module named 'flow.envs'
/opt/conda/lib/python3.8/site-packages/glfw/__init__.py:906: GLFWError: (65544) b'X11: The DISPLAY environment variable is missing'warnings.warn(message, GLFWError)
/opt/conda/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative usesimport imp
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.'nearest': pil_image.NEAREST,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.'bilinear': pil_image.BILINEAR,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.'bicubic': pil_image.BICUBIC,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.'hamming': pil_image.HAMMING,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.'box': pil_image.BOX,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.'lanczos': pil_image.LANCZOS,
Warning: CARLA failed to import. Set the environment variable D4RL_SUPPRESS_IMPORT_ERROR=1 to suppress this message.
No module named 'agents.navigation'
/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py:415: UserWarning: WARN: The `registry.env_specs` property along with `EnvSpecTree` is deprecated. Please use `registry` directly as a dictionary instead.logger.warn(
pybullet build time: May 20 2022 19:44:17