TorchRL 安装记录（亲测有效）

全部历史命令记录：

(torchrl_1) C:\Users\XCOS>doskey /history
conda env list
conda create -n torchrl_1 python=3.10 -y
conda activate torchrl_1
python -m pip install --upgrade pip setuptools wheel
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia -y
python -c "import torch; print('torch', torch.__version__); print('cuda_available', torch.cuda.is_available())"
python -m pip install "torchrl[utils]" tqdm tensorboard hydra-submitit-launcher
python -m pip install "gym[atari]" "gym[accept-rom-license]" pygame
python -m pip install dm_control
python -m pip install "moviepy<2.0.0"
python -m pip install wandb
pip list
doskey /history

环境：Anaconda、Cuda 12.1、两张3090

一、创建并激活虚拟环境

conda create -n torchrl_1 python=3.10 -y
conda activate torchrl_1

二、升级 pip / setuptools / wheel （非常重要）

python -m pip install --upgrade pip setuptools wheel   # 旧 pip 会导致 extras / PEP508 / 本地编译时出错 ——摘自Github

三、安装 PyTorch 及 Cuda （建议用win+conda）

如果你有 NVIDIA GPU，想用CUDA 12.1 则：

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia -y

如果没有 GPU ，则

conda install pytorch torchvision torchaudio cpuonly -c pytorch -y

四、验证 PyTorch 是否安装成功

python -c "import torch; print('torch', torch.__version__); print('cuda_available', torch.cuda.is_available())"

如果 cuda_available 为 True，表示 PyTorch 能访问 GPU，否则是 CPU-only（或驱动安装有问题）

五、安装 TorchRL （推荐通过 pip 在已激活的 conda 环境里执行）

只安装 core + utils，此为基本安装！

python -m pip install "torchrl[utils]"  

若想安装 TorchRL 的更多可选包（完整功能/渲染/环境/日志），可以把 README 列出的可选依赖一并装上（下面是常见组合的示例）：

常见额外工具（hydra, tqdm, tensorboard, submitit 等）

python -m pip install "torchrl[utils]" tqdm tensorboard hydra-submitit-launcher

如果要 Gym + Atari（经典游戏环境）：

python -m pip install "gym[atari]" "gym[accept-rom-license]" pygame

如果要 DeepMind Control Suite：

python -m pip install dm_control

渲染（moviepy）：

python -m pip install "moviepy<2.0.0"

可选：wandb（权重与偏差）：

python -m pip install wandb

六、安装后的验证

python -c "import torch, torchrl; print('torch', torch.__version__); import importlib; print('torchrl import ok', importlib.import_module('torchrl'))"
python -c "import torch; print('cuda', torch.cuda.is_available(), torch.cuda.device_count())"

输出显示如下：

证明 TorchRL安装成功！

七、程序样例运行

修改pycharm解释器为虚拟环境 torchrl_1 下的 python.exe，直接运行参考网址[3]中的python程序即可：

import torch
from tensordict.nn import TensorDictModule
from tensordict.nn.distributions import NormalParamExtractor
from torch import nnfrom torchrl.collectors import SyncDataCollector
from torchrl.data.replay_buffers import TensorDictReplayBuffer, \LazyTensorStorage, SamplerWithoutReplacement
from torchrl.envs.libs.gym import GymEnv
from torchrl.modules import ProbabilisticActor, ValueOperator, TanhNormal
from torchrl.objectives import ClipPPOLoss
from torchrl.objectives.value import GAEenv = GymEnv("Pendulum-v1")
model = TensorDictModule(nn.Sequential(nn.Linear(3, 128), nn.Tanh(),nn.Linear(128, 128), nn.Tanh(),nn.Linear(128, 128), nn.Tanh(),nn.Linear(128, 2),NormalParamExtractor()),in_keys=["observation"],out_keys=["loc", "scale"]
)
critic = ValueOperator(nn.Sequential(nn.Linear(3, 128), nn.Tanh(),nn.Linear(128, 128), nn.Tanh(),nn.Linear(128, 128), nn.Tanh(),nn.Linear(128, 1),),in_keys=["observation"],
)
actor = ProbabilisticActor(model,in_keys=["loc", "scale"],distribution_class=TanhNormal,distribution_kwargs={"min": -1.0, "max": 1.0},return_log_prob=True)
buffer = TensorDictReplayBuffer(LazyTensorStorage(1000),SamplerWithoutReplacement())
collector = SyncDataCollector(env,actor,frames_per_batch=1000,total_frames=1_000_000)
loss_fn = ClipPPOLoss(actor, critic, gamma=0.99)
optim = torch.optim.Adam(loss_fn.parameters(), lr=2e-4)
adv_fn = GAE(value_network=critic, gamma=0.99, lmbda=0.95, average_gae=True)
for data in collector:  # collect datafor epoch in range(10):adv_fn(data)  # compute advantagebuffer.extend(data.view(-1))for i in range(20):  # consume datasample = buffer.sample(50)  # mini-batchloss_vals = loss_fn(sample)loss_val = sum(value for key, value in loss_vals.items() ifkey.startswith("loss"))loss_val.backward()optim.step()optim.zero_grad()print(f"avg reward: {data['next', 'reward'].mean().item(): 4.4f}")

程序样例运行成功：

（其中，Gym尚有问题，若需要，请自行解决）

参考网址：

[1] https://github.com/pytorch/rl
[2] https://chatgpt.com/c/68f8b81e-8518-8320-bebe-f0d98c22f18a
[3] https://blog.csdn.net/HGGshiwo/article/details/133698735