当前位置： 首页 > news >正文

【深度学习】强化学习（RL）-A3C（Asynchronous Advantage Actor-Critic）

news 来源：原创 2025/9/13 16:01:03

A3C（Asynchronous Advantage Actor-Critic）详解

A3C（Asynchronous Advantage Actor-Critic） 是 深度强化学习（Deep Reinforcement Learning, DRL） 领域的重要算法，由 DeepMind 在 2016 年提出。它是一种异步并行的 Actor-Critic 方法，用于解决强化学习中收敛慢、数据利用率低等问题，适用于复杂的高维度环境（如 Atari 游戏、机器人控制等）。

1. A3C 的核心思想

A3C 是 Actor-Critic 方法的增强版，主要改进点是：

  • 通过异步（Asynchronous）并行地训练多个代理（Agent），提高训练效率，减少计算资源的浪费。
  • 采用 优势函数（Advantage Function） 来减少估计值的方差，从而提高训练的稳定性和收敛速度。
  • 结合 策略梯度方法（Policy Gradient） 和 值函数估计（Value Function Estimation），平衡探索（Exploration）和利用（Exploitation）。

A3C 相比 DQN（Deep Q-Network） 的最大不同点在于：

  1. 无经验回放（No Experience Replay）：A3C 不需要存储和采样经验数据，而是让多个 Agent 在不同环境中异步探索，提高了训练稳定性。
  2. 异步更新（Asynchronous Updates）：多个 Agent 以不同的时间间隔独立探索环境，减少了训练数据的相关性，提高了探索的多样性。

2. A3C 的网络架构

A3C 采用 Actor-Critic 结构，包含两个关键的网络：

  1. Actor（策略网络，Policy Network）：

    • 负责学习策略（Policy），即在每个状态下采取不同动作的概率分布 \pi(a|s;\theta)。
    • 通过梯度上升优化策略，使得高奖励的动作更可能被选中。

  2. Critic（值函数网络，Value Network）：

    • 负责学习状态值函数 V(s;\theta_v)，即在当前状态 s 下，未来可能获得的长期奖励。
    • 计算 优势函数（Advantage Function），用于指导 Actor 更新策略。

A3C 采用的目标函数

A3C 采用 Actor-Critic 方法，即：

  • 策略更新（Actor）：

    \nabla_{\theta} J = \mathbb{E} \left[ \nabla_{\theta} \log \pi(a | s; \theta) A(s, a) \right]

    其中，优势函数（Advantage Function）：

    A(s, a) = R - V(s;\theta_v)

    这里，RRR 是从当前状态出发获得的实际奖励，V(s;\theta_v) 是 Critic 估计的状态值。

  • Critic 更新（使用均方误差）：

    L_v(\theta_v) = (R - V(s;\theta_v))^2

    Critic 通过最小化均方误差来优化值函数，使其能够更准确地估计状态值。

3. A3C 的训练流程

A3C 采用 多个异步环境并行运行，每个环境都有自己的 Actor 和 Critic，它们共享全局参数 θ 和 \theta_v，训练过程如下：

  1. 初始化全局网络参数 θ 和 \theta_v​。
  2. 创建多个 Worker（子代理），每个 Worker 独立运行：
    • 与环境交互，收集数据（状态、动作、奖励）。
    • 计算优势函数和目标值。
    • 计算梯度并更新全局网络参数。
  3. 所有 Worker 异步执行，独立探索，最终收敛到较优的策略。

A3C 通过 多线程并行执行，使得不同 Worker 在不同环境中探索，提高了数据效率和稳定性。

4. A3C 代码实现（PyTorch）

以下是 A3C 的简化版本，包括 Actor-Critic 网络和训练流程。

Actor-Critic 网络

import torch
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
import gym
import numpy as np# 定义全局 Actor-Critic 网络
class ActorCritic(nn.Module):def __init__(self, input_dim, action_dim):super(ActorCritic, self).__init__()self.fc1 = nn.Linear(input_dim, 128)self.actor = nn.Linear(128, action_dim)self.critic = nn.Linear(128, 1)def forward(self, x):x = torch.relu(self.fc1(x))policy_logits = self.actor(x)value = self.critic(x)return policy_logits, value

Worker 进程

# Worker 进程
def worker(global_model, optimizer, env_name, global_episode_counter):env = gym.make(env_name)state_dim = env.observation_space.shape[0]action_dim = env.action_space.nlocal_model = ActorCritic(state_dim, action_dim)local_model.load_state_dict(global_model.state_dict())while global_episode_counter.value < 1000:  # 限制训练轮数state_info = env.reset()if isinstance(state_info, tuple):  # 适配新的 Gym 版本state = state_info[0]  # 只获取 state，不要 infoelse:state = state_infostate = np.array(state, dtype=np.float32)done = Falseepisode_reward = 0while not done:with torch.no_grad():  # 关闭梯度计算state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)policy_logits, _ = global_model(state_tensor)action_prob = torch.softmax(policy_logits, dim=1)action = torch.multinomial(action_prob, num_samples=1).item()# Gym 新版 step() 返回 5 个值：state, reward, terminated, truncated, infostep_result = env.step(action)if len(step_result) == 4:  # 旧版 Gymnext_state_info, reward, done, _ = step_resultelse:  # 新版 Gymnext_state_info, reward, terminated, truncated, _ = step_resultdone = terminated or truncated  # 兼容新版本next_state = np.array(next_state_info, dtype=np.float32)state = next_stateepisode_reward += reward# 更新全局 episode 计数with global_episode_counter.get_lock():global_episode_counter.value += 1print(f"Episode: {global_episode_counter.value}, Reward: {episode_reward}")

主训练循环

# 主训练逻辑
if __name__ == "__main__":env_name = "CartPole-v1"env = gym.make(env_name)state_dim = env.observation_space.shape[0]action_dim = env.action_space.nglobal_model = ActorCritic(state_dim, action_dim)global_model.share_memory()  # 共享内存optimizer = optim.Adam(global_model.parameters(), lr=0.001)global_episode_counter = mp.Value('i', 0)  # 多进程共享变量num_workers = 4processes = []for _ in range(num_workers):p = mp.Process(target=worker, args=(global_model, optimizer, env_name, global_episode_counter))p.start()processes.append(p)for p in processes:p.join()

运行结果: 

Episode: 1, Reward: 24.0
Episode: 2, Reward: 11.0
Episode: 3, Reward: 16.0
Episode: 4, Reward: 13.0
Episode: 5, Reward: 33.0
Episode: 6, Reward: 18.0
Episode: 7, Reward: 12.0
Episode: 8, Reward: 17.0
Episode: 9, Reward: 11.0
Episode: 10, Reward: 20.0
Episode: 11, Reward: 14.0
Episode: 12, Reward: 20.0
Episode: 13, Reward: 12.0
Episode: 14, Reward: 22.0
Episode: 15, Reward: 17.0
Episode: 16, Reward: 14.0
Episode: 17, Reward: 18.0
Episode: 18, Reward: 12.0
Episode: 19, Reward: 25.0
Episode: 20, Reward: 12.0
Episode: 21, Reward: 14.0
Episode: 22, Reward: 22.0
Episode: 23, Reward: 21.0
Episode: 24, Reward: 12.0
Episode: 25, Reward: 11.0
Episode: 26, Reward: 13.0
Episode: 27, Reward: 10.0
Episode: 28, Reward: 13.0
Episode: 29, Reward: 10.0
Episode: 30, Reward: 17.0
Episode: 31, Reward: 10.0
Episode: 32, Reward: 18.0
Episode: 33, Reward: 26.0
Episode: 34, Reward: 15.0
Episode: 35, Reward: 19.0
Episode: 36, Reward: 17.0
Episode: 37, Reward: 14.0
Episode: 38, Reward: 9.0
Episode: 39, Reward: 10.0
Episode: 40, Reward: 12.0
Episode: 41, Reward: 23.0
Episode: 42, Reward: 15.0
Episode: 43, Reward: 29.0
Episode: 44, Reward: 11.0
Episode: 45, Reward: 28.0
Episode: 46, Reward: 10.0
Episode: 47, Reward: 13.0
Episode: 48, Reward: 13.0
Episode: 49, Reward: 21.0
Episode: 50, Reward: 14.0
Episode: 51, Reward: 15.0
Episode: 52, Reward: 16.0
Episode: 53, Reward: 47.0
Episode: 54, Reward: 14.0
Episode: 55, Reward: 18.0
Episode: 56, Reward: 9.0
Episode: 57, Reward: 22.0
Episode: 58, Reward: 16.0
Episode: 59, Reward: 11.0
Episode: 60, Reward: 14.0
Episode: 61, Reward: 17.0
Episode: 62, Reward: 32.0
Episode: 63, Reward: 13.0
Episode: 64, Reward: 8.0
Episode: 65, Reward: 18.0
Episode: 66, Reward: 19.0
Episode: 67, Reward: 26.0
Episode: 68, Reward: 19.0
Episode: 69, Reward: 39.0
Episode: 70, Reward: 29.0
Episode: 71, Reward: 26.0
Episode: 72, Reward: 26.0
Episode: 73, Reward: 13.0
Episode: 74, Reward: 23.0
Episode: 75, Reward: 15.0
Episode: 76, Reward: 13.0
Episode: 77, Reward: 50.0
Episode: 78, Reward: 11.0
Episode: 79, Reward: 21.0
Episode: 80, Reward: 81.0
Episode: 81, Reward: 9.0
Episode: 82, Reward: 20.0
Episode: 83, Reward: 14.0
Episode: 84, Reward: 17.0
Episode: 85, Reward: 25.0
Episode: 86, Reward: 10.0
Episode: 87, Reward: 21.0
Episode: 88, Reward: 24.0
Episode: 89, Reward: 18.0
Episode: 90, Reward: 15.0
Episode: 91, Reward: 15.0
Episode: 92, Reward: 11.0
Episode: 93, Reward: 16.0
Episode: 94, Reward: 21.0
Episode: 95, Reward: 14.0
Episode: 96, Reward: 13.0
Episode: 97, Reward: 17.0
Episode: 98, Reward: 11.0
Episode: 99, Reward: 20.0
Episode: 100, Reward: 11.0
Episode: 101, Reward: 11.0
Episode: 102, Reward: 20.0
Episode: 103, Reward: 26.0
Episode: 104, Reward: 29.0
Episode: 105, Reward: 11.0
Episode: 106, Reward: 11.0
Episode: 107, Reward: 12.0
Episode: 108, Reward: 15.0
Episode: 109, Reward: 11.0
Episode: 110, Reward: 15.0
Episode: 111, Reward: 12.0
Episode: 112, Reward: 16.0
Episode: 113, Reward: 19.0
Episode: 114, Reward: 13.0
Episode: 115, Reward: 12.0
Episode: 116, Reward: 16.0
Episode: 117, Reward: 11.0
Episode: 118, Reward: 16.0
Episode: 119, Reward: 13.0
Episode: 120, Reward: 16.0
Episode: 121, Reward: 22.0
Episode: 122, Reward: 11.0
Episode: 123, Reward: 11.0
Episode: 124, Reward: 14.0
Episode: 125, Reward: 12.0
Episode: 126, Reward: 12.0
Episode: 127, Reward: 10.0
Episode: 128, Reward: 18.0
Episode: 129, Reward: 62.0
Episode: 130, Reward: 22.0
Episode: 131, Reward: 9.0
Episode: 132, Reward: 15.0
Episode: 133, Reward: 23.0
Episode: 134, Reward: 13.0
Episode: 135, Reward: 12.0
Episode: 136, Reward: 14.0
Episode: 137, Reward: 10.0
Episode: 138, Reward: 20.0
Episode: 139, Reward: 18.0
Episode: 140, Reward: 23.0
Episode: 141, Reward: 28.0
Episode: 142, Reward: 38.0
Episode: 143, Reward: 18.0
Episode: 144, Reward: 25.0
Episode: 145, Reward: 17.0
Episode: 146, Reward: 41.0
Episode: 147, Reward: 10.0
Episode: 148, Reward: 18.0
Episode: 149, Reward: 11.0
Episode: 150, Reward: 33.0
Episode: 151, Reward: 25.0
Episode: 152, Reward: 25.0
Episode: 153, Reward: 15.0
Episode: 154, Reward: 16.0
Episode: 155, Reward: 20.0
Episode: 156, Reward: 13.0
Episode: 157, Reward: 33.0
Episode: 158, Reward: 13.0
Episode: 159, Reward: 60.0
Episode: 160, Reward: 27.0
Episode: 161, Reward: 14.0
Episode: 162, Reward: 13.0
Episode: 163, Reward: 37.0
Episode: 164, Reward: 11.0
Episode: 165, Reward: 12.0
Episode: 166, Reward: 18.0
Episode: 167, Reward: 11.0
Episode: 168, Reward: 28.0
Episode: 169, Reward: 13.0
Episode: 170, Reward: 13.0
Episode: 171, Reward: 11.0
Episode: 172, Reward: 20.0
Episode: 173, Reward: 9.0
Episode: 174, Reward: 10.0
Episode: 175, Reward: 15.0
Episode: 176, Reward: 29.0
Episode: 177, Reward: 33.0
Episode: 178, Reward: 12.0
Episode: 179, Reward: 12.0
Episode: 180, Reward: 13.0
Episode: 181, Reward: 26.0
Episode: 182, Reward: 43.0
Episode: 183, Reward: 33.0
Episode: 184, Reward: 21.0
Episode: 185, Reward: 23.0
Episode: 186, Reward: 22.0
Episode: 187, Reward: 25.0
Episode: 188, Reward: 9.0
Episode: 189, Reward: 37.0
Episode: 190, Reward: 10.0
Episode: 191, Reward: 12.0
Episode: 192, Reward: 16.0
Episode: 193, Reward: 28.0
Episode: 194, Reward: 17.0
Episode: 195, Reward: 16.0
Episode: 196, Reward: 29.0
Episode: 197, Reward: 35.0
Episode: 198, Reward: 12.0
Episode: 199, Reward: 10.0
Episode: 200, Reward: 14.0
Episode: 201, Reward: 14.0
Episode: 202, Reward: 13.0
Episode: 203, Reward: 13.0
Episode: 204, Reward: 10.0
Episode: 205, Reward: 15.0
Episode: 206, Reward: 14.0
Episode: 207, Reward: 15.0
Episode: 208, Reward: 9.0
Episode: 209, Reward: 18.0
Episode: 210, Reward: 31.0
Episode: 211, Reward: 21.0
Episode: 212, Reward: 47.0
Episode: 213, Reward: 19.0
Episode: 214, Reward: 10.0
Episode: 215, Reward: 12.0
Episode: 216, Reward: 20.0
Episode: 217, Reward: 18.0
Episode: 218, Reward: 10.0
Episode: 219, Reward: 9.0
Episode: 220, Reward: 16.0
Episode: 221, Reward: 19.0
Episode: 222, Reward: 11.0
Episode: 223, Reward: 12.0
Episode: 224, Reward: 28.0
Episode: 225, Reward: 11.0
Episode: 226, Reward: 34.0
Episode: 227, Reward: 11.0
Episode: 228, Reward: 12.0
Episode: 229, Reward: 12.0
Episode: 230, Reward: 10.0
Episode: 231, Reward: 15.0
Episode: 232, Reward: 15.0
Episode: 233, Reward: 24.0
Episode: 234, Reward: 18.0
Episode: 235, Reward: 11.0
Episode: 236, Reward: 42.0
Episode: 237, Reward: 12.0
Episode: 238, Reward: 12.0
Episode: 239, Reward: 15.0
Episode: 240, Reward: 10.0
Episode: 241, Reward: 21.0
Episode: 242, Reward: 15.0
Episode: 243, Reward: 13.0
Episode: 244, Reward: 11.0
Episode: 245, Reward: 33.0
Episode: 246, Reward: 44.0
Episode: 247, Reward: 20.0
Episode: 248, Reward: 31.0
Episode: 249, Reward: 39.0
Episode: 250, Reward: 27.0
Episode: 251, Reward: 16.0
Episode: 252, Reward: 9.0
Episode: 253, Reward: 21.0
Episode: 254, Reward: 13.0
Episode: 255, Reward: 11.0
Episode: 256, Reward: 26.0
Episode: 257, Reward: 23.0
Episode: 258, Reward: 14.0
Episode: 259, Reward: 12.0
Episode: 260, Reward: 13.0
Episode: 261, Reward: 28.0
Episode: 262, Reward: 26.0
Episode: 263, Reward: 19.0
Episode: 264, Reward: 34.0
Episode: 265, Reward: 14.0
Episode: 266, Reward: 26.0
Episode: 267, Reward: 13.0
Episode: 268, Reward: 80.0
Episode: 269, Reward: 20.0
Episode: 270, Reward: 12.0
Episode: 271, Reward: 11.0
Episode: 272, Reward: 14.0
Episode: 273, Reward: 19.0
Episode: 274, Reward: 27.0
Episode: 275, Reward: 14.0
Episode: 276, Reward: 16.0
Episode: 277, Reward: 18.0
Episode: 278, Reward: 23.0
Episode: 279, Reward: 18.0
Episode: 280, Reward: 13.0
Episode: 281, Reward: 11.0
Episode: 282, Reward: 20.0
Episode: 283, Reward: 16.0
Episode: 284, Reward: 10.0
Episode: 285, Reward: 12.0
Episode: 286, Reward: 47.0
Episode: 287, Reward: 18.0
Episode: 288, Reward: 17.0
Episode: 289, Reward: 24.0
Episode: 290, Reward: 10.0
Episode: 291, Reward: 19.0
Episode: 292, Reward: 12.0
Episode: 293, Reward: 13.0
Episode: 294, Reward: 11.0
Episode: 295, Reward: 32.0
Episode: 296, Reward: 12.0
Episode: 297, Reward: 27.0
Episode: 298, Reward: 10.0
Episode: 299, Reward: 18.0
Episode: 300, Reward: 14.0
Episode: 301, Reward: 21.0
Episode: 302, Reward: 12.0
Episode: 303, Reward: 13.0
Episode: 304, Reward: 14.0
Episode: 305, Reward: 13.0
Episode: 306, Reward: 15.0
Episode: 307, Reward: 15.0
Episode: 308, Reward: 12.0
Episode: 309, Reward: 27.0
Episode: 310, Reward: 20.0
Episode: 311, Reward: 16.0
Episode: 312, Reward: 12.0
Episode: 313, Reward: 9.0
Episode: 314, Reward: 24.0
Episode: 315, Reward: 41.0
Episode: 316, Reward: 16.0
Episode: 317, Reward: 20.0
Episode: 318, Reward: 11.0
Episode: 319, Reward: 15.0
Episode: 320, Reward: 41.0
Episode: 321, Reward: 9.0
Episode: 322, Reward: 42.0
Episode: 323, Reward: 35.0
Episode: 324, Reward: 10.0
Episode: 325, Reward: 11.0
Episode: 326, Reward: 11.0
Episode: 327, Reward: 10.0
Episode: 328, Reward: 23.0
Episode: 329, Reward: 26.0
Episode: 330, Reward: 14.0
Episode: 331, Reward: 19.0
Episode: 332, Reward: 27.0
Episode: 333, Reward: 9.0
Episode: 334, Reward: 11.0
Episode: 335, Reward: 22.0
Episode: 336, Reward: 17.0
Episode: 337, Reward: 17.0
Episode: 338, Reward: 16.0
Episode: 339, Reward: 14.0
Episode: 340, Reward: 15.0
Episode: 341, Reward: 22.0
Episode: 342, Reward: 12.0
Episode: 343, Reward: 28.0
Episode: 344, Reward: 21.0
Episode: 345, Reward: 12.0
Episode: 346, Reward: 27.0
Episode: 347, Reward: 16.0
Episode: 348, Reward: 13.0
Episode: 349, Reward: 15.0
Episode: 350, Reward: 8.0
Episode: 351, Reward: 13.0
Episode: 352, Reward: 17.0
Episode: 353, Reward: 19.0
Episode: 354, Reward: 9.0
Episode: 355, Reward: 14.0
Episode: 356, Reward: 9.0
Episode: 357, Reward: 19.0
Episode: 358, Reward: 10.0
Episode: 359, Reward: 12.0
Episode: 360, Reward: 23.0
Episode: 361, Reward: 26.0
Episode: 362, Reward: 15.0
Episode: 363, Reward: 34.0
Episode: 364, Reward: 37.0
Episode: 365, Reward: 28.0
Episode: 366, Reward: 11.0
Episode: 367, Reward: 19.0
Episode: 368, Reward: 10.0
Episode: 369, Reward: 15.0
Episode: 370, Reward: 32.0
Episode: 371, Reward: 30.0
Episode: 372, Reward: 32.0
Episode: 373, Reward: 17.0
Episode: 374, Reward: 8.0
Episode: 375, Reward: 12.0
Episode: 376, Reward: 11.0
Episode: 377, Reward: 36.0
Episode: 378, Reward: 16.0
Episode: 379, Reward: 14.0
Episode: 380, Reward: 13.0
Episode: 381, Reward: 13.0
Episode: 382, Reward: 13.0
Episode: 383, Reward: 16.0
Episode: 384, Reward: 13.0
Episode: 385, Reward: 13.0
Episode: 386, Reward: 14.0
Episode: 387, Reward: 16.0
Episode: 388, Reward: 17.0
Episode: 389, Reward: 28.0
Episode: 390, Reward: 17.0
Episode: 391, Reward: 16.0
Episode: 392, Reward: 10.0
Episode: 393, Reward: 18.0
Episode: 394, Reward: 12.0
Episode: 395, Reward: 18.0
Episode: 396, Reward: 28.0
Episode: 397, Reward: 31.0
Episode: 398, Reward: 16.0
Episode: 399, Reward: 15.0
Episode: 400, Reward: 22.0
Episode: 401, Reward: 28.0
Episode: 402, Reward: 22.0
Episode: 403, Reward: 10.0
Episode: 404, Reward: 36.0
Episode: 405, Reward: 13.0
Episode: 406, Reward: 17.0
Episode: 407, Reward: 14.0
Episode: 408, Reward: 19.0
Episode: 409, Reward: 16.0
Episode: 410, Reward: 18.0
Episode: 411, Reward: 16.0
Episode: 412, Reward: 22.0
Episode: 413, Reward: 17.0
Episode: 414, Reward: 13.0
Episode: 415, Reward: 15.0
Episode: 416, Reward: 17.0
Episode: 417, Reward: 15.0
Episode: 418, Reward: 14.0
Episode: 419, Reward: 25.0
Episode: 420, Reward: 12.0
Episode: 421, Reward: 46.0
Episode: 422, Reward: 21.0
Episode: 423, Reward: 15.0
Episode: 424, Reward: 11.0
Episode: 425, Reward: 9.0
Episode: 426, Reward: 18.0
Episode: 427, Reward: 23.0
Episode: 428, Reward: 24.0
Episode: 429, Reward: 16.0
Episode: 430, Reward: 29.0
Episode: 431, Reward: 11.0
Episode: 432, Reward: 16.0
Episode: 433, Reward: 16.0
Episode: 434, Reward: 11.0
Episode: 435, Reward: 10.0
Episode: 436, Reward: 10.0
Episode: 437, Reward: 11.0
Episode: 438, Reward: 11.0
Episode: 439, Reward: 22.0
Episode: 440, Reward: 17.0
Episode: 441, Reward: 9.0
Episode: 442, Reward: 11.0
Episode: 443, Reward: 14.0
Episode: 444, Reward: 11.0
Episode: 445, Reward: 16.0
Episode: 446, Reward: 16.0
Episode: 447, Reward: 16.0
Episode: 448, Reward: 21.0
Episode: 449, Reward: 20.0
Episode: 450, Reward: 27.0
Episode: 451, Reward: 13.0
Episode: 452, Reward: 33.0
Episode: 453, Reward: 22.0
Episode: 454, Reward: 16.0
Episode: 455, Reward: 18.0
Episode: 456, Reward: 30.0
Episode: 457, Reward: 13.0
Episode: 458, Reward: 10.0
Episode: 459, Reward: 14.0
Episode: 460, Reward: 16.0
Episode: 461, Reward: 18.0
Episode: 462, Reward: 11.0
Episode: 463, Reward: 18.0
Episode: 464, Reward: 10.0
Episode: 465, Reward: 19.0
Episode: 466, Reward: 27.0
Episode: 467, Reward: 16.0
Episode: 468, Reward: 15.0
Episode: 469, Reward: 12.0
Episode: 470, Reward: 14.0
Episode: 471, Reward: 18.0
Episode: 472, Reward: 16.0
Episode: 473, Reward: 9.0
Episode: 474, Reward: 25.0
Episode: 475, Reward: 29.0
Episode: 476, Reward: 13.0
Episode: 477, Reward: 23.0
Episode: 478, Reward: 17.0
Episode: 479, Reward: 15.0
Episode: 480, Reward: 20.0
Episode: 481, Reward: 11.0
Episode: 482, Reward: 11.0
Episode: 483, Reward: 18.0
Episode: 484, Reward: 21.0
Episode: 485, Reward: 15.0
Episode: 486, Reward: 12.0
Episode: 487, Reward: 11.0
Episode: 488, Reward: 18.0
Episode: 489, Reward: 20.0
Episode: 490, Reward: 14.0
Episode: 491, Reward: 11.0
Episode: 492, Reward: 18.0
Episode: 493, Reward: 23.0
Episode: 494, Reward: 29.0
Episode: 495, Reward: 12.0
Episode: 496, Reward: 11.0
Episode: 497, Reward: 12.0
Episode: 498, Reward: 16.0
Episode: 499, Reward: 10.0
Episode: 500, Reward: 14.0
Episode: 501, Reward: 9.0
Episode: 502, Reward: 32.0
Episode: 503, Reward: 12.0
Episode: 504, Reward: 16.0
Episode: 505, Reward: 21.0
Episode: 506, Reward: 15.0
Episode: 507, Reward: 14.0
Episode: 508, Reward: 28.0
Episode: 509, Reward: 14.0
Episode: 510, Reward: 12.0
Episode: 511, Reward: 15.0
Episode: 512, Reward: 19.0
Episode: 513, Reward: 14.0
Episode: 514, Reward: 12.0
Episode: 515, Reward: 13.0
Episode: 516, Reward: 13.0
Episode: 517, Reward: 10.0
Episode: 518, Reward: 18.0
Episode: 519, Reward: 18.0
Episode: 520, Reward: 18.0
Episode: 521, Reward: 12.0
Episode: 522, Reward: 27.0
Episode: 523, Reward: 11.0
Episode: 524, Reward: 9.0
Episode: 525, Reward: 24.0
Episode: 526, Reward: 47.0
Episode: 527, Reward: 17.0
Episode: 528, Reward: 10.0
Episode: 529, Reward: 22.0
Episode: 530, Reward: 24.0
Episode: 531, Reward: 13.0
Episode: 532, Reward: 14.0
Episode: 533, Reward: 29.0
Episode: 534, Reward: 11.0
Episode: 535, Reward: 17.0
Episode: 536, Reward: 17.0
Episode: 537, Reward: 28.0
Episode: 538, Reward: 9.0
Episode: 539, Reward: 27.0
Episode: 540, Reward: 13.0
Episode: 541, Reward: 16.0
Episode: 542, Reward: 16.0
Episode: 543, Reward: 12.0
Episode: 544, Reward: 20.0
Episode: 545, Reward: 26.0
Episode: 546, Reward: 11.0
Episode: 547, Reward: 20.0
Episode: 548, Reward: 21.0
Episode: 549, Reward: 38.0
Episode: 550, Reward: 26.0
Episode: 551, Reward: 11.0
Episode: 552, Reward: 18.0
Episode: 553, Reward: 13.0
Episode: 554, Reward: 14.0
Episode: 555, Reward: 15.0
Episode: 556, Reward: 28.0
Episode: 557, Reward: 11.0
Episode: 558, Reward: 9.0
Episode: 559, Reward: 16.0
Episode: 560, Reward: 19.0
Episode: 561, Reward: 13.0
Episode: 562, Reward: 23.0
Episode: 563, Reward: 12.0
Episode: 564, Reward: 9.0
Episode: 565, Reward: 18.0
Episode: 566, Reward: 14.0
Episode: 567, Reward: 11.0
Episode: 568, Reward: 12.0
Episode: 569, Reward: 60.0
Episode: 570, Reward: 26.0
Episode: 571, Reward: 50.0
Episode: 572, Reward: 16.0
Episode: 573, Reward: 27.0
Episode: 574, Reward: 46.0
Episode: 575, Reward: 10.0
Episode: 576, Reward: 31.0
Episode: 577, Reward: 14.0
Episode: 578, Reward: 14.0
Episode: 579, Reward: 42.0
Episode: 580, Reward: 36.0
Episode: 581, Reward: 46.0
Episode: 582, Reward: 21.0
Episode: 583, Reward: 11.0
Episode: 584, Reward: 34.0
Episode: 585, Reward: 10.0
Episode: 586, Reward: 13.0
Episode: 587, Reward: 10.0
Episode: 588, Reward: 16.0
Episode: 589, Reward: 16.0
Episode: 590, Reward: 16.0
Episode: 591, Reward: 12.0
Episode: 592, Reward: 17.0
Episode: 593, Reward: 11.0
Episode: 594, Reward: 23.0
Episode: 595, Reward: 11.0
Episode: 596, Reward: 11.0
Episode: 597, Reward: 23.0
Episode: 598, Reward: 21.0
Episode: 599, Reward: 18.0
Episode: 600, Reward: 15.0
Episode: 601, Reward: 9.0
Episode: 602, Reward: 22.0
Episode: 603, Reward: 20.0
Episode: 604, Reward: 11.0
Episode: 605, Reward: 12.0
Episode: 606, Reward: 16.0
Episode: 607, Reward: 15.0
Episode: 608, Reward: 14.0
Episode: 609, Reward: 27.0
Episode: 610, Reward: 19.0
Episode: 611, Reward: 34.0
Episode: 612, Reward: 18.0
Episode: 613, Reward: 12.0
Episode: 614, Reward: 11.0
Episode: 615, Reward: 12.0
Episode: 616, Reward: 20.0
Episode: 617, Reward: 31.0
Episode: 618, Reward: 14.0
Episode: 619, Reward: 20.0
Episode: 620, Reward: 15.0
Episode: 621, Reward: 17.0
Episode: 622, Reward: 24.0
Episode: 623, Reward: 49.0
Episode: 624, Reward: 13.0
Episode: 625, Reward: 12.0
Episode: 626, Reward: 12.0
Episode: 627, Reward: 11.0
Episode: 628, Reward: 24.0
Episode: 629, Reward: 13.0
Episode: 630, Reward: 21.0
Episode: 631, Reward: 36.0
Episode: 632, Reward: 28.0
Episode: 633, Reward: 60.0
Episode: 634, Reward: 9.0
Episode: 635, Reward: 10.0
Episode: 636, Reward: 17.0
Episode: 637, Reward: 15.0
Episode: 638, Reward: 14.0
Episode: 639, Reward: 22.0
Episode: 640, Reward: 24.0
Episode: 641, Reward: 19.0
Episode: 642, Reward: 26.0
Episode: 643, Reward: 15.0
Episode: 644, Reward: 17.0
Episode: 645, Reward: 15.0
Episode: 646, Reward: 17.0
Episode: 647, Reward: 14.0
Episode: 648, Reward: 17.0
Episode: 649, Reward: 16.0
Episode: 650, Reward: 10.0
Episode: 651, Reward: 13.0
Episode: 652, Reward: 15.0
Episode: 653, Reward: 34.0
Episode: 654, Reward: 14.0
Episode: 655, Reward: 11.0
Episode: 656, Reward: 9.0
Episode: 657, Reward: 15.0
Episode: 658, Reward: 11.0
Episode: 659, Reward: 12.0
Episode: 660, Reward: 16.0
Episode: 661, Reward: 24.0
Episode: 662, Reward: 11.0
Episode: 663, Reward: 14.0
Episode: 664, Reward: 46.0
Episode: 665, Reward: 14.0
Episode: 666, Reward: 24.0
Episode: 667, Reward: 28.0
Episode: 668, Reward: 16.0
Episode: 669, Reward: 15.0
Episode: 670, Reward: 12.0
Episode: 671, Reward: 13.0
Episode: 672, Reward: 25.0
Episode: 673, Reward: 23.0
Episode: 674, Reward: 13.0
Episode: 675, Reward: 27.0
Episode: 676, Reward: 24.0
Episode: 677, Reward: 18.0
Episode: 678, Reward: 15.0
Episode: 679, Reward: 12.0
Episode: 680, Reward: 9.0
Episode: 681, Reward: 11.0
Episode: 682, Reward: 17.0
Episode: 683, Reward: 9.0
Episode: 684, Reward: 20.0
Episode: 685, Reward: 10.0
Episode: 686, Reward: 12.0
Episode: 687, Reward: 40.0
Episode: 688, Reward: 24.0
Episode: 689, Reward: 25.0
Episode: 690, Reward: 9.0
Episode: 691, Reward: 16.0
Episode: 692, Reward: 9.0
Episode: 693, Reward: 13.0
Episode: 694, Reward: 17.0
Episode: 695, Reward: 15.0
Episode: 696, Reward: 18.0
Episode: 697, Reward: 13.0
Episode: 698, Reward: 10.0
Episode: 699, Reward: 56.0
Episode: 700, Reward: 13.0
Episode: 701, Reward: 17.0
Episode: 702, Reward: 11.0
Episode: 703, Reward: 23.0
Episode: 704, Reward: 33.0
Episode: 705, Reward: 24.0
Episode: 706, Reward: 25.0
Episode: 707, Reward: 26.0
Episode: 708, Reward: 18.0
Episode: 709, Reward: 10.0
Episode: 710, Reward: 9.0
Episode: 711, Reward: 11.0
Episode: 712, Reward: 12.0
Episode: 713, Reward: 21.0
Episode: 714, Reward: 16.0
Episode: 715, Reward: 12.0
Episode: 716, Reward: 24.0
Episode: 717, Reward: 23.0
Episode: 718, Reward: 16.0
Episode: 719, Reward: 32.0
Episode: 720, Reward: 25.0
Episode: 721, Reward: 14.0
Episode: 722, Reward: 9.0
Episode: 723, Reward: 11.0
Episode: 724, Reward: 30.0
Episode: 725, Reward: 21.0
Episode: 726, Reward: 13.0
Episode: 727, Reward: 14.0
Episode: 728, Reward: 12.0
Episode: 729, Reward: 32.0
Episode: 730, Reward: 12.0
Episode: 731, Reward: 9.0
Episode: 732, Reward: 32.0
Episode: 733, Reward: 15.0
Episode: 734, Reward: 11.0
Episode: 735, Reward: 14.0
Episode: 736, Reward: 13.0
Episode: 737, Reward: 16.0
Episode: 738, Reward: 23.0
Episode: 739, Reward: 14.0
Episode: 740, Reward: 17.0
Episode: 741, Reward: 12.0
Episode: 742, Reward: 43.0
Episode: 743, Reward: 36.0
Episode: 744, Reward: 16.0
Episode: 745, Reward: 13.0
Episode: 746, Reward: 28.0
Episode: 747, Reward: 11.0
Episode: 748, Reward: 10.0
Episode: 749, Reward: 24.0
Episode: 750, Reward: 14.0
Episode: 751, Reward: 27.0
Episode: 752, Reward: 13.0
Episode: 753, Reward: 23.0
Episode: 754, Reward: 20.0
Episode: 755, Reward: 9.0
Episode: 756, Reward: 19.0
Episode: 757, Reward: 32.0
Episode: 758, Reward: 12.0
Episode: 759, Reward: 18.0
Episode: 760, Reward: 12.0
Episode: 761, Reward: 25.0
Episode: 762, Reward: 15.0
Episode: 763, Reward: 29.0
Episode: 764, Reward: 9.0
Episode: 765, Reward: 14.0
Episode: 766, Reward: 17.0
Episode: 767, Reward: 11.0
Episode: 768, Reward: 15.0
Episode: 769, Reward: 10.0
Episode: 770, Reward: 23.0
Episode: 771, Reward: 10.0
Episode: 772, Reward: 14.0
Episode: 773, Reward: 18.0
Episode: 774, Reward: 13.0
Episode: 775, Reward: 29.0
Episode: 776, Reward: 74.0
Episode: 777, Reward: 13.0
Episode: 778, Reward: 12.0
Episode: 779, Reward: 21.0
Episode: 780, Reward: 14.0
Episode: 781, Reward: 12.0
Episode: 782, Reward: 15.0
Episode: 783, Reward: 38.0
Episode: 784, Reward: 9.0
Episode: 785, Reward: 15.0
Episode: 786, Reward: 24.0
Episode: 787, Reward: 50.0
Episode: 788, Reward: 14.0
Episode: 789, Reward: 28.0
Episode: 790, Reward: 14.0
Episode: 791, Reward: 12.0
Episode: 792, Reward: 18.0
Episode: 793, Reward: 10.0
Episode: 794, Reward: 11.0
Episode: 795, Reward: 10.0
Episode: 796, Reward: 11.0
Episode: 797, Reward: 24.0
Episode: 798, Reward: 13.0
Episode: 799, Reward: 11.0
Episode: 800, Reward: 22.0
Episode: 801, Reward: 14.0
Episode: 802, Reward: 51.0
Episode: 803, Reward: 11.0
Episode: 804, Reward: 10.0
Episode: 805, Reward: 20.0
Episode: 806, Reward: 13.0
Episode: 807, Reward: 23.0
Episode: 808, Reward: 12.0
Episode: 809, Reward: 11.0
Episode: 810, Reward: 13.0
Episode: 811, Reward: 14.0
Episode: 812, Reward: 10.0
Episode: 813, Reward: 23.0
Episode: 814, Reward: 15.0
Episode: 815, Reward: 34.0
Episode: 816, Reward: 27.0
Episode: 817, Reward: 17.0
Episode: 818, Reward: 19.0
Episode: 819, Reward: 14.0
Episode: 820, Reward: 15.0
Episode: 821, Reward: 22.0
Episode: 822, Reward: 19.0
Episode: 823, Reward: 16.0
Episode: 824, Reward: 17.0
Episode: 825, Reward: 12.0
Episode: 826, Reward: 13.0
Episode: 827, Reward: 15.0
Episode: 828, Reward: 22.0
Episode: 829, Reward: 18.0
Episode: 830, Reward: 16.0
Episode: 831, Reward: 21.0
Episode: 832, Reward: 20.0
Episode: 833, Reward: 20.0
Episode: 834, Reward: 37.0
Episode: 835, Reward: 11.0
Episode: 836, Reward: 15.0
Episode: 837, Reward: 11.0
Episode: 838, Reward: 22.0
Episode: 839, Reward: 18.0
Episode: 840, Reward: 33.0
Episode: 841, Reward: 19.0
Episode: 842, Reward: 14.0
Episode: 843, Reward: 11.0
Episode: 844, Reward: 10.0
Episode: 845, Reward: 15.0
Episode: 846, Reward: 22.0
Episode: 847, Reward: 13.0
Episode: 848, Reward: 10.0
Episode: 849, Reward: 11.0
Episode: 850, Reward: 16.0
Episode: 851, Reward: 23.0
Episode: 852, Reward: 28.0
Episode: 853, Reward: 26.0
Episode: 854, Reward: 9.0
Episode: 855, Reward: 11.0
Episode: 856, Reward: 18.0
Episode: 857, Reward: 18.0
Episode: 858, Reward: 19.0
Episode: 859, Reward: 16.0
Episode: 860, Reward: 11.0
Episode: 861, Reward: 9.0
Episode: 862, Reward: 16.0
Episode: 863, Reward: 15.0
Episode: 864, Reward: 20.0
Episode: 865, Reward: 25.0
Episode: 866, Reward: 14.0
Episode: 867, Reward: 12.0
Episode: 868, Reward: 13.0
Episode: 869, Reward: 16.0
Episode: 870, Reward: 16.0
Episode: 871, Reward: 27.0
Episode: 872, Reward: 15.0
Episode: 873, Reward: 14.0
Episode: 874, Reward: 17.0
Episode: 875, Reward: 11.0
Episode: 876, Reward: 30.0
Episode: 877, Reward: 10.0
Episode: 878, Reward: 26.0
Episode: 879, Reward: 11.0
Episode: 880, Reward: 22.0
Episode: 881, Reward: 13.0
Episode: 882, Reward: 29.0
Episode: 883, Reward: 13.0
Episode: 884, Reward: 18.0
Episode: 885, Reward: 15.0
Episode: 886, Reward: 12.0
Episode: 887, Reward: 11.0
Episode: 888, Reward: 25.0
Episode: 889, Reward: 12.0
Episode: 890, Reward: 14.0
Episode: 891, Reward: 23.0
Episode: 892, Reward: 48.0
Episode: 893, Reward: 14.0
Episode: 894, Reward: 19.0
Episode: 895, Reward: 10.0
Episode: 896, Reward: 10.0
Episode: 897, Reward: 13.0
Episode: 898, Reward: 13.0
Episode: 899, Reward: 14.0
Episode: 900, Reward: 10.0
Episode: 901, Reward: 16.0
Episode: 902, Reward: 20.0
Episode: 903, Reward: 30.0
Episode: 904, Reward: 11.0
Episode: 905, Reward: 18.0
Episode: 906, Reward: 17.0
Episode: 907, Reward: 17.0
Episode: 908, Reward: 17.0
Episode: 909, Reward: 33.0
Episode: 910, Reward: 18.0
Episode: 911, Reward: 17.0
Episode: 912, Reward: 12.0
Episode: 913, Reward: 18.0
Episode: 914, Reward: 15.0
Episode: 915, Reward: 26.0
Episode: 916, Reward: 19.0
Episode: 917, Reward: 13.0
Episode: 918, Reward: 21.0
Episode: 919, Reward: 17.0
Episode: 920, Reward: 13.0
Episode: 921, Reward: 14.0
Episode: 922, Reward: 8.0
Episode: 923, Reward: 10.0
Episode: 924, Reward: 30.0
Episode: 925, Reward: 15.0
Episode: 926, Reward: 15.0
Episode: 927, Reward: 11.0
Episode: 928, Reward: 11.0
Episode: 929, Reward: 13.0
Episode: 930, Reward: 13.0
Episode: 931, Reward: 11.0
Episode: 932, Reward: 37.0
Episode: 933, Reward: 30.0
Episode: 934, Reward: 13.0
Episode: 935, Reward: 12.0
Episode: 936, Reward: 14.0
Episode: 937, Reward: 24.0
Episode: 938, Reward: 10.0
Episode: 939, Reward: 32.0
Episode: 940, Reward: 18.0
Episode: 941, Reward: 18.0
Episode: 942, Reward: 15.0
Episode: 943, Reward: 14.0
Episode: 944, Reward: 18.0
Episode: 945, Reward: 13.0
Episode: 946, Reward: 10.0
Episode: 947, Reward: 27.0
Episode: 948, Reward: 17.0
Episode: 949, Reward: 13.0
Episode: 950, Reward: 14.0
Episode: 951, Reward: 13.0
Episode: 952, Reward: 21.0
Episode: 953, Reward: 24.0
Episode: 954, Reward: 21.0
Episode: 955, Reward: 21.0
Episode: 956, Reward: 33.0
Episode: 957, Reward: 15.0
Episode: 958, Reward: 13.0
Episode: 959, Reward: 45.0
Episode: 960, Reward: 13.0
Episode: 961, Reward: 11.0
Episode: 962, Reward: 12.0
Episode: 963, Reward: 18.0
Episode: 964, Reward: 15.0
Episode: 965, Reward: 10.0
Episode: 966, Reward: 11.0
Episode: 967, Reward: 25.0
Episode: 968, Reward: 17.0
Episode: 969, Reward: 13.0
Episode: 970, Reward: 16.0
Episode: 971, Reward: 11.0
Episode: 972, Reward: 21.0
Episode: 973, Reward: 13.0
Episode: 974, Reward: 12.0
Episode: 975, Reward: 12.0
Episode: 976, Reward: 28.0
Episode: 977, Reward: 18.0
Episode: 978, Reward: 10.0
Episode: 979, Reward: 13.0
Episode: 980, Reward: 15.0
Episode: 981, Reward: 16.0
Episode: 982, Reward: 13.0
Episode: 983, Reward: 12.0
Episode: 984, Reward: 18.0
Episode: 985, Reward: 22.0
Episode: 986, Reward: 29.0
Episode: 987, Reward: 24.0
Episode: 988, Reward: 12.0
Episode: 989, Reward: 13.0
Episode: 990, Reward: 11.0
Episode: 991, Reward: 11.0
Episode: 992, Reward: 13.0
Episode: 993, Reward: 41.0
Episode: 994, Reward: 18.0
Episode: 995, Reward: 11.0
Episode: 996, Reward: 15.0
Episode: 997, Reward: 14.0
Episode: 998, Reward: 9.0
Episode: 999, Reward: 19.0
Episode: 1000, Reward: 14.0
Episode: 1001, Reward: 22.0
Episode: 1002, Reward: 16.0
Episode: 1003, Reward: 18.0

代码说明：

  • 采用 多进程 来创建多个 Worker。
  • 每个 Worker 进行独立的环境交互，并异步更新全局 Actor-Critic 网络。
  • 使用 共享内存 来同步多个进程的参数。

5. A3C 的优势

  • 提高训练效率：多个 Worker 并行执行，避免了 DQN 的经验回放机制，提高了数据利用率。
  • 稳定性强：异步更新减少了数据相关性，提高了训练稳定性。
  • 收敛速度快：相比传统 DQN，A3C 需要更少的训练时间即可获得较优策略。

6. A3C 的改进版本

由于 A3C 仍然存在计算资源消耗较高的问题，后续研究提出了 A2C（Advantage Actor-Critic），它采用 同步更新（Synchronous Updates），可以进一步提高训练的稳定性和效率。

7. 总结

特点A3C 关键点
核心思想结合 Actor-Critic，采用异步更新
网络结构包括 Actor（策略）和 Critic（值函数）
训练方法多线程异步训练，提高效率
目标函数采用 Advantage Function 改进策略梯度
优点提高数据效率、收敛快、适用于复杂任务
改进版本A2C（同步版本）

A3C 在强化学习领域具有里程碑意义，为后续的 PPO、SAC 等强化学习算法 提供了重要的理论基础！

相关文章：

【深度学习】强化学习（RL）-A3C（Asynchronous Advantage Actor-Critic）

A3C&#xff08;Asynchronous Advantage Actor-Critic&#xff09;详解 A3C&#xff08;Asynchronous Advantage Actor-Critic&#xff09; 是 深度强化学习&#xff08;Deep Reinforcement Learning, DRL&#xff09; 领域的重要算法&#xff0c;由 DeepMind 在 2016 年提出。…...

编程日记 2025/9/13 16:01:03

DeepSeek-v1到DeepSeek-v3再到DeepSeek-R1的变迁和进化史，创新点，值得大家学习，DeepSeek系列干货

DeepSeek-v1 1.高质量的数据构建&#xff1a;2T tokens中英文数据集&#xff08;数据去重、过滤和重混&#xff09;&#xff1b; 2. 模型架构参考LlaMa&#xff1b; 3.数据并行、张量并行、超参数设置等&#xff1a; 衍生&#xff1a;DeepSeek-Coder、deepseek MoE、DeepSe…...

编程日记 2025/9/13 16:00:25

基于多层感知机(MLP)实现MNIST手写体识别

实现步骤 下载数据集处理好数据集确定好模型&#xff08;初始化模型参数等等&#xff09;确定优化函数&#xff08;损失函数也称为目标函数&#xff09;和优化方法&#xff08;一般选用随机梯度下降 SDG &#xff09;进行模型的训练进行模型的评估 import torch import torch…...

编程日记 2025/9/8 7:48:45

windows下安装pyenv+virtualenv+virtualenvwrapper

1、下载pyenv 进入git官网&#xff0c;打包下载zip到本地 2、解压到安装目录 解压下载好的pyenv-win-master.zip到自己的安装目录&#xff0c;如D:\Program Files 3、配置环境变量 右击桌面 此电脑 --> 属性 --> 高端系统设置 --> 环境变量 --> 新建系统变量…...

编程日记 2025/9/9 5:56:51

C# 确保程序只有一个实例运行

常规需求 C#程序只能运行一次,不能多开: using System; using System.Collections.Generic; using System.Linq; using System.Windows.Forms; using System.Threading; using System.Runtime.InteropServices; using System.Security.Principal; namespace BallLocation {sta…...

编程日记 2025/9/12 3:20:46

MySQL 的存储引擎有哪些？它们之间有什么区别？

MySQL 支持多种存储引擎&#xff0c;每种存储引擎都有其独特的特性和适用场景。以下是 MySQL 中常见的存储引擎及其主要区别&#xff1a; 1.常见存储引擎及其特点 (1)InnoDB • 事务支持&#xff1a;支持完整的 ACID 特性&#xff0c;适用于需要事务处理的场景。 • 锁机制&…...

编程日记 2025/9/9 18:58:59

大语言模型的评测

大语言模型评测是评估这些模型在各种任务和场景下的性能和能力的过程。 能力 1. 基准测试&#xff08;Benchmarking&#xff09; GLUE&#xff08;General Language Understanding Evaluation&#xff09;&#xff1a;包含多个自然语言处理任务&#xff0c;如文本分类、情感分…...

编程日记 2025/9/8 22:21:36

《从零到全栈：Vue2入门宝典》

1. Vue 简介 1.1 什么是 Vue&#xff1f; Vue 是一套用于构建用户界面的渐进式框架。Vue 的核心库专注于视图层&#xff0c;易于与其他库或现有项目集成。Vue 的目标是通过尽可能简单的 API 实现数据双向绑定和组件化开发。 1.2 什么是“渐进式”&#xff1f; “渐进式”意味…...

编程日记 2025/9/7 16:37:32

next.js-学习3

next.js-学习3 6. 设置数据库1. 传代码到github https://github.com/2. github和Vercel链接&#xff0c;用Vercel预览和部署3. 创建数据库4. 初始化数据库 7.读取数据1. 在/app/lib/data.ts中这两行是连接postgres的2. 在/app/dashboard/page.tsx中使用3.在/app/dashboard/page…...

编程日记 2025/9/13 16:00:25

Hadoop第2课（伪分布式集群的搭建）

jdk和hadoop安装包&#xff1a; hadoop-2.9.2.t......等2个文件官方版下载丨最新版下载丨绿色版下载丨APP下载-123云盘 1、用XFTP发送hadoop安装包和jdk到/home/hadoop/目录下&#xff08;hadoop用户的主目录&#xff09; 2、解压jdk安装包到~目录 卸载jdk的命令&#xff1a;r…...

编程日记 2025/9/12 22:29:32

Linux——进程池

前言&#xff1a;大佬写博客给别人看&#xff0c;菜鸟写博客给自己看&#xff0c;我是菜鸟。 1.实现思路 思路&#xff1a;通过创建匿名管道&#xff0c;来实现父子进程之间的通信 注1&#xff1a;父写&#xff0c;子读 注2&#xff1a;匿名管道只能用来进行具有血管关系的进程…...

编程日记 2025/9/11 8:41:44

JavaScript 简单类型与复杂类型-简单类型的内存分配

深入理解JavaScript中的简单类型&#xff08;基本数据类型&#xff09;和复杂类型&#xff08;引用数据类型&#xff09;如何在内存中存储对于编写高效、无误的代码至关重要。本文将专注于探讨简单类型的内存分配机制&#xff0c;即栈&#xff08;Stack&#xff09;内存&#x…...

编程日记 2025/9/11 12:48:10

深度生成模型（一）——具身智能综述与算法分类简介

具身智能对于机器人的控制可以分为端到端模型和非端到端模型&#xff1a; 端到端模型&#xff1a;具身模型&#xff08;如 ACT 和 DP&#xff09;将视觉感知&#xff08;Vision&#xff09;与动作生成&#xff08;Action&#xff09;整合为单一神经网络&#xff0c;直接实现从…...

编程日记 2025/9/10 20:34:27

Vue 中，使用模板（Template） 和 Render 函数编写组件的区别

在 Vue 2 中&#xff0c;模板&#xff08;Template&#xff09; 和 Render 函数 是两种不同的组件编写方式&#xff0c;它们各有特点和适用场景。以下是它们的核心区别和实际应用场景分析&#xff1a; 1. 基本区别 特性模板&#xff08;Template&#xff09;Render 函数语法形…...

编程日记 2025/9/12 16:39:50

【笔记】论文阅读方法（AI大模型）

1 为什么读论文 构建知识体系&#xff1a;通过Related Works快速了解该方向研究现状&#xff0c;追踪经典论文 紧跟前沿技术&#xff1a;了解领域内新技术及效果&#xff0c;快速借鉴到自身项目 培养科研逻辑&#xff1a;熟悉论文体系&#xff0c;了解如何创造新事物&#x…...

编程日记 2025/9/8 7:49:13

JWT+redis实现令牌刷新优化方案

令牌刷新优化方案的详细实现步骤&#xff1a; 1. 令牌服务层改造 1.1 JWT工具类增强 // JwtUtils.java 新增方法 public class JwtUtils {// 生成带动态过期时间的令牌public static String createToken(String subject, String userId, String username, long expirationMi…...

编程日记 2025/9/9 19:03:25

安全面试5

文章目录 sql的二次注入在linux下&#xff0c;现在有一个拥有大量ip地址的txt文本文档&#xff0c;但是里面有很多重复的&#xff0c;如何快速去重&#xff1f;在内网渗透中&#xff0c;通过钓鱼邮件获取到主机权限&#xff0c;但是发现内网拦截了tcp的出网流量&#xff0c;聊一…...

编程日记 2025/9/9 1:49:36

vim临时文件泄露

##解题思路 感觉ctfshow的题目都挺有意思的&#xff0c;大家可以去做做 首先题目提示vim临时文件泄露&#xff0c;一般在vim编辑的时候&#xff0c;会有个swp的中间文件生成&#xff0c;根据这个特性&#xff0c;从而可以猜测&#xff0c;我们可以通过访问一个swp文件路径&am…...

编程日记 2025/8/28 0:10:06

使用Docker将ros1自定义消息通过rosjava_bootstrap生成jar包

文章目录 预准备环境rosjava_bootstrap坏消息好消息 环境安装docker安装rosjava_bootstrap仓库rosjava_center仓库修改rosjava_bootstrap代码拉取docker镜像放置自己的自定义消息 启动docker编译 预准备环境 rosjava_bootstrap rosjava_bootstrap是将自定义的ROS消息生成java…...

编程日记 2025/9/10 1:41:34

本地快速搭建一套AI人脸识别技术研究学习的实验环境

如果你想在本地搭建一套学习和研究AI人脸识别技术的框架&#xff0c;建议使用开源工具和框架进行实验&#xff0c;因为它们通常提供了较为丰富的文档和社区支持&#xff0c;能够帮助你深入理解人脸识别的核心原理。以下是一套可行性强且综合性的方案&#xff0c;涵盖了人脸检测…...

编程日记 2025/9/8 16:09:12

SpringBoot项目连接Oracle视图报错整理

在若依框架中增加连接Oracle视图报错 工具测试连接 通过使用plsql连接数据库测试&#xff0c;连接成功 1. 相关配置内容 连接配置 url: jdbc:oracle:thin:192.168.0.210:1521:HIS username: portal_his password: XXXXXX driver-class-name: oracle.jdbc.driver.OracleDr…...

编程日记 2025/9/13 0:14:13

【我的 PWN 学习手札】House of Husk

House of Husk House of Husk是利用格式化输出函数如printf、vprintf在打印输出时&#xff0c;会解析格式化字符如%x、%lld从而调用不同的格式化打印方法&#xff08;函数&#xff09;。同时C语言还提供了注册自定义格式化字符的方法。注册自定义格式化字符串输出方法&#xf…...

编程日记 2025/9/7 21:48:21

面试-JVM：JVM的组成及作用

JVM包含两个子系统和两个组件&#xff1a; 两个子系统&#xff1a; 类加载子系统&#xff08;ClassLoader Subsystem&#xff09; 作用&#xff1a;根据给定的全限定类名&#xff08;如&#xff1a;java.long.Object&#xff09;来装载class文件至运行时数据区的方法区。 核心…...

编程日记 2025/9/9 21:38:28

Node.js项目启动流程以及各个模块执行顺序详解

Node.js项目启动流程以及各个模块执行顺序的问题。首先&#xff0c;我需要仔细阅读并理解我搜索到的资料&#xff0c;从中提取关键信息&#xff0c;然后综合这些信息组织成一个结构化的回答。 首先&#xff0c;根据我搜索到的资料都详细描述了Node.js的启动流程&#xff0c;涉及…...

编程日记 2025/9/13 1:43:01

obj离线加载（vue+threejs）+apk方式浏览

demo需求&#xff1a;移动端&#xff0c;实现obj本地离线浏览 结合需求&#xff0c;利用&#xff08;vue2threejs173&#xff09;进行obj的加载&#xff0c;然后采用apk方式&#xff08;hbuilderX打包发布&#xff09;移动端浏览&#xff1b; https://github.com/bianbian886/…...

编程日记 2025/9/12 13:01:56

【Python 语法】Python 数据结构

线性结构&#xff08;Linear Structures&#xff09;1. 顺序存储列表&#xff08;List&#xff09;元组&#xff08;Tuple&#xff09;字符串&#xff08;String&#xff09; 2. 线性存储栈&#xff08;Stack&#xff09;队列&#xff08;Queue&#xff09;双端队列&#xff08…...

编程日记 2025/9/11 4:37:33

Flutter - 布局Widget

Flutter的布局主要分为 单子组件 和 多子组件 两大类&#xff1a; Container、Padding、Align这些属于单子组件&#xff0c;而Row、Column、ListView这些则是多子组件。 单子组件 Align组件 Align 是一个用于控制子组件位置的单子布局组件。它通过指定对齐方式&#xff08…...

编程日记 2025/9/11 12:17:41

AD从原理图到PCB超详细教程

AD超详细教程 前言一、建立一个工程模板二、原理图1.设计原理图。2.使用AD自带库和网上开源原理图库3.画原理图库4.编译原理图三、PCB1.确定元器件尺寸大小2.绘制PCB Library①使用元器件向导绘制元件库②原理图与PCB的映射3.绘制PCB①更新PCB②调整元件位置③布线④漏线检查⑤…...

编程日记 2025/9/7 15:54:22

视频级虚拟试衣技术在淘宝的产品化实践

作为一种新的商品表现形态&#xff0c;内容几乎存在于手淘用户动线全流程&#xff0c;例如信息流种草内容、搜索消费决策内容、详情页种草内容等。通过低成本、高时效的AIGC内容生成能力&#xff0c;能够从供给端缓解内容生产成本高的问题&#xff0c;通过源源不断的低成本供给…...

编程日记 2025/9/13 7:26:22

Redis安装及其AnotherRedisDesktopManagera安装使用

一、Redis安装 1. 下载Redis安装包 通过网盘分享的文件&#xff1a;Redis 链接: https://pan.baidu.com/s/1elAT8mk3EIoYQQ3WoVVoNg?pwd7yrz 提取码: 7yrz 2. 解压Redis安装包 下载完成后&#xff0c;将Redis安装包解压到一个指定的目录&#xff0c;例如&#xff1a;C:\Re…...

编程日记 2025/9/10 3:56:21

2025系统架构师（一考就过）：案例之四：架构复用、架构评估、特定架构（DSSA）、架构开发方法（ABSD）

二、软件架构复用 ◆软件产品线是指一组软件密集型系统&#xff0c;它们共享一个公共的、可管理的特性集&#xff0c;满足某个特定市场或任务的具体需要&#xff0c;是以规定的方式用公共的核心资产集成开发出来的。即围绕核心资产库进行管理复用、集成新的系统。 ◆软件架构…...

编程日记 2025/9/9 3:45:54

软件工程应试复习（考试折磨版）

针对学校软件工程考试&#xff0c;参考教材《软件工程导论&#xff08;第6版&#xff09;》1-8章 学习的艺术&#xff1a;不断地尝试&#xff0c;我一定会找到高效用的方法&#xff0c;让学习变成一门艺术&#xff0c;从应试备考中解救出我的时间同胞们。 好嘞&#xff01;既然…...

编程日记 2025/9/12 14:36:15

Kafka可视化工具EFAK(Kafka-eagle)安装部署

Kafka Eagle是什么&#xff1f; Kafka Eagle是一款用于监控和管理Apache Kafka的开源系统&#xff0c;它提供了完善的管理页面&#xff0c;例如Broker详情、性能指标趋势、Topic集合、消费者信息等。 源代码地址&#xff1a;https://github.com/smartloli/kafka-eagle 前置条件…...

编程日记 2025/9/12 23:06:02

本地部署SenseVoice（包括离线设备操作）

Anaconda3 Anaconda Installers and Packages SenseVoice 魔搭社区 FFmpeg FFmpeg 安装Anaconda3&#xff08;Windows&#xff09; 下载完成后&#xff0c;自定义路径后安装。在开始菜单中出现Anaconda Prompt说明安装成功。 安装Anaconda3&#xff08;Linux&#xff…...

编程日记 2025/9/13 3:24:26

sql server 复制从备份初始化数据

参考 &#xff1a; 从备份初始化订阅&#xff08;事务&#xff09; - SQL Server | Microsoft Learn sql server 复制默认是用快照初始化数据的&#xff0c;也支持从备份初始化数据&#xff0c;参考如上...

编程日记 2025/8/25 20:39:24

MySQL 复合索引

MySQL 复合索引详解 引言 在实际业务场景中&#xff0c;多条件组合查询是最常见的操作之一。例如&#xff0c;根据“用户ID 时间范围”查询订单&#xff0c;或根据“商品分类 价格区间”筛选商品。此时&#xff0c;单列索引可能无法满足性能需求&#xff0c;而**复合索引&am…...

编程日记 2025/9/12 11:16:07

蓝桥杯备赛（C/C++组）

README&#xff1a; 本笔记是自己的备考笔记&#xff0c;按照官网提纲进行复习&#xff01;适合有基础&#xff0c;复习用。 一、总考点 试题考查选手解决实际问题的能力&#xff0c;对于结果填空题&#xff0c;选手可以使用手算、软件、编程等方法解决&#xff0c;对于编程大…...

编程日记 2025/9/6 15:34:22

人类驾驶的人脑两种判断模式（反射和预判）--＞自动驾驶两种AI模式

一种模式是直觉模式&#xff0c;判断是基于条件反射&#xff0c;视觉感知 触发到 直接条件反射&#xff08;从经历中沉淀形成的神经信息闭环&#xff09;&#xff0c;类似现在自动驾驶技术的传统AI模式。 另一种模式是物理时空图式推理模式&#xff0c;判断是基于预判预测&…...

编程日记 2025/9/11 11:05:58

C语言机试编程题

编写版本&#xff1a;vc2022 1.求最大/小值 #include<stdio.h> int main(){int a[50],n;int max, min;printf("请输入您要输入几个数");scanf_s("%d", &n);printf("请输入您要比较的%d个数\n",n);for (int i 0; i<n; i) {scanf_…...

编程日记 2025/9/11 13:59:19

SmartMediakit之音视频直播技术的极致体验与广泛应用

引言 在数字化时代&#xff0c;音视频直播技术已经深入到各个行业和领域&#xff0c;成为信息传递和交流的重要手段。视沃科技自2015年成立以来&#xff0c;一直致力于为传统行业提供极致体验的音视频直播技术解决方案&#xff0c;其旗下的大牛直播SDK凭借强大的功能和卓越的性…...

编程日记 2025/9/8 21:36:22

NVIDIA GEFORCE GTX1050显卡如何搭建AI人脸识别技术环境

NVIDIA GEFORCE GTX1050显卡如何搭建AI人脸识别技术环境！实际上，这个显卡虽然是入门级的，但是依然可以满足你的入门学习要求。 你的显卡是 NVIDIA GTX 1050，显存为 2GB，虽然它的性能不如高端显卡（如RTX 3060、3090等），但对于学习和研究 人脸识别技术，尤其是进行基础的…...

编程日记 2025/9/11 21:48:31

华为数通Datacom认证体系详解：从HCIA到HCIE的进阶路径

华为数通Datacom&#xff08;Data Communication&#xff09;课程是华为认证体系中的核心方向之一&#xff0c;聚焦企业网络通信与数据通信技术&#xff0c;适合从事网络规划、部署和运维的人员。 一、数通Datacom课程体系 华为数通Datacom认证分为 三个级别&#xff0c;逐级递…...

编程日记 2025/9/13 4:32:05

【运维】内网服务器借助通过某台可上外网的服务器实现公网访问

背景&#xff1a; 内网服务器无法连接公网,但是办公电脑可以连接内网服务器又可以连接公网。 安装软件 1、frp 2、ccproxy 配置 1、内网服务器 # 内网服务器启动frp服务配置文件参考vi frps.ini# frps.ini [common] bind_port 7000# 备注: bind_port端口可以随意配置。配置完…...

编程日记 2025/9/10 18:46:45

Spring之Bean的生命周期过程中调用的方法

1。这个部分除了6,9都在这了 package com.example.springbootdemo3.lifebeean;import org.springframework.beans.BeansException; import org.springframework.beans.factory.*; import org.springframework.beans.factory.annotation.Value; import org.springframework.con…...

编程日记 2025/9/9 15:50:26

Vue 中动态实现进度条

在 Vue 中动态实现进度条&#xff0c;基本上有两种常见的方法&#xff1a;直接通过 Vue 数据绑定控制样式&#xff0c;或者利用外部库来实现更复杂的功能。我们会深入探讨这两种方式&#xff0c;并且详细说明每种方法的实现步骤、优缺点以及使用场景。 1. 使用 Vue 数据绑定来…...

编程日记 2025/9/8 21:12:44

Qt互斥锁(QMutex)的使用、QMutexLocker的使用

Qt互斥锁【QMutex】的使用、QMutexLocker的使用 Chapter1 Qt互斥锁(QMutex)的使用、QMutexLocker的使用一、QMutexLocker和QMutex实现示例图二、QMutex和QMutexLocker的关系&#xff08;个人理解&#xff09;三、QMutex使用和QMutexLocker使用1.QMutex的使用2.QMutexLocker的使…...

编程日记 2025/9/10 6:18:19

Python开发 Flask框架面试题及参考答案

目录 Flask 的核心设计理念是什么？与 Django 相比有哪些显著差异？ 解释 Flask 框架的核心理念及其作为 “微框架” 的优缺点 Flask 的依赖库有哪些？简述 Werkzeug 和 Jinja2 的作用 什么是 WSGI？Flask 如何基于 WSGI 实现服务端与应用的交互 解释 RESTful API 的设计原…...

编程日记 2025/9/11 3:23:50

DeepSeek模型昇腾部署优秀实践

2024年12月26日&#xff0c;DeepSeek-V3横空出世&#xff0c;以其卓越性能备受瞩目。该模型发布即支持昇腾&#xff0c;用户可在昇腾硬件和MindIE推理引擎上实现高效推理&#xff0c;但在实际操作中&#xff0c;部署流程与常见问题困扰着不少开发者。本文将为你详细阐述昇腾 De…...

编程日记 2025/9/10 23:56:25

【cv】vs2022配置opencv

release下配置包含目录和库目录 E:\sdk\sdk_cuda12.3\opencv490\include E:\sdk\sdk_cuda12.3\opencv490\include\opencv2 E:\sdk\sdk_cuda12.3\opencv490\lib release下配置包含链接器输入的依附依赖项 opencv_world490.lib release编译文件夹下需手动复制opencv_world49…...

编程日记 2025/9/13 0:48:04

RabbitMQ系列（五）基本概念之Queue

在 RabbitMQ 中&#xff0c;Queue&#xff08;队列&#xff09; 是存储消息的容器&#xff0c;也是消息传递的核心载体。以下是其核心特性与作用的全方位解析&#xff1a; 一、Queue 的定义与核心作用 消息存储容器 Queue 是 RabbitMQ 中实际存储消息的实体&#xff0c;生产者…...

编程日记 2025/9/13 5:02:16