RL【6】：Stochastic Approximation and Stochastic Gradient Descent

系列文章目录

Fundamental Tools

RL【1】：Basic Concepts
RL【2】：Bellman Equation
RL【3】：Bellman Optimality Equation

Algorithm

RL【4】：Value Iteration and Policy Iteration
RL【5】：Monte Carlo Learning
RL【6】：Stochastic Approximation and Stochastic Gradient Descent

Method

---

---

前言

本系列文章主要用于记录 B站 赵世钰老师的【强化学习的数学原理】的学习笔记，关于赵老师课程的具体内容，可以移步：
B站视频：【【强化学习的数学原理】课程：从零开始到透彻理解（完结）】
GitHub 课程资料：Book-Mathematical-Foundation-of-Reinforcement-Learning

Tips：本文主要记录了一些扩展知识，涉及一些复杂的数学推导，故在此先做简单的记录，后续会进一步完善

---

Robbins-Monro Algorithm

Problem statement

Suppose we would like to find the root of the equation $g(w)=0$, where $w\in \mathbb{R}$ is the variable to be solved and $g:\mathbb{R}\to \mathbb{R}$ is a function.

• Many problems can be eventually converted to this root finding problem. For example, suppose $J(w)$ is an objective function to be minimized. Then, the optimization problem can be converted to

  $g(w)={\nabla }_{w}J(w)=0.$

• Note that an equation like $g(w)=c$ with $c$ as a constant can also be converted to the above equation by rewriting $g(w)-c$ as a new function.

The Robbins–Monro (RM) algorithm can solve this problem:

$w_{k+1}=w_k-a_k\tilde{g}(w_k,{\eta }_k),k=1,2,3,\dots$

• where
  • $w_k$ is the $k$-th estimate of the root.
  • $\tilde{g}(w_k,{\eta }_k)=g(w_k)+{\eta }_k$ is the $k$-th noisy observation.
  • $a_k$ is a positive coefficient.
• The function $g(w)$ is a black box!
  • Input sequence: {wk}\{ w_k \}{wk​}
  • Noisy output sequence: {g~(wk,ηk)}\{ \tilde{g}(w_k, \eta_k) \}{g~​(wk​,ηk​)}

Robbins-Monro Theorem

In the Robbins–Monro algorithm, if

1. $0<c_1\le {\nabla }_{w}g(w)\le c_2\text{for all }w;$
2. ${\sum }_{k=1}^{\infty }{a}_k=\infty \text{and}{\sum }_{k=1}^{\infty }{a}_k^2<\infty ;$
3. $\mathbb{E}[{\eta }_k\mid {\mathcal{H}}_k]=0\text{and}\mathbb{E}[{\eta }_k^2\mid {\mathcal{H}}_k]<\infty ;$

where Hk={wk,wk−1,…}\mathcal{H}k = \{ w_k, w{k-1}, \ldots \}Hk={wk​,wk−1,…}, then $w_k$ converges with probability 1 (w.p.1) to the root $w^{\ast }$ satisfying $\ast g(w^{\ast })=0\ast$.

Convergence properties

1. $0<c_1\le {\nabla }_{w}g(w)\le c_2\text{for all }w$
   • This condition indicates $g$ to be monotonically increasing, which ensures that the root of $g(w)=0$ exists and is unique.
   • The gradient is bounded from above.
2. ${\sum }_{k=1}^{\infty }{a}_k=\infty \text{and}{\sum }_{k=1}^{\infty }{a}_k^2<\infty$
   • The condition ${\sum }_{k=1}^{\infty }{a}_k^2<\infty$ ensures that $a_k$ converges to zero as $k\to \infty$.
   • The condition ${\sum }_{k=1}^{\infty }{a}_k=\infty$ ensures that $a_k$ do not converge to zero too fast.
   • Why is this condition important?

     • Summarizing $w_2=w_1-a_1\tilde{g}(w_1,{\eta }_1),w_3=w_2-a_2\tilde{g}(w_2,{\eta }_2),\dots ,w_{k+1}=w_k-a_k\tilde{g}(w_k,{\eta }_k)$ leads to

       $w_1-w_{\infty }={\sum }_{k=1}^{\infty }{a}_k\tilde{g}(w_k,{\eta }_k).$

     • Suppose $w_{\infty }=w^{\ast }$. If ${\sum }_{k=1}^{\infty }{a}_k<\infty$, then ${\sum }_{k=1}^{\infty }{a}_k\tilde{g}(w_k,{\eta }_k)$ may be bounded. Then, if the initial guess $w_1$ is chosen arbitrarily far away from $w^{\ast }$, then the above equality would be invalid.

3. $\mathbb{E}[{\eta }_k\mid {\mathcal{H}}_k]=0\text{and}\mathbb{E}[{\eta }_k^2\mid {\mathcal{H}}_k]<\infty$
   • A special yet common case is that {ηk}\{\eta_k\}{ηk​} is an iid stochastic sequence satisfying $\mathbb{E}[{\eta }_k]=0$ and $\mathbb{E}[{\eta }_k^2]<\infty$.
   • The observation error ${\eta }_k$ is not required to be Gaussian.
4. What {ak}\{a_k\}{ak​} satisfies the two conditions?
   • One typical sequence is $a_k=\frac{1}{k}$.
   • It holds that ${\lim }_{n\to \infty }\left({\sum }_{k=1}^n\frac{1}{k}-\ln n\right)=\kappa$,
     • where $\kappa \approx 0.577$ is called the Euler–Mascheroni constant (also called Euler’s constant)
   • It is notable that ${\sum }_{k=1}^{\infty }\frac{1}{k^2}=\frac{{\pi }^2}{6}<\infty .$
     • The limit ${\sum }_{k=1}^{\infty }\frac{1}{k^2}$ also has a specific name in the number theory

---

Stochastic Gradient Descent

Algorithm description

• Suppose we aim to solve the following optimization problem:

  ${\min }_{w}J(w)=\mathbb{E}[f(w,X)]$

  • $w$ is the parameter to be optimized.
  • $X$ is a random variable. The expectation is with respect to $X$.
  • $w$ and $X$ can be either scalars or vectors. The function $f(\cdot )$ is a scalar.

• Three Methods

  • Method 1: gradient descent (GD)

    $w_{k+1}=w_k-{\alpha }_k{\nabla }_w\mathbb{E}[f(w_k,X)]=w_k-{\alpha }_k\mathbb{E}[{\nabla }_{w}f(w_k,X)]$

    • Drawback: the expected value is difficult to obtain.

  • Method 2: batch gradient descent (BGD)

    $\mathbb{E}[{\nabla }_{w}f(w_k,X)]\approx \frac{1}{n}{\sum }_{i=1}^n{\nabla }_{w}f(w_k,x_i),$

    $w_{k+1}=w_k-{\alpha }_k\frac{1}{n}{\sum }_{i=1}^n{\nabla }_{w}f(w_k,x_i).$

    • Drawback: it requires many samples in each iteration for each $w_k$.

  • Method 3: stochastic gradient descent (SGD)

    $w_{k+1}=w_k-{\alpha }_k{\nabla }_{w}f(w_k,x_k),$

    • Compared to the gradient descent method: Replace the true gradient
      $\mathbb{E}[{\nabla }_{w}f(w_k,X)]$ by the stochastic gradient ${\nabla }_{w}f(w_k,x_k)$.
    • Compared to the batch gradient descent method: let $n=1$.

Convergence analysis

• From GD to SGD:

  $w_{k+1}=w_k-{\alpha }_k\mathbb{E}[{\nabla }_{w}f(w_k,X)]⟹w_{k+1}=w_k-{\alpha }_k{\nabla }_{w}f(w_k,x_k)$

  • ${\nabla }_{w}f(w_k,x_k)$ can be viewed as a noisy measurement of
    $\mathbb{E}[{\nabla }_{w}f(w,X)]$:

    ${\nabla }_{w}f(w_k,x_k)=\mathbb{E}[{\nabla }_{w}f(w,X)]+\underset{\eta }{\underbrace{({\nabla }_{w}f(w_k,x_k)-\mathbb{E}[{\nabla }_{w}f(w,X)])}}.$

  • Since

    ${\nabla }_{w}f(w_k,x_k)\ne \mathbb{E}[{\nabla }_{w}f(w,X)],$

• SGD as RM Algorithm

  • Then, the convergence naturally follows. The aim of SGD is to minimize

    $J(w)=\mathbb{E}[f(w,X)].$

  • This problem can be converted to a root-finding problem:

    ${\nabla }_{w}J(w)=\mathbb{E}[{\nabla }_{w}f(w,X)]=0.$

  • Let

    $g(w)={\nabla }_{w}J(w)=\mathbb{E}[{\nabla }_{w}f(w,X)].$

  • Then, the aim of SGD is to find the root of $g(w)=0$.

• SGD = RM

  • What we can measure is

    $\tilde{g}(w,\eta )={\nabla }_{w}f(w,x)=\underset{g(w)}{\underbrace{\mathbb{E}[{\nabla }_{w}f(w,X)]}}+\underset{\eta }{\underbrace{{\nabla }_{w}f(w,x)-\mathbb{E}[{\nabla }_{w}f(w,X)]}}.$

    • It is exactly the SGD algorithm.
    • Therefore, SGD is a special RM algorithm.

• Convergence of SGD

  • In the SGD algorithm, if
    1. $0<c_1\le {\nabla }_w^{2}f(w,X)\le c_2$;
    2. ${\sum }_{k=1}^{\infty }{a}_k=\infty \text{and}{\sum }_{k=1}^{\infty }{a}_k^2<\infty$;
    3. {xk}k=1∞\{x_k\}_{k=1}^\infty{xk​}k=1∞​ is iid;
  • then $w_k$ converges to the root of ${\nabla }_w\mathbb{E}[f(w,X)]=0$ with probability $1$.

A deterministic formulation

• Consider the optimization problem:

  ${\min }_{w}J(w)=\frac{1}{n}{\sum }_{i=1}^{n}f(w,x_i),$

  • $f(w,x_i)$ is a parameterized function.
  • $w$ is the parameter to be optimized.
  • {xi}i=1n\{x_i\}_{i=1}^n{xi​}i=1n​ is a set of real numbers, where $x_i$ does not have to be a sample of any random variable.

• The gradient descent algorithm for solving this problem is

  $w_{k+1}=w_k-{\alpha }_k{\nabla }_{w}J(w_k)=w_k-{\alpha }_k\frac{1}{n}{\sum }_{i=1}^n{\nabla }_{w}f(w_k,x_i).$

• Suppose the set is large and we can only fetch a single number every time. In this case, we can use the following iterative algorithm:

  $w_{k+1}=w_k-{\alpha }_k{\nabla }_{w}f(w_k,x_k).$

• Questions:

  • Is this algorithm SGD? It does not involve any random variables or expected values.
  • How should we use the finite set of numbers {xi}i=1n\{x_i\}_{i=1}^n{xi​}i=1n​? Should we sort these numbers in a certain order and then use them one by one? Or should we randomly sample a number from the set?

• Answer:

  • A quick answer to the above questions is that we can introduce a random variable manually and convert the deterministic formulation to the stochastic formulation of SGD.

  • In particular, suppose $X$ is a random variable defined on the set {xi}i=1n\{x_i\}_{i=1}^n{xi​}i=1n​.

  • Suppose its probability distribution is uniform such that

    $p(X=x_i)=\frac{1}{n}.$

  • Then, the deterministic optimization problem becomes a stochastic one:

    ${\min }_{w}J(w)=\frac{1}{n}{\sum }_{i=1}^{n}f(w,x_i)=\mathbb{E}[f(w,X)].$

    • The last equality in the above equation is strict instead of approximate. Therefore, the algorithm is SGD.
    • The estimate converges if $x_k$ is uniformly and independently sampled from {xi}i=1n\{x_i\}{i=1}^n{xi​}i=1n. Note that $x_k$ may repeatedly take the same number in {xi}i=1n\{x_i\}{i=1}^n{xi​}i=1n since it is sampled randomly.

---

BGD, MBGD, and SGD

• Suppose we would like to minimize $J(w)=\mathbb{E}[f(w,X)]$ given a set of random samples {xi}i=1n\{x_i\}_{i=1}^n{xi​}i=1n​ of $X$. The BGD, SGD, and MBGD algorithms solving this problem are, respectively,
  • $w_{k+1}=w_k-{\alpha }_k\frac{1}{n}{\sum }_{i=1}^n{\nabla }_{w}f(w_k,x_i),\text{(BGD)}$
    • In the BGD algorithm, all the samples are used in every iteration. When $n$ is large, $\frac{1}{n}{\sum }_{i=1}^n{\nabla }_{w}f(w_k,x_i)\approx \mathbb{E}[{\nabla }_{w}f(w_k,X)]$.
  • $w_{k+1}=w_k-{\alpha }_k\frac{1}{m}{\sum }_{j\in \mathcal{I}k}{\nabla }_{w}f(w_k,x_j),\text{(MBGD)}$
    • In the MBGD algorithm, ${\mathcal{I}}_k$ is a subset of {1,…,n}\{1, \ldots, n\}{1,…,n} with size $|{\mathcal{I}}_k|=m$. The set $\mathcal{I}k$ is obtained by $m$ i.i.d. samplings.
  • $\ast w_{k+1}=w_k-{\alpha }_k{\nabla }_{w}f(w_k,x_k).\text{(SGD)}\ast$
    • In the SGD algorithm, $x_k$ is randomly sampled from {xi}i=1n\{x_i\}{i=1}^n{xi​}i=1n at time $k$.
• Compare MBGD with BGD and SGD:
  • Compared to SGD, MBGD has less randomness because it uses more samples instead of just one as in SGD.
  • Compared to BGD, MBGD does not require to use all the samples in every iteration, making it more flexible and efficient.
  • If $m=1$, MBGD becomes SGD.
  • If $m=n$, MBGD does NOT become BGD strictly speaking because MBGD uses randomly fetched $n$ samples whereas BGD uses all $n$ numbers. In particular, MBGD may use a value in {xi}i=1n\{x_i\}_{i=1}^n{xi​}i=1n​ multiple times whereas BGD uses each number once.

---

Summary

• Mean estimation: compute $\mathbb{E}[X]$ using {xk}\{x_k\}{xk​}

  $w_{k+1}=w_k-\frac{1}{k}(w_k-x_k).$

• RM algorithm: solve $g(w)=0$ using {g~(wk,ηk)}\{\tilde{g}(w_k, \eta_k)\}{g~​(wk​,ηk​)}

  $w_{k+1}=w_k-a_k\tilde{g}(w_k,{\eta }_k).$

• SGD algorithm: minimize $J(w)=\mathbb{E}[f(w,X)]$ using {∇wf(wk,xk)}\{\nabla_w f(w_k, x_k)\}{∇w​f(wk​,xk​)}

  $w_{k+1}=w_k-a_k{\nabla }_{w}f(w_k,x_k).$

---

总结