增广拉格朗日乘子法（Augmented Lagrange Method）

转载自：增广拉格朗日乘子法（Augmented Lagrange Method）

增广拉格朗日乘子法的作用是用来解决等式约束下的优化问题，

假定需要求解的问题如下：

　　　　minimize　　 f(X)

　　　　s.t.:　　　　　h(X)=0

其中，f:Rⁿ->R; h:Rⁿ->R^m

朴素拉格朗日乘子法的解决方案是：

　　　　L(X,λ)=f(X)+μh(X);　　μ:R^m

　　　　此时，求解L对X和μ的偏导同时为零就可以得到最优解了。

增广拉格朗日乘子法的解决方案是：

　　　　L_c(x,λ)=f(X)+μh(X)+^₁/₂c|h(X)|²

　　　　每次求出一个xi，然后按照梯度更新参数μ，c每次迭代逐渐增大（使用ALM方法好像还有一些假设条件）

　　　　整个流程只需要几步就可以完成了，一直迭代就可得到最优解了。

参考文献：

　　[1]Multiplier and Gradient Methods,1969

　　[2]constrained optimization and lagrange multiplier methods(page 104),1982

wiki：https://en.wikipedia.org/wiki/Augmented_Lagrangian_method

Let us say we are solving the following constrained problem:

subject to

This problem can be solved as a series of unconstrained minimization problems. For reference, we first list the penalty method approach:

The penalty method solves this problem, then at the next iteration it re-solves the problem using a larger value of  (and using the old solution as the initial guess or "warm-start").

The augmented Lagrangian method uses the following unconstrained objective:

and after each iteration, in addition to updating, the variable  is also updated according to the rule

where  is the solution to the unconstrained problem at the kth step, i.e. 

The variable  is an estimate of the Lagrange multiplier, and the accuracy of this estimate improves at every step. The major advantage of the method is that unlike the penalty method, it is not necessary to take  in order to solve the original constrained problem. Instead, because of the presence of the Lagrange multiplier term,  can stay much smaller.

The method can be extended to handle inequality constraints. For a discussion of practical improvements, see.^[4]