Locally deforming continuation method based on a shooting method for a class of optimal control problems

ABSTRACT This paper proposes a new continuation method for solving optimal control problems. The proposed method is based on a shooting method. In the proposed method, a cost function of an optimal control problem is locally deformed to find the solution of the problem in a stable way. This paper also analyses a relationship between the variation of the continuation parameter and the proximity of the solutions before and after a deformation in the proposed method. The obtained relation provides guidance on how to deform the continuation parameter. The effectiveness of this method is confirmed through numerical examples.


Introduction
Optimal control has been widely used in various fields, and various methods for solving optimal control problems exist. In optimal control problems for linear systems, the solution is often found analytically and is relatively easy to solve. On the other hand, in the case of nonlinear systems, the optimal solution is rarely found analytically, and therefore the solution is calculated numerically. There are two main methods for solving nonlinear optimal control problems numerically: the direct method and the indirect method [1]. The direct method first approximates the input of the solution of the optimal control problem with a function. Then it updates the function to minimize the cost function. On the other hand, in the indirect method, the optimal control problem is replaced by a two-point boundary value problem (TPBVP), and the TPBVP is solved numerically [2]. Generally speaking, the indirect method is more accurate than the direct method. Several indirect methods, such as the shooting methods [3,4] and the collocation methods [5], are known. The shooting methods are one of the most popular indirect methods. In the methods, the TPBVP is solved as an initial value problem, and they have an advantage that the computational complexity is very small because the search parameters are only the initial values. However, the shooting methods have a disadvantage that the convergence region is small, and therefore a good estimate of the solution has to be given beforehand. The continuation method is one of the ways to mitigate this difficulty.
The continuation method is a method to solve a difficult problem by iteratively deforming the problem continuously with a continuation parameter, and various methods have been proposed [6][7][8][9]. The idea of the method is to find a solution in a stable way by using the solution before deformation as an initial guess of the current solution after deformation. In the method, an easy optimal control problem is set up and solved first. Next, the cost function of the easy problem is deformed slightly towards the original problem, and the deformed optimal control problem is solved by using the previously obtained solution as an initial guess. By repeating this procedure and transforming the problem into the original problem in the end, the solution of the original problem is obtained. If each deformation is small, the solutions before and after a deformation are expected to be close enough, hence the method can solve the original problem in a stable way. The continuation method is widely used for solving various difficult problems such as obstacle avoidance problems [10], model predictive control problems [11], and sparse optimal control problems like an L 1 /L 2 -optimal control problem [12]. However, there is no continuation method that is effective for all problems, and it is unclear how the change of the cost function affects the closeness of the solutions before and after the deformation in those conventional methods.
In this paper, we propose a new continuation method, named a locally deforming continuation method, based on a shooting method. The idea of this method is to transform the cost function of the optimal control problem locally during the iterations of the continuation method. In addition, we provide a relation between the variation of the continuation parameter and the closeness of the solutions before and after the deformation in the form of an upper bound of the errors in the shooting method. This relation gives clear guidance on how to change the continuation parameter during the algorithm for stably solving relatively difficult problems.
The rest of paper is organized as follows. In Section 2, we introduce the problem formulation, and in Section 3, we briefly revisit some previous results. Next, Section 4 proposes the locally deforming continuation method and a modified shooting method and the results of our method are illustrated in Section 5. We summarize this article in Section 6.

Notation:
The Euclidean norm of the vector x is expressed as x . The symbol x i represents the ith element of the vector x. The function min(a) returns the minimum element of the vector a and max returns the maximum element of the vector a.

Problem setting
Here we treat the optimal control problem to find the input which minimizes the cost function subject to the following dynamics and input constraints: where x ∈ R n is the state vector, u ∈ R m is the input vector, u b ∈ R m is a positive constant vector, t f ∈ R + is a terminal time, x 0 ∈ R n is an initial state vector, x f ∈ R n is a terminal state vector, f : R n → R n , g : R n → R n×m , and l : R n × R m × R → R + . In this paper, we assume that l is continuously differentiable for x. If the dynamics (2) is nonlinear, the input cannot be obtained analytically in general. There are several ways to solve the problem, and these methods are mainly classified into the direct method or the indirect method. In this study, we use a shooting method classified into the indirect method for solving this problem. When the cost is the L 1 norm of the input, the problem becomes a sparse optimal control problem, and the resulting optimal control input is sparse, i.e. the L 0 norm of the input is much smaller than t f [12]. However, in the case of nonlinear systems, problems such as L 1 -optimal control problems are challenging to solve. The continuation method is one of the techniques to overcome this difficulty. The next section briefly reviews a conventional method.

Preliminaries
This section briefly reviews an existing way to solve an optimal control problem with a shooting method.

Transformation of the optimal control problem
Define a Hamiltonian function as follows: It follows from Pontryagin's minimum principle that the optimal control problem with (1)-(4) is reduced to a problem of solving the following simultaneous differential equation: with λ(t) ∈ R n that is the costate of x(t). This problem is called a TPBVP. Assume that (8) can be rewritten as the following by using a function of x(t), λ(t), and t.
where f u : R n × R n × R → R m . In this case, the TPBVP is rewritten as follows with ξ(t) = (x(t) , λ(t) ) .
Here the defined function F is F : There are several ways to solve this TPBVP, such as the shooting methods and the collocation methods. Among them, the shooting methods have an advantage that the number of the search parameters is small. The next section briefly explains a standard shooting method.

Shooting method
If x(0) and λ(0) are given, x(t), λ(t) satisfying (10) is uniquely determined. The shooting method takes advantage of this property and treats the TPBVP as an initial value problem. Since x(0) is already given as the boundary condition, the method searches for λ(0) that satisfies (10) and x(0) = x 0 . (11) Define the solution of the problem as λ * (0). In the algorithm, first an initial guess of the solution λ * (0) is given as λ g , and then update λ(0) so that the error x decreases. The difficulty of this method is that the error cannot be reduced to 0 if the initial guess λ g is far from λ * (0). One of the way to mitigate this difficulty is to apply the continuation method. In the continuation method, a relatively easy optimal control problem is solved first. Then, the cost function is slightly changed towards the original problem, and solve the changed problem by using the previous solution as the initial guess. Since the previous solution is expected to be close to the solution of the current problem, the shooting method successfully finds the solution with the given initial guess. Iterating this procedure and finally we can obtain the solution of the original problem. There are several continuation methods for solving an optimal control problem [13]. However, there is no continuation method that is effective for all problems, and it is unclear how the change of the cost function affects the closeness of the solutions before and after the deformation in those conventional methods.
This paper proposes a new continuation method based on a shooting method and analyses the relation between the variation of the continuation parameter and the deviation of the initial estimate from the solution. This relation gives a clear guidance on how to change the continuation parameter during the algorithm for stably solving relatively difficult problems. The next section explains the proposed method.

Proposed method
The proposed method solves the problem iteratively by successively transforming the problem from a relatively easy one to the original problem, in the same way as the existing methods. The characteristic feature of the proposed method is that the cost function is locally deformed during the iterations. Define the cost function of a relatively easy problem as and define the cost function of the original problem as with Basically, the previous continuation method gradually changes the cost function by transforming the continuation parameter c from 0 to 1, where the cost function is defined as In our new method, we define a cost function as The function l( and continuously differentiable at any t ∈ [0, t f ]. Figure  1 shows an example of the shape of l E (x(t), u(t), t), l O (x(t), u(t), t), and l(x(t), u(t), t, c), where the horizontal axis is time and the vertical axis is the value of the function l(x(t), u(t), t). The black solid line Our new continuation method transforms the cost function locally by changing the value of c. Note that we assume that each optimal control problem transformed iteratively has a solution. In the following, we define the solution of the optimal control problem with the cost function (15) The next section explains a shooting method used in the proposed continuation method. In the shooting method, the condition (11) is modified.

Modified shooting method
Define a Hamiltonian function as for the optimal control problem defined in the previous section. Then, from Pontryagin's minimum principle, and (4), Here we modify the shooting method in Section 3.2 as in the following definition. and for the error J c (t mid ) defined as with Here ξ c,+ (t) and ξ c,− (t) are defined as The difference between this method and the shooting method introduced in Section 3.2 is that there are two parameters to search for, not only λ(0) but also λ(t f ), and the boundary conditions are the continuity of x(t) and λ(t) at t = t mid . When the initial guess of the solution is different from the solution, J c (t mid ) takes a value greater than zero, and if the norm of the error is sufficiently small for the initial guess, the guess is expected to be close enough to the solution. We define the error for the initial guess as "initial error." In the next section, we explain the new continuation method and provides a clear relation between the variation of the continuation parameter c and the initial error I J c (t mid ) .

Locally deforming continuation method
The idea of the locally deforming continuation method is to transform the cost function locally. In the existing methods, the cost function is deformed by changing the continuation parameter c from 0 to 1 as in Figure 2. On the other hand, the locally deforming continuation method locally changes the cost function by changing c from t f to −h, where the function l(x(t), u(t), t, c) changes as shown in Figure 3. Both Figures 2 and 3 are depicted in the same way as in Figure 1. The algorithm of the locally deforming continuation method with the modified shooting method is shown in Algorithm 1.
Here t mid is set as The parameters t f , x 0 , x f , h, δ are given before executing the algorithm and assume that x J E (t), u J E (t), and  λ J E (t) are obtained. Here h and δ are scalars satisfying 0 < δ ≤ h, and δ represents the variation of the continuation parameter c.

end if 11: end loop
In the next section, we analyse the relation between δ and the initial error I J c (t mid ) with respect to the proposed algorithm.

Analysis of the relation between δ and I J c (t mid )
First, we prove the following lemmas for I J c (t mid ) and then we introduce the relation between δ and I J c (t mid ).
We omit the proof since it is proved in the same way as Lemma 4.1. In addition to Lemmas 4.1-4.3, under certain conditions, the following theorem holds.
Proof: This proof refers the proof of Theorem 2.1 in [14]. Consider the case δ ≤ c ≤ t f − h and define (x c,+ , λ c, Hence the following relation d dt holds. By multiplying the integrating factor e −(α 1 +α 2 )t to (44), we have From Lemma 4.2, the initial error is gained by integratingξ c−δ,+ −ξ c,+ with (37) and (38) from c − δ to c + h. The integration of (45) from c − δ to c + h leads to Since x c−δ,+ (c − δ) = x c,+ (c − δ) and λ c−δ,+ (c − δ) = λ c,+ (c − δ) hold from (33), it follows from (46) that and the relation In the same way, for the case holds since t f − (c − δ) ≤ h + δ, and for the case −h ≤ c < δ, This completes the proof. For example, the theorem shows that both h and δ should be decreased for reducing the initial error, that is, reducing h + δ means the guess of the solution in each iteration becomes close to the solution and thus makes the algorithm more likely to succeed. Hence, h should be chosen so that it is as large as δ, and vice versa.
In addition, if h + δ is sufficiently small, the upper bound of the initial error I J c−δ (f t mid (c)) varies linearly with respect to h + δ and I J c−δ (f t mid (c)) also expected to change linearly.
The next section shows the effectiveness of the proposed method through numerical examples.

Numerical example
In this example, we apply an L 1 /L 2 -optimal control [12] to a two-wheeled rover depicted in Figure 4, where the horizontal axis is X, the vertical axis is Y , x 1 is the angle of the rover, and x 2 and x 3 are x-position and y-position of the rover, respectively. The system of the rover is denoted by and the input is restricted by u b,1 = u b,2 = 5. Define an optimal control problem with the cost function and a boundary condition The resultant trajectory of this optimal control problem is known as a hands-off trajectory [15], where the input u 1 (t) takes zero in a large part of the control sequence.

Solve the problem without continuation method
In this example, we solve the problem with the shooting method without using the continuation method. Since the shooting method requires an initial guess of the solution, we first solve an L 2 -optimal control problem with the cost function and set its solution as an initial guess for solving the L 1 /L 2 -optimal control problem. Figures 5 and 6 show the generated trajectories and inputs respectively. In Figure 5, the first row figure shows x 1 (t), the second figure shows x 2 (t), and the third figure shows x 3 (t). The horizontal axis is time and the vertical axis is the state. The plus sign markers show the boundary conditions of each state. In Figure 6, the blue dash-dotted line is u 1 (t) and the blue dashed line is u 2 (t), where the horizontal axis is time and the vertical axis is the input. As this result shows, the generated trajectories do not satisfy the boundary conditions, that is, the shooting method fails to solve the L 1 /L 2 -optimal control problem.  In the next problem, we use the locally deforming continuation method for solving the L 1 /L 2 -optimal control problem.

Solve the problem with the locally deforming continuation method
In this example, we use the locally deforming continuation method for solving the L 1 /L 2 -optimal control problem. In the method, J E and J O are defined as (48) and (50), and J c is defined by (15) and (16) with To confirm the effectiveness of our method, we performed Algorithm 1 under several combinations of h and δ listed in Table 1. Here we define h 0 = 0.500 and δ 0 = 0.125. Figure 7 shows the log-log plot of the relation between the mean of the initial errors of each case and h + δ. The horizontal axis is h + δ, and the vertical axis is the mean of the Euclidean norm of the initial errors of the shooting method during Algorithm 1. The green line shows the line of y = 0.2x, where y is the mean of the initial errors and x is h + δ. From Figure 7, we can see that the error decreases linearly with respect to h + δ if h + δ is small enough. Figures 8 and 9 show the generated trajectory and input in Case 4 in the same way as Figures 5 and 6. As can be seen from Figures 8-9, the proposed method successfully generates trajectories satisfying the boundary conditions and the generated input u 1 (t) takes zero in large part. These results show the locally deforming continuation method is effective   Table 1.  for solving a relatively difficult optimal control problem like an L 1 /L 2 -optimal control problem.

Comparison between the proposed method and the conventional method
In the example, we compare the initial errors of the conventional method and the proposed method in five cases, in which the number of iterations of each continuation method is the same in each case. Here the boundary condition at t f = 2 is set to x(2) = (1, 1, 1) . The conventional method with the cost function (14) is solved by the shooting method introduced in Section 3.2, and the initial error in this method is defined as follows. In addition, the continuation parameter c in (14) is changed in each iteration as c ← c −δ from 0 towards 1.
In Figure 10, the upper figure shows the means of the initial errors of the conventional method, and the lower figure shows those of the proposed one, where the cases that succeeded to find the solutions are marked in blue and the failed cases are coloured in red. At each case,δ and h + δ take the values as in horizontal axes, wherē δ 0 = 1/2, h 0 = 2, δ 0 = 2 and h = δ. The vertical axes are the mean of the Euclidean norm of the initial errors. The iteration number of each case is listed in Table 2. In the conventional method, if the iteration number is  increased, the estimate of the solution is expected to become close to the solution. However, as shown in the upper figure of Figure 10, the means of the initial errors in Cases 1 and 2 are almost the same in the conventional methods. On the other hand, the means of the initial errors of the proposed method decrease as in the lower figure of Figure 10, as mentioned in Remark 4.1.
Though there is no difference in the success or failure of searching the solution, this example shows that the proposed method is superior to the conventional method in terms that it provides a guideline on how to change the continuation parameter for successfully obtaining the solution.

Conclusion
In this study, we proposed a new continuation method based on the modified shooting method for optimal control problems. We clarify the relationship between the continuation parameter and the proximity of the solutions before and after a deformation in terms of the error in the shooting method, which has been unclear in the conventional continuation methods. This relation is useful in solving the optimal control problem in the sense that it provides a guideline on how to change the continuation parameter when applying the locally deforming continuation method. Our future work is to work on more difficult problems like L 0 -optimal control problems [16] or CLOT optimal control problems [17] by using the locally deforming continuation method.

Disclosure statement
No potential conflict of interest was reported by the author(s).