Generalized logarithmic penalty function method for solving smooth nonlinear programming involving invex functions

Abstract In this paper, we have reviewed some penalty function methods for solving constrained optimization problems in the literature and proposed a continuously differentiable logarithmic penalty function which consists of the proposed logarithmic penalty function and modified Courant-Beltrami penalty function for equality and inequality constraints, respectively. Furthermore, we hybridized the two and came up with the general form of both (equality and inequality) constraints. However, in the first part, the equivalence between the sets of optimal solutions in the original optimization problem and its associated penalized logarithmic optimization problem constituted by invex functions with equality and inequality constraints has been established. In the second part, we have validated the general form of the logarithmic penalty function and compared the results with absolute value penalty function results by solving nine small problems from Hock-Schittkowski collections of test problems with different classifications. The experiments were carried out via quasi-newton algorithm using a fminunc routine function in matlab2018a. The general form yields a better objective value compared to absolute value penalty function.


Introduction
In this paper, the general form of constrained optimization involving equality and inequality constraints is considered: Minimize f ðxÞ Subject to h j x ð Þ ¼ 0; j 2 J ¼ f1; 2; :::::::; sg; g i x ð Þ 0; i 2 I ¼ 1; 2; :::::::; m f g ; x 2 X; (P) Where f : X ! R and h j : X ! R; j 2 J ; g i : X ! R; i 2 I; are differentiable functions on a non-empty open subset of real number X: Let's introduce some notation that will frequently be used throughout this paper. Let ¼ fx 2 X : h j x ð Þ ¼ 0; j 2 J; g i ðxÞ 0; i 2 Ig be the set of all feasible solutions of the optimization problem (P). I x ð Þ ¼ i 2 I : g i x ð Þ ¼ 0 È É denotes the set of active constraints at an optimal point x: In recent years, an approach to solve a non-linear optimization problem (P) is not limited to the conventional strategies for solving a constrained optimization problem only. Several methods have come into existence in the last few decades. One of the attractive approaches is the use of the penalty function, and it is usually defined in terms of the constraints h and g: The constraints are incorporated into the objective function by adding a penalty term that penalized any violations of the single unconstrained optimization problem. This approach guaranteed that the solution could be found through unconstrained minimization technique.
Based on the existing approach of penalty function methods, it has been divided into two forms: a standard penalty function and an exact penalty function which can also be subdivided into differentiable and non-differentiable. Zangwill (1967) was the first to introduce non-differentiable exact penalty function and presented an algorithm that can be used to solve a non-linear programming problem, and the method appears to be more useful in the concave case. Primarily, a penalty function has been used to transform a problem (P) into a single unconstrained problem or a finite sequence of the unconstrained optimization problems. The non-differentiable exact penalty function introduced by Zangwill (1967) for the problem (P) was; Note that: g þ i x ð Þ ¼ maxf0; g x ð Þg: The work of Fletcher (1973) comes up with a continuously differentiable exact penalty function regarding the problem (P) for inequality constraints; according to Fletcher's work it is possible to construct an exact penalty function which is sufficiently smooth to grant conventional techniques for solving the problem (P) to locate the local minimum for the transformed unconstrained problem. Other researchers (Bazaraa, Sherali, & Shetty, 1991;Bertsekas & Koksal, 2000;Charalambous, 1978;Conn, 1973) further investigated continuously differentiable and non-differentiable exact penalty function.
However, the idea of convexity plays a dominant role in almost all kinds of penalty function approaches (see for example Bazaraa et al. (1991), Bertsekas and Koksal (2000), Charalambous (1978), Conn (1973) and Mangasarian (1985)). In the last few years, some various convex function generalizations have been derived which give room for extending optimality condition and some classical duality results, formerly restricted to convex programmes, to the larger classes of optimization problems. The notion of invexity introduced by Hanson (1981) and named by Craven (1981) was among the types. The work of Hanson applied the extended concept of convex functions to prove optimality conditions and duality results for the non-linear constrained optimization problem. Antczak (2009) presented some new results on the exact penalty function methods; the work of Antczak characterized a differentiable non-convex optimization problem with mixed constraints as in problem (P) via the following exact penalty function with q ¼ 1; where q is a positive integer, the equivalence between sets of optimal solutions in the problem (P) and the following penalized optimization problem under suitable invexity assumption is established: where c is a penalty parameter. Liu and Feng (2010) constructed a classical exponential penalty function method for multi-objective programming problems (MOPP) and its convergence is proved. Antczak (2011) introduced the l 1 exact exponential penalty function method, which is constructed based on the optimization problem (P) with both constraints (equality and inequality). The l 1 exact exponential penalty function method is used to solve an optimization problem constituted by r-invex functions (with respect to the same function g), the penalty function is of the following form: where r is a finite real number not equal to 0. Note that the function 1 r ðe rg þ i x ð Þ À 1Þ is defined by Obviously, (5) has the penalty features relative to the single constraint g i ðxÞ 0; that is 0 for all values of x that satisfy the constraint and result in large values whenever the constraint is violated; the penalty function (4) is considered to be a classical (1) if r ¼ 0 that was defined by Pietrzykowski (1969) and also by Han and Mangasarian (1979). Further, the results have been proved through the classical l 1 exact penalty function method under r-invexity assumption by Antczak (2010) for both equality and inequality constraints. Jayswal and Choudhury (2014) were able to extend the work of Antczak (2011) and Liu and Feng (2010) to multiobjective fractional programming problems and examine the convergence of the method. Morrison (1968) proposes another penalized problem of the form; where M is an estimated optimized objective function f ðx Ã Þ and hðxÞ is an equality constraint. The Morrison function (6) can also be considered as an exact penalty function. Another, similar to that of the Morrison method has been proposed by Meng, Hu, Dang, and Yang (2004) and Meng, Dang, Jiang, Xu, and Shen (2013). Luenberger (1973) explored an unconstrained problem that works in the space R mþn with respect to the objective and constraint functions in (6) of the following form; where k T is the transpose of LaGrange multiplier vector; this approach does not require successive minimization solution. Nevertheless, this approach admits disadvantage of higher dimension. Filter based approach to solve the same constrained optimization problem were introduced by Fletcher and Leyffer (2002). The concept is implemented by minimizing two functions f x ð Þ; CðxÞ simultaneously. The function CðxÞ possesses the basic penalty function property, i.e.
It is a list of pairs f l ; C l À Á whereas no pair will be allowed to dominate another. Nie (2007) modified the original filter method specifically for the equality constraint only, and the process is called a sequential penalty quadratic programming filter method (SIQP). His approach replaced the objective function by the penalized function of the form: where r is a fixed parameter that does not need to be updated at each step. According to Nie, this approach is advantageous compared to the original filter method. Utsch De Freitas Pinto and Martins Ferreira (2014) propose an exact penalty function based on matrix projection, and the constructed unconstrained problem is of the form: where d x ð Þ ¼ P x ð Þr T f ðxÞ (gradient projection vector used by Rosen (1960Rosen ( , 1961) r T f ðxÞ is the transpose of rf ðxÞ and P Þ (projection matrix over the constrained tangent subspace of the considered problem).
Some of the observed setbacks of this approach include; all the stationary points of the considered problem are local minima and the difficulty of the matrix inversion required to compute P x ð Þ: Moreover, none of the above-listed penalty functions approach the authors' claim to be perfect on any formulations. Another critical point to be taken into consideration is that the composition of any optimization problem may vary from one form to another, apart from the notion of convex, invex, non-convex optimization problems; there are also regularity, irregularity, linear objective with non-linear constraints functions, non-linear objective with linear constraints functions, quadratic and polynomial functions.
In this paper, we propose another penalty function in accordance with existing penalty function methods. The proposed penalty function is constructed according to the general form of mathematical programming problems (P), since it is suitable for the optimization problems with mixed constraints (equality and inequality), unlike some of the existing ones that are either restricted to equality constraints or inequality constraints only. Furthermore, it is among the category of the continuously differentiable penalty function. The entire presentation is organized as follows: In Section 2, the method of the proposed logarithmic penalty function is introduced. In Section 3, we use the first order necessary optimality conditions to derive the KKT multipliers for the proposed logarithmic penalty function. In Section 4, the equivalence between the sets of optimal solutions of the original optimization problem and its associated penalized problem constituted by invex functions (with respect to the same function g) was established. In Section 5, we validated the proposed general form of logarithmic penalty function methods via a quasi-Newton algorithm using the fminunc routine function in matlab2018a. Finally, in the last section, the results obtained are discussed followed by the conclusion of the paper.

Method of logarithmic penalty function
Transforming a constrained problem into a single or a sequence of unconstrained optimization problems can be done through penalty methods. The primary responsibility of the penalty term is to incur a decisive penalty for infeasible points, while there is no penalty for feasible points.
For the considered non-linear optimization problem (P), its penalized optimization problem is defined by P x; c ð Þ ¼ f x ð Þ þ cp x ð Þ: p represents a suitable penalty function, and c is a positive penalty parameter. Now, the newly constructed logarithmic penalty function consists of the following parts: (i) Proposed logarithmic penalty function for equality constraints (ii) Modified Courant-Beltrami penalty function for inequality constraints If we consider an optimization problem (P) with equality constraints only, we construct the following penalty function regarding (i); It is obvious that (10) possesses the penalty function features relative to the equality constraints h j x ð Þ ¼ 0 as presented by Hassan and Baharum (2019); the penalized optimization based on (10) should be constructed as follows; To make the proposed logarithmic penalty function more general, we modified the Courant-Beltrami penalty function as presented in (Byrne, 2008;Ernst & Volle, 2013), and the Courant-Beltrami penalty function restricted to inequality constraints is constructed based on the exact penalty function (2) with q ¼ 2 as follows; Therefore, we modified (12) in logarithmic form like the proposed penalty function (10) as follows: Combining (10) and (13) will lead to generalized logarithmic penalty function regarding the problem (P), i.e.
For simplicity and convenience, upon substitution of (11) and (13) right hand sides into (14), the generalized logarithmic penalty function can be expressed as follows: These terms can also be denoted by can simply be rewritten as follows: Both functions u and / are continuous functions satisfying the following conditions: :::::::m

Logarithmic penalty function Karush-Khun-Tucker multipliers
In optimization problems, the first order necessary conditions for a non-linear optimization problem to be optimal are Karush-Kuhn-Tucker (KKT) conditions, or Kuhn-Tucker conditions, if some of the constraints' qualifications are satisfied. Nevertheless, Courant-Beltrami penalty function may not be differentiable at a point g i x ð Þ ¼ 0 for some i 2 I: But for the constrained optimization both objective function and constraints may be partially differentiable on R n while the penalized problem is not, as differentiability is not among the properties of maxf0; g i x ð Þg: According to proposition 1 (Ernst & Volle, 2013), some additional assumptions may be imposed on the constraint function g i x ð Þ; i.e. if the constraint g i x ð Þ has continuous first-order partial derivatives on R n ; for this reason g þ i x ð Þ À Á 2 admit the same. Therefore, where l ¼ 1; 2; ::::; n; representing the number of variables. Considering equation ( Ã ), if p 2 x ð Þ : R n ! R is a modified Courant-Beltrami penalty function (MCBPF) and the constraints g i x ð Þ have continuous first order partial derivative on R n ; then In the same way, for equality constrained let p 1 x ð Þ : R n ! R; and assume that p x ð Þ is continuously differentiable at x; then Theorem 1. Let x Ã be an optimal solution to the problem (P) and it satisfies the first-order necessary optimality conditions of the constrained problem (P). Moreover, let a suitable CQ be satisfied at x Ã ; then x Ã is a solution to the penalized problem.
Proof. If x Ã is a feasible point which satisfies the first-order necessary optimality conditions of the problem, then Then (16) can be replaced with the following equation; where n Ã i and l Ã j are vectors of KKT multiplier and n Ã i ! 0 for all i 2 I; l Ã j ! 0 for all j 2 J: Theorem 2. Let x Ã be an optimal solution to the problem (P) and assume that any constraint qualification is satisfied at x Ã : Then there exist LaGrange multipliers l Ã j 2 R s and n Ã i 2 R m such that Definition 2. (Hanson, 1981) Let f : X ! R be a differentiable function on X & R n and u 2 R n : If there exists a vector-valued function g : R n Â R n ! R n such that, 8 x 2 X; the following inequality holds, then the function f is said to be an invex (strictly invex) function with respect to g at u on X: If (17) holds at each point u 2 R n ; then f is said to be an invex (strictly invex) function with respect to g on R n : Definition 3. (Hanson, 1981) Let f : X ! R be a differentiable function on X & R n and u 2 R n : If there exists a vector-valued function g : R n Â R n ! R n such that, 8 x 2 X; the following inequality holds, then the function f is said to be an incave (strictly incave) function with respect to g at u on X: If (18) holds at each point u 2 R n ; then f is said to be an incave (strictly incave) function with respect to g on R n :

Logarithmic penalty function method for invex optimization problem
In this section, we consider the case whereby all the functions constituting an optimization problem are invex (with respect to the same function g). Under invexity assumption imposed on the functions constituting the problem (P) and the relevant constraints qualifications (Bazaraa et al., 1991), for a suitably large penalty parameter c exceeding some suited Threshold, a KKT point minimizes the problem (P) if and only if it minimizes the problem defined by: The subsequent theorem establishes the equivalence between an optimal solution in the original optimization problem (P) and its logarithmic penalty function minimizer under assumptions that the problem (P) is an invex optimization problem.
Theorem 3. Let x Ã be a feasible point in the considered optimization problem (P) and KKT necessary optimality conditions (i)-(iii) fulfilled at x Ã : Partitioning the set of indexes for equality constraints h j into J þ and J À i.e. J þ ¼ fj 2 J : l Ã j > 0g and J À ¼ fj 2 J : l Ã j < 0g: Moreover, suppose that the function f ; g i ; i 2 I x Ã ð Þ; h j ; j 2 J þ is invex at x Ã on X with respect to the same function g and the function h j ; j 2 J À are incave at x Ã on X with respect to the same function g: If c is sufficiently large (setting c ! maxfn Ã i ; i 2 I; l Ã j ; j 2 Jg).

Proof.
By hypothesis, x Ã is feasible in the problem (P) and KKT conditions when (i)-(iii) are satisfied at the feasible point x Ã with the Lagrange multiplier l Ã j ; j 2 J; n Ã i ; i 2 I; for equality constraints and inequality constraints, respectively.
Our goal is to show that x Ã is also a minimizer of the penalized problem (19) with a sufficiently large penalty parameter c: According to the problem (19), we have Moreover, On the other hand From KKT condition (ii), we have From KKT condition (i), we get Since f is also invex at x Ã on X with respect to the same function g and by invex definition with the feasibility of the problem (P) at x Ã ; we have From Equation (20) to Equation (28), we conclude that the following inequality holds for any This shows that x Ã is a minimizer of the penalized optimization problem (19). This completes the proof of the theorem.
Corollary 4. (This is a corollary to Theorem 3) Let x Ã be a point in the optimization problem (P), and the suitable constraint qualifications are satisfied at x Ã for the problem (P). Moreover, suppose that the function f ; g i ; i 2 I x Ã ð Þ; h j ; j 2 J þ are invex at x Ã on X with respect to the same function g and the function h j ; j 2 J À are incave at x Ã on X with respect to the same function g: If c is sufficiently large (setting c ! maxfn Ã i ; i 2 I; l Ã j ; j 2 Jg), where both l Ã j ; j ¼ 1; 2; ::::; s; n Ã i ; i ¼ 1; 2; : ; m are LaGrange multipliers associated with the equality and inequality constraints h j and g i respectively, then x Ã is also a minimizer of the penalized problem (19).
Proposition 5. Let the point x Ã be a minimizer of the logarithmic penalty function (15) in the penalized problem (19). Then the following inequality holds for all x 2 F: Suppose that x Ã minimizes the logarithmic penalty function (15) in the penalized problem (19). Since x Ã is an optimal solution in the problem (19), then the following inequality holds for all x 2 F: Hence, by definition of g þ i x ð Þ the following inequality is true for all x 2 F: This concludes the proof of the proposition. Now, the following theorem establishes contradictory results without any assumption that the functions constituting the problem (P) are invex.
Theorem 6. Let x Ã be a minimizer of logarithmic penalty function (15) in the penalized problem (19). Furthermore, let the inequality P x; c ð Þ ! P x Ã ; c ð Þ be satisfied for any c ! c Ã and for all x 2 F: Then x Ã is also an optimal solution in the original optimization problem (P).

Proof.
Let us assume that x Ã minimizes the logarithmic penalty function (15). By Proposition 5, the following inequality holds for all x 2 F: i.e.
The inequality (29) indicates that the function's values are bounded below on the feasible set F in the problem (P). Now, our goal is to prove that x Ã is also optimal in the original optimization problem (P). Accordingly, we will first show that x Ã is feasible in the original problem (P).
By contradiction, suppose that x Ã is not feasible to the problem (P). Hence, by (1), we have Based on the hypothesis, P x; c ð Þ ! P x Ã ; c ð Þ is satisfied for any c ! c Ã and for all x 2 F: Let us assume that x be any feasible solution in (P). So, So c because x 2 F: Combining (33) and (34) we get which is a contradiction to (30). Hence, x Ã is feasible in the problem (P) and its optimality follows from (29). Obviously, Theorem 6 has been well-established even though no invexity assumption is required on the functions constituting the optimization problem (P). Now, let us consider the case when invexity assumption is imposed on the problem constituted by invex functions, and we are to show that there exists the edge of a penalty parameter c; above which the result is true.
Theorem 7. Let x Ã be a minimizer of the logarithmic penalty function (1) in the penalized optimization problem (19). Moreover, let the functions f; g i ; i 2 I x ð Þ; h j ; j 2 J þ x ð Þ :¼ f j 2 J : l j > 0g are invex at any optimal point x in the problem (P) on X with respect to the same function g; and the function h j ; j 2 J À x ð Þ : ¼ f j 2 J : l j < 0g are incave at any optimal point x in (P) on X with respect to the same function g; the set of all feasible points in the original optimization problem is compact, and the penalty parameter c is considered to be sufficiently large (we set c > max n i ; i 2 I; È l j ; j2Jg where n i ; i¼1;2;::::;m:; l j ; j¼1;2;::::;s: are LaGrange multipliers satisfying KKT necessary optimality conditions at x) . Therefore, x Ã is also optimal to the problem (P).
Proof. We are required to show that x Ã is also optimal in the optimization problem (P). First, we need to show that x Ã is among the feasible points in the problem (P). By contradiction, let us assume that x Ã is not feasible in the problem (P). By weierstrass theorem, f admits it minimum x on a compact set F (since f is a continuous function bounded below on the set F). Therefore, the original optimization problem (P) has an optimal solution x: Based on this hypothesis, the functions f ; g i ; i 2 I x ð Þ; h j ; j 2 J þ x ð Þ are invex at x on X with respect to the same function g and the function h j ; j 2 J À x ð Þ are incave at x on X with respect to the same function g: Therefore, by definitions of invex and incave functions, we have the following: Since x is optimal in the optimization problem (P), there exist Lagrange multipliers n 2 R m þ and l 2 R s such that the KKT necessary optimality conditions (i)-(iii) hold. Then, by (19)-(21), Upon adding the both sides of (35), (39) and (40), we get Observe that x Ã ¼ ð0; 0Þ is feasible in the considered penalized problem and KKT necessary optimality conditions (i)-(iii) are fulfilled at x Ã ¼ ð0; 0Þ with multipliers n Ã 1 ¼ n Ã 2 ¼ 1: We set the penalty parameter c > max n i ; i 2 I; È É ; as stated in Theorem 3. Figures 1-3 represented the considered problem in its original form, absolute value penalty function form and logarithmic penalty function form, respectively. The results of all the three forms indicate that x Ã ¼ ð0; 0Þ is an optimal solution in the considered non-linear programming problem. Furthermore, the stated sufficient conditions for convex optimization problem in Theorem 9.3.1 (Bazaraa et al., 1991) are not applicable, since not all functions constituting the problem (P) are convex but the functions constituting the problem (P) are invex with respect to the same function g on the set X: Therefore, it follows that sufficient conditions in Theorem 3 are essential for the non-convex optimization problem.
Example 2. Consider the following optimization problem: Obviously, the objective function is not an invex function with respect to any function g on R defined by g :  and x Ã ¼ À1 is an optimal solution in the considered problem. Therefore, we use the proposed logarithmic penalty function (15). Then we have the following single unconstrained optimization problem; min P x; c ð Þ ¼ x 3 þ cln max 0; Àx À 1 f g ð Þ 2 þ 1 P x; c ð Þ does not have a minimizer at x Ã ¼ À1 for any values of c > 0 as depicted in Figures 4-6, where c ¼ 10: Therefore, no equivalence exists between the original optimization problem, its exact penalized optimization problem via the absolute value penalty function and its penalized problem with proposed logarithmic penalty function since the downward growth of f exceeds that of the g upward growth at x Ã : Indeed, the invexity assumption is essential to ensure the equivalence between the sets of optimal solutions in the original optimization problem and its transformed proposed unconstrained problem with logarithmic penalty function.

Numerical test
In this section, some small problems were tested with different classifications to justify the proposed logarithmic penalty function efficiency. All the problems were selected from Hock and Schittkowski (1981) collections set of continuous problems with equality and inequality constraints. The following     fminunc routine function in the matlab2018a Their classifications are also included in the last column of the table.

The classification
The classification of each problem provides more detailed information regarding the problem and consists of letters, i.e. OCD. O-information about objective function C-information about constraint function D-information about the regularity of the problem

Results and discussions
Based on the results obtained in the first part, it was observed that the problem solved by Antczak (2009) reiterated that invexity assumptions are essential to establish an equivalence between the sets of optimal solutions in the original optimization problem and its associated penalized problem as depicted in the Figures 1-3 for the original problem, penalized problem via absolute value penalty function and penalized problem via proposed logarithmic penalty function, respectively; x Ã ¼ ð0; 0Þ happens to be the optimal solution for all the three methods. On the other hand, in the subsequent problem the objective is not an invex function on R with respect to any function g; and this yields to different optimal solutions as illustrated in Figures 4-6 for the original constrained problem, penalized problem via exact absolute value penalty function and penalized problem by means of proposed logarithmic penalty function, respectively. However, in the second part, it can be observed that for the problems with classifications QQR, QPR, PLR and PPR, the proposed logarithmic penalty proved to be more efficient compared to the original formulation of the constrained optimization problem and penalized problem with exact absolute value penalty function. Nevertheless, for QPR fminunc stopped because it cannot decrease the objective function along the current search direction, even though its number of iterations is higher than that of GLP, while in the remaining GLP it admits more numbers of iterations compared to exact absolute value penalty function formulation; the same conclusions go for LGI and PGR.
Moreover, in the last two problems with classification GPR, the absolute value penalty function method via the same algorithm, problems HS080 and HS081 turn out to be unbounded while the proposed logarithmic penalty function method was able to solve the problems.

Conclusion
This paper reviewed some of the existing penalty function methods for solving constrained optimization problem and proposed a continuously differentiable penalty function called logarithmic penalty function method; the proposed approach is the hybridization of logarithmic penalty function and modified Courant-Beltrami penalty function for equality and inequality constraints, respectively. The equivalence between the sets of the optimal solutions for the original optimization problem constituted by invex functions with respect to the same function g and its associated penalized problem via the proposed penalty function has been established. According to the results obtained, the concept of invexity is essential to determine the equivalence between the sets of the optimal solution in the original problem and its associated penalized problem.
Few problems with mixed (equality and inequality) constraints from the Hock-Schittkowski collection were solved by means of quasi-Newton algorithm via routine function fminunc (MATLAB2018a) as in Table 1; the objective values obtained for the proposed logarithmic penalty function turn out to be more advantageous to decision makers from a practical point of view, even though there is a need to address some issues regarding the number of iterations (Table 2). Future work will focus on inventing a suitable algorithm for this type of formulation and also to solve practical problems.

Disclosure statement
No potential conflict of interest was reported by the authors.