How to check whether a loss function is convex or not

In optimization problems, determining whether a loss function is convex is crucial. Convexity simplifies the search for optimal solutions because it ensures the presence of a unique global minimum. (A “minimum” refers to the lowest value of a function within a specified domain or set of possible inputs. The “global minimum” is the lowest value of the function across its entire domain, meaning it’s the smallest value that the function can attain for any input.)

What is convexity?

A function $f(x)$ is convex if, for any two points $x_{1}$ and $x_{2}$ in its domain, and for any $λ$ in the range [0, 1], the following inequality holds:

$f(λx_1 + (1 - λ)x_2) ≤ λf(x_1) + (1 - λ)f(x_2)$

In short, the line segment linking any two locations on a function’s graph must be above the graph to be convex.

We can determine the convexity using the following techniques:

1. Second derivative test

We can determine convexity using the second derivative if our function is twice-differentiable:

If a function’s second derivative is non-negative for every $x$ in its domain, it’s considered convex.
If the second derivative of a function $f(x)$ is non-positive for all $x$ in the domain, the function is concave.

Note: The function is neither convex nor concave if the second derivative has a variable sign, i.e., if it’s occasionally positive and occasionally negative.

2. Jensen’s inequality

Jensen’s inequality is a useful tool for determining convexity. It specifically aids in determining whether a function is convex by looking at how it interacts with expectations (averages) and the convexity of its subcomponents.

In relation to convexity and loss functions:

A loss function $L(θ)$ is convex if it satisfies Jensen’s inequality, meaning that, if it can be expressed as an expectation (average) of some value $g(θ)$ , and $g(θ)$ is a convex function, then $L(θ)$ is convex.

In short, if $L(θ)=E[g(θ)]$ and $g(θ)$ is convex, then $L(θ)$ is convex.

To elaborate on Jensen’s inequality further:

If $f(x)$ is convex and $g$ is any arbitrary function, then the following is true for every random variable $X$ :

$f(E[X])≤E[f(X)]$

This inequality ensures that the expected value of a convex function applied to a random variable is always greater than or equal to the convex function applied to the expected value of that random variable. It’s a powerful concept often used in the context of convex analysis and optimization.

3. Graphical analysis

Our loss function’s graph can be plotted to reveal some graphic insights. The function is probably convex if its curve is consistently above its tangent lines.

Remember: This isn’t a strict test, particularly for complicated functions.

4. Numerical tests

Calculating $f(λx_1 +(1−λ)x_2)$ and $λf(x_1)+(1−λ)f(x_2)$ for various values of $λ$ , $x_1$ , and $x_2$ allows us to check the convexity of our loss function at various points. Convexity is suggested if the inequality holds for these values.

Coding example

Here’s a simple Python code example that will show us how to use the second derivative test to determine whether a function is convex:

import numpy as np
import matplotlib.pyplot as plt
def loss_function(x):
  return x**2  # Replace this with your own loss function
def is_convex(func, interval):
  x = np.linspace(interval[0], interval[1], 100)
  second_derivative = np.gradient(np.gradient(func(x), x), x)
  if np.all(second_derivative >= 0):
    return True
  elif np.all(second_derivative <= 0):
    return False
  else:
    return None
interval = (-6, 6)  # Define the interval where you want to check convexity
result = is_convex(loss_function, interval)
if result is None:
  print("The function is neither convex nor concave.")
elif result:
  print("The function is convex.")
else:
  print("The function is not convex.")
# Plot the function and its second derivative
x = np.linspace(interval[0], interval[1], 100)
plt.plot(x, loss_function(x), label="Loss Function")
plt.plot(x, np.gradient(np.gradient(loss_function(x), x), x), label="Second Derivative")
plt.legend()
plt.xlabel("x")
plt.ylabel("y")
plt.title("Convexity Check")
plt.grid(True)
plt.savefig('output/output.png')
plt.show()

Explanation

Lines 1–2: Import the NumPy and Matplotlib libraries for numerical operations and plotting.
Lines 4–5: Define a placeholder loss function (e.g., $f(x) = x^2$ ).
Line 8: Generate 100 x-coordinates within the interval (-6, 6).
Line 9: Calculate the second derivative of the loss function at each x-coordinate.
Lines 11–16: If the second derivative is non-negative at all points, the function is convex. If it’s non-positive at all points, the function is concave. Otherwise, it’s neither convex nor concave.
Line 18: Define the interval for convexity checking.
Line 19: Check if the loss function is convex in the defined interval.
Lines 21–26: Print whether the function is convex, concave, or neither.
Lines 29–38: Plot the function and its second derivative using Matplotlib, and display the plot.

Conclusion

Keep in mind that verifying convexity can be challenging, particularly for complex functions. Convexity can be taken for granted in many practical situations depending on the nature of the issue or specialized knowledge.

Learn in-demand tech skills in half the time

PRODUCTS

Mock Interview

New

Courses

Skill Paths

Projects

Assessments