☑️ Ensure Workflow retry policy catches relevant errors only
retryStrategy is an optional field of the Workflow CRD that provides control over how to retry a workflow step. One of its fields is
retryPolicy, which defines the policy of NodePhase statuses that will be retried (NodePhase is the condition of a node at the current time).
The value of
retryPolicy can be either: Always, OnError, or OnTransientError. In addition, the user can use the expression key to control more of the retries.
What’s the catch?
retryPolicy=Always is too much. The user only wants to retry on system-level errors (eg, the node dying or being preempted), but not on errors occurring in user-level code since these failures indicate a bug. In addition, this option is more suitable for long-running containers than workflows which are jobs.
retryPolicy=OnError doesn't handle preemptions: retryPolicy=OnError handles some system-level errors like the node disappearing or the pod being deleted. However, during graceful Pod termination, the kubelet assigns a Failed status and a Shutdown reason to the terminated Pods. As a result, node preemptions result in node status "Failure", not "Error" so preemptions aren't retried.
retryPolicy=OnError doesn't handle transient errors: Classify preemption failure message as a transient error is allowed however, this requires retryPolicy=OnTransientError. (see also, TRANSIENT_ERROR_PATTERN).
It is recommended to use Always, along with this expression that filters out unnecessary errors:
lastRetry.status == "Error" or (lastRetry.status == "Failed" and asInt(lastRetry.exitCode) not in )
Targeted objects by this rule (types of kind): Workflow
Complexity: medium (What does this mean?)
Policy as code identifier: ARGO_WORKFLOW_ENSURE_RETRY_ON_BOTH_ERROR_AND_TRANSIENT_ERROR
This rule will fail
If the retryPolicy key is set with a value of Always, but the aforementioned expression is not set:
Rule output in the CLI
$ datree test *.yaml
>> File: failExample.yaml
❌ Ensure Workflow retry policy catches relevant errors only [1 occurrence]
💡 Incorrect value for key `retryPolicy` - the expression should include retry on steps that failed either on transient or Argo controller errors
How to fix this failure
When using a
Always, set the
expression key with the following value:
expression: 'lastRetry.status == "Error" or (lastRetry.status == "Failed" and asInt(lastRetry.exitCode) not in )'