• Executions in pending state, even though there are sufficient available Celery workers to handle them
• Executions stuck in cancelling or force_cancelling state

Stuck executions can prevent the deleting or modifying a deployment. Even worse, stuck system workflow can bring all workflows on a Cloudify Manager grinding to a halt.
To recover from that, the trick is to update the execution status manually to a “completed” status (such as terminated, cancelled or failed).

Method 1: Using PostgreSQL

• Log in to PostgreSQL as follows:
  
  • Log into Cloudify Manager with SSH.
    • If you work in an HA environment, SSH into the active manager.

• If you have sudo access, execute the following command:

  sudo -u postgres psql cloudify_db

• If you don’t have sudo access, execute the following command:

  psql -U cloudify -W -d cloudify_db -h localhost

• Query the database to ensure that you know what you’re doing:

  select status from executions where id='execution-id';

  (Replace execution-id with the bad execution ID. Keep the apostrophes.)

• Update the execution:

  update executions set status='cancelled' where id='execution-id';

  That will update the execution to have a cancelled status.

Method 2: Using the REST API

You can send an HTTP PATCH request as follows:

curl --request PATCH -H "Content-Type: application/json" -H "Tenant: <tenant-name>" -u <username>:<password> -d '{"status": "cancelled"}' https://<manager-ip>/api/v3.1/executions/<execution-id>

curl --request PATCH -H "Content-Type: application/json" -d '{"status": "cancelled"} 'https://<manager-ip>/api/v2.1/executions/<execution-id>

• Replace username and password with a username and password who can be authenticated against the REST service.
• Replace manager-ip with the IP / hostname of the manager.
• Replace https with http if your REST service is not SSL-protected (which should never really happen).
• Replace execution-id with the execution’s ID.
• Replace tenant-name with the name of the tenant to which the execution belongs.