Agfdhyk

Over the past week I've been seeing the number of instances on my GAE Flexible Environment fall to 0, with no new instance spinning up. My understanding of the Flexible environment is that this shouldn't be possible... (https://cloud.google.com/appengine/docs/the-appengine-environments)

I was wondering if anyone else has been seeing these issues, or if they've solved the problem on their end before. My one hypothesis is that this might be an issue with my health monitoring endpoints, but haven't seen anything that jumps out as a problem when I review the code.

This hasn't been a problem for me until last week, and now it seems like I have to redeploy my environment (with no changes) every couple of days just to "reset" the instances. It's worth noting that I have two services under this same App Engine project, both running flexible versions. But I only seem to have this issue with one of the services (what I call the worker service).

Screenshot from App Engine UI:

Screenshot from Logs UI that shows the SIGTERM being sent:

PS - Could this have anything to do with the recent Google Compute issues that have been coming up... https://news.ycombinator.com/item?id=18436187

Edit: Adding the yaml file for "worker" service. Note that I'm using Honcho to add an endpoint to monitor health of the worker service via Flask. I added those code examples as well.

yaml File

service: worker

runtime: python

threadsafe: yes

env: flex

entrypoint: honcho start -f /app/procfile worker monitor



runtime_config:

  python_version: 3



resources:

  cpu: 1

  memory_gb: 4

  disk_size_gb: 10



automatic_scaling:

  min_num_instances: 1

  max_num_instances: 20

  cool_down_period_sec: 120

  cpu_utilization:

    target_utilization: 0.7

Procfile for Honcho

default: gunicorn -b :$PORT main:app

worker: python tasks.py

monitor: python monitor.py /tmp/psq.pid

monitor.py

import os

import sys



from flask import Flask





# The app checks this file for the PID of the process to monitor.

PID_FILE = None





# Create app to handle health checks and monitor the queue worker. This will

# run alongside the worker, see procfile.

monitor_app = Flask(__name__)





@monitor_app.route('/_ah/health')

def health():

    """

    The health check reads the PID file created by tasks.py main and checks the proc

    filesystem to see if the worker is running.

    """

    if not os.path.exists(PID_FILE):

        return 'Worker pid not found', 503



    with open(PID_FILE, 'r') as pidfile:

        pid = pidfile.read()



    if not os.path.exists('/proc/{}'.format(pid)):

        return 'Worker not running', 503



    return 'healthy', 200





@monitor_app.route('/')

def index():

    return health()





if __name__ == '__main__':

    PID_FILE = sys.argv[1]

    monitor_app.run('0.0.0.0', 8080)