Details
-
Type: Task
-
Status: Open
-
Priority: Major
-
Resolution: Unresolved
-
Affects Version/s: Systems & Operations
-
Fix Version/s: Systems & Operations
-
Component/s: None
-
Labels:None
-
Environment:
2.0.2 and up.
Description
This is in response to AML-1146
It has been observed that the update order status service (currently running every 60 seconds) on Tomcat can freeze and stop checking for orders to update status on. To the end user, it will look like all of their orders are stuck in the submitted state. To someone checking the services, the update order service has stopped polling, in fact, logging seems to cease. Restarting the tomcat service, processes all order statuses in the queue and continues on as normal.
We need a process to detect this condition and alert aml-ops that the system needs to be restarted. There may be a better way to handle this. Worst case, we need to let someone know there is an issue. Best case is that we are alerted to an issue and the system fixes itself.