Incorrect use of log with system logs is characterized by absence of the latest records about events in the System Logs section and can be caused by internal Redis or Celery failure.
To restore log with system logs:
Restart mobile platform components:
When using standalone configuration of Foresight Mobile Platform, restart the fmp_celeryworker_1, fmp_redis_1 containers.
When using cluster configuration of Foresight Mobile Platform, restart the fmp-celeryworker, fmp-redis pods.
Check the volume of allocated system hard disk resources for Elasticsearch:
When using standalone configuration of Foresight Mobile Platform, check volume of allocated disk system resources for the fmp_elasticsearch_1 container. The fmp_elasticsearch_1 container is part of the event logging system on mobile platform server.
When using cluster configuration of Foresight Mobile Platform, check volume of allocated disk system resources for the fmp-elasticsearch-data pod.
For example, for the cluster deployed on OKD/OCP open the Storage > Persistent Volume Claims subsection in the OKD application:
The Capacity column contains the volume of allocated system resources, the Used column contains the volume of used system resources.
If required, add more system resources on mobile platform server or in cluster.
To add more system resources for the fmp_elasticsearch_1 container on mobile platform server, use operating system tools or virtualization tools. Check health of the fmp_elasticsearch_1 container. To do this, restart Docker containers:
% docker-compose -f docker-compose.standalone.yml up
If after executing the command the "elastic is down" message is displayed, the fmp_elasticsearch_1 container was not started. To start the fmp_elasticsearch_1 container and restore log with system logs, clear container volume.
IMPORTANT. When the fmp_elasticsearch_1 container volume is being cleared, mobile platform server is unavailable. It is recommended to clear container volume when system load is minimal. After the volume is cleared, all event records are cleared in the System Logs subsection.
To add more system resources in cluster, see the Adding System Resources in Cluster subsection.
After executing the operations, the log with system logs is restored.
To clear volume of the fmp_elasticsearch_1 container on mobile platform server:
Delete the running Docker components with data saving:
% docker-compose -f docker-compose.standalone.yml down
Delete the fmp_es_data volume:
% docker volume rm fmp_es_data
Restart the containers:
% docker-compose -f docker-compose.standalone.yml up -d
After executing the operations the volume of the fmp_elasticsearch_1 container is cleared.
To add more system resources PV of the fmp-elasticsearch-data pod in the OKD/OCP cluster:
IMPORTANT. Select the time with minimum cluster user load because adding the volume of system resources blocks execution of background tasks. If required, export logs to file. After executing the operations with the pod, system logs will be deleted.
Reduce the minimum number of replicas in Horizontal Pod Autoscalers (HPA) for the fmp-celeryworker-hpa pod. To do this, open the HPA configuration for edit and set the minReplicas parameter to 0:
kubectl edit hpa fmp-celeryworker-hpa -n fmp
NOTE. It is relevant for Foresight Mobile Platform 23.12 or later.
Stop writing of logs in Elasticsearch. To do this, decrease the number of fmp-celeryworker pods to zero.
Decrease the number of fmp-elasticsearch-data pods to zero.
Determine which PV are used by fmp-elasticsearch-data pods and delete Persistent Volume Claim (PVC) for them.
Increase the volume of released PV to 30G.
Write storageClass named es-data for increased PV.
Write storageClass named es-data for fmp-elasticsearch-data pods:
Change the yaml file using the OKD application (to apply changes in the current release) according to the structure:
spec:
volumeClaimTemplates:
spec:
storageClassName: es-data
Change the values.production.yaml file (to apply changes in the following updates/release reinstallations) according to the structure:
elasticsearch:
data:
persistence:
storageClass: es-data
Increase the number of fmp-elasticsearch-data pods to 3 and make sure that all pods are started and successfully create new PVC based on the updated PV.
Prepare Elasticsearch. To do this, go to the terminal of one of the fmp-dashboard pods and start the init.sh script from any place.
Resume writing of logs in Elasticsearch. To do this, restore the number of fmp-celeryworker pods to 1.
Restore the minimum number of replicas in HPA for the fmp-celeryworker-hpa pod as in the "a" step. To do this, open the HPA configuration for edit and set the minReplicas parameter to 1.
NOTE. It is relevant for Foresight Mobile Platform 23.12 or later.
After executing the operations the volume of PV system resources use of the fmp-elasticsearch-data pod is added in the OKD/OCP cluster.
See also: