Incorrect use of log with system logs is characterized by absence of the latest records about events in the System Logs section.
To restore log with system logs:
Restart mobile platform components on the server or in the cluster if a fault-tolerant cluster based on OKD/OCP is used:
The hyperhive_celeryworker_1, hyperhive_redis_1 containers on mobile platform server.
The fmp-celeryworker, fmp-redis pods in the cluster.
Check the volume of allocated system hard disk resources for Elasticsearch:
The hyperhive_elasticsearch_1 container on mobile platform server.
Persistent Volumes (PV) of the fmp-elasticsearch-data pod in the cluster. To do this, in the OKD application open the Storage > Persistent Volume Claims subsection:
The Capacity column contains the volume of allocated system resources, the Used column contains the volume of used system resources.
If required, add more system resources for the hyperhive_elasticsearch_1 container on mobile platform server using operating system tools or using virtualization.
To add more system resources PV of the fmp-elasticsearch-data pod in the cluster:
IMPORTANT. Select the time with minimum cluster user load because adding the volume of system resources blocks execution of background tasks. If required, export logs to file. After executing the operations with the pod, system logs will be deleted.
Stop writing of logs in Elasticsearch. To do this, decrease the number of fmp-celeryworker pods to zero.
Decrease the number of fmp-elasticsearch-data pods to zero.
Determine which PV are used by fmp-elasticsearch-data pods and delete Persistent Volume Claim (PVC) for them.
Increase the volume of released PV to 30G.
Write storageClass named es-data for increased PV.
Write storageClass named es-data for fmp-elasticsearch-data pods:
Change the yaml file using the OKD application (to apply changes in the current release) according to the structure:
spec:
volumeClaimTemplates:
spec:
storageClassName: es-data
Change the values.production.yaml file (to apply changes in the following updates/release reinstallations) according to the structure:
elasticsearch:
data:
persistence:
storageClass: es-data
Increase the number of fmp-elasticsearch-data pods to 3 and make sure that all pods are started and successfully create new PVC based on the updated PV.
Prepare Elasticsearch. To do this, go to the terminal of one of the fmp-dashboard pods and start the init.sh script from any place.
Resume writing of logs in Elasticsearch. To do this, restore the number of fmp-celeryworker pods to 1.
After executing the operations, the log with system logs is restored.
See. also: