Restoring Database Replica Availability

A database replica unavailability is characterized with:

Unavailability of response during data read request execution - the 500 error is returned.
Increase in response time during data read request execution for several database replicas.
State of database pod:

Running, but not all containers are running.
Error or Pending.

Absence of database response may be caused by desynchronization of database replicas.

To restore database replica availability, check pod work on each cluster node. Restart the pod if it works incorrectly. If after restarting the pod the incorrect behavior remains, recreate database replicas.

Recreating Database Replicas

To recreate database replicas:

Authenticate in the OKD cluster:

oc login -u your_username -p your_password your_host

Get pod name and specify port for access to the PostgreSQL DBMS administration panel:

POD_NAME=$(kubectl get pods --namespace stackgres -l "app=stackgres-restapi" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward "$POD_NAME" 8443:9443 --namespace stackgres

Get credentials to enter to the PostgreSQL DBMS administration panel:

kubectl get secret -n stackgres stackgres-restapi --template '{{ printf "username = %s\npassword = %s\n" (.data.k8sUsername | base64decode) ( .data.clearPassword | base64decode) }}'

Specify URL of the PostgreSQL DBMS administration panel in the browser field:

https://<IP address or DNS server name>:8443/

Enter the credentials obtained at Step 3 to the Username and Password boxes and log in to the system.
Select SGCluster in the fmp namespace:

Select the required database in the StackGres Cluster column:

Click the EDIT CLUSTER button:

Set the 1 value in the Number of Instances box and click the UPDATE CLUSTER button:

NOTE. If several replicas are used for data reading (pods with the Replica role) and one of them works correctly, data read requests will be handled successfully. In this case response time can be increased if database loading is increased. When Number of Instances is decreased to one, data read requests will stop being handled until database replicas are recreated.

Make sure that the list of pods contains one pod with the Primary role. The pod with the Replica role should be deleted.
Click the EDIT CLUSTER button and set the value equal to the original number of replicas in the Number of Instances box.
Make sure that the list of pods contains a pod with the Replica role:

NOTE. The replicas synchronization time depends on the volume of data and the speed of connection between cluster nodes.

After executing the operations, database replicas are recreated and synchronized.

To reconnect the fmp-socketserver pod to the database in the OKD application:

Open the Workloads > Deployments subsection.
Select fmp-socketserver in the mobile platform project, for example, fmp:

Decrease the number of pods to zero in the Deployment Detailsm section and then increase the number to the previous value to recreate instance of the fmp-socketserver pod:

After executing the operations the fmp-socketserver pod is reconnected to the database.