Fix problem of OSD down when performing DOR in both controllers

When there is no active controller during host startup, the variable
$service_enabled in ceph.pp assumes its default value of False, which
causes Puppet to skip the section responsible for mounting the OSD.

This scenario happens, for example, when a DOR is performed in both
controllers in AIO-DX simultaneously.

In this scenario, the OSD mount is expected to be triggered by the OSD
start in /etc/init.d/ceph and not by puppet. However, since the OSD
mountpoint is not available, /etc/init.d/ceph cannot find the
.osd_configured file, and the OSD start is skipped. As a result, the OSD
remains unmounted, leading to the Ceph OSD being down.

To fix this, /etc/init.d/ceph now checks for the .osd_configured
flag only if the OSD is already mounted. This allows the script to
proceed and mount the OSD as expected.

Additionally, during OSD startup in /etc/init.d/ceph, before mounting,
the /dev/sdx1 device for a OSD, the PARTLABEL is verified. If it does
not match "ceph data", it indicates the OSD has not yet been created by
Puppet, and the start process is skipped to prevent the OSD from being
started prematurely which could lead to errors during Puppet OSD
configuration.
During a restore in progress, the OSD is already mounted but not yet
defined in ceph.conf. In this state, $first_dev cannot be resolved,
which would cause the PARTLABEL check to fail. Since the restore process
must start the OSD, the PARTLABEL check is skipped while a restore is in
progress.

Test-Plan:
  PASS: build-pkgs and build-image
  PASS: [SX, DX, STD] Fresh install with ceph bare metal
  PASS: [SX, DX, STD] DOR tests ensuring the OSD is mounted/started
    - Resetting all hosts at the same time
    - Resetting one host at a time
  PASS: [SX, DX, STD] host-swact
  PASS: [SX, DX, STD] lock-unlock standby controller
  PASS: [SX, DX, STD] Add OSDs at runtime
  PASS: Make /etc/init.d/ceph be called for new OSD before puppet
        configuration and observe the OSD being skipped in ceph
        script
  PASS: Make /etc/init.d/ceph be called for a already configured
        OSD before puppet configuration and observe the OSD is not
        affected
  PASS: Make /etc/init.d/ceph be called for a new OSD
        during puppet configuration and observe the OSD being skipped
        in ceph script
  PASS: Make /etc/init.d/ceph be called right before final puppet OSD
        mount and observe the OSD being skipped in ceph script
  PASS: Confirm PARTLABEL check works for SATA, NVMe and multipath
  PASS: [SX, DX, STD] Perform BnR
  PASS: [DX] Perform Platform Upgrade to the built image

Closes-Bug: 2127198
Related-Bug: 2125764
Change-Id: I81e466c70027a88814ca97d6e67a24b72baef531
Signed-off-by: Ítalo Vieira <italo.gomesvieira@windriver.com>

ceph/ceph/debian/deb_folder/ceph-base.ceph.init

@@ -746,11 +746,19 @@ daemon_is_running() {
         exit 1  # pid is something else" "" "okfail"
 }
 
+osd_is_mounted() {
+    findmnt -no SOURCE "/var/lib/ceph/osd/ceph-$1" > /dev/null 2>&1
+}
+
 osd_is_configured() {
     # Flag .osd_configured is created once the OSD is configured by puppet
     [ -f "/var/lib/ceph/osd/ceph-$1/.osd_configured" ]
 }
 
+is_restore_in_progress() {
+  [ -f /etc/platform/.restore_in_progress ]
+}
+
 stop_daemon() {
     name=$1
     daemon=$2
@@ -984,7 +992,9 @@ for name in $what; do
 
     get_conf pid_file "$run_dir/$type.$id.pid" "pid file"
 
-    if [ "$type" = "osd" ] && ! osd_is_configured "$id"; then
+    # Skip unconfigured OSDs only if they are already mounted.
+    # If not mounted, let this script proceed so it can handle the mount.
+    if [ "$type" = "osd" ] && osd_is_mounted "$id" && ! osd_is_configured "$id"; then
         log "$name" "INFO" "Skipping Ceph $name as it is not configured"
         continue
     fi
@@ -1052,6 +1062,16 @@ for name in $what; do
 
     case "$command" in
 	start|onestart)
+            # Check if the OSD device has the expected "ceph data" partition label.
+            # If not, Puppet hasn’t finished creating it, so skip startup to avoid errors.
+            if [ "$type" = "osd" ] && ! is_restore_in_progress; then
+              partlabel=$(lsblk -no PARTLABEL "$first_dev")
+              if [ "$partlabel" != "ceph data" ]; then
+                log "$name" "INFO" "Skipping Ceph $name as it is not created"
+                continue
+              fi
+            fi
+
 	    # Increase max_open_files, if the configuration calls for it.
 	    get_conf max_open_files "32768" "max open files"
 	    # Remove stale admin socket