Edit

Troubleshoot health check failures for Azure Local

Applies to: Hyperconverged deployments of Azure Local

This article describes how to troubleshoot and fix health check problems that your Azure Local instance reports.

Health and readiness checks

Health and readiness checks are essential to ensure lifecycle operations such as deployment and update go smoothly. They also maintain correct system functionality and ensure high availability for your workloads. Many different scenarios perform and report health and readiness checks separately:

  • Before deployment:

  • Scheduled daily checks:

    • System health checks run once every 24 hours to provide regular updates on health issues. Address these issues to maintain correct system functionality and ensure high availability for your workloads.
  • Before update:

    • Update readiness checks run as part of the update preparation phase. This phase runs after downloading the update content and before beginning the installation.
  • Before solution upgrade:

  • Before add or repair node:

    • A health check of the new node is performed as one of the first steps of the add node or repair node process. This check ensures the new node is healthy and a homogeneous match for the existing servers in the cluster. When you're monitoring add or repair progress, evaluate failures during the EnvironmentValidatorScaleOut steps similar to other health check failures discussed in this article.

The Azure Local health check framework runs various tests, sometimes referred to as validators or health checks. It runs tests that are appropriate for the lifecycle stage of your system. These checks confirm the system is healthy, has appropriate connectivity, and is ready to move on to the next step (for example, deploy, update, and add or repair node). The daily system health check confirms the system is healthy and ready for production workloads.

Differences between preupdate and other health checks

The results of the daily system health checks and preupdate readiness checks might differ. Both system and preupdate readiness checks perform similar validations. However, preupdate checks use the latest version of the tests from the update that you’re preparing to install. Due to improvements in the new update, a health check that is currently failing in the daily system check might pass in the preupdate check. Because of this difference, run the update preparation before your planned update to ensure your system is ready for the update. For more information, see:

Health check severity levels

Regardless of which type of health or readiness check you’re performing, Azure Local categorizes all validation results using the same severity levels: Critical, Warning, and Informational.

  • Critical indicates a serious health concern and indicates a problem that you should address to avoid impacting the system health. If you’re attempting to perform a lifecycle operation such as deploy, update, add, repair, or upgrade, this severity indicates problems that you must resolve before proceeding.

  • Warning indicates potential problems that you should evaluate. Warning severity failures might also prevent you from performing a lifecycle operation.

  • Informational indicates health check results for your awareness and might not require immediate action. Informational severity health check failures don’t block lifecycle operations. This severity provides information about the system’s state and any potential problems that shouldn’t affect the operation of the system directly.

Bypass warnings

  1. The Warning status indicates potential problems that might not be severe, but that you should review to confirm they won't impact the update before you bypass them.

    Screenshot of Update Progress page showing warnings summary.

  2. To investigate warnings, select Validate that the system is ready.

    Screenshot of Update Progress page showing an expanded warning.

  3. Select View details for the warning in question.

    Screenshot of Update Progress page showing warning details.

  4. If you determine the warning doesn't impact the update, you can bypass the warning, either by using PowerShell, or through the portal:

    Screenshot of Proceed anyway prompt.

Viewing health check or readiness results

The method to view the health check results differs depending on which scenario the tests are running.

Health checks when planning for deployment or upgrade

Use the AzStackHci.EnvironmentChecker module to assess environment readiness when planning to deploy a new cluster or upgrade an existing cluster with Azure Local. This standalone module is regularly updated to perform many of the same health check validation tests that it regularly performs before and after Azure Local deployments.

This module uses a simple text interface to view the results prior to completion of the Azure Local deployment or upgrade. While the results format might look different from other (deploy and update) health checks, the result properties for each test match the equivalent health checks tests of the same name that are performed in later lifecycle stages. As a result, you can use the Resources to troubleshoot failed results in this article to identify a known issue. However, when you're evaluating readiness to upgrade, you should first see Validate solution upgrade readiness for remediation guidance.

Health checks just before deployment

The final step before deployment starts is to perform a validation as discussed in Validate and deploy the system. This step performs validations on all systems that you selected for deployment. Compared to the earlier assessment of environment readiness, this step might uncover health check failures that didn't appear earlier, such as problems with the selected servers that don't match each other, or problems with the proposed Active Directory, networking, or Solution Builder Extension (SBE) configuration. This difference exists because the Azure Local health check framework knows more about your configuration at this stage.

You can view these predeployment validation results from the portal as described in Validate and deploy the system. For more information about troubleshooting deployment validation issues, see Troubleshoot deployment validation issues.

Health checks after deployment

How you view health checks after deployment depends on whether you're using the Azure portal, Azure PowerShell with Azure Local release 2610 and later, or PowerShell with Azure Local release 2509 and earlier.

Daily system health checks

You can view the most recent daily system health check result in Azure Update Manager:

  1. In the system list, view the Critical state of Update readiness.

    Screenshot of Update Manager page.

  2. Select one or more systems from the list, and then select One-time Update.

  3. On the Check readiness page, review the list of readiness checks and their results.

    1. Select the View details links under Affected systems.

    2. When the details box opens, you can view more details, individual system results, and the Remediation for failing health checks.

    Screenshot of Install updates page.

    To resolve the failures, follow the remediation instructions.

    Note

    The system health checks run every 24 hours, so it might take up to 24 hours for the new results to sync to the Azure portal after remediating the failures. To initiate a new system health check immediately or further troubleshoot, see the PowerShell pivot for this section.

You can also view daily system health check results by selecting multiple clusters in the Azure Update Manager and selecting Install now:

Screenshot of selecting multiple clusters in Azure Update Manager.

For example, suppose you update three clusters. For two clusters, the daily system health checks succeeded; for one cluster, they failed.

Screenshot of Install updates failure details pane.

System daily health check provide a daily snapshot of the health of your clusters, but are not required to be successful and should not deter you from attempting to prepare an update if you believe the issue reported has been recently addressed. As part of preparing to install the update or staring the update, the preupdate health check will run the latest validation tests to determine if it is appropriate to install the update.

In the Install updates page, the Next button is enabled even if there are failed system daily health check results:

Screenshot of Install updates page when there are daily system health check failures.

If you select Next, you're prompted to continue without resolving the failed system daily health checks:

Screenshot of Proceed anyway prompt.

Preupdate health checks

To view preupdate health check results, view the Check readiness details when installing and tracking system updates in Azure Update Manager:

  1. In History, select the failed update run from the list.

  2. On the Check readiness page, review the list of readiness checks and their results.

    1. Select the View details links under Affected systems.

    2. When the details box opens, you can view more details, individual system results, and the Remediation for failing health checks.

    Screenshot of Update progress page.

    Follow the remediation instructions to resolve the failures. Then select Try again to retry the preupdate readiness checks and Resume the update.

    To further troubleshoot, see the Powershell pivot for this section.

Note

For Azure Local release 2610 and later, we recommend using the Get-SolutionHealthSummary command to view system health check results. For previous releases, use the Get-SolutionUpdateEnvironment or Get-SolutionUpdate commands.

Azure Local release 2610 and later

To view system or preupdate health check results through PowerShell, complete the following steps.

  1. To validate that the health checks failed, run the following command on one of the machines in your system:

    Get-SolutionHealthSummary
    

    Here's a sample output for a failed daily system health check:

    Status  : FAILED
    Info    : {
                  HealthCheckSource     : Daily system health check for 12.2610.1004.37
                  HealthCheckDate       : 10/13/2026 02:11:43
                  Time since last check : 1.2h ago
                  Statistics            : 974 total checks: 3 critical, 1 warning, 4 informational, 966 passed
              }
    Results : {
                  Result       : Result 1
                  Severity     : Critical
                  DisplayName  : Storage Job Health Check
                  Title        : Storage Job Status Check
                  Description  : Check to verify the Storage Job Health[FAILURE] Storage job [UserStorage_3-Repair] is in
                                 [Suspended] state.
                  Remediation  : Troubleshoot Storage Spaces and Storage Spaces Direct health and operational states:
                  https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-states.
                                 Troubleshoot Storage Spaces Direct:
                  https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/troubleshooting-storage-spaces
                  ...
              }
    
    Most Recent Check Timestamp: 10/13/2026 02:11:43 UTC (1.2h ago)
    FAILED
    974 total checks: 3 critical, 1 warning, 4 informational, 966 passed
    
  2. Review the Status on your system and view the FAILED or WARNING value.

  3. Review the HealthCheckSource field to see whether it contains Daily system health check or Pre-update health check. In either case, when a health check fails, the default Get-SolutionHealthSummary syntax shows the details of the blocking failures.

  4. You can filter the results as follows.

    • To filter for critical failures, add the -CriticalFailures flag:
      Get-SolutionHealthSummary -CriticalFailures
      
    • To filter for warnings, add the -WarningFailures flag:
      Get-SolutionHealthSummary -WarningFailures
      
    • To filter for informational messages, add the -IncludeInformational flag:
      Get-SolutionHealthSummary -IncludeInformational
      
  5. Review the Remediation property for each of the failed tests and take action as appropriate to resolve the failures.

  6. If you require more diagnostic information to determine the cause of the failed tests, add the -Detailed flag:

    Get-SolutionHealthSummary -Detailed
    

    Here's a sample output for a failed daily system health check:

    Status  : FAILED
    Info    : {
                  HealthCheckSource     : Daily system health check for 12.2610.1004.37
                  HealthCheckDate       : 10/13/2026 02:11:43
                  Time since last check : 1.3h ago
                  Statistics            : 974 total checks: 3 critical, 1 warning, 4 informational, 966 passed
              }
    Results : {
                  Result       : Result 1
                  Severity     : Critical
                  DisplayName  : Storage Services Physical Disks Health Check
                  Title        : Storage Services Physical Disks Check
                  Description  : Check to verify the Storage Services Physical Disk Health[FAILURE] The disk
                                 {1}\\example-cluster\root/Microsoft/Windows/Storage/Providers_v2\SPACES_PhysicalDisk.ObjectId=
                                 "{00000000-0000-0000-0000-000000000000}:PD:{00000000-0000-0000-0000-000000000000}"
                                 on node Node19 is in Unhealthy/Transient Error.
                  Remediation  : Troubleshoot Storage Spaces and Storage Spaces Direct health and operational states:
                  https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-states.
                                 Troubleshoot Storage Spaces Direct:
                  https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/troubleshooting-storage-spaces
                  Name         : Storage Services Physical Disks Summary
                  Status       : Failed
                  TargetResourceID   : Cluster/example-cluster/Disk/eui.00000000000000000000000000000000
                  TargetResourceName : eui.00000000000000000000000000000000
                  TargetResourceType : Disk
                  Timestamp    : 10/13/2026 05:20:23
                  HealthCheckSource  : Test-EnvironmentReadiness
                  Tags         : {[Member: Physical Disks], [Group: StorageSvcsSummary]}
                  AdditionalData     : {
                      ScaleUnit: example-cluster,
                      CanPool: False,
                      SBLDiskState: 6,
                      PercentageSpaceUsed: 4.63065227232033,
                      BusType: NVMe,
                      Model: MO001600KXNZV,
                      WriteLatencyMax: ,
                      Firmware: HPS7,
                      CannotPoolReason: In a Pool,
                      Failures: The disk
                                {1}\\example-cluster\root/Microsoft/Windows/Storage/Providers_v2\SPACES_PhysicalDisk.ObjectId="
                                {00000000-0000-0000-0000-000000000000}:PD:{00000000-0000-0000-0000-000000000000}"
                                on node Node19 is in Unhealthy/Transient Error.,
                      OperationalStatus: Transient Error,
                      StorageEnclosure: ,
                      PhysicalLocation: PCI Slot 11 : Bus 71 : Device 0 : Function 0 : Adapter 2,
                      SerialNumber: 0000_0000_0000_0000_0000_0000_0000_0000.,
                      ReadErrorsCorrected: ,
                      SizeInGB: 1490.42,
                      ReadErrorsUncorrected: ,
                      FriendlyName: MO001600KXNZV,
                      WriteErrorsUncorrected: ,
                      MediaType: SSD,
                      Temperature: 23,
                      Usage: Auto-Select,
                      Host: Node19,
                      WriteErrorsCorrected: ,
                      HealthStatus: Unhealthy,
                      ObjectId: {1}\\example-cluster\root/Microsoft/Windows/Storage/Providers_v2\SPACES_PhysicalDisk.ObjectId="
                      {00000000-0000-0000-0000-000000000000}:PD:{00000000-0000-0000-0000-000000000000}",
                      UniqueId: eui.00000000000000000000000000000000,
                      ReadLatencyMax: 0,
                      SBLHealthCounters: ,
                      Manufacturer: ,
                      AllocatedSizeInGB: 69.02
              ...
              }
    
    Most Recent Check Timestamp: 10/13/2026 02:11:43 UTC (1.3h ago)
    FAILED
    974 total checks: 3 critical, 1 warning, 4 informational, 966 passed
    
  7. If present, use the diagnostic information shown in AdditionalData property, such as FailedMachines, Source, and ExceptionMessage to determine which physical machines are causing the test failure, together with information provided in the link from the Remediation property to resolve the failures.

  8. After resolving the failures, run the health checks again.

    • For system health checks, run the following command.

      Invoke-SolutionUpdatePrecheck -SystemHealth
      
    • For preupdate health checks, run the following command.

      Get-SolutionUpdate | ?{$_.Status -eq "HealthCheckFailed"} | Start-SolutionUpdate -PrepareOnly
      
  9. Run Get-SolutionHealthSummary again to confirm the failing tests are resolved. It might take a few minutes for the health checks to run. To confirm whether you're seeing updated or previous results, look at the Info field of the output from Get-SolutionHealthSummary, and compare the value of Time since last check to the value from your previous run of Get-SolutionHealthSummary.

    Info    : {
                  HealthCheckSource     : Daily system health check for 12.2610.1004.37
                  HealthCheckDate       : 10/13/2026 02:11:43
                  Time since last check : 1.3h ago
                  Statistics            : 974 total checks: 3 critical, 1 warning, 4 informational, 966 passed
              }
    

Azure Local release 2609 and earlier

For Azure Local release 2509 and earlier, use different PowerShell commands to view daily system health checks and preupdate health checks.

Note

For Azure Local release 2610 and later, we recommend using the Get-SolutionHealthSummary command to view system and preupdate health check results.

Daily system health checks

  1. To check if the daily system health checks failed, run the following command on one of the machines in your system:

    Get-SolutionUpdateEnvironment
    

    Here's a sample output:

    PS C:\Users\lcmuser> Get-SolutionUpdateEnvironment
    ResourceId        : redmond
    SbeFamily         : VirtualForTesting
    HardwareModel     : Virtual Machine
    LastChecked       : 9/12/2023 10:34:42 PM
    PackageVersions   : {Solution: 10.2309.0.20, Services: 10.2309.0.20, Platform: 1.0.0.0, SBE: 4.0.0.0}
    CurrentVersion    : 10.2309.0.20
    CurrentSbeVersion : 4.0.0.0
    LastUpdated       :
    State             : AppliedSuccessfully
    HealthState       : Failure
    HealthCheckResult : {Storage Pool Summary, Storage Services Physical Disks Summary, Storage Services Physical Disks Summary, Storage Services Physical Disks Summary...}
    HealthCheckDate   : 9/12/2023 7:03:32 AM
    AdditionalData    : {[SBEAdditionalData, Solution Builder extension is partially installed. Please install the latest Solution Builder Extension provided by your hardware vendor. For more information, see https://aka.ms/SBE.]}
    
  2. Review the HealthState on your system and view the Failure or Warning value.

  3. To filter the HealthCheckResult property to identify failing tests, run the following command:

    $result = Get-SolutionUpdateEnvironment
    
    $result.HealthCheckResult | Where-Object {$_.Status -ne "SUCCESS"} | Format-List Title, Status, Severity, Description, Remediation
    

    Here's a sample output:

    Title       : The machine proxy on each failover cluster node should be set to a local proxy server
    Status      : FAILURE
    Severity    : INFORMATIONAL
    Description : Validating cluster setup for update.
    Remediation : `https://learn.microsoft.com/en-us/windows-server/failover-clustering/cluster-aware-updating-requirements#tests-for-cluster-updating-readiness`
    
    Title       : The CAU clustered role should be installed on the failover cluster to enable self-updating mode
    Status      : FAILURE
    Severity    : INFORMATIONAL
    Description : Validating cluster setup for update.
    Remediation : `https://learn.microsoft.com/en-us/windows-server/failover-clustering/cluster-aware-updating-requirements#tests-for-cluster-updating-readiness`
    
  4. Review the Remediation property for the failed tests and take action as appropriate to resolve the failures.

  5. If you require more diagnostic information to determine the cause of the failed tests, examine the AdditionalData property by using the -FullHealthCheckDetails parameter:

    $FullResults = Get-SolutionUpdateEnvironment -FullHealthCheckDetails
    
    $Failures = $FullResults.HealthCheckResult | Where-Object { $_.Status -ne "SUCCESS" -and $_.Severity -ne "INFORMATIONAL" }
    
    $Failures | Format-List *
    
  6. If present, use the diagnostic information shown in AdditionalData property, such as FailedMachines, Source, and ExceptionMessage to determine which physical machines are causing the test failure, together with information provided in the link from the Remediation property to resolve the failures.

  7. After resolving the failures, invoke the system health checks again by running the following command:

    Invoke-SolutionUpdatePrecheck -SystemHealth
    
  8. Use Get-SolutionUpdateEnvironment to confirm the failing tests are resolved. It might take a few minutes for the system health checks to run.

    Here's a sample output:

    PS C:\Users\lcmuser> Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckResult, HealthCheckDate
    
    HealthState       : InProgress
    HealthCheckResult :
    HealthCheckDate   : 1/1/0001 12:00:00 AM
    
    PS C:\Users\lcmuser> Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckResult, HealthCheckDate
    
    HealthState       : Success
    HealthCheckResult : {Storage Pool Summary, Storage Subsystem Summary, Storage Services Summary, Storage Services Summary...}
    HealthCheckDate   : 10/18/2024 11:56:49 PM
    

Preupdate health checks

To view preupdate health check results, use the Get-SolutionUpdate cmdlet. This cmdlet provides the results from the preupdate health checks that use the newer tests included in the update that you prepared or attempted to install. For a health check failure, Get-SolutionUpdate reports the update with a value of HealthCheckFailed in the status property.

To view the details of the failed preupdate health check via PowerShell:

  1. To identify the ResourceId of the updates that you previously selected to prepare or install, run the following command.

    Get-SolutionUpdate | ?{$_State -match "(Health)|(Prep)|(Download)|(ToInstall)|(Scheduled)" | Format-Table ResourceId, State, Version
    

    Here's a sample output:

    PS C:\Users\lcmuser> Get-SolutionUpdate | Format-Table ResourceId, State, Version
    
    ResourceId                     State                  Version
    
    ----------                     -----                  -------
    
    redmond/Solution10.2405.2.7    HealthCheckFailed      10.2405.2.7
    
  2. Look at the output from the previous step to identify which updates are in the HealthCheckFailed state. For a complete list of update phases, see Azure Local update phases.

  3. Using the ResourceId of the update you're attempting to install, run the following command. Replace the Id parameter with the ResourceId of your update, from the output of the previous command:

    Get-SolutionUpdate -Id <RESOURCE_ID> | FL ResourceId, State, HealthState, HealthCheckResult HealthCheckDate
    

    Here's a sample output:

    ResourceId            : example-domain/example-id
    State                 : HealthChecking
    HealthState           : Failed
    HealthCheckResult     : {Storage Services Physical Disks Summary, Storage Services Physical Disks Summary, Storage
                            Services Physical Disks Summary, Storage Services Physical Disks Summary...}
    HealthCheckDate       : 10/13/2026 6:46:58 AM
    
  4. Review the HealthState, HealthCheckDate, and HealthCheckResult values.

    Note

    If the HealthCheckDate value isn't recent, or the HealthState is Unknown, it might mean you didn't attempt to prepare or start the update. Before you investigate the HealthCheckResults fully, consider if you should first start a new health check. For more information, see Understand update phases.

  5. If there's a value in the HealthCheckResults property, filter the property to identify failing tests. Run the following command. Replace the Id value with the ResourceId of your update, identified from the earlier command:

    $result = Get-SolutionUpdate -Id <RESOURCE_ID>
    
    $result.HealthCheckResult | Where-Object {$_.Status -ne "SUCCESS"} | Format-List Title, Status, Severity, Description, Remediation
    

    Here's a sample output:

    Title       : The machine proxy on each failover cluster node should be set to a local proxy server
    
    Status      : FAILURE
    
    Severity    : INFORMATIONAL
    
    Description : Validating cluster setup for update.
    
    Remediation : https://learn.microsoft.com/en-us/windows-server/failover-clustering/cluster-aware-updating-requirements#tests-for-cluster-updating-readiness
    
    Title       : The CAU clustered role should be installed on the failover cluster to enable self-updating mode
    
    Status      : FAILURE
    
    Severity    : INFORMATIONAL
    
    Description : Validating cluster setup for update.
    
    Remediation : https://learn.microsoft.com/en-us/windows-server/failover-clustering/cluster-aware-updating-requirements#tests-for-cluster-updating-readiness
    
  6. Using the link shown in the Remediation property of the failed test, review the article by using a device with a web browser to determine appropriate actions to resolve the failures.

  7. If you need more diagnostic information to determine the cause of the failed tests, examine the AdditionalData property by using the -FullHealthCheckDetails parameter:

    $FullResults = Get-SolutionUpdate -Id <RESOURCE_ID> -FullHealthCheckDetails
    
    $Failures = $FullResults.HealthCheckResult | Where-Object { $_.Status -ne "SUCCESS" -and $_.Severity -ne "INFORMATIONAL" }
    
    $Failures | Format-List *
    

    If present, use the diagnostic information shown in the AdditionalData property, such as FailedMachines, Source, and ExceptionMessage to determine which physical machines are causing the test failure, together with information provided in the link from the Remediation property, to resolve the failures.

  8. After resolving the failures, invoke the update readiness checks again by running the following command:

    Get-SolutionUpdate -Id <RESOURCE_ID> | Start-SolutionUpdate -PrepareOnly
    

Resources to troubleshoot failed results

If the health check results don't immediately offer insights into what the issue is or how to fix it, see the following resources: