Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[windows] Don't fail Windows Update test if an update failed but later succeeded #11253

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

AtOMiCNebula
Copy link
Contributor

Description

Bug fix

I've been trying to build the Windows Server 2025 image this month, and eight of nine attempts to build have failed due to Windows Update failures (the ninth was a fluke failure during VS installation). I kept the OS disk and debugged into it, and the issue was that the "Windows Malicious Software Removal Tool x64 - v5.130 (KB890830)" update was failing due to the machine being rebooted. It appears the install was starting around the time Install-PowerShellCore.ps1 was running. On a hunch, I tried updating the PS installer to set ENABLE_MU=0/USE_MU=0 but the failures didn't go away.

Broken Test Output (note duplicate entries for MSRT):

azure-arm.image: Describing Windows Updates
azure-arm.image:   [+] WindowsUpdateDone.txt should exist 5ms (2ms|3ms)
azure-arm.image:   [+] 2024-11 Cumulative Update for .NET Framework 3.5 and 4.8.1 for Microsoft server operating system version 24H2 for x64 (KB5045934) 57ms (56ms|1ms)
azure-arm.image:   [+] Security Intelligence Update for Microsoft Defender Antivirus - KB2267602 (Version 1.421.1098.0) - Current Channel (Broad) 4ms (2ms|2ms)
azure-arm.image:   [+] Security Intelligence Update for Microsoft Defender Antivirus - KB2267602 (Version 1.421.1101.0) - Current Channel (Broad) 2ms (1ms|1ms)
azure-arm.image:   [+] Security Update for Microsoft OLE DB Driver for SQL Server (KB5040711) 2ms (1ms|1ms)
azure-arm.image:   [+] Update for Microsoft Defender Antivirus antimalware platform - KB4052623 (Version 4.18.24090.11) - Current Channel (Broad) 2ms (1ms|1ms)
azure-arm.image:   [-] Windows Malicious Software Removal Tool x64 - v5.130 (KB890830) 175ms (173ms|2ms)
azure-arm.image:    Expected collection 'Installed' to contain 'Failed', but it was not found.
azure-arm.image:    at $State | Should -BeIn $expect, C:\image\tests\WindowsFeatures.Tests.ps1:76
azure-arm.image:    at <ScriptBlock>, C:\image\tests\WindowsFeatures.Tests.ps1:76
azure-arm.image:   [+] Windows Malicious Software Removal Tool x64 - v5.130 (KB890830) 3ms (2ms|1ms)

To address this, I've updated the WU logic to ignore an earlier failure for a given update if it is followed by a successful attempt.

I don't know if there is a larger issue here...should WU be running in the background at this point? I only encountered this issue on the WS2025 image, and have also been building WS2022 and WS2019 a bunch without issue, so maybe something's still off with the new image?

Related issue:

I haven't created one, but can if you'd like me to.

Check list

  • Related issue / work item is attached
  • Tests are written (if applicable)
  • Documentation is updated (if applicable)
  • Changes are tested and related VM images are successfully generated

@bearycool11
Copy link

here, let me help you finish this, as this adds onto @intelligenceq's stub work from 3 days ago/10 minutes ago.

Look at my version as a prototype for this repo
https://github.com/bearycool11/pmll_blockchain/blob/main/.devcontainer/internal.windows-2025.json

@@ -66,10 +66,10 @@ Describe "Windows Updates" {
"$env:windir\WindowsUpdateDone.txt" | Should -Exist
}

$testCases = Get-WindowsUpdateStates | Sort-Object Title | ForEach-Object {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I see, yeah this jSON is going to be really long I got you one second.
internal.windows-2025.json

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to start we going to take this

{
"NodeType": "HeaderNode",
"Title": "Windows Server 2025",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "OS Version:",
"Version": "10.0.26100 Build 2605"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Image Version:",
"Version": "20241215.1.0"
},
{
"NodeType": "HeaderNode",
"Title": "Windows updates",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Update Title",
"Version": "Some Update",
"TimeCreated": "2024-12-30T12:34:56Z"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Another Update",
"Version": "Another Update Version",
"TimeCreated": "2024-12-30T12:34:56Z"
}
]
}
]
}

and combine it into the prototype.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
"NodeType": "HeaderNode",
"Title": "Windows Server 2025",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "OS Version:",
"Version": "10.0.26100 Build 2605"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Image Version:",
"Version": "20241215.1.0"
},
{
"NodeType": "HeaderNode",
"Title": "Windows updates",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Update Title",
"Version": "Some Update",
"TimeCreated": "2024-12-30T12:34:56Z"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Another Update",
"Version": "Another Update Version",
"TimeCreated": "2024-12-30T12:34:56Z"
}
]
}
]
}

thank you CoPIlot for intergrating this by the way!
https://github.com/bearycool11/pmll_blockchain/blob/main/.devcontainer/internal.windows-2025.json

now turns into this repo's version of this.

Copy link

@bearycool11 bearycool11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
"NodeType": "HeaderNode",
"Title": "Windows Server 2025",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "OS Version:",
"Version": "10.0.26100 Build 2605"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Image Version:",
"Version": "20241215.1.0"
},
{
"NodeType": "HeaderNode",
"Title": "Windows updates",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Update Title",
"Version": "Some Update",
"TimeCreated": "2024-12-30T12:34:56Z"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Another Update",
"Version": "Another Update Version",
"TimeCreated": "2024-12-30T12:34:56Z"
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Windows features",
"Children": {
"NodeType": "ToolVersionNode",
"ToolName": "Windows Subsystem for Linux (WSLv1):",
"Version": "Enabled"
}
},
{
"NodeType": "HeaderNode",
"Title": "Installed Software",
"Children": [
{
"NodeType": "HeaderNode",
"Title": "Language and Runtime",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Bash",
"Version": "5.2.37(1)-release"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Go",
"Version": "1.23.4"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Julia",
"Version": "1.10.5"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Kotlin",
"Version": "2.1.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "LLVM",
"Version": "19.1.5"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Node",
"Version": "22.12.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Perl",
"Version": "5.40.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "PHP",
"Version": "8.3.14"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Python",
"Version": "3.9.13"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Ruby",
"Version": "3.3.6"
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Package Management",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Chocolatey",
"Version": "2.4.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Composer",
"Version": "2.8.4"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Helm",
"Version": "3.16.2"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Miniconda",
"Version": "24.9.2 (pre-installed on the image but not added to PATH)"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "NPM",
"Version": "10.9.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "NuGet",
"Version": "6.12.1.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "pip",
"Version": "24.3.1 (python 3.9)"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Pipx",
"Version": "1.7.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "RubyGems",
"Version": "3.5.22"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Vcpkg",
"Version": "(build from commit c26eabb23)"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Yarn",
"Version": "1.22.22"
},
{
"NodeType": "HeaderNode",
"Title": "Environment variables",
"Children": {
"NodeType": "TableNode",
"Headers": "Name|Value",
"Rows": [
"VCPKG_INSTALLATION_ROOT|C:\vcpkg",
"CONDA|C:\Miniconda"
]
}
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Project Management",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Ant",
"Version": "1.10.14"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Gradle",
"Version": "8.11"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Maven",
"Version": "3.9.9"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "sbt",
"Version": "1.10.6"
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Tools",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "7zip",
"Version": "24.09"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "aria2",
"Version": "1.37.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "azcopy",
"Version": "10.27.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Bazel",
"Version": "8.0.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Bazelisk",
"Version": "1.25.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Bicep",
"Version": "0.32.4"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Cabal",
"Version": "3.12.1.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "CMake",
"Version": "3.31.2"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "CodeQL Action Bundle",
"Version": "2.20.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Docker",
"Version": "26.1.3"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Docker Compose v2",
"Version": "2.27.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Docker-wincred",
"Version": "0.8.2"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "ghc",
"Version": "9.10.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Git",
"Version": "2.47.1.windows.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Git LFS",
"Version": "3.6.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "ImageMagick",
"Version": "7.1.1-41"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "jq",
"Version": "1.7.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Kind",
"Version": "0.25.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Kubectl",
"Version": "1.32.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "gcc",
"Version": "14.2.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "gdb",
"Version": "14.2"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "GNU Binutils",
"Version": "2.42"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Newman",
"Version": "6.2.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "OpenSSL",
"Version": "3.4.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Packer",
"Version": "1.11.2"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Pulumi",
"Version": "3.143.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "R",
"Version": "4.4.2"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Service Fabric SDK",
"Version": "10.1.2493.9590"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Stack",
"Version": "3.1.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Swig",
"Version": "4.1.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "VSWhere",
"Version": "3.1.7"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "WinAppDriver",
"Version": "1.2.2009.02003"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "WiX Toolset",
"Version": "3.14.1.8722"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "yamllint",
"Version": "1.35.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "zstd",
"Version": "1.5.6"
}
]
},
{
"NodeType": "HeaderNode",
"Title": "CLI Tools",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "AWS CLI",
"Version": "2.22.17"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "AWS SAM CLI",
"Version": "1.132.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "AWS Session Manager CLI",
"Version": "1.2.694.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Azure CLI",
"Version": "2.67.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Azure DevOps CLI extension",
"Version": "1.0.1"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "GitHub CLI",
"Version": "2.63.2"
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Rust Tools",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Cargo",
"Version": "1.83.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Rust",
"Version": "1.83.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Rustdoc",
"Version": "1.83.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Rustup",
"Version": "1.27.1"
},
{
"NodeType": "HeaderNode",
"Title": "Packages",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Clippy",
"Version": "0.1.83"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Rustfmt",
"Version": "1.8.0"
}
]
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Browsers and Drivers",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Google Chrome",
"Version": "131.0.6778.140"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Chrome Driver",
"Version": "131.0.6778.108"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Microsoft Edge",
"Version": "131.0.2903.99"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Microsoft Edge Driver",
"Version": "131.0.2903.99"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Mozilla Firefox",
"Version": "133.0.3"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Gecko Driver",
"Version": "0.35.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "IE Driver",
"Version": "4.14.0.0"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Selenium server",
"Version": "4.27.0"
},
{
"NodeType": "HeaderNode",
"Title": "Environment variables",
"Children": {
"NodeType": "TableNode",
"Headers": "Name|Value",
"Rows": [
"CHROMEWEBDRIVER|C:\SeleniumWebDrivers\ChromeDriver",
"EDGEWEBDRIVER|C:\SeleniumWebDrivers\EdgeDriver",
"GECKOWEBDRIVER|C:\SeleniumWebDrivers\GeckoDriver",
"SELENIUM_JAR_PATH|C:\selenium\selenium-server.jar"
]
}
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Java",
"Children": {
"NodeType": "TableNode",
"Headers": "Version|Environment Variable",
"Rows": [
"8.0.432+6|JAVA_HOME_8_X64",
"11.0.25+9|JAVA_HOME_11_X64",
"17.0.13+11 (default)|JAVA_HOME_17_X64",
"21.0.5+11.0|JAVA_HOME_21_X64"
]
}
},
{
"NodeType": "HeaderNode",
"Title": "Shells",
"Children": {
"NodeType": "TableNode",
"Headers": "Name|Target",
"Rows": [
"gitbash.exe|C:\Program Files\Git\bin\bash.exe",
"msys2bash.cmd|C:\msys64\usr\bin\bash.exe",
"wslbash.exe|C:\Windows\System32\bash.exe"
]
}
},
{
"NodeType": "HeaderNode",
"Title": "MSYS2",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Pacman",
"Version": "6.1.0"
},
{
"NodeType": "HeaderNode",
"Title": "Notes",
"Children": {
"NodeType": "NoteNode",
"Content": "Location: C:\msys64\n\nNote: MSYS2 is pre-installed on image but not added to PATH."
}
}
]
},
{
"NodeType": "HeaderNode",
"Title": "Cached Tools",
"Children": [
{
"NodeType": "ToolVersionsListNode",
"ToolName": "Go",
"Versions": [
"1.21.13",
"1.22.10",
"1.23.4"
],
"MajorVersionRegex": "^\d+\.\d+",
"ListType": "List"
},
{
"NodeType": "ToolVersionsListNode",
"ToolName": "Node.js",
"Versions": [
"18.20.5",
"20.18.1",
"22.12.0"
],
"MajorVersionRegex": "^\d+",
"ListType": "List"
},

@@ -66,10 +66,10 @@ Describe "Windows Updates" {
"$env:windir\WindowsUpdateDone.txt" | Should -Exist
}

$testCases = Get-WindowsUpdateStates | Sort-Object Title | ForEach-Object {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to start we going to take this

{
"NodeType": "HeaderNode",
"Title": "Windows Server 2025",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "OS Version:",
"Version": "10.0.26100 Build 2605"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Image Version:",
"Version": "20241215.1.0"
},
{
"NodeType": "HeaderNode",
"Title": "Windows updates",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Update Title",
"Version": "Some Update",
"TimeCreated": "2024-12-30T12:34:56Z"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Another Update",
"Version": "Another Update Version",
"TimeCreated": "2024-12-30T12:34:56Z"
}
]
}
]
}

and combine it into the prototype.

@@ -66,10 +66,10 @@ Describe "Windows Updates" {
"$env:windir\WindowsUpdateDone.txt" | Should -Exist
}

$testCases = Get-WindowsUpdateStates | Sort-Object Title | ForEach-Object {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
"NodeType": "HeaderNode",
"Title": "Windows Server 2025",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "OS Version:",
"Version": "10.0.26100 Build 2605"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Image Version:",
"Version": "20241215.1.0"
},
{
"NodeType": "HeaderNode",
"Title": "Windows updates",
"Children": [
{
"NodeType": "ToolVersionNode",
"ToolName": "Update Title",
"Version": "Some Update",
"TimeCreated": "2024-12-30T12:34:56Z"
},
{
"NodeType": "ToolVersionNode",
"ToolName": "Another Update",
"Version": "Another Update Version",
"TimeCreated": "2024-12-30T12:34:56Z"
}
]
}
]
}

thank you CoPIlot for intergrating this by the way!
https://github.com/bearycool11/pmll_blockchain/blob/main/.devcontainer/internal.windows-2025.json

now turns into this repo's version of this.

Copy link

@bearycool11 bearycool11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#11254

to fully intergrated the install helpers and runners

@AtOMiCNebula
Copy link
Contributor Author

@bearycool11, I don't think your issue is at all related to mine. My change is unrelated to devcontainer or the software-report.

@shamil-mubarakshin
Copy link
Contributor

Hey @AtOMiCNebula, thanks for contributing. We are aware of this issue and working on the same solution.
I am currently testing a simpler approach to skipping failed updates that succeeded later, but due to sporadic nature of this error I am rarely hitting it with some debug code so it takes time

@AtOMiCNebula
Copy link
Contributor Author

Hey @shamil-mubarakshin, thanks for the follow-up. Unfortunately for me, I've been hitting this issue constantly. 😦 Happy for your team to land your own fix for this, but this worked well to unblock me in my private testing, and I felt it was pretty safe, so I wanted to offer it up as an option.

Some additional changes I made to build with (that might help you hit it more consistently) are using the Standard_D8ds_v5 SKU in East US 2, and also I bumped the image to the Gen2 equivalent (2025-Datacenter-g2). Not sure how far that'll get you, but I figured it can't hurt. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants