Skip to content

Correctly detect installed SDK versions #79614

Closed

Description

@jkloth
BPO 35433
Nosy @pfmoore, @db3l, @vstinner, @tjguk, @jkloth, @zware, @zooba, @pablogsal, @miss-islington
PRs
  • bpo-35433: Properly detect installed SDK versions #11009
  • [3.7] bpo-35433: Properly detect installed SDK versions (GH-11009) #11058
  • [3.6] bpo-35433: Properly detect installed SDK versions (GH-11009) #11059
  • [3.6] bpo-35433: Properly detect installed SDK versions (GH-11009) #11060
  • Files
  • msbuild-win10-bad.log: Failing build log
  • msbuild-win10-good.log: Working build log
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2018-12-11.16:06:19.104>
    created_at = <Date 2018-12-07.01:03:10.727>
    labels = ['3.8', 'build', '3.7', 'OS-windows']
    title = 'Correctly detect installed SDK versions'
    updated_at = <Date 2018-12-11.16:06:19.103>
    user = 'https://github.com/jkloth'

    bugs.python.org fields:

    activity = <Date 2018-12-11.16:06:19.103>
    actor = 'jkloth'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2018-12-11.16:06:19.104>
    closer = 'jkloth'
    components = ['Build', 'Windows']
    creation = <Date 2018-12-07.01:03:10.727>
    creator = 'jkloth'
    dependencies = []
    files = ['47985', '47986']
    hgrepos = []
    issue_num = 35433
    keywords = ['patch']
    message_count = 37.0
    messages = ['331258', '331262', '331263', '331303', '331315', '331320', '331324', '331329', '331388', '331389', '331437', '331462', '331463', '331466', '331467', '331471', '331513', '331533', '331541', '331544', '331545', '331546', '331547', '331548', '331551', '331552', '331554', '331555', '331556', '331557', '331558', '331559', '331561', '331562', '331563', '331611', '331635']
    nosy_count = 10.0
    nosy_names = ['paul.moore', 'db3l', 'vstinner', 'tim.golden', 'jkloth', 'jeremy.kloth', 'zach.ware', 'steve.dower', 'pablogsal', 'miss-islington']
    pr_nums = ['11009', '11058', '11059', '11060']
    priority = 'normal'
    resolution = None
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue35433'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    Activity

    1. jkloth commented on Dec 7, 2018

      @jkloth
      ContributorAuthor

      In the process of eliminating compiler warnings on my buildbot, I needed to update VS2015 to the latest toolset (VS2015 Update 3). This in turn now causes an error due about not having the required version of Windows SDK installed.

      It seems that the detection logic for that uses a hard-coded list which may not be up-to-date (and possibly incorrect for some installs). Referenced PR fixes this.

    2. jkloth commented on Dec 7, 2018

      @jkloth
      ContributorAuthor

      Also, my buildbot *should* fail until this change is merged.

    3. zooba commented on Dec 7, 2018

      @zooba
      Member

      Looks good to me. I can't apply tags from my phone, apparently, so someone else will have to merge and mark the backport versions.

    4. vstinner commented on Dec 7, 2018

      @vstinner
      Member

      Is this issue related to bpo-34977: Add Windows App Store package?

    5. zooba commented on Dec 7, 2018

      @zooba
      Member

      No, it just automates something that's been manual for a while. Installing VS 2017 on build bots is still a manual process ;)

    6. jkloth commented on Dec 7, 2018

      @jkloth
      ContributorAuthor

      Sorry Victor, I should have added more context when I nosy'ed you to this issue.

      Since you seem to be the "buildbot keeper" as of late, I wanted to give you a pointer to the reason for my failing buildbot. Which, as I surmised, is failing compile due to the missing SDK error that is fixed by this PR ;)

    7. vstinner commented on Dec 7, 2018

      @vstinner
      Member

      After I reverted bpo-34977 change, most Windows buildbots are back to green.

      In the process of eliminating compiler warnings on my buildbot, I needed to update VS2015 to the latest toolset (VS2015 Update 3). This in turn now causes an error due about not having the required version of Windows SDK installed.

      Is it a warning or an error? What is the warning/error message? What is your buildbot?

      Right now, it seems like only "AMD64 Windows7 SP1 3.x" is still red:

      "2>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 10.0.17134.0 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]"

      https://buildbot.python.org/all/#/builders/40/builds/1322

      Are you talking about this error?

    8. jeremykloth commented on Dec 7, 2018

      jeremyklothmannequin
      Mannequin

      Is it a warning or an error? What is the warning/error message? What is your buildbot?

      It is a compiler error as you point out below (with message). By buildbot is:

      https://buildbot.python.org/all/#/workers/12

      Right now, it seems like only "AMD64 Windows7 SP1 3.x" is still red:

      "2>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 10.0.17134.0 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]"

      https://buildbot.python.org/all/#/builders/40/builds/1322

      Are you talking about this error?

      All the builders on my buildbot can no longer find a matching SDK
      version due to how the SDK version detection is handled. Previously,
      my buildbot had VS2015.2 installed whose build tools (version 1.2)
      didn't use the property DefaultWindowsSDKVersion to select the SDK
      version and just defaulted to Windows 8.1 SDK. Now that I've updated
      to VS2015.3 (includes build tools 1.4) it does use this property to
      select the SDK version. However, I have a newer SDK installed than is
      listed in the hard-coded list so it defaults to highest one in the
      list causing the missing SDK error.

      This PR changes the detection logic to simply accept what SDK the user
      has installed as long as it is newer (or the same) as the minimum
      allowed version.

      For those versed in MSBuild, this should be fairly self-evident, but
      hopefully this helps those who are not :)

    9. jkloth commented on Dec 8, 2018

      @jkloth
      ContributorAuthor

      Not to be impatient or anything, but this change is keeping my buildbot from being useful. Would it be possible for someone to merge as it has already been approved (msg331263).

    10. zooba commented on Dec 8, 2018

      @zooba
      Member

      Did you see my comment about converting the version numbers to System.Version before comparing? (I didn't know it could be done the first time I looked :) )

    11. jkloth commented on Dec 9, 2018

      @jkloth
      ContributorAuthor

      Sorry for the delay, but I wanted to get an environment that still had an older VS2015 install to test against. VS2015 prior to Update 3 use a different heuristic to determine the SDK version for building.

      I've made the following changes:

      • version checking uses System.Version,
      • WindowsTargetPlatformVersion is set to the detected Win10 SDK (this is the offical setting anyway, but required for VS2015 prior to Update 3)
      • added display of the detected SDK to ShowVersionInfo target (build.bat -V)
    12. 18 remaining items

    13. db3l commented on Dec 10, 2018

      @db3l
      Contributor

      (sorry for the rapid updates)

      I'm also fairly sure that none of my workers have update 3 for VS2015. They do however all have VS2017 - but I think VS2015 still gets picked for 3.6 if both are present, right? So that's another variable, in that my workers wouldn't have run into the update 3 issue with the older configuration that Jeremy did.

      Not sure if there's a way to have the build setup to work for both cases.

    14. zooba commented on Dec 11, 2018

      @zooba
      Member

      So before the change, the 16299 SDK wasn't being detected either, but perhaps the 10240 one was? That had some known issues in the debug UCRT, which is why the version gets printed out towards the end of the build.

      The only thing I can think of would be the WindowsTargetPlatformVersion setting, which is new. It may be that it is used differently on different MSBuild versions (which roughly match to VS versions) by the built-in target files.

      Are you able to run "MSBuild pythoncore.vcxproj /t:ShowVersionInfo /flp:v=diag" and upload the MSBuild.log file (or email it to me)?

    15. db3l commented on Dec 11, 2018

      @db3l
      Contributor

      So before the change, the 16299 SDK wasn't being detected either, but perhaps the 10240 one was?

      So I'm just confused....

      It does seems likely that 10240 of the UCRT was being used (based on the attached msbuild logs). Howevr, the UCRT warning in the build process was fine, and said I was using 16299 (see https://buildbot.python.org/all/#/builders/31/builds/717/steps/1/logs/stdio for the last working build on Win10).

      I'm not sure what to trust - the msbuild log or whatever the ucrtused test is doing for detection?

      In either case, this issue commmit does seem to force selection of the non-existent 15063 SDK, at least based on msbuild.log - the process doesn't get far enough to run the UCRT version check.

      Using Win10 for the moment, under "Program Files (x86)" I do seem to have some older UCRT SDKs (in "Microsoft SDKs/Windows Kits/10/ExtensionSDKs/Microsoft.UniversalCRT.Debug") as 10.0.10150.0, 10.0.10240.0 and 10.0.16299.0. Whereas only 10.0.16299.0 exists in the "Windows Kits/10/Platforms/UAP" directory.

      So I guess msbuild is not strictly picking the "earliest" for UCRT as 10240 isn't quite the earliest.

      In comparison, my Windows 7 worker also has the 15063 UCRT SDK so the enforced minimum works there for the UCRT path. Of course, the ucrtused test in the build process seems to always say 16299.

      I ran the requested msbuild command on the Win10 worker as part of PCBuild\build.bat just after the first find_msbuild.bat call. I'm attaching the log files as msbuild-win10-good.log (commit prior to this change) and msbuild-win10-bad.log (after this change).

      So I guess the question is exactly what do we need to enforce in the build process and/or provide at a minimum on a worker? Should I just install the older 15063 SDK on Win8/10, or should we include some more recent versions in the build time version check? What is a new user trying to build Python likely to have (assuming they're still using VS2015).

    16. db3l commented on Dec 11, 2018

      @db3l
      Contributor

      (and the working log)

    17. jkloth commented on Dec 11, 2018

      @jkloth
      ContributorAuthor

      A quick look at the good build log does indicate that the 8.1 SDK was being used (vars UseWindows81SDK=true and WindowsTargetPlatformVersion=8.1) which is the default behavior for VS2015 if a matching (from python.props) Win10SDK cannot be found.

      It seems that the build tools you have installed do not give an error when the selected SDK (WindowsTargetPlatformVersion) isn't found on the system.

      You've said that you have the Build Tools for VS2015 installed, a quick look at the download date indicates that they were released prior to VS2015 Update 1 (which includes 10.0.10586.0).

    18. jkloth commented on Dec 11, 2018

      @jkloth
      ContributorAuthor

      I forgot to mention that the presence of UseWindows81SDK in the build log indicates that the Build Tools are at most at version 1.2 (included with VS2015 Update 1) which should still work (it's what I tested against), but the difference may be the standalone Build Tools might not include an error for missing SDK whereas the VS2015 Tools do.

    19. db3l commented on Dec 11, 2018

      @db3l
      Contributor

      Hmm, VS2015 started as a full installation (with UI), probably right from its initial release. The build tools only installation (v141) is for VS2017.

      Best I can tell I'm at update 1 - my update version in the registry is 14.0.24720 (plus I have a .1 installer still in the filesystem). Not sure why I don't have 10586 if that came with update 1. But I wouldn't be surprised if I'm no further. The last time I used the VS2015 installer it completely messed up one of my workers and took a bunch of time to recover from.

      When switching to VS2017 I think I thought it was going to be preferred for the 3.x series when available, so I probably treated the VS2015 version as less critical. I guess that didn't end up happening for earlier than 3.7.

      Anyway, best next steps to close out this issue...

      I assume my installing any matching SDK (say 15063) would allow the workers to match the build script. Alternatively, including more recent SDKs (16299 is the next in line) in the build script would work with the existing workers. I'm fine either way.

    20. jkloth commented on Dec 11, 2018

      @jkloth
      ContributorAuthor

      Well, if VS2015 is installed, the simplest way to have the required SDK(s) is to go to Control Panel -> Uninstall -> Microsoft Visual Studio (Community) 2015

      Click Modify.
      Expand "Windows and Web Development"
      Expand "Universal Windows App Development Tools"
      Select at least "Windows 10 SDK (10.0.10586)"

    21. db3l commented on Dec 11, 2018

      @db3l
      Contributor

      Oh, it's not the installation itself, I'm just wondering if allowing a newer version is ok too?

      Of course, it doesn't preclude expanding the build script in the future, so I've installed 15063 to both Win8/10 workers.

      I don't currently have access to restart builds through the buildbot web interface any more, but if someone who does wants to rerun the last 3.6 build on each builder we can make sure this resolves the issue on those workers.

      I've done a quick manual build that compiles with the exception of not finding midl.exe for pyshellext, but that seems separate (every SDK has the binary) so I'm assuming it's environmental outside of the regular build environment.

    22. jeremykloth commented on Dec 11, 2018

      jeremyklothmannequin
      Mannequin

      Oh, it's not the installation itself, I'm just wondering if allowing a newer version is ok too?

      The original PR (included in 3.7, 3.x) uses the latest discovered SDK,
      but Steve stated that that logic broke the Pipelines build, so I
      reworked to PR to just check for the versions already hard-coded.

      Of course, it doesn't preclude expanding the build script in the future, so I've installed 15063 to both Win8/10 workers.

      For 3.6, this is the last hurrah. It goes into security mode after
      tonight. Although, it does seem that we might need to add a check for
      the SDK ourselves instead of relying on MSBuild to do it (it may be a
      full Visual Studio only thing, not Build Tools).

      I don't currently have access to restart builds through the buildbot web interface any more

      If anyone with the power to do so is watching, this is something that
      I miss now, too. It seems that the buildbot owners are not given the
      proper permissions for this task.

    23. db3l commented on Dec 11, 2018

      @db3l
      Contributor

      Ah, got it (and see the pipelines comment by Steve).

      Jeremy, I suspect you might actually be able to restart the most recent 3.6 builds on my builders since you were the committer. It changed in Sep to only allow python-core users and the "owner" of the build. Though I don't think I learned for sure if owner was the author of the commit or just the actual committer.

      I think Zachary was going to look into it time permitting. We probably need a different GitHub group for buildbot owners or something.

      Separately, I think in the context of this issue I think it's fair to say that the commit need not be rolled back. I have to imagine that anyone trying to build 3.6 using VS 2015 at this point is going to be better represented by Jeremy's worker than mine (in terms of VS point release and default SDK installation), and I expect to be able to resolve any remaining issues worker-side once some 3.6 builds go through.

    24. vstinner commented on Dec 11, 2018

      @vstinner
      Member

      The 3.6 change broke the compilation 2 Windows buildbots: Jemery Kloth is investigating the regression:
      #11009 (comment)

    25. jkloth commented on Dec 11, 2018

      @jkloth
      ContributorAuthor

      All the Windows 3.6 windows buildbots are happy! Thanks to all!

    26. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions