Loop through folder of multiple wim's

Discussion in 'Windows 8' started by hypedave, Nov 11, 2014.

  1. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    #181 s1ave77, Dec 4, 2014
    Last edited by a moderator: Apr 20, 2017
    Better use the complete script i offered last. I made some changes and added several variables

    Code:
    ::===============================================================================================================
    :: SET COMPARE FILES
    set "mountfolder=f:\mount"
    set "wimfolder=f:\wim"
    set "backupfolder=f:\backup"
    set "masterfolder=f:\master"
    set "sourcefolder=f:\source"
    set "wimfile=f:\wim\wimlist.txt"
    set "deltafile=f:\wim\deltalist.txt"
    set "packages=f:\images\packages.txt"
    set "folderpackages=f:\images\folderpackages.txt"
    set "integratepackages=f:\images\integratepackages.txt"
    if exist %wimfile% del /s /q %wimfile% >nul
    if exist %deltafile% del /s /q %deltafile% >nul
    ::===============================================================================================================
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    #182 hypedave, Dec 4, 2014
    Last edited by a moderator: Apr 20, 2017
    (OP)
    Getting errors

    Code:
    ===============================================================================
    Backing up to f:\backup\backup_12-04-2014-15-45.wim
    ===============================================================================
    Exporting Compare-Object : Cannot bind argument to parameter 'ReferenceObject' because it is null. to f:\backup\backup.wim
    
    Deployment Image Servicing and Management tool
    Version: 6.3.9600.17031
    
    Error: 2
    
    Here is the script. Replaced f:\wim tp f:\images and add setpath for driver store;

    Code:
    [@echo off
    title þ IMAGE SERVICING TOOL þ v.0.20
    ::Idea/Motivation: hypedave, Code: s1ave77 (forums.mydigitallife.net)
    ::===============================================================================================================
    :: GET ADMIN RIGHTS
    (NET FILE||(powershell -command Start-Process '%0' -Verb runAs -ArgumentList '%* '&EXIT /B))>NUL 2>&1
    ::===============================================================================================================
    setLocal EnableDelayedExpansion
    :================================================================================================================
    ::===============================================================================================================
    :: GET TIMESTAMP DATA: will show month.day.year.hour.min.sec
    for /f "skip=1 delims=" %%x in ('wmic os get localdatetime') do if not defined X set X=%%x
    :: Timestamp with seconds
    :: set timestamp=%X:~4,2%-%X:~6,2%-%X:~0,4%-%X:~8,2%-%X:~10,2%-%X:~12,2%
    :: Timestamp without seconds
    set timestamp=%X:~4,2%-%X:~6,2%-%X:~0,4%-%X:~8,2%-%X:~10,2%
    ::===============================================================================================================
    :: EXCLUDE WIMS FROM 
    set textfile=f:\images\exclusions.txt
    ::===============================================================================================================
    :: Path to update packages group 1
    set updates1=f:\updates\Windows7-x64
    :: Path to update packages group 2
    set updates2=f:\updates\Windows7-x86
    :: Path to update packages group 3
    set updates3=f:\updates\Windows8-x64
    :: Path to update packages group 4
    set updates4=f:\updates\Windows8-x86
    :: Path to update packages group 5
    set updates5=f:\updates\Windows8.1-x64
    :: Path to update packages group 6
    set updates6=f:\updates\Windows8.1-x86
    ::===============================================================================================================
    :: Path to Windows 7 x64 Make & Model Driver Store
    set drivers1=f:\drivers\Windows 7 x64
    :: Path to Windows 8 x64 Make & Model Driver Store
    set drivers2=f:\drivers\Windows 7 x86
    :: Path to Windows 8 x64 Make & Model Driver Store
    set drivers3=f:\drivers\Windows 8 x64
    :: Path to Windows 8 x86 Make & Model Driver Store
    set drivers4=f:\drivers\Windows 8 x86
    :: Path to Windows 8.1 x64 Make & Model Driver Store
    set drivers5=f:\drivers\Windows 8.1 x64
    :: Path to Windows 8.1 x86 Make & Model Driver Store
    set drivers6=f:\drivers\Windows 8.1 x86
    ::===============================================================================================================
    :: SET COMPARE FILES
    set "mountfolder=f:\mount"
    set "wimfolder=f:\images"
    set "backupfolder=f:\backup"
    set "masterfolder=f:\master"
    set "sourcefolder=f:\source"
    set "wimfile=f:\images\wimlist.txt"
    set "deltafile=f:\images\deltalist.txt"
    set "packages=f:\images\packages.txt"
    set "folderpackages=f:\images\folderpackages.txt"
    set "integratepackages=f:\images\integratepackages.txt"
    if exist %wimfile% del /s /q %wimfile% >nul
    if exist %deltafile% del /s /q %deltafile% >nul
    ::===============================================================================================================
    :: GET WIM NAMES FROM FOLDER
    call :CheckFolder %wimfolder%, wim, %wimfile%
    :: compare files and write difference to %deltafile%
    ::===============================================================================================================
    call :CompareWithPowershell %textfile%, %wimfile%, %deltafile%
    ::===============================================================================================================
    if not exist %deltafile% (
    echo No files to process found.
    echo:
    pause
    if exist %wimfile% del /s /q %wimfile% >nul
    exit
    )
    :================================================================================================================
    ::===============================================================================================================
    :: PATH TO FILES COPY
    set path1=f:\files
    set target=f:\mount
    :================================================================================================================
    ::===============================================================================================================
    ::START PROCEDURE
    ::BACKUP ALL WIMS TO f:\backup\backup.wim
    echo:
    echo ===============================================================================
    echo Backing up to f:\backup\backup_%timestamp%.wim
    echo ===============================================================================
    echo.
    set "p=0"
    for /f "tokens=* delims= " %%a in (%deltafile%) do (
    echo Exporting %%a to f:\backup\backup.wim
    echo:
    Dism /Export-Image /SourceImageFile:"%wimfolder%\%%a.wim" /SourceIndex:1 /DestinationImageFile:"%backupfolder%\backup_%timestamp%.wim" /compress:max /CheckIntegrity
    )
    :================================================================================================================
    ::===============================================================================================================
    ::MOUNT SINGLE WIMs TO f:\mount, ADD UPDATES AND UNMOUNT/COMMIT
    echo:
    echo ===============================================================================
    echo Mounting to f:\mount
    echo Adding Updates
    echo Unmount/Commit
    echo ===============================================================================
    echo:
    ::===============================================================================================================
    ::EXECUTE LOOP FOR DELTAS ONLY
    for /f "tokens=* delims= " %%a in (%deltafile%) do (
    echo Processing %%a
    echo:
    DISM /mount-wim /wimfile:"f:\images\%%a.wim" /index:1 /mountdir:"%mountfolder%"
    echo %%a | find "w81u1x64_Lenovo_ThinkPad_Yoga" 1>nul && Dism /Image:f:\mount /Add-Driver /Driver:"%drivers4%\Lenovo\Thinkpad Yoga" /Recurse /ForceUnsigned
    powershell.exe remove-item "%target%\Users\Public\Desktop\Encore Login.url" -Force -Recurse -ErrorAction SilentlyContinue -verbose
    powershell.exe remove-item %target%\lanrev* -Force -Recurse -ErrorAction SilentlyContinue -verbose
    powershell.exe Remove-Item “%target%\programdata\pole position software\lanrev agent\PowerOnHistory.plist” -Force -Recurse -ErrorAction SilentlyContinue -verbose
    powershell.exe Remove-Item “%target%\programdata\pole position software\lanrev agent\To Be Downloaded Packages.plist” -Force -Recurse -ErrorAction SilentlyContinue -verbose
    powershell.exe Remove-Item “%target%\programdata\pole position software\lanrev agent\cacheddata*” -Force -Recurse -ErrorAction SilentlyContinue -verbose
    xcopy.exe %path1%\*.* %target%\*.* /s /d /e /h /r /k /y
    DISM /Image:%mountfolder% /Import-DefaultAppAssociations:%target%\Windows\AppAssoc.xml
    echo %%a | find "w7sp1x64" 1>nul && (
    call :CheckMountedWIM %mountfolder%, %packages%
    call :CheckUpdateFolder %updates1%, msu, %folderpackages%
    call :CompareWithPowershell %packages%, %folderpackages%, %integratepackages%
    if exist %integratepackages% call :IntegratePackages %integratepackages%, %updates1%, msu, %mountfolder%
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    )
    echo %%a | find "w7sp1x86" 1>nul && (
    call :CheckMountedWIM %mountfolder%, %packages%
    call :CheckUpdateFolder %updates2%, msu, %folderpackages%
    call :CompareWithPowershell %packages%, %folderpackages%, %integratepackages%
    if exist %integratepackages% call :IntegratePackages %integratepackages%, %updates2%, msu, %mountfolder%
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    )
    echo %%a | find "w8x64" 1>nul && (
    call :CheckMountedWIM %mountfolder%, %packages%
    call :CheckUpdateFolder %updates3%, msu, %folderpackages%
    call :CompareWithPowershell %packages%, %folderpackages%, %integratepackages%
    if exist %integratepackages% call :IntegratePackages %integratepackages%, %updates3%, msu, %mountfolder%
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    )
    echo %%a | find "w8x86" 1>nul && (
    call :CheckMountedWIM %mountfolder%, %packages%
    call :CheckUpdateFolder %updates4%, msu, %folderpackages%
    call :CompareWithPowershell %packages%, %folderpackages%, %integratepackages%
    if exist %integratepackages% call :IntegratePackages %integratepackages%, %updates4%, msu, %mountfolder%
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    )
    echo %%a | find "w81u1x64" 1>nul && (
    call :CheckMountedWIM %mountfolder%, %packages%
    call :CheckUpdateFolder %updates5%, msu, %folderpackages%
    call :CompareWithPowershell %packages%, %folderpackages%, %integratepackages%
    if exist %integratepackages% call :IntegratePackages %integratepackages%, %updates5%, msu, %mountfolder%
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    )
    echo %%a | find "w81u1x86" 1>nul && (
    call :CheckMountedWIM %mountfolder%, %packages%
    call :CheckUpdateFolder %updates6%, msu, %folderpackages%
    call :CompareWithPowershell %packages%, %folderpackages%, %integratepackages%
    if exist %integratepackages% call :IntegratePackages %integratepackages%, %updates6%, msu, %mountfolder%
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    )
    DISM /unmount-wim /mountdir:"%mountfolder%" /commit
    echo %%a>>%textfile%
    )
    :================================================================================================================
    ::===============================================================================================================
    :: MERGE SINGLE WIMS TO f:\master\install.wim
    echo:
    echo ===============================================================================
    echo Exporting to f:\master
    echo ===============================================================================
    echo:
    for /f "tokens=* delims= " %%a in (%deltafile%) do (
    echo Exporting %%a to f:\master\install.wim
    echo:
    Dism /Export-Image /SourceImageFile:"%wimfolder%\%%a.wim" /SourceIndex:1 /DestinationImageFile:"%masterfolder%\install.wim" /compress:max /CheckIntegrity
    )
    if exist %wimfile% del /s /q %wimfile% >nul
    if exist %deltafile% del /s /q %deltafile% >nul
    :================================================================================================================
    ::===============================================================================================================
    :: SPLIT MASTER WIM
    echo:
    echo ===============================================================================
    echo Splitting WIM to f:\sources
    echo ===============================================================================
    echo:
    Dism /Split-Image /ImageFile:"%masterfolder%\install.wim" /SWMFile:"%sourcefolder%\install.swm" /FileSize:256
    :================================================================================================================
    ::===============================================================================================================
    :: END PROCEDURE
    :: script will pause here
    echo.
    echo ===============================================================================
    echo DONE ...
    echo ===============================================================================
    echo.
    pause
    :================================================================================================================
    ::===============================================================================================================
    :: EXIT
    endlocal
    exit
    :================================================================================================================
    ::===============================================================================================================
    ::COMPARE LISTS WITH POWERSHELL
    :CompareWithPowershell
    for /f "tokens=* delims= " %%a in ('"powershell Compare-Object -ReferenceObject (Get-Content %~1) -DifferenceObject (Get-Content %~2) ^| where-object SideIndicator -eq =^> ^| select -Expand InputObject ^| fl"') do echo %%a>>%~3
    goto:eof
    ::===============================================================================================================
    ::CHECK FOLDER
    :CheckFolder
    for /r "%~1" %%a in (*.%~2) do echo %%~na>>%~3
    goto:eof
    ::===============================================================================================================
    ::CHECK FOLDER II
    :CheckUpdateFolder
    for /r "%~1" %%i in (*.%~2) do (
        set "updatetemp=%%~ni"
        for /f "tokens=2 delims=-" %%m in ("!updatetemp!") do echo %%m>>%~3
    )
    goto:eof
    ::===============================================================================================================
    ::CHECK WIM
    :CheckMountedWIM
    if exist %wimfile% del /s /q %wimfile% >nul
    for /f "tokens=5,6,7 delims=:_~ " %%a in ('"dism /image:%~1 /Get-Packages /english" ^| findstr /i Package') do if not %%a equ amd64 if not %%a equ x86 echo %%a>>%~2
    goto:eof
    ::===============================================================================================================
    ::INTEGRATE PACKAGES
    :IntegratePackages
    for /f "tokens=* delims= " %%a in (%~1) do (
    for /r "%~2" %%i in (*.%~3) do (
    echo %%~ni | findstr /i "%%a" 1>nul && DISM /image:%~4 /Add-Package /PackagePath:%%i
    ))
    if exist %packages% del /s /q %packages% >nul
    if exist %folderpackages% del /s /q %folderpackages% >nul
    if exist %integratepackages% del /s /q %integratepackages% >nul
    goto:eof
     
  3. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    #183 s1ave77, Dec 4, 2014
    Last edited: Dec 4, 2014
    Hmm ... seems a file , should be deltafile, contains powershell error output instead of the WIM names, so dism is failing on that.

    Delete all but not exclusions.txt, so the script will create them freshly and test again. If it still fails we need to check wimfile, textfile and deltafile.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    #184 hypedave, Dec 4, 2014
    Last edited by a moderator: Apr 20, 2017
    (OP)
    See here is the weird thing. I deleted all files accept for the exclusions file and it ran successfully. First run only text file that was present was exlcusions.txt. Do I not need to worry about manually creating that file? Windows Update weirdness. I took a really old image from September and it only required one update from the general folder. It did not process any from the Hotfix and Security folder. All updates are current using the WHDownloader tool.

    Also this is what the first line of exclusions.txt looks like
    Code:
    Compare-Object : Cannot bind argument to parameter 'ReferenceObject' because it is null.
     
  5. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    #185 s1ave77, Dec 4, 2014
    Last edited: Dec 4, 2014
    #1. When it failed, only textfile, wimfile and deltafile should be present.

    #. Check file naming scheme of the update files, the script assumes them being like 'Windows8.1-KBXXXXXXX-x64' (with optional '-vX version string).
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    The MSU files are named like this

    Windows8.1-KB0000000-x64.msu
    Windows8.1-KB0000000-vX-x64
    Windows8.1-KB2976978-vX-x64_2c929c86c75c2ae2c7e5ddafb841e7be4b92efc8

    replace X with real actual numbers.


    Something really funky is going on with the exclusions.txt file. If I start from scratch with a blank txt exclusions.txt file it errors out. I then go a delete the other generated txt files, execute again it works. What am I doing wrong?
     
  7. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    #187 s1ave77, Dec 4, 2014
    Last edited: Dec 4, 2014
    Yeah, should have mentioned that textfile cannot be empty in our scenario, that case isn't covered and therefor will screw the rest.



    EDIT:
    When it runs, we should add a little menu, you're right with this :good3:. Options to start a new Process, set new data to the variables and delete the temp files :cool2:.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    Okay so i'm not loosing my mind here. Okay well do you want me to layout a design for the menu?
     
  9. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    Yep, if you like to do, do it :good3:.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    Well im gonna need to wrap my head around the menu concept idea I have in mind. I'm so amazed by Murphy's diskpart script that I am trying to build it around that flow.
     
  11. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    How's it coming? Haven't had a chance to do a menu?
     
  12. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    Sorry, don't get this ... please elaborate.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    Ok memory refresh :)

     
  14. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    And what's the current problem :g:?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  15. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    If the exclusions.txt file is completely blank then it would throw errors during script execution. For this the script would have to be modified to clean up the environment before executing the script. As a quick fix I made an exclusion.txt file with one content on the first line. It starts fine.
     
  16. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    Ahhh, needed a moment to catch up. Atm working on WU TOOL and my own, will check out solution aferwards.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    Ok sounds like a plan. I need to look into some bigger hardware. It took almost 12hrs to backup 43 wim files and then another 8hrs to mount and inject updates.
     
  18. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    Yeah, i know :D. Exporting WIMs needs lots of RAM, CPU power for delta processing (first index for similiar systems i.e. Win 8.1 Core/Pro/ProWMC is complete where the others do only contain the differences as hardlinks [afaik]) and a fast HDD or better SSD/RevoDrive. The Mount and Servicing mostly needs only a fast drive.


    EDIT: Best stabilty test i found so far is WIM<>ESD conversion :cool2:.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. hypedave

    hypedave MDL Member

    Oct 14, 2014
    130
    30
    10
    True, but doesn't it also require a lot of CPU to convert wim to ESD? I've thought about changing the path of the mount folder to one of my SSD drives to see if that speeds things up. I'm moving the files now from my iscsi lun to a regular netword raid 10 share. I tried to send you a PM but your box is full.
     
  20. s1ave77

    s1ave77 Has left at his own request

    Aug 15, 2012
    16,092
    24,400
    340
    Box should be empty enough now :D.

    Yep, that's why i said stability test. Conversion uses all available CPU power and as much RAM as WIM is big, needing ~25 mins for updated Win 8.1 WIM.

    HDD speed isn't a point here, as tested SSD doesn't speed this up.

    Unfortunately writing huge amounts of data, while mounting and unmounting WIM, is the death sentence for SSDs so far. No clue if a Revo Drive will stand this torture a while :g:.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...