[DISCUSSION] UUP dump - download Windows 10/11 UUPs with ease

Discussion in 'MDL Projects and Applications' started by whatever127, Sep 2, 2017.

  1. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    you know what, separate fileinfo and packs scripts are better because I can just add an option to call packs generation when it's done
    "generate metadata" on the website is stable enough

    I can't write GUIs and personally I think it's unnecessary because I don't like adding more clicks to this, but anyway maybe @abbodi1406 can make something
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. Enthousiast

    Enthousiast MDL Tester

    Oct 30, 2009
    54,966
    125,939
    450
    I mean a setup gui, where you can set all up and with GUI i mean something like the KMS_VL_ALL AiO cmd.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    #5163 Paul Mercer, Sep 18, 2023
    Last edited: Sep 18, 2023
    people got lost in two scripts on Discord server, I better not split fileinfo+packs script

    did some testing with the 1.5.1 version:
    1. got 49 builds in 1 minute
    2. metadata was then generated in 6 minutes

    looks fine to me
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. jmtella

    jmtella MDL Senior Member

    Nov 18, 2012
    295
    166
    10
    One question: if I have the server started, does it automatically refresh new builds as well as uudump.net?

    If not, what do I have to do for this?
     
  5. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    only uupdump-get-fileinfo+packs.cmd can get new builds, on Linux I'm using cron to autorun it once in a day (ofc script is ported to Linux shell script)
    on Windows you can use Task Scheduler, but I never tried it myself
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. redsunset

    redsunset MDL Member

    Mar 16, 2022
    198
    98
    10
    I'm testing an iso made with SkipApps = 1
    Sideloaded Photos, Paint, Clock and WSA (and their respective dependencies) with Powershell. While WSA launches and works without issues, the other 3 don't. Is there some kind of hidden dependency or registry that could prevent launching those 3 apps?
     
  7. Carlos Detweiller

    Carlos Detweiller Emperor of Ice-Cream
    Staff Member

    Dec 21, 2012
    8,076
    10,298
    270
    Code:
    [2023-09-18 18:06:44 UTC] UUP dump v3.62.2
    [2023-09-18 18:06:44 UTC] UUP dump API v1.40.2
    [2023-09-18 18:06:44 UTC] Parsing database info...
    [2023-09-18 18:06:44 UTC] Done parsing database info.
    [2023-09-18 18:06:44 UTC] Generating packs for 95ef7fd2-51b3-473d-aead-7689435f4c08...
    [2023-09-18 18:06:44 UTC] Fetching information from the server...
    [2023-09-18 18:06:44 UTC] Information has been successfully fetched.
    [2023-09-18 18:06:44 UTC] Parsing information...
    [2023-09-18 18:06:44 UTC] An error has occurred
    [2023-09-18 18:06:44 UTC] Generating packs for dfe05d0e-cb89-4c7c-827e-e09385a7f330...
    [2023-09-18 18:06:44 UTC] Fetching information from the server...
    [2023-09-18 18:06:45 UTC] Information has been successfully fetched.
    [2023-09-18 18:06:45 UTC] Parsing information...
    [2023-09-18 18:06:45 UTC] An error has occurred
    [2023-09-18 18:06:45 UTC] Generating packs for 45974967-f2bf-429c-9f5b-cd2e7b9c6c6a...
    [2023-09-18 18:06:45 UTC] Fetching information from the server...
    [2023-09-18 18:06:45 UTC] Information has been successfully fetched.
    [2023-09-18 18:06:45 UTC] Parsing information...
    [2023-09-18 18:06:45 UTC] An error has occurred
    [2023-09-18 18:06:45 UTC] Generating packs for 08e36e9c-738f-46cc-811e-6f62260616f9...
    [2023-09-18 18:06:45 UTC] Fetching information from the server...
    [2023-09-18 18:06:45 UTC] Information has been successfully fetched.
    [2023-09-18 18:06:45 UTC] Parsing information...
    [2023-09-18 18:06:45 UTC] An error has occurred
    [2023-09-18 18:06:45 UTC] Generating packs for 1f27828e-0fad-4d53-8e5f-ff389a7a88d2...
    [2023-09-18 18:06:45 UTC] Fetching information from the server...
    [2023-09-18 18:06:46 UTC] Information has been successfully fetched.
    [2023-09-18 18:06:46 UTC] Parsing information...
    [2023-09-18 18:06:46 UTC] An error has occurred
    New method chokes on old builds in the database. The errors didn't get away until I manually deleted those. How about offering to delete such obsolete builds right away?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    well, I already merged packsgen with fetchdetect in v1.5.0, v1.5.1 gives the same results in a different order
    in your case packs weren't generated for some reason, shouldn't happen again with v1.5.1 (packsgen -> fetchupd -> packsgen)

    about deleting old builds - we can't check for sure, sometimes WU throws errors for no reason
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. Carlos Detweiller

    Carlos Detweiller Emperor of Ice-Cream
    Staff Member

    Dec 21, 2012
    8,076
    10,298
    270
    These error messages did not get away on their own and occurred two times on every run, once before and once after. Probably that's because the builds are removed from MS servers.

    OK, I never used 1.5.0.

    Also noticed you moved the database. Unfortunately only after it started to build a completely new one. Old db was in .\uup-dump\website\, new is in .\uup-dump\.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. Carlos Detweiller

    Carlos Detweiller Emperor of Ice-Cream
    Staff Member

    Dec 21, 2012
    8,076
    10,298
    270
    The old version's script generated packs only for the entries it downloaded fileinfo for, so, there wasn't any need to run the dedicated packs script. Maybe upon download, they had no packs or there was an error at that time. Removed now, should not happen again, packsgen now runs plenty.

    Maybe, but a warning or one-time migration to the new location (moving packs and fileinfo directories) would have been nice.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    #5172 Paul Mercer, Sep 18, 2023
    Last edited: Sep 18, 2023
    technically there's no difference:
    fetchdetect - it's getting fileinfo and packs for each build one by one, should get packs if it was missing
    packsgen - it's getting packs for all builds in fileinfo
    fetchupd - it's getting fileinfo only

    previously it was fetchdetect only in one script and packsgen in the second one
    the catch: you may miss packs due to some WU error if fetchdetect didn't do its job correctly

    now it's one script in 3 steps:
    1. packsgen - generates missing packs, takes less than a second to parce db on empty db
    2. fetchupd - very fast operation to get all builds (fileinfo), so users can see the result immediately
    3. packsgen - generates packs for recently added builds
    and it regenerates metadata on each run

    new way is more simple, safe and takes the same time on new DBs
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. Carlos Detweiller

    Carlos Detweiller Emperor of Ice-Cream
    Staff Member

    Dec 21, 2012
    8,076
    10,298
    270
    I used the old "uupdump-get-fileinfo+packs.cmd", it downloaded the fileinfo for the newly detected builds and then immediately generated the packs for them. There was no general packsgen over all builds unless you used the dedicated script (which I never had to as the builds I wanted worked fine), otherwise the errors would have shown up already (some thing were from March).
    Now, it will even post-generate packs for the ones that failed earlier (which unfortunately happens).

    Just a note to the new feature, update-only packs. I tested it and the appropriate updates were fetched fine (no useless "baseless" stuff, cab over msu). However, there's one thing I miss, as I keep checksum lists. Could you generate a text file with the SHA1 of the downloaded updates? As it stands, I still have to visit the KB page to fetch the SHA1 sums from the list.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. abbodi1406

    abbodi1406 MDL KB0000001

    Feb 19, 2011
    18,610
    100,025
    340
    update-only download pack is just like the general download / download&convert packs
    none included sha1 checksum file by default

    however, the download script could be modified to keep the temporary aria2_script.*.txt which include the files hashes
     
  15. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. Carlos Detweiller

    Carlos Detweiller Emperor of Ice-Cream
    Staff Member

    Dec 21, 2012
    8,076
    10,298
    270
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    I just checked it and it's not deleting for me, it's still in the files folder
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. Paul Mercer

    Paul Mercer MDL Guru

    Apr 6, 2018
    2,084
    4,363
    90
    #5178 Paul Mercer, Sep 20, 2023
    Last edited: Sep 21, 2023
    Updated UUP dump local server to version 1.5.2

    Changelog:
    - All new AIO scripts by @abbodi1406
    - Added support to download checksums_xxxx.sha1 file



    Once again thanks to @abbodi1406, I hope this time everyone will be satisfied with AIO scripts

    If you do not prefer the AIO script then there are "uupdump-get.cmd" and "uupdump-run.cmd" scripts in the "scripts" folder, and these scripts also work in the root folder so you can move it

    "Remove failed builds" option is based on any WU error so it's pretty dangerous to me, use it with caution @Carlos Detweiller


    upd: internal MSIT channel is excluded from ALL option, please redownload
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. Enthousiast

    Enthousiast MDL Tester

    Oct 30, 2009
    54,966
    125,939
    450
    When he is nagged enough he will create new gems :D
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. jmtella

    jmtella MDL Senior Member

    Nov 18, 2012
    295
    166
    10
    upload_2023-9-20_19-2-50.png