Working Deduplication files for Microsoft Windows 10

Discussion in 'Windows 10' started by dreamss, Oct 4, 2014.

  1. robnitro

    robnitro MDL Novice

    Jun 11, 2010
    3
    0
    0
    #401 robnitro, Apr 16, 2018
    Last edited: Apr 16, 2018
    You are correct!
    I just checked using the windows 10 1709 boot dvd. Files could not be read that were deduped. Some files were readable though. I was checking via copy command on command line from the dvd recovery.

    Some other oddities with dedupe in NTFS.

    If you compressed files using xpress or LZX, dedupe doesn't handle those files.
    Uncompressed or NTFS compressed files are deduped.

    Case test on 16GB test partition on SSD:
    5GB of game files.
    LZX compressed, used 2.2 GB (13.8 GB free)
    Dedupe of that, no change in free space.

    (garbage collection to clean up unused dedupe files)

    NTFS compressed the files, used 2.5 GB (13.5GB free)
    Dedupe of that, Free space went up to 14.1 GB.

    (Dedupe alone without NTFS compression gave me 13.9 GB free, so the NTFS compression does help a lot)
     
  2. abuabdo

    abuabdo MDL Novice

    Oct 29, 2012
    13
    1
    0
    #402 abuabdo, Apr 16, 2018
    Last edited: Apr 16, 2018
    can some1 confirm if Dedup16299 is working on windows 10 1709 16299.371
    When I try to start a job, nothing happens either stays in queue or initialize
     
  3. kibkalo

    kibkalo MDL Member

    Sep 8, 2009
    124
    10
    10
    Wrong
    Any Win10 would be able to read duduplicated volume. Even my MacOS can read it. They would write all blocks as new and won’t run despite tasks of course.
     
  4. Tito

    Tito Super Mod / Adviser
    Staff Member

    Nov 30, 2009
    17,707
    15,968
    340
    Please read this post - any OS (macOS inclusive) that comes with decent NTFS read(/write) driver can read a deduplicated volume, but only non-deduplicated portions can be read properly.
     
  5. Wishbringer

    Wishbringer MDL Junior Member

    Aug 21, 2009
    82
    26
    0
    In 17134.1 the known Dedup_16299.15_Dism Package doesn't work anymore.
    Deduped files can't be accessed.
     
  6. maeglin

    maeglin MDL Novice

    Dec 14, 2012
    9
    1
    0
    Thanks for the heads up.
    Anyone with a successful setup? In case above was a system specific issue and not a general incompatibility.
     
  7. intergalactic

    intergalactic MDL Novice

    Sep 5, 2014
    5
    5
    0
    I can confirm this. Just upgraded to 1803 and dedup doesn't work anymore. This was expected. Hope that someday Windows 10 has native dedup.
     
  8. intergalactic

    intergalactic MDL Novice

    Sep 5, 2014
    5
    5
    0
    Windows Server version 1803 will be generally available on May 7. I think we can get new working dedup package for Windows 10 1803 from there.
     
  9. cinescope

    cinescope MDL Novice

    Apr 21, 2018
    33
    14
    0
    Bear in mind that 16299+ will convert any existing deduped disk to its own newer format, which is NOT backward compatible with older duplication. So if you need to dual boot or you need to rollback, you're screwed, just like W10 dedup V.S. W8.x dedup.

    It's really a shame that MS never said a word or a warning about those forced format upgrades.
     
  10. ebegemot

    ebegemot MDL Novice

    Aug 5, 2016
    3
    7
    0
    upgraded to 1803 as well and got non-working dedup
    can't wait till WS RS4 release so I just installed Dedup_16299_Manual.7z on top of W10 1803. and it does work. not DISM package, but manual package.
     
  11. intergalactic

    intergalactic MDL Novice

    Sep 5, 2014
    5
    5
    0
    Thanks for the tip. I installed it as well and it seems to be working.

    Link for the post where 16299 manual package download links can be found

    https://forums.mydigitallife.net/th...crosoft-windows-10.57189/page-19#post-1380333
     
  12. alessio.71

    alessio.71 MDL Novice

    Aug 24, 2011
    22
    4
    0
    Thank you! This worked for me too
    Alex
     
  13. chtse53

    chtse53 MDL Novice

    Aug 23, 2015
    8
    0
    0
    Have you tested that the deduplication processes were effectively executed? That would be great if the 16299 package is fully compatible with 17134.
     
  14. alessio.71

    alessio.71 MDL Novice

    Aug 24, 2011
    22
    4
    0
    I launched an optimization and a garbage collection job and did run without problems, on Monday I'll check if the deduplication job did "really" something (I haven't check if the new files did get really duplicated).
    Alex
     
  15. DiproXin

    DiproXin MDL Novice

    Apr 8, 2017
    1
    2
    0
    In my case, installing the 16299 manual package on 1803 has restored reading access. However launching optimization or garbage collection jobs didn't have any effects. (Deleting +200Gb of files didn't free any space on my volume)
     
  16. chtse53

    chtse53 MDL Novice

    Aug 23, 2015
    8
    0
    0
    I created a duplicate file copy and set minimumfileagedays to 0 and started an optimization job. The jobstatus showed "initialization" and then ended within a minute or two. get-dedupstatus showed no new deduplication was achieved. Apparently, installing 16229 package on 17134 only restores the access of the deduped files without other dedup functions.
     
  17. ebegemot

    ebegemot MDL Novice

    Aug 5, 2016
    3
    7
    0
    could be. I use it for Hyper-V, so not that much changes I have.

    anyway, it's better than nothing until WS RS4 files will be released to public.
     
  18. alessio.71

    alessio.71 MDL Novice

    Aug 24, 2011
    22
    4
    0
    I confirm, read only, no new deduplication:

    PS C:\> Get-DedupVolume |Start-DedupJob -wait -Type Optimization

    Type ScheduleType StartTime Progress State Volume
    ---- ------------ --------- -------- ----- ------
    Optimization Manual 0 % Failed E:
     
  19. abbodi1406

    abbodi1406 MDL KB0000001

    Feb 19, 2011
    10,503
    42,475
    340
  20. ebegemot

    ebegemot MDL Novice

    Aug 5, 2016
    3
    7
    0
    installer.cmd still checks for the old version
    for /f "tokens=6 delims=[]. " %%G in ('ver') do set winbuild=%%G
    if %winbuild% NEQ 16299 (
    set MESSAGE=ERROR: This package is for Windows 10 ver 1803 - Build 17134
    goto :END

    just changed the build number and optimization seems to be working now