Working Deduplication files for Microsoft Windows 10

Discussion in 'Windows 10' started by dreamss, Oct 4, 2014.

  1. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    747
    684
    30
  2. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    Was anyone able to resolve this?
    I have 2 volumes using DeDupe. After upgrading to the latest DeDup build, one of the volumes works, the other fails with the same errors as above.
     
  3. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    747
    684
    30
    It seems to be hit or miss as far as Data Deduplication running on 19041/2. Bear in mind, that running this Windows Server feature on Client builds is an unsupported scenario.
     
  4. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    Thanks bfoos - Having this retrofitted into W10 is great work, and before using it i understood it would be far from a perfect fit, so no complaints here.

    I have some inklings of whats going wrong and will do some further testing. Luckily, the volume that is failing for me is a removable USB drive so i can move it to a Windows Server 2016 machine with de-dupe running, if needed.
     
  5. Mamabun

    Mamabun MDL Novice

    Aug 13, 2016
    1
    0
    0
    It seems dedup no longer works on removable drives as build 2004. Looks like it's enforcing the fixed disk/non-removable disk requirement now. It works fine on my E: drive which is internal but not my P: drive which is connected using USB 3
     
  6. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    #646 King_Frat, Jul 31, 2020
    Last edited: Jul 31, 2020
    So done some testing ... In summary I found even the latest build of Windows Server 2016 was unable to access the duplicated files including do an un-optimize, full scrub, etc. However, Server 2019 was able to access the deduped files.

    I suspect one of two causes:
    A) The libraries used in the Windows10 1909 dedup build possibly came from a Server 2019 distro(??) and are not downward compatible to the 2016/10 builds - Im not sure if this is even possible, but just a thought.
    B) As Mamabun states, its possible this could be something to do with removable media status. I say this as for my 2019 Server test, I mounted the USB drive as a 'SCSI Physical' in Hyper-V. I can not test the same approach with 2016 Server as that machine is a physical for me.

    FYI - I did try making a disk image of the USB and mounting it as a virtual drive in Win10, but I was still unable to access the compressed files. Im not sure if in that scenario Win10 still handles the virtual disk like removable media.

    Bottom line is that for anyone getting the below errors, your inaccessible files are not lost, but you likely need Server 2019 to access them.

    Windows 10 error: [Error 0x80565309, "A required filter driver is either not installed, not loaded, or not ready for service."]
    Server 2016 error: [Error 0x80070519, "The revision level is unknown."]

    Hope this helps.

    **UPDATE**

    So in trying further testing I realized I had a 2019 VM defined on my Win10 PC running under VirtualBOX which already had dedup enabled. I have literally dozens of trial/test Virtual Servers running on a multitude of Hypervisors and cant recall what I created or was using that one for. I haven't used that VM in a long time but it is possible that at some point I booted that VM and gave it access to the physical USB devices from the W10 host that had dedup running.
    Maybe, once the standard Server2019 dedup job kicked off, it updated that volume so it was no longer compatible with the Dedup build from Server 2016. (makes the [Error 0x80070519, "The revision level is unknown."] error message reported by Server 2016, seem to make sense now!

    Bottom line is that my problems could be self inflicted!

    Not sure if the others on this forum that are also reporting issues with the newest dedup package, have also had similar circumstances and had their dedup'd volumes accessed by Win Server 2019 ... ?
     
  7. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    #647 King_Frat, Jul 31, 2020
    Last edited: Jul 31, 2020
    Found this on MSFT's website regarding Data Dedup Interoperability. The Robocopy bit caught my attention as i had no idea (i use it all the time) so thought I'd share...

    Windows 10 (client OS)
    Data Deduplication is not supported on Windows 10. There are several popular blog posts in the Windows community describing how to remove the binaries from Windows Server 2016 and install on Windows 10, but this scenario has not been validated as part of the development of Data Deduplication. Vote for this item for Windows 10 vNext on the Windows Server Storage UserVoice.

    Windows Search
    Windows Search doesn't support Data Deduplication. Data Deduplication uses reparse points, which Windows Search can't index, so Windows Search skips all deduplicated files, excluding them from the index. As a result, search results might be incomplete for deduplicated volumes. Vote for this item for Windows Server vNext on the Windows Server Storage UserVoice.

    Robocopy
    Running Robocopy with Data Deduplication is not recommended because certain Robocopy commands can corrupt the Chunk Store. The Chunk Store is stored in the System Volume Information folder for a volume. If the folder is deleted, the optimized files (reparse points) that are copied from the source volume become corrupted because the data chunks are not copied to the destination volume.

    Unfortunately, I don't have enough posts in this forum to include links to the article for the UserVoice pages for voting
     
  8. luzea

    luzea MDL Addicted

    Apr 25, 2019
    673
    697
    30
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. Rainmaker52

    Rainmaker52 MDL Novice

    Mar 25, 2018
    5
    2
    0
    I'm currently running W10 19041.421 with the package mentioned above. The binaries in there seem to be for 19041.1.
    Could there have been a 2004 update which broke deduplication for these binaries? Would it be possible to update your W2019 VM to a newer revision and extract those files?

    Or, could you share how to get those files? I'll create a VM and update it myself.
     
  10. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    747
    684
    30
    #650 bfoos, Aug 7, 2020 at 23:42
    Last edited: Aug 7, 2020 at 23:48
    The dedup files on server, regardless of CU are always .1 (there are no newer binaries outside of vNext insider builds) and never updated until a new first half SAC build comes along. As I have stated numerous times in this thread. Data Deduplication is part of a Windows Server, File Server role and not intended to be installed on client builds and certainly not supported.

    As far as an CU breaking Data Deduplication? Highly unlikely as Windows Server and Client builds share the same CU's. If a CU broke things on Server, MS would likely fix it and the same fix would be installed on Client's with the dedup package installed.