Working Deduplication files for Microsoft Windows 10

Discussion in 'Windows 10' started by dreamss, Oct 4, 2014.

  1. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    757
    701
    30
  2. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    Was anyone able to resolve this?
    I have 2 volumes using DeDupe. After upgrading to the latest DeDup build, one of the volumes works, the other fails with the same errors as above.
     
  3. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    757
    701
    30
    It seems to be hit or miss as far as Data Deduplication running on 19041/2. Bear in mind, that running this Windows Server feature on Client builds is an unsupported scenario.
     
  4. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    Thanks bfoos - Having this retrofitted into W10 is great work, and before using it i understood it would be far from a perfect fit, so no complaints here.

    I have some inklings of whats going wrong and will do some further testing. Luckily, the volume that is failing for me is a removable USB drive so i can move it to a Windows Server 2016 machine with de-dupe running, if needed.
     
  5. Mamabun

    Mamabun MDL Novice

    Aug 13, 2016
    1
    0
    0
    It seems dedup no longer works on removable drives as build 2004. Looks like it's enforcing the fixed disk/non-removable disk requirement now. It works fine on my E: drive which is internal but not my P: drive which is connected using USB 3
     
  6. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    #646 King_Frat, Jul 31, 2020
    Last edited: Jul 31, 2020
    So done some testing ... In summary I found even the latest build of Windows Server 2016 was unable to access the duplicated files including do an un-optimize, full scrub, etc. However, Server 2019 was able to access the deduped files.

    I suspect one of two causes:
    A) The libraries used in the Windows10 1909 dedup build possibly came from a Server 2019 distro(??) and are not downward compatible to the 2016/10 builds - Im not sure if this is even possible, but just a thought.
    B) As Mamabun states, its possible this could be something to do with removable media status. I say this as for my 2019 Server test, I mounted the USB drive as a 'SCSI Physical' in Hyper-V. I can not test the same approach with 2016 Server as that machine is a physical for me.

    FYI - I did try making a disk image of the USB and mounting it as a virtual drive in Win10, but I was still unable to access the compressed files. Im not sure if in that scenario Win10 still handles the virtual disk like removable media.

    Bottom line is that for anyone getting the below errors, your inaccessible files are not lost, but you likely need Server 2019 to access them.

    Windows 10 error: [Error 0x80565309, "A required filter driver is either not installed, not loaded, or not ready for service."]
    Server 2016 error: [Error 0x80070519, "The revision level is unknown."]

    Hope this helps.

    **UPDATE**

    So in trying further testing I realized I had a 2019 VM defined on my Win10 PC running under VirtualBOX which already had dedup enabled. I have literally dozens of trial/test Virtual Servers running on a multitude of Hypervisors and cant recall what I created or was using that one for. I haven't used that VM in a long time but it is possible that at some point I booted that VM and gave it access to the physical USB devices from the W10 host that had dedup running.
    Maybe, once the standard Server2019 dedup job kicked off, it updated that volume so it was no longer compatible with the Dedup build from Server 2016. (makes the [Error 0x80070519, "The revision level is unknown."] error message reported by Server 2016, seem to make sense now!

    Bottom line is that my problems could be self inflicted!

    Not sure if the others on this forum that are also reporting issues with the newest dedup package, have also had similar circumstances and had their dedup'd volumes accessed by Win Server 2019 ... ?
     
  7. King_Frat

    King_Frat MDL Novice

    Mar 11, 2018
    4
    0
    0
    #647 King_Frat, Jul 31, 2020
    Last edited: Jul 31, 2020
    Found this on MSFT's website regarding Data Dedup Interoperability. The Robocopy bit caught my attention as i had no idea (i use it all the time) so thought I'd share...

    Windows 10 (client OS)
    Data Deduplication is not supported on Windows 10. There are several popular blog posts in the Windows community describing how to remove the binaries from Windows Server 2016 and install on Windows 10, but this scenario has not been validated as part of the development of Data Deduplication. Vote for this item for Windows 10 vNext on the Windows Server Storage UserVoice.

    Windows Search
    Windows Search doesn't support Data Deduplication. Data Deduplication uses reparse points, which Windows Search can't index, so Windows Search skips all deduplicated files, excluding them from the index. As a result, search results might be incomplete for deduplicated volumes. Vote for this item for Windows Server vNext on the Windows Server Storage UserVoice.

    Robocopy
    Running Robocopy with Data Deduplication is not recommended because certain Robocopy commands can corrupt the Chunk Store. The Chunk Store is stored in the System Volume Information folder for a volume. If the folder is deleted, the optimized files (reparse points) that are copied from the source volume become corrupted because the data chunks are not copied to the destination volume.

    Unfortunately, I don't have enough posts in this forum to include links to the article for the UserVoice pages for voting
     
  8. Rainmaker52

    Rainmaker52 MDL Novice

    Mar 25, 2018
    14
    3
    0
    I'm currently running W10 19041.421 with the package mentioned above. The binaries in there seem to be for 19041.1.
    Could there have been a 2004 update which broke deduplication for these binaries? Would it be possible to update your W2019 VM to a newer revision and extract those files?

    Or, could you share how to get those files? I'll create a VM and update it myself.
     
  9. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    757
    701
    30
    #650 bfoos, Aug 7, 2020
    Last edited: Aug 7, 2020
    The dedup files on server, regardless of CU are always .1 (there are no newer binaries outside of vNext insider builds) and never updated until a new first half SAC build comes along. As I have stated numerous times in this thread. Data Deduplication is part of a Windows Server, File Server role and not intended to be installed on client builds and certainly not supported.

    As far as an CU breaking Data Deduplication? Highly unlikely as Windows Server and Client builds share the same CU's. If a CU broke things on Server, MS would likely fix it and the same fix would be installed on Client's with the dedup package installed.
     
  10. Rainmaker52

    Rainmaker52 MDL Novice

    Mar 25, 2018
    14
    3
    0
    Yes, I know it's not supported. But it "used to work" and now I'm sort of stuck with ~ 40 unreadable Hyper-V VMs hosted on a deduplicated volume, without any diskspace to unoptimize them. I can probably map the volume raw to a W2019 VM, and try and read the data that way (data is not lost). I know running dedup on W10 was a risk, and this was a risk I intentionally took.

    As per the replies in this thread, it works for some people, but not for others. I'm trying to think of reasons why it would not work, and maybe, possibly, fix it. Also to have my own VMs back of course :).

    Hmm, this is probably true if I updated to 19041.1, installed dedup pack, and then updated to 19041.421. But, what if I went from an older version straight to 19041.421? The CU updates would not "know" the dedup binaries would be there, and they would not be updated.
     
  11. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    757
    701
    30
    #652 bfoos, Aug 10, 2020
    Last edited: Aug 10, 2020
    The dedup binaries don't get updated by CU's. The only time new dedup binaries are released is when first half SAC builds drop (again, outside of vNext insider builds). And being a Server specific feature, MS isn't tweaking it on a regular basis and dropping fixes in CU's. The only time they would likely consider doing that would be for showstopping bugs and I am guessing that these issues simply aren't affecting Server customers, otherwise they would be addressing it.

    If you went from say 17763 or 18362 to 19041, installing the 19041 dedup pack is all that is required. As for why it works for some people (including me) and not for others? I have no idea and nobody has really posted any kind of information that could be used to debug the issue. So... ¯\_(ツ)_/¯

    Something to keep in mind is that the required cab files are pulled directly from Server builds with no modifications. Also, cumulative updates are just that... Cumulative. As in Each new CU supersedes the prior CU by including its fixes along with whatever new bits the new CU addresses, so your supposition is based on false logic.
     
  12. Rainmaker52

    Rainmaker52 MDL Novice

    Mar 25, 2018
    14
    3
    0
    OK. So I read your reply too late, and tried it anyway.

    I've now updated my W10 to 19041.423 (coming from 19041.421). .423 is currently available as an "Optional update" via Windows update, as this CU is in preview.
    Tried running a dedup job, still immediately returned "Failed".

    However, looking in Event Viewer, in Microsoft-Windows-Deduplication/Operational, it does show a different error.
    421:
    423:
    fltmc still does not show an instance for the filter driver Dedup for my D: drive though. It does show it for 2 other drives, which do not have deduplication enabled:
    Obviously, I've tried re-enabling dedup on the D: drive, but with no effect. Surprising is also that PowerShell doesn't throw an error (as it does when I start-dedupjob for L:\), but the error is only shown in Event Viewer.
     
  13. Rainmaker52

    Rainmaker52 MDL Novice

    Mar 25, 2018
    14
    3
    0
    #655 Rainmaker52, Aug 10, 2020
    Last edited: Aug 11, 2020
    I couldn't troubleshoot much longer, and had to start recovery due to a particular VM I'll need tomorrow.

    I thought it would be interesting on how to recover. I've tried reading the disk using 2019 vNext 20185, but I couldn't read the files. Instead, I had to go back to a VM running 10.0.17763.737. Probably newer versions will work as well, but this one is easily obtained from the Windows Server evaluation page.

    To recover my files, I basically setup a Windows Server evaluation VM. Then I created a new VHD file using Disk Management, created a volume and assigned a drive letter.

    Then, I used dd for Windows to copy the partition block for block;
    Code:
    dd if="\\.\D:" of="\\.\H:" bs=4M
    This VHDX (which I needed to run from an external harddisk) can then be attached to the VM. I did have to Install--WindowsFeature -Name FS-Data-Deduplication on the VM.
    Then I basically made a UNC connection between my host and my guest, and copied the files over after formatting my original deduplicated volume.

    This is rather slow, but again, I needed the data.

    I'll enable deduplication on my newly formatted volume after I'm done with the VM.

    I'm not sure when I originally enabled dedup, but it might be as long ago as 2017. It almost seems as if the newer versions of Windows are no longer compatible with this layout. I happen to know a number of intermediate versions altered the way Windows handles reparse points. Also to allow for better support for containers.

    *edit; after formatting the volume and re-enabling deduplication with the dedup package in this thread, everything works again. It really does seem there is incompatibility between the volume format and the recent versions of Windows.
     
  14. btgdelta

    btgdelta MDL Novice

    Jul 14, 2014
    2
    0
    0
    Hey bfoos,
    just wanted to share an interesting experience with dedup under Windows 10 and have your thoughts.
    You probably won't remember, but I posted a couple of years ago about having bluescreens caused by dedup.sys. After a while I found out those bluescreens were caused by file access to certain files. You could not read those files, but you could delete them (so probably some kind of chunk storage problem). I kinda learned to live with it, thinking that some a windows reinstall or update would fix this issue.
    Now last week I finally got a new system, installed a fresh OS, installed the dedup packets for 2004, spun up the deduped drive and got the same bluescreens. Fun part: scrubbing/garbageCollection didn't help but worked, unoptimization produced the same bluescreen. This annoyed me in a major way, I installed Server 2019 on the same system, added the dedup feature and was able to read all files without a problem. Unoptimization works like a charm too.
    Any ideas what could be causing this major difference in handling the drive?
     
  15. bfoos

    bfoos MDL Guide Dog

    Jun 15, 2008
    757
    701
    30
    I'm beginning to wonder if the chunklibrary and vdsinterop packages included in the client builds of 19041/2 are to blame here.
     
  16. Rainmaker52

    Rainmaker52 MDL Novice

    Mar 25, 2018
    14
    3
    0
    That's actually I had something which I had in my previous reply, before editing it :)

    Interesting development here as well; the volume worked fine. Now, 48 hours later (probably, 1 deduplication schedule run later), it is no longer accessible. With the same "filter driver" error.

    The plot deepens.
     
  17. Cronustx

    Cronustx MDL Novice

    Oct 20, 2015
    1
    0
    0
    Anyone get this to work yet? 19041.450. Same filter driver error with mine.
     
  18. smashed

    smashed MDL Junior Member

    Feb 6, 2008
    56
    15
    0
    I stopped using Dedup on Windows 10 2004 because it's unreliable and I have the space available right now.