Working Deduplication files for Microsoft Windows 10

Discussion in 'Windows 10' started by dreamss, Oct 4, 2014.

  1. maeglin

    maeglin MDL Novice

    Dec 14, 2012
    10
    1
    0
    I have a weird problem with deduplication. I have installed the latest package "Dedup_10.0.14393.0" and I can access all deduped files without any problem, but I cannot dedup new files.
    OptimizedFiles and InPolicyFiles numbers stay the same, no matter I delete old files or add new files.
    What could possible be the problem?
    I have tried uninstalling and reinstalling the dedup package, but it did not solve the problem.
     
  2. halasz

    halasz MDL Member

    Jan 13, 2013
    116
    35
    10
    #302 halasz, Sep 12, 2016
    Last edited by a moderator: Apr 20, 2017
  3. TheKingDavid

    TheKingDavid MDL Novice

    Aug 22, 2016
    9
    1
    0
    #303 TheKingDavid, Sep 13, 2016
    Last edited: Sep 16, 2016
    +1 same problems here.
    Access to dedup files but nothing else is working.:confused:

    I'm using a french version. A clue !
     
  4. halasz

    halasz MDL Member

    Jan 13, 2013
    116
    35
    10
    Not sure - the package includes only en-US - but I'm using en-US and I can't dedup new files either.
     
  5. JJ2SMILY

    JJ2SMILY MDL Novice

    Mar 10, 2012
    24
    40
    0
    #305 JJ2SMILY, Sep 16, 2016
    Last edited by a moderator: Apr 20, 2017
    Possible Solution

    For those having issues with running Windows deduplication, try one of the followings. It might help to solve your issue.

    Asumming I'm running deduplication on d drive, so adjust d to whatever your drive letter or mounted folder is.


    1) Run the following command(Read-only mode) to check if there are any errors on your disk. If any, fix them.

    Code:
    chkdsk d:

    2) Run PowerShell as Administrator, and use this command.

    Code:
    get-dedupvolume | fl 
    Check the values of these entries: MinimumFileAgeDays, MinimumFileSize, NoCompress, ExcludeFolder, ExcludeFileType, and NoCompressionFileType to see if they are properly set.


    3) Run PowerShell as Administrator, and use this command.

    Code:
    get-dedupstatus | fl
    Check the last result and result message for optimization, garbagecollection, and scrubbing to see if there are any errors. If so, use the error number to find a solution on Internet.


    4) Run PowerShell as Administrator, and try the following command.

    Code:
    start-dedupjob -volume "d:" -type optimization -memory 20 -priority low -wait 
    If this fails, most likely you don't have enough system resources to complete the job. By my experience, more memory is required. So restart your computer and wait for a minute to complete the startup process, and then disable Windows Defender or any other antivirus/malware software. Also close unnecessary programs running in the background. Then run the above command again. Even if it fails again, give a few more tries.


    5) Run full garbagecollection and scrubbing, and then run optimization.
    Code:
    start-dedupjob -volume "d:" -type garbagecollection -full -wait
    start-dedupjob -volume "d:" -type scrubbing -full -wait
    start-dedupjob -volume "d:" -type optimization -memory 20 -wait

    6) Move some files to other drives to make more disk space for the optimization.



    Useful readings

    Code:
    Data Deduplication Overview
    https: //technet.microsoft.com/en-us/windows-server-docs/storage/data-deduplication/overview
    
    Deduplication Cmdlets
    https: //technet.microsoft.com/en-us/library/hh848450.aspx
    
    Troubleshooting Data Deduplication Corruptions
    http: //social.technet.microsoft.com/wiki/contents/articles/31038.troubleshooting-data-deduplication-corruptions.aspx

    Good luck!
     
  6. JJ2SMILY

    JJ2SMILY MDL Novice

    Mar 10, 2012
    24
    40
    0
    halasz,

    It doesn't seem the issue is the deduplication in your case. More like something else.

    You said you had Windows Server 2016 TP5 and then switched to Windows 10 Pro, right? Does the drive/files you're trying to run deduplication have write permission from your current OS/user? Check that first and if you still have the issue, try this.

    1) Create a vhdx file with the size 20GB on that drive. (Open up Computer Management and then right-click on Disk Management under Storage. Choose Create VHD.)
    2) Attach the vhdx file. (Almost the same steps in 1, but choose Attach VHD.)
    3) Copy a 2-3GB file to the attached vhdx file. (Any file is okay, but just watch out its size so that a plenty of space is left for the optimization.)
    4) Make a copy of that file within the vhdx file.
    4) Run the optimization against the vhdx file, not the drive.

    See if the job can be completed.

    Good luck!
     
  7. T-S

    T-S MDL Guru

    Dec 14, 2012
    3,984
    1,331
    120
    I tested the filter driver are working and the deduped files are accessible, but I haven't tried thoroughly the service part, being mostly on a real win server, but Initially I had the impression that the dedup part wasn't working as well. Then others reported it as working so I didn't investigate further, but is very likely there is something wrong/missing in the 14393 dedup package.
     
  8. halasz

    halasz MDL Member

    Jan 13, 2013
    116
    35
    10
    #309 halasz, Sep 18, 2016
    Last edited by a moderator: Apr 20, 2017
    I have write permission, yes. I tried making a vhdx as you describe but get the same 0x80565342 error
    Code:
    PS C:\Windows\system32> dir e:
    
        Directory: E:\
    
    Mode                LastWriteTime         Length Name
    ----                -------------         ------ ----
    -a----       17/09/2016     18:03     3382538240 Win10_1607_English_x32 - Copy.iso
    -a----       17/09/2016     18:03     3382538240 Win10_1607_English_x32.iso
    
    PS C:\Windows\system32> enable-dedupvolume e:
    Enabled            UsageType          SavedSpace           SavingsRate          Volume
    -------            ---------          ----------           -----------          ------
    True               Default            0 B                  0 %                  E:
    
    PS C:\Windows\system32> set-dedupvolume -volume e: -minimumfileagedays 0
    PS C:\Windows\system32> start-dedupjob -volume "e:" -type optimization -memory 20 -priority low -wait
    Type               ScheduleType       StartTime              Progress   State                  Volume
    ----               ------------       ---------              --------   -----                  ------
    Optimization       Manual                                    0 %        Failed                 e:
    
    PS C:\Windows\system32> get-dedupstatus | fl
    
    Volume                             : D:
    VolumeId                           : \\?\Volume{f90625c9-b832-4b8c-b069-205223038076}\
    Capacity                           : 55.43 GB
    FreeSpace                          : 23.3 GB
    UsedSpace                          : 32.13 GB
    UnoptimizedSize                    : 32.13 GB
    SavedSpace                         : 0 B
    SavingsRate                        : 0 %
    OptimizedFilesCount                : 0
    OptimizedFilesSize                 : 0 B
    OptimizedFilesSavingsRate          : 0 %
    InPolicyFilesCount                 : 0
    InPolicyFilesSize                  : 0 B
    LastOptimizationTime               : 18/09/2016 17:47:07
    LastOptimizationResult             : 0x80565342
    LastOptimizationResultMessage      : The job was terminated while in a handshake pending state.
    LastGarbageCollectionTime          : 17/09/2016 15:15:23
    LastGarbageCollectionResult        : 0x80565342
    LastGarbageCollectionResultMessage : The job was terminated while in a handshake pending state.
    LastScrubbingTime                  : 17/09/2016 15:19:23
    LastScrubbingResult                : 0x80565342
    LastScrubbingResultMessage         : The job was terminated while in a handshake pending state.
    Volume                             : E:
    VolumeId                           : \\?\Volume{5c9767f1-4ac3-481d-9e5e-73bf5b8e21a3}\
    Capacity                           : 19.87 GB
    FreeSpace                          : 13.52 GB
    UsedSpace                          : 6.35 GB
    UnoptimizedSize                    : 6.35 GB
    SavedSpace                         : 0 B
    SavingsRate                        : 0 %
    OptimizedFilesCount                : 0
    OptimizedFilesSize                 : 0 B
    OptimizedFilesSavingsRate          : 0 %
    InPolicyFilesCount                 : 0
    InPolicyFilesSize                  : 0 B
    LastOptimizationTime               : 18/09/2016 18:06:17
    LastOptimizationResult             : 0x80565342
    LastOptimizationResultMessage      : The job was terminated while in a handshake pending state.
    LastGarbageCollectionTime          :
    LastGarbageCollectionResult        :
    LastGarbageCollectionResultMessage :
    LastScrubbingTime                  :
    LastScrubbingResult                :
    LastScrubbingResultMessage         :
    
    PS C:\Windows\system32> get-dedupvolume | fl
    
    Volume                   : D:
    VolumeId                 : \\?\Volume{f90625c9-b832-4b8c-b069-205223038076}\
    Enabled                  : True
    UsageType                : Default
    DataAccessEnabled        : True
    Capacity                 : 55.43 GB
    FreeSpace                : 23.3 GB
    UsedSpace                : 32.13 GB
    UnoptimizedSize          : 32.13 GB
    SavedSpace               : 0 B
    SavingsRate              : 0 %
    MinimumFileAgeDays       : 0
    MinimumFileSize          : 32768
    NoCompress               : False
    ExcludeFolder            :
    ExcludeFileType          :
    ExcludeFileTypeDefault   : {edb, jrs}
    NoCompressionFileType    : {asf, mov, wma, wmv...}
    ChunkRedundancyThreshold : 100
    Verify                   : False
    OptimizeInUseFiles       : False
    OptimizePartialFiles     : False
    InputOutputScale         : 0
    Volume                   : E:
    VolumeId                 : \\?\Volume{5c9767f1-4ac3-481d-9e5e-73bf5b8e21a3}\
    Enabled                  : True
    UsageType                : Default
    DataAccessEnabled        : True
    Capacity                 : 19.87 GB
    FreeSpace                : 13.52 GB
    UsedSpace                : 6.35 GB
    UnoptimizedSize          : 6.35 GB
    SavedSpace               : 0 B
    SavingsRate              : 0 %
    MinimumFileAgeDays       : 0
    MinimumFileSize          : 32768
    NoCompress               : False
    ExcludeFolder            :
    ExcludeFileType          :
    ExcludeFileTypeDefault   : {edb, jrs}
    NoCompressionFileType    : {asf, mov, wma, wmv...}
    ChunkRedundancyThreshold : 100
    Verify                   : False
    OptimizeInUseFiles       : False
    OptimizePartialFiles     : False
    InputOutputScale         : 0
    
    PS C:\Windows\system32>
    
    

    It would certainly be interesting to know if anyone else can dedup new files - whether it is a problem with the 14393 package or something specific to me.
     
  9. maeglin

    maeglin MDL Novice

    Dec 14, 2012
    10
    1
    0
    I get this error while trying to install this package.

    Error: 0x800f081e​
    The specified package is not applicable to this image.​

    I'm on Windows 10.14393.187
     
  10. abbodi1406

    abbodi1406 MDL KB0000001

    Feb 19, 2011
    17,201
    90,736
    340
    The new MUI one? or both?
     
  11. maeglin

    maeglin MDL Novice

    Dec 14, 2012
    10
    1
    0
    Both. I could install previous packages with no apparent errors (except for the aforementioned not being able to dedup new files).
     
  12. abbodi1406

    abbodi1406 MDL KB0000001

    Feb 19, 2011
    17,201
    90,736
    340
    Well, i don't know the reason
     
  13. TheKingDavid

    TheKingDavid MDL Novice

    Aug 22, 2016
    9
    1
    0
    Same errors here :eek:
    Windows 10.14393.187
     
  14. fbifido

    fbifido MDL Member

    Jun 6, 2007
    199
    26
    10
    #316 fbifido, Sep 26, 2016
    Last edited by a moderator: Apr 20, 2017
  15. maeglin

    maeglin MDL Novice

    Dec 14, 2012
    10
    1
    0
    I installed this package, but cannot dedup new files nor I can use garbage collection for deleted files.
     
  16. TheKingDavid

    TheKingDavid MDL Novice

    Aug 22, 2016
    9
    1
    0
    I already use this package :-S
    But I removed the previous one and installed this one.
    [h=3]Unfortunately same behavior. [/h]
     
  17. TheKingDavid

    TheKingDavid MDL Novice

    Aug 22, 2016
    9
    1
    0
    #319 TheKingDavid, Sep 26, 2016
    Last edited: Sep 26, 2016
    Windows Server 2016 RTM build 14393.0.160911-2111 is out

    May be, we will have new version with final Deduplication and FileServer Package?

    :confused: Sad new : I've just download Windows Server 2016 RTM and we can forget it, the files provided have the same date, the same size ...
     
  18. abbodi1406

    abbodi1406 MDL KB0000001

    Feb 19, 2011
    17,201
    90,736
    340
    Nothing new is out

    Windows Server 2016 RTM build 14393.0.160911-2111 = Windows Server 2016 RTM 14393.0.160715-1616 + some cumulative update