Data deduplication for Windows 8 x64

Discussion in 'Windows 8' started by 100, Jun 8, 2012.

  1. 100

    100 MDL Expert

    May 17, 2011
    1,346
    1,544
    60
    #1 100, Jun 8, 2012
    Last edited by a moderator: Apr 20, 2017
  2. meShell

    meShell MDL Novice

    Jan 27, 2009
    25
    9
    0
    If that is working in RTM that would be really great as I really wanted to have it on my workstation! Thx for sharing! Will try it!
     
  3. kocoman

    kocoman MDL Senior Member

    May 16, 2007
    329
    4
    10
    Is there any way to make a software raid 1 mirror or software jbod mirror on the with data duplication enabled (like a backup) .. In case a drive fails? Thanks
     
  4. 100

    100 MDL Expert

    May 17, 2011
    1,346
    1,544
    60
    Yes, works for me on mirrored volumes on Windows 8. JBOD is not a mirror.
     
  5. SuperBubble

    SuperBubble MDL Member

    Nov 18, 2011
    148
    255
    10
    Hurrah 100! Neat!

    A while back, I tried something similar with the Single Instance Storage service from WDS, and it was a pain in the neck to install, set up, and the SIS Groveller ate CPU like a hundred hungry Windows Search indexers. I gave up on it very quickly.

    This is a vast improvement. Have you considered getting the mods to put this up as a full HOWTO?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. kocoman

    kocoman MDL Senior Member

    May 16, 2007
    329
    4
    10
    I have one 1TB drive, and 3 320GB drive, I want to make JBOD a 1TB from the 3 smaller drives, then mirror it with the 1TB single drive, then run data deduplication on that. If it does not work that way then I need to buy another 1TB I guess...
     
  7. 100

    100 MDL Expert

    May 17, 2011
    1,346
    1,544
    60
    I don't think Windows will do that. Why not use storage spaces on Windows 8? It's a much more flexible option than RAID or JBOD across differently-sized drives. I've done a quick test and it enabled and deduped files on a storage space volume just as well.
     
  8. 100

    100 MDL Expert

    May 17, 2011
    1,346
    1,544
    60
    Updated for RTM.
     
  9. msimonsen

    msimonsen MDL Novice

    Aug 24, 2012
    1
    0
    0
    Hello. How did you extract the packages?
     
  10. Ozymandias

    Ozymandias MDL Junior Member

    Jun 17, 2008
    65
    25
    0
    I tried to install data deduplication on Server 2012 Essentials (as deduplication has been removed from Essentials) but it comes up with:

    Error: 0x800f081e
    The specified package is not applicable to this image.

    I'm pretty much out of my depth with dism etc - any thoughts to getting it to work?
     
  11. 100

    100 MDL Expert

    May 17, 2011
    1,346
    1,544
    60
    I haven't checked out Essentials RC yet, but since it's not RTM it probably has a different build number than Windows 8/Server 2012. The dedup package extracted from RTM has a dependency on build 9200, so most likely that is why dism cannot install it. What's the build number of Essentials RC? If it's still 8400 (same as Server 2012 RC and Essentials beta) the files extracted from Server 2012 RC will probably work.


    Currently I'm using a rather messy script I've written. All the package manifests are in %systemroot%\servicing\packages (*.mum), and the actual files are in %systemroot%\winsxs, but figuring out which components from winsxs are actually required for a package isn't straightforward, unfortunately. That is what the script tries to do.
    I may rework it into a proper application some time and publish it, if there's any interest.
     
  12. Mr Jinje

    Mr Jinje MDL Expert

    Aug 19, 2009
    1,773
    1,090
    60
    #12 Mr Jinje, Aug 25, 2012
    Last edited: Aug 25, 2012
    Is 'deduplication' the best name they could come up with ?

    What happens to my de-dup HDD after I wipe my OS and reinstall, are they transferable without issues. Or does it have machine specific anything going on.
     
  13. Ozymandias

    Ozymandias MDL Junior Member

    Jun 17, 2008
    65
    25
    0
    Ah yes, essentials RC has a build number of 8400 (on desktop and winver).
     
  14. 100

    100 MDL Expert

    May 17, 2011
    1,346
    1,544
    60
    #14 100, Aug 25, 2012
    Last edited by a moderator: Apr 20, 2017
    (OP)
  15. Ozymandias

    Ozymandias MDL Junior Member

    Jun 17, 2008
    65
    25
    0
    #15 Ozymandias, Aug 25, 2012
    Last edited: Aug 25, 2012
    Seems to install OK & found the GUI.

    Adding another virtual disk (E drive) I can enable deduplication - excellent.
    Many thanks - good job :)

    btw loads of guys on the MS forums are bemoaning the lack of deduplication in Essentials along with WSUS not being allowed as well.
     
  16. dreamss

    dreamss MDL Junior Member

    Apr 7, 2009
    99
    48
    10
    woot enabled it on my ssd game drive...

    the eval tool results:
    Processed files: 12590
    Processed files size: 40.94 GB
    Optimized files size: 33.86 GB
    Space savings: 7.08 GB
    Space savings percent: 17

    once i ran it:
    PS C:\Windows\system32> Get-DedupStatus

    FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
    --------- ---------- -------------- ------------- ------
    19.9 GB 7.03 GB 12582 12602 D:


    PS C:\Windows\system32> Get-DedupJob
    PS C:\Windows\system32> Get-DedupVolume

    Enabled SavedSpace SavingsRate Volume
    ------- ---------- ----------- ------
    True 7.03 GB 16 % D:
     
  17. romprod

    romprod MDL Novice

    Aug 16, 2012
    7
    1
    0
    #17 romprod, Aug 27, 2012
    Last edited by a moderator: Apr 20, 2017
    Hi, I've tried the first command above but I get the following error message back. I'm running activated Win8 Pro 64bit RTM. Can anyone help?

    Code:
    Deployment Image Servicing and Management tool
    Version: 6.2.9200.16384
    
    
    Error: 11
    
    You cannot service a running 64-bit operating system with a 32-bit version of DISM.
    Please use the version of DISM that corresponds to your computer's architecture.
    
    The DISM log file can be found at C:\Windows\Logs\DISM\dism.log
    PS C:\Users\User\Downloads\w8rcdedup>
     
  18. romprod

    romprod MDL Novice

    Aug 16, 2012
    7
    1
    0
    Ignore that, I was running it in a Power Shell instead of the command prompt.... doh!
     
  19. Rakeesh

    Rakeesh MDL Novice

    Feb 25, 2012
    21
    35
    0
    Deduplication isn't a windows thing, it's something that the storage industry has been using for years already. I don't really see why they would name it anything else - the purpose of deduplication isn't for compression (although it can have that effect) rather its primary goal is to preserve data integrity.

    Really the name fits quite well because it describes perfectly what it does, in the IT world you don't give things fancy names, because you aren't trying to sell somebody something. That would be like saying we should rename TCP/IP to "Packet Switching Magic" to make it more consumer friendly.
     
  20. Stannieman

    Stannieman MDL Guru

    Sep 4, 2009
    2,233
    1,808
    90
    #20 Stannieman, Aug 31, 2012
    Last edited by a moderator: Apr 20, 2017
    Hmm
    Code:
    Deployment Image Servicing and Management tool
    Version: 6.2.9200.16384
    
    Image Version: 6.2.9200.16384
    
    Processing 1 of 6 - Adding package Microsoft-Windows-VdsInterop-Package~31bf3856
    ad364e35~amd64~~6.2.9200.16384
    [==========================100.0%==========================]
    Processing 2 of 6 - Adding package Microsoft-Windows-VdsInterop-Package~31bf3856
    ad364e35~amd64~en-US~6.2.9200.16384
    [==========================100.0%==========================]
    Processing 3 of 6 - Adding package Microsoft-Windows-FileServer-Package~31bf3856
    ad364e35~amd64~~6.2.9200.16384
    [================           29.0%                          ]
    An error occurred - Microsoft-Windows-FileServer-Package Error: 0x80073701
    Processing 4 of 6 - Adding package Microsoft-Windows-FileServer-Package~31bf3856
    ad364e35~amd64~en-US~6.2.9200.16384
    [==========================100.0%==========================]
    Processing 5 of 6 - Adding package Microsoft-Windows-Dedup-Package~31bf3856ad364
    e35~amd64~~6.2.9200.16384
    [==========================100.0%==========================]
    Processing 6 of 6 - Adding package Microsoft-Windows-Dedup-Package~31bf3856ad364
    e35~amd64~en-US~6.2.9200.16384
    [==========================100.0%==========================]
    
    Error: 14081
    
    The specified package is not applicable to this image.
    The specified package is not applicable to this image.
    The specified package is not applicable to this image.
    The command completed with errors.
    For more information, refer to the log file.
    
    The DISM log file can be found at C:\Windows\Logs\DISM\dism.log
    Buildnumber of the packages is correct. I'm running Enterprise RTM x64, any ideas?
    The 2 VdsInterop packages are installed. Then there are 4 that fail, but "The specified package is not applicable to this image." is only 3 times written to the console, so what happened to the 4th? Or is the last package the cause of the 14081...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...