Magical DeDup VHD Archives

Discussion in 'Windows Server' started by murphy78, May 1, 2014.

  1. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    #1 murphy78, May 1, 2014
    Last edited by a moderator: Apr 20, 2017
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. Stars

    Stars MDL Novice

    Apr 27, 2014
    21
    2
    0
    I was curious about this so tried installing Server 2012 R2 with update and double clicked on Server2012-R2.vhd
    Gave me an error though and couldn't mount it - something I missed? (hash checked out ok)
     
  3. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    Did you install the dedup feature first?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. Stars

    Stars MDL Novice

    Apr 27, 2014
    21
    2
    0
    I also tried running the script in Win8-Data-DeDup.7z - but that is probably not what you meant.
    I simply just installed a clean new Server 2012 R2 with update and tried opening the .vhd file.
     
  5. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    Did you hash check it before you used it?
    A lot of people who use javascript downloader helpers report file contents are fubar
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. Stars

    Stars MDL Novice

    Apr 27, 2014
    21
    2
    0
    yes hash checked out ok - one more thing though:
    The Server 2012 R2 was a vmware machine and the .vhd was on the host machine - used the "shared folder technique" to acces the .vhd from the vmware machine.
    That shouldn't give any problems mounting the .vhd right?

    So basicly I should be able to just open those .vhd files directly on a server 2012 R2 without any further installation/customization?
     
  7. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    Hmm... I don't know. mounting virtual disks from inside a virtual machine might cause issues, but I don't know.
    Did you install the vmware tools?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Stars

    Stars MDL Novice

    Apr 27, 2014
    21
    2
    0
    Tried installing win 8.1 in with update in vmware and and ran the script from an elevated cmd and the .vhd files inside the virtual machine - that worked :)
    So probably the "shared folder technique" that didn't work.
     
  9. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    added a few mirrors to firedrive and updated a few of the vhds
    moved the storage server 2008 r2 essentials to the 2011 vhd since it's part of that pack and contains much of the same features. It takes a gig less.

    I included an updated version of the 8.1 with Update service like I posted in moderate's thread.

    I'm going to see if I can up all of these vhds to firedrive since mega sucks now.
    It might take a while. Firedrive likes to start uploading and then pause and do nothing forever sometimes.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. filox

    filox MDL Novice

    Sep 15, 2010
    16
    0
    0
    It strikes me how to make the vhd, you could make a little tuto to learn to do one more thing ?

    thank you very much
     
  11. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    #12 murphy78, Jun 10, 2014
    Last edited by a moderator: Apr 20, 2017
    (OP)
    Firstly you enable File Server deduplication options in Server. If you are using Win8.x you need to use a script to add the cab files and dedup-core feature.
    You can search the board for dedup on win8/8.1/8.1 with update

    So, after you install dedup feature, you have new powershell options.
    I'm not sure if you need to import anything on server, but on the win8.x ports, I was able to just fire up powershell and start using the commands.

    So what you need is 2 command windows: one for powershell and one for diskpart
    In diskpart you first need to create a dynamic vhd.
    For diskpart, all numbers that are not related to offsets are in megabytes, so 500 would be 500MB.

    Code:
    create vdisk file="c:\vhd\windows-archive.vhd" type=expandable maximum=500000
    select vdisk file="c:\vhd\windows-archive.vhd"
    attach vdisk
    format fs=ntfs quick label="Windows-Dedup-Archive"
    assign
    Now go to your powershell cmd (I don't use the powershell program, I just type 'powershell' in admin prompt)
    Now if you're not going to use dedup for any sort of weekly scheduled parsing like a long-term file server, you might want to disable the 3 default weekly passes.
    It has a hotspot scrubbing task, a garbage collection pass, and an optimization pass.
    If you are JUST working with vhd files, you don't want these as they only run weekly and chances are that you won't want them to be running every time your system idles after missing a schedule.
    So to disable these run:
    Code:
    set-dedupschedule -name "WeeklyGarbageCollection -Enabled $false
    set-dedupschedule -name "WeeklyScrubbing" -Enabled $false
    set-dedupschedule -name "BackgroundOptimization" -Enabled $false
    If you ever want to turn them back on, you just set them to $true

    Now that you're primed, you can load up your vhd.
    You can shrink it multiple times after dedup optimizing, so don't be afraid to add a bunch of iso files, optimize, re-add a bunch, optimize, etc

    To Optimize:
    So firstly, on your new vhd, you need to enable the dedup.
    This is also done in powershell
    In this example, our attached vhd has the drive letter H:
    Code:
    enable-dedupvolume H:
    set-dedupvolume H: -minimumfileagedays 0
    start-dedupjob -volume H: -type Optimization
    Once you've initially set the enable and minfiledays, you don't need to run them for every optimize/scrub/garbagecollection

    If you set -type GarbageCollection it will collect chunks that are no longer used. This is only true if you've deleted a file from your vhd or drive. If you haven't deleted anything, it won't do anything.
    -type Scrub should be used with the -full argument when it is used, but for vhd you probably will never use it. Since we haven't set our volume to use hotspot data, it won't ever have anything to actually scrub.
    If you were making a high-end dedup file server, you might use the redundant hotspot data so that when one chunk is damaged, a lot of important files would not be lost. It's just a failsafe, though, like parity.

    So after optimizing, you can now detach and reattach your vhd in readonly mode to compact it.
    This is done in your diskpart cmd window:
    Code:
    select vdisk file="c:\vhd\windows-archive.vhd"
    detach vdisk
    attach vdisk readonly
    compact vdisk
    That's about all there is to it.
    It's very handy when using windows iso files as they often contain about 90% of the same data for the same build versions.

    It's not as efficient as using delta files, but then delta files don't automatically parse themselves for network file transfers like dedup does.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. filox

    filox MDL Novice

    Sep 15, 2010
    16
    0
    0
    Practicing with all this in a PowerShell cmd did not come out with anything just >> and wait and wait to cap it off and now the laptop will not boot or bios boot screen completely in black, any suggestions please

    thanks
     
  13. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    What OS are you using and what method did you install the dedup feature?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. filox

    filox MDL Novice

    Sep 15, 2010
    16
    0
    0
    I have three partitions on the hard disk in which a is 8.1x86, one for testing and another 8.1x64 where dedup install and made a vhd as you indicate me in C: \ by diskpark and then in cmd with administrator privileges PowerShell and then put the other comados indicated by you, as did nothing after a while methylene finger on the power button, which pump.The again after a while the screen goes black and no options bios, how could retrieve the laptop please.

    thanks
     
  15. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    bios problems usually indicate hardware failure, not software issues.
    If you cannot even boot to your x86 os, then you might have either a cpu, motherboard, or hard disk failure.
    None of these things would be affected by installing this dedup feature on an x64 os.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. filox

    filox MDL Novice

    Sep 15, 2010
    16
    0
    0
    hahaha .. I start, I say what I did remove the battery and any external power for a while longer, WHILE we were doing this post and start my regular bios again, that scare. Could be some cache Memory.
    When can I capture and cmd commands that were not doing anything and I put it to bad to see and I was wrong ..

    Furthermore sorry for my English translation I'm Spanish.

    Thank you very much for your trouble
     
  17. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    when you do the dedup optimize command, it doesn't give you a display progress since it runs in the background.
    You can check it with the powershell command: get-dedupjob
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. filox

    filox MDL Novice

    Sep 15, 2010
    16
    0
    0
    #19 filox, Jun 11, 2014
    Last edited by a moderator: Apr 20, 2017
    Let's start.
    Console cmd with administrator privilejios
    Code:
     create vdisk file="c:\vhd\windows-archive.vhd" type=expandable maximum=500000
    
    select vdisk file="c:\vhd\windows-archive.vhd"
    
    attach vdisk
    
    create partition primary
    
    select partition 1
    
    active
    
    format fs=ntfs quick label="Windows-Dedup-Archive"
    
    assign letter=H 
    Then copy the folder with their isos in VHD
    Cmd:
    Code:
    @echo off
    color 1A
    
    Powershell 
    
    set-dedupschedule -name "WeeklyGarbageCollection -Enabled $false
    
    set-dedupschedule -name "WeeklyScrubbing" -Enabled $false
    
    set-dedupschedule -name "BackgroundOptimization" -Enabled $false
    
    enable-dedupvolume H:
    set-dedupvolume H: -minimumfileagedays 0
    start-dedupjob -volume H: -type Optimization
    And so that gives:

    I can not send pictures I have only 12 post sorry
    What am I doing wrong?

    Installed and testing SOS 8.1x64
    Code:
    If you're wanting to use these vhd files on win8.1 x64 you'll need:
     File: Win81-2919355-Data-DeDup.7z 
    thanks
     
  19. murphy78

    murphy78 MDL DISM Enthusiast

    Nov 18, 2012
    6,787
    10,365
    210
    #20 murphy78, Jun 11, 2014
    Last edited by a moderator: Apr 20, 2017
    (OP)
    for your diskpart you need to run diskpart... it wasn't specified in your script there:
    Code:
    if not exist c:\vhd\ mkdir c:\vhd
    
    echo create vdisk file="c:\vhd\windows-archive.vhd" type=expandable maximum=500000>c:\vhd\diskpart.txt
    echo select vdisk file="c:\vhd\windows-archive.vhd">>c:\vhd\diskpart.txt
    echo attach vdisk>>c:\vhd\diskpart.txt
    echo create partition primary>>c:\vhd\diskpart.txt
    echo format fs=ntfs quick label="Windows-Dedup-Archive">>c:\vhd\diskpart.txt
    echo assign letter=H>>c:\vhd\diskpart.txt
    diskpart /s c:\vhd\diskpart.txt
    
    ahh... well I've never initialized powershell that same way.
    You might try just moving powershell to the front of each cmd like so:
    Code:
    @echo off
    color 1A
    
    Powershell set-dedupschedule -name "WeeklyGarbageCollection -Enabled $false
    Powershell set-dedupschedule -name "WeeklyScrubbing" -Enabled $false
    Powershell set-dedupschedule -name "BackgroundOptimization" -Enabled $false
    
    
    Powershell enable-dedupvolume H:
    Powershell set-dedupvolume H: -minimumfileagedays 0
    Powershell start-dedupjob -volume H: -type Optimization
    You're scripting a lot of things that I never mentioned to script. with scripting you have to handle arguments differently and run powershell commands differently
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...