Magical DeDup VHD Archives

You need to login to view this posts content.

 

Did you install the dedup feature first?

 

Did you hash check it before you used it?
A lot of people who use javascript downloader helpers report file contents are fubar

 

Hmm... I don't know. mounting virtual disks from inside a virtual machine might cause issues, but I don't know.
Did you install the vmware tools?

 

added a few mirrors to firedrive and updated a few of the vhds
moved the storage server 2008 r2 essentials to the 2011 vhd since it's part of that pack and contains much of the same features. It takes a gig less.

I included an updated version of the 8.1 with Update service like I posted in moderate's thread.

I'm going to see if I can up all of these vhds to firedrive since mega sucks now.
It might take a while. Firedrive likes to start uploading and then pause and do nothing forever sometimes.

 

Firstly you enable File Server deduplication options in Server. If you are using Win8.x you need to use a script to add the cab files and dedup-core feature.
You can search the board for dedup on win8/8.1/8.1 with update

So, after you install dedup feature, you have new powershell options.
I'm not sure if you need to import anything on server, but on the win8.x ports, I was able to just fire up powershell and start using the commands.

So what you need is 2 command windows: one for powershell and one for diskpart
In diskpart you first need to create a dynamic vhd.
For diskpart, all numbers that are not related to offsets are in megabytes, so 500 would be 500MB.

Code:

create vdisk file="c:\vhd\windows-archive.vhd" type=expandable maximum=500000
select vdisk file="c:\vhd\windows-archive.vhd"
attach vdisk
format fs=ntfs quick label="Windows-Dedup-Archive"
assign

Now go to your powershell cmd (I don't use the powershell program, I just type 'powershell' in admin prompt)
Now if you're not going to use dedup for any sort of weekly scheduled parsing like a long-term file server, you might want to disable the 3 default weekly passes.
It has a hotspot scrubbing task, a garbage collection pass, and an optimization pass.
If you are JUST working with vhd files, you don't want these as they only run weekly and chances are that you won't want them to be running every time your system idles after missing a schedule.
So to disable these run:

Code:

set-dedupschedule -name "WeeklyGarbageCollection -Enabled $false
set-dedupschedule -name "WeeklyScrubbing" -Enabled $false
set-dedupschedule -name "BackgroundOptimization" -Enabled $false

If you ever want to turn them back on, you just set them to $true

Now that you're primed, you can load up your vhd.
You can shrink it multiple times after dedup optimizing, so don't be afraid to add a bunch of iso files, optimize, re-add a bunch, optimize, etc

To Optimize:
So firstly, on your new vhd, you need to enable the dedup.
This is also done in powershell
In this example, our attached vhd has the drive letter H:

Code:

enable-dedupvolume H:
set-dedupvolume H: -minimumfileagedays 0
start-dedupjob -volume H: -type Optimization

Once you've initially set the enable and minfiledays, you don't need to run them for every optimize/scrub/garbagecollection

If you set -type GarbageCollection it will collect chunks that are no longer used. This is only true if you've deleted a file from your vhd or drive. If you haven't deleted anything, it won't do anything.
-type Scrub should be used with the -full argument when it is used, but for vhd you probably will never use it. Since we haven't set our volume to use hotspot data, it won't ever have anything to actually scrub.
If you were making a high-end dedup file server, you might use the redundant hotspot data so that when one chunk is damaged, a lot of important files would not be lost. It's just a failsafe, though, like parity.

So after optimizing, you can now detach and reattach your vhd in readonly mode to compact it.
This is done in your diskpart cmd window:

Code:

select vdisk file="c:\vhd\windows-archive.vhd"
detach vdisk
attach vdisk readonly
compact vdisk

That's about all there is to it.
It's very handy when using windows iso files as they often contain about 90% of the same data for the same build versions.

It's not as efficient as using delta files, but then delta files don't automatically parse themselves for network file transfers like dedup does.

 

What OS are you using and what method did you install the dedup feature?

 

bios problems usually indicate hardware failure, not software issues.
If you cannot even boot to your x86 os, then you might have either a cpu, motherboard, or hard disk failure.
None of these things would be affected by installing this dedup feature on an x64 os.

 

when you do the dedup optimize command, it doesn't give you a display progress since it runs in the background.
You can check it with the powershell command: get-dedupjob

 

for your diskpart you need to run diskpart... it wasn't specified in your script there:

Code:

if not exist c:\vhd\ mkdir c:\vhd

echo create vdisk file="c:\vhd\windows-archive.vhd" type=expandable maximum=500000>c:\vhd\diskpart.txt
echo select vdisk file="c:\vhd\windows-archive.vhd">>c:\vhd\diskpart.txt
echo attach vdisk>>c:\vhd\diskpart.txt
echo create partition primary>>c:\vhd\diskpart.txt
echo format fs=ntfs quick label="Windows-Dedup-Archive">>c:\vhd\diskpart.txt
echo assign letter=H>>c:\vhd\diskpart.txt
diskpart /s c:\vhd\diskpart.txt

ahh... well I've never initialized powershell that same way.
You might try just moving powershell to the front of each cmd like so:

Code:

@echo off
color 1A

Powershell set-dedupschedule -name "WeeklyGarbageCollection -Enabled $false
Powershell set-dedupschedule -name "WeeklyScrubbing" -Enabled $false
Powershell set-dedupschedule -name "BackgroundOptimization" -Enabled $false


Powershell enable-dedupvolume H:
Powershell set-dedupvolume H: -minimumfileagedays 0
Powershell start-dedupjob -volume H: -type Optimization

You're scripting a lot of things that I never mentioned to script. with scripting you have to handle arguments differently and run powershell commands differently