257 replies to this topic

[web ferret]

Hi Brian, I ran the tool and it finished after 32 minutes and nothing has changed on my disks. So the good news is that the app itself works but the bad news is that it didn't work for me. This last could be the rather small amount of data on my WHS?

[Beefcake]

Hi Brian, I'm still at work but in my lunch I downloaded the new version to read the walkthrough.txt. Good suggestions in there but I have some questions concerning the choices one must make and what effect these choices have.
For example; should I include D:\ in the balancing effort? I have no clue to be honest. Same thing for to choose whether (in step 2) I want to clear data off the landing zone. I really wouldn't know what to pick!

In short: I don't know what the best choices are in step 1 and 2. Perhaps someone here can shed some light on this (prefferably in Drive balancer for Dummies style )

Another thing; I reckon all the fake files will be deleted when the process has finished?

When I am at home I'll give it another try but that is probably during saturday or sunday.

Oh, another thing popped my mind... What happens if I would install a new drive or remove a broken drive. Should I run DB again?

No problem..here's the answer.....

D: = Landing Zone except the name Landing Zone is no longer true.

So, your choice (since you only have D: and one pooled drive is you HAVE to include the D: in the balancing effort. As far as D: is concerned, you have 3 choices to make

1) Do you want to include it in the balancing effort
2) If yes, you don't clear it since you want data to be on it
2a) If no, make the choice if you want the data (if any) currently on D: to be moved onto the other pooled drives before balancing starts.

So, since you only have D: and the other pooled drives, you need to select "yes" to #1 and then you get no option to clear it out first since that wouldn't make any sense.

[Beefcake]

Would this app need to be run every now and again to rebalance the pool?

I believe it's going to be weekly run for me as I am using WHS primarily as a SageTV server. It does not keep things balanced in the future, so you can run it as often as you'd like.

-Brian

[Beefcake]

Hi Brian, I ran the tool and it finished after 32 minutes and nothing has changed on my disks. So the good news is that the app itself works but the bad news is that it didn't work for me. This last could be the rather small amount of data on my WHS?

Yeah, this is my best guess. If you are interested in trying a new option, I can whip up a new exe for you to test that would be more aggressive in making DE move around files. Let me know if you'd like to try this.

-Brian

[Beefcake]

Yeah, this is my best guess. If you are interested in trying a new option, I can whip up a new exe for you to test that would be more aggressive in making DE move around files. Let me know if you'd like to try this.

-Brian

I found some more reasons why drive balancer may not be able to do it's job as well as you might want.....
*****************************************************
Things that can cause a less than stellar results

1) Look at the amount of Shadows as a whole on the drive pool. If this number is greater that 0, it will affect the final outcome of drive balance. I found that I had 100GB+ of Shadows during my first run of drive balancer. After the first run completed, an overnight run, the shadows were gone. I then reran drive balancer and it finished up the job.

2) The amount of duplicated folders can may make it difficult to get a "perfect" balance. I am not suggesting turning off duplication on anything, I'm just pointing out that duplication will limit the choices DE has when it moves around files. If you have 4+ disks in the pool, this becomes less of an issue since DE will have more choices.

[f00kie]

I found some more reasons why drive balancer may not be able to do it's job as well as you might want.....
*****************************************************
Things that can cause a less than stellar results

1) Look at the amount of Shadows as a whole on the drive pool. If this number is greater that 0, it will affect the final outcome of drive balance. I found that I had 100GB+ of Shadows during my first run of drive balancer. After the first run completed, an overnight run, the shadows were gone. I then reran drive balancer and it finished up the job.

2) The amount of duplicated folders can may make it difficult to get a "perfect" balance. I am not suggesting turning off duplication on anything, I'm just pointing out that duplication will limit the choices DE has when it moves around files. If you have 4+ disks in the pool, this becomes less of an issue since DE will have more choices.

Hey Beefcake,

Thanks for the great app, it was much needed. Unfortunately, I started writing my own version a while back, in C#, based on Brent's code (from the other thread), and have it basically the way I want it, I think (yours obviously has more features). I was wondering one thing though: would you be able to say how you get the balancing to start right away (if that is what happens -- can't test now as I'm running my own!). Is it as simple as restarting a service?

Thanks.

[Beefcake]

Hey Beefcake,

Thanks for the great app, it was much needed. Unfortunately, I started writing my own version a while back, in C#, based on Brent's code (from the other thread), and have it basically the way I want it, I think (yours obviously has more features). I was wondering one thing though: would you be able to say how you get the balancing to start right away (if that is what happens -- can't test now as I'm running my own!). Is it as simple as restarting a service?

Thanks.

Of course...I have no trade secrets!

static public void RestartDriveExtender()
		{
			Console.WriteLine("\nRestarting the Drive Extender Service....\n");
			System.Diagnostics.Process.Start("cmd", @"/c net stop ""Drive Extender Migrator Service"" ");
			Thread.Sleep(5000);
			System.Diagnostics.Process.Start("cmd", @"/c net start ""Drive Extender Migrator Service"" ");
		}

There is a whole fancy way of telling when a process ends, but I decided not to do it and just wait seconds. I came up with this after speaking with a user over at the SageTV forums called S_M_E.

Note: It doesn't hurt to stop the service if it's actually moving data. I call this function after 10 minutes of no changes in the free space on the "crunch" drives - those which don't have enough current free space to create all the fake files.

[f00kie]

Of course...I have no trade secrets!

static public void RestartDriveExtender()
		{
			Console.WriteLine("\nRestarting the Drive Extender Service....\n");
			System.Diagnostics.Process.Start("cmd", @"/c net stop ""Drive Extender Migrator Service"" ");
			Thread.Sleep(5000);
			System.Diagnostics.Process.Start("cmd", @"/c net start ""Drive Extender Migrator Service"" ");
		}

There is a whole fancy way of telling when a process ends, but I decided not to do it and just wait seconds. I came up with this after speaking with a user over at the SageTV forums called S_M_E.

Note: It doesn't hurt to stop the service if it's actually moving data. I call this function after 10 minutes of no changes in the free space on the "crunch" drives - those which don't have enough current free space to create all the fake files.

Great, thanks a lot. I've put it into my code; will test it out tonight when I add another drive in, hehe.
If you need some ideas on some of the code (I'm sure we have a lot of the same code), I've posted a version of mine here: http://forum.wegotse...o...ost&p=40595

[Beefcake]

Great, thanks a lot. I've put it into my code; will test it out tonight when I add another drive in, hehe.
If you need some ideas on some of the code (I'm sure we have a lot of the same code), I've posted a version of mine here: http://forum.wegotse...o...ost&p=40595

I will definitely take a look to see if I could get any ideas from it. Also, I foudn that using fsutil to not be a good idea since you can't easily tell when it's finished. As a result, look at the way LZreallocator ceates files....it rocks and is super quick.

TO ALL:
I found a bug on my own system where WHS is reporting the wrong amount of free space for D:, so if you are using the D: in the balancing effort, you may not be getting good balancing results. Basically, WHS is reporting the free space for C and D when you look at D: So, the fix was to subtract out the C: free space all by itself (which oddly enough works).

So, the bug is in WHS, not my code. This is also a function of that January update where the size of the pool is now "correctly" returned when you map a share. I will be releasing a fix for this hopefully this evening. I also plan to through in some error exception handling so that I can help debug any other bugs that may arise.

See you all tonight.

[Beefcake]

TO ALL:
I found a bug on my own system where WHS is reporting the wrong amount of free space for D:, so if you are using the D: in the balancing effort, you may not be getting good balancing results. Basically, WHS is reporting the free space for C and D when you look at D: So, the fix was to subtract out the C: free space all by itself (which oddly enough works).

So, the bug is in WHS, not my code. This is also a function of that January update where the size of the pool is now "correctly" returned when you map a share. I will be releasing a fix for this hopefully this evening. I also plan to through in some error exception handling so that I can help debug any other bugs that may arise.

See you all tonight.

Updated code with:
1) bug work around
2) improved balancing algorithm
3) error exception handling

...is in the OP

[peter_h]

Thanks for this. I Had 3 TB this morning and added another TB for a total of 4tb. One drive was at 97%. After running this, everything is nice and spread out.

However, it did not complete the last step. It completed and told me to hit enter to del the files but after 10 min, the files were still there. The drives in the pool were all balanced out but the OS drive was still at critical 99%. I had to manually go in and delete the fake folder. Other than that, great app.

[Moskus]

I'm definitively going to try this, but I feel like I've missed something:

What are the benefits of having the all the drives balanced?

[Drashna Jaelre (WGS)]

Peaceful nights rest.

Honestly, nothing IMO. A lot of people like having the files evenly spread across the disk for various reasons. But I think the main reason is to move data off the system disk, first and foremost. There is definitely the possibility for a performance gain by moving the data off the system disk.

[Beefcake]

I'm definitively going to try this, but I feel like I've missed something:

What are the benefits of having the all the drives balanced?

Not everyone will want this....no doubt.

All reasons are based on increasing the "average" performance of the system as a whole when having multiple clients access different files. There are also a few things about sharing the workload amongst all your drives rather than one (to increase a drive's MTBF in your system).

It basically creates some kind of neatness feeling in me .

[Beefcake]

Thanks for this. I Had 3 TB this morning and added another TB for a total of 4tb. One drive was at 97%. After running this, everything is nice and spread out.

However, it did not complete the last step. It completed and told me to hit enter to del the files but after 10 min, the files were still there. The drives in the pool were all balanced out but the OS drive was still at critical 99%. I had to manually go in and delete the fake folder. Other than that, great app.

That's odd....can I assume that the program exited "normally"..... i.e. - without a crash or anything, it just either disappeared or returned to a command line.

Right?

-Brian

[GaMeR]

Beefcake:

Would it be possible to create some form of GUI for this program with error outputs and logs? I think that would be very nice to have, so when the program is done you get to see if it all was successfull or not.

[poleary2000]

Based on what I have read, this app spreads out the files amongst all drives using "fake" files. I do like the ability that is shows to spread out the data evenly.

Maybe this would be a completely different app, but I think I would like the ability to force a share completely onto one drive. For example, today I might have:

Pictures
Holiday
Christmas
Picture 1

[Beefcake]

Beefcake:

Would it be possible to create some form of GUI for this program with error outputs and logs? I think that would be very nice to have, so when the program is done you get to see if it all was successfull or not.

I am thinking about making a GUI for it, but to be honest, I have ZERO experience programming up a GUI. I don't it would be that difficult, just time consuming. Unfortunately, time is something I don't have a lot of the next few days. I may be able to look at it later in the week, though.

[Beefcake]

Based on what I have read, this app spreads out the files amongst all drives using "fake" files. I do like the ability that is shows to spread out the data evenly.

Maybe this would be a completely different app, but I think I would like the ability to force a share completely onto one drive. For example, today I might have:

Pictures
Holiday
Christmas
Picture 1

That is a different app all together. I too, would like to be able to force which files go on which drives. This is on my "need to look into" list. MS specifically says not to mess with moving files around, so I am a bit reluctant. None the less, I'll just create some "fake" files and try moving those around in some way.

This app works by not actually moving anything around. It relies on DE to do that work for it.

-Brian

[poleary2000]

Based on what I have read, this app spreads out the files amongst all drives using "fake" files. I do like the ability that is shows to spread out the data evenly. However, I am wondering if it exacorbates an issue I already have with DE. See below.

Maybe this would be a completely different app, but I think I would like the ability to force a share completely onto one drive. For example, today I might have:

Pictures
Holiday
Christmas Orig File Dup File
Picture 1.jpg Drive 2 Drive 3
Picture 2.jpg Drive 4 Drive 2
Picture 3.jpg Drive 4 Drive 2
Picture 3.jpg Drive 1 Drive 4

I think it would be nice to have the ability to force the entire share onto one drive, as such:

Pictures
Holiday
Christmas Orig File Dup File
Picture 1.jpg Drive 2 Drive 3
Picture 2.jpg Drive 2 Drive 3
Picture 3.jpg Drive 2 Drive 3
Picture 3.jpg Drive 2 Drive 3

In that way, I wouldn't be so concerned about the drives being used in a uniform manner, but rather a complete share set on one drive. That way, if something catastrophic occurred (my OS failed, I couldn't reinstall, and didn't have a backup), I would be able to plug the drive into any machine and quickly reassembly my share. Otherwise, the process would be painstaking to say the least to assembly thousands of folders with even more files.

I wish DE worked this way to begin with, but would love an app that did this.


EDIT: NOT SURE WHAT I DID WITH THE POST PREVIOUSLY. I NEVER HIT SUBMIT.