28 replies to this topic

[MJ-bos]

I'm getting this message every 30 - 60 seconds (preceded by several total disk throughput messages (4-9) and the same "% done" reported for each drive (59%, 0%, 0%, 67%, 74%).

I've had this running for about 6 hours with the same messages over and over... there is disk activity being reported in the Disk Management plugin.

Any ideas... should I kill it and clean up?

Thanks in advance,
MJ (also in Mass.)

[zzmegad]

this is the same problem I am having. what size is the drive you WHS is installed on?

[honsoworld]

I am having the same probelm. Maybe this is a Power Pack 2 issue?...

[Beefcake]

I am having the same probelm. Maybe this is a Power Pack 2 issue?...

hrmm.. this was happening before power pack 2, so I don't think that's it. It seems like the system just gets in a situation where DE can't (or won't) move the data around. I haven' hit this problem yet, so it's difficult for me to "fix". My best guess at this point is that file duplication is affecting DE's ability to mvoe files, although that really doesn't make much sense.

Could you guys comment on whether or not you include the "D" drive in the balancing? I have a theory that I'm filling up D too much and perhaps then DE can't update the tombstone for the files it wants to move, so it doesn't move anything.

I could whip up a version that does behaves a little differently for you all to try....any takers? I'm going to be busy today, but I could get it done tonight around 8pm EST.

-Brian

[honsoworld]

hrmm.. this was happening before power pack 2, so I don't think that's it. It seems like the system just gets in a situation where DE can't (or won't) move the data around. I haven' hit this problem yet, so it's difficult for me to "fix". My best guess at this point is that file duplication is affecting DE's ability to mvoe files, although that really doesn't make much sense.

Could you guys comment on whether or not you include the "D" drive in the balancing? I have a theory that I'm filling up D too much and perhaps then DE can't update the tombstone for the files it wants to move, so it doesn't move anything.

I could whip up a version that does behaves a little differently for you all to try....any takers? I'm going to be busy today, but I could get it done tonight around 8pm EST.

-Brian

I have tried this on two WHSs and I have encountered the same problem. I did include the D drive for both runs.

Other comments about my setups:
*All of my folders are using file duplication
*Both of my servers have PP2 installed
*When I ran this tool I used option 1

I would love to try a different version once you have the time. This seams like a great program, and would love to get it working on my setups.

[revengineer]

Could you guys comment on whether or not you include the "D" drive in the balancing? I have a theory that I'm filling up D too much and perhaps then DE can't update the tombstone for the files it wants to move, so it doesn't move anything.

Same problem here. Nothing is moving. I have duplication on for most folders. I initially did include D in balancing.Now I am trying without and it is sleeping multiple times for 1 min whiel data is moved of D. AT least the drives are shoveling data now. Not sure whether it will balance in the end.

[revengineer]

After playing around with this program for the better part of the day, I conclude that this program is doing many thing but balancing the drives. I just hope destroying my data is not one of the tasks accomplished. The tuning must be completely hosed. Will be back for v2.1.

[honsoworld]

I have been tinkering around with this and have yet to find a “good” solution. I have however found a method that comes pretty close.

NOTE: I have only tested this with D included, and option 1.

I run this tool normally and let it calculate the drive specs and create its fake files. As soon as I start getting these errors I stop the tool by using Ctrl+C. I do not immediately remove the fake files. I left theses fake files and came back several (in my case 5) hours later. After waiting 5 hours I deleted the fake files and to my surprise my drives where balanced within 5%.

I repeated this process on my other home server at a differently location and had the same results.

Now, I did not actually watch the entire process for 5 hours, so I don’t know what the disk usage was during that time period, but it would appear that either 1) the DE service takes more than 30 seconds to get going, or 2) that even when the DE service has low activity, it is still doing something.

I have a suggestion to the programmer to maybe add an option to disable the “DE Service Restarts Due to Low Disk Activity” flag that gets thrown. Obviously in some cases giving the DE service more time to do its things pays off.

It should also be noted that when I did not stop this tool, and let it sit for 5 hours while constantly restarting the DE service, I did not get any results, which could mean that restarting the DE service in some cases can actually have a negative effect.

[Beefcake]

I have been tinkering around with this and have yet to find a “good” solution. I have however found a method that comes pretty close.

NOTE: I have only tested this with D included, and option 1.

I run this tool normally and let it calculate the drive specs and create its fake files. As soon as I start getting these errors I stop the tool by using Ctrl+C. I do not immediately remove the fake files. I left theses fake files and came back several (in my case 5) hours later. After waiting 5 hours I deleted the fake files and to my surprise my drives where balanced within 5%.

I repeated this process on my other home server at a differently location and had the same results.

Now, I did not actually watch the entire process for 5 hours, so I don’t know what the disk usage was during that time period, but it would appear that either 1) the DE service takes more than 30 seconds to get going, or 2) that even when the DE service has low activity, it is still doing something.

I have a suggestion to the programmer to maybe add an option to disable the “DE Service Restarts Due to Low Disk Activity” flag that gets thrown. Obviously in some cases giving the DE service more time to do its things pays off.

It should also be noted that when I did not stop this tool, and let it sit for 5 hours while constantly restarting the DE service, I did not get any results, which could mean that restarting the DE service in some cases can actually have a negative effect.

Hrmm.. perhaps letting the user specify the amount of time between trying to restart DE would be a good solution. It also seems definitely like folder duplication is the reason files aren't moving. I think I may know how to solve that one as well, but will need testers since it doesn't happen on my system.

Sorry about the delays, I've been really busy with family things lately. I'll to get to this Sat. or Sunday.

-Brian

[honsoworld]

...will need testers since it doesn't happen on my system...

I would love to help. Send any test builds my way ;-)

[htguy]

I have a WHS PP2 server with the following drives:
Seagate 1TB (SYS/D)
Seagate 1TB (1ST POOL)
Seagate 750GB (2ND POOL)
Seagate 500GB (3RD POOL)
Seagate 500GB (4TH POOL)

Out of concern for a failure of the 1TB drives (I had a WHS console warning that the 1st pool drive was missing, it returned after a reboot though) I wanted to lower my risk by using the balancing utility. I started it around seven pm last night and while I was monitoring it with disk management it kept on constantly stopping / restarting the DE migrator service due to low disk activity. Drive 0 and Drive 1 went from 7% and resp 0% to 71% and 43% over night. I tried pausing the application with CTRL-S and when I did that I am seeing disk activity in the regions of 7 MB/S to 40 MB/S in contrast to around 2 - 3MB/S when the application is running.

My scenario was with 88% full SYS/D drive and 95% full 1st pool drive. I have had it up and running for several days with DE doing it's thing by it self.

[Beefcake]

I have a WHS PP2 server with the following drives:
Seagate 1TB (SYS/D)
Seagate 1TB (1ST POOL)
Seagate 750GB (2ND POOL)
Seagate 500GB (3RD POOL)
Seagate 500GB (4TH POOL)

Out of concern for a failure of the 1TB drives (I had a WHS console warning that the 1st pool drive was missing, it returned after a reboot though) I wanted to lower my risk by using the balancing utility. I started it around seven pm last night and while I was monitoring it with disk management it kept on constantly stopping / restarting the DE migrator service due to low disk activity. Drive 0 and Drive 1 went from 7% and resp 0% to 71% and 43% over night. I tried pausing the application with CTRL-S and when I did that I am seeing disk activity in the regions of 7 MB/S to 40 MB/S in contrast to around 2 - 3MB/S when the application is running.

My scenario was with 88% full SYS/D drive and 95% full 1st pool drive. I have had it up and running for several days with DE doing it's thing by it self.

Long awaited update is ready for testing.

From Readme -

Revision History:
v1.05
- Attempting to solve problem where no files are moved on the system when D:\is filled
prior to running Drive Balancer. Please read UPDATE NOTE for potential solution
- Fixed bug where calculation of free space on D: was not "worked around". Refer to v1.03
history notes below. This fix will help systems that either
a) don't include D:\ in balancing
tried to clear the landing zone
- Added programmable time interval for checking disk activity
- Added programmable disk activity threshold for restarting Drive Extender service

UPDATED NOTE:
Some people have been having trouble not getting any data to move when their D:\ has lots of data on it. Before doing
anything else, please run Drive Balancer to clear the landing zone only. To do this, choose "n" to include d:, then
choose "y" to clear landing zone, and the "5" to exit. After this initial run (which can take hours), rerun Drive
Balancer the with the options you want. I believe this will help the problem of no data being moved on some people's
systems.

[FloridaGuy]

I installed v1.05 and started the process. But I'm getting the "De Service is being restarted Due to low disk Activity" message about every 30 seconds and it keeps saying Drive 2 88% done

I've had this running now for about 24 hours with the same messages over and over... there is disk activity being reported in the Disk Management plugin. It just does not seem to be going past Drive 2 and 88% done.

Any ideas?

FloridaGuy

[Beefcake]

Add up the disk activity you see in Disk management and set your activity threshold below this number. Drive Balancer also spits out disk bandwidth totals....what is it reporting? Try setting your threshold below that number as well. Does Disk Management and Drive Balancer report the same total disk bandwidth? They should......

You could also try a longer timeout between disk activity checks. That would change the overall length of time between trying to start DE. By itself, this won't solve anything, but will help in conjunction with above option.


Another thought...you may want to check out your error logs to make sure your system hasn't suffered any kidn of error on the SATA drive/controller. My own system had some issues when I ran the ICH10R controller on the motherboard in anything BUT IDE enhanced mode (changeable in the BIOS). Both AHCI or SATA mode functioned for a while, but I eventually had errors getting logged against the controller and my throughput went way down to like 2-3MB/sec. That is way to slow for ANY sata based drive.

For now, hit Ctrl-C and perform the cleanup ritual specified in the readme and respond to my questions and check your error logs. After that, I can help you diagnose the problem and help you get Drive Balancer running perfectly for you (hopefully).

-Brian

I installed v1.05 and started the process. But I'm getting the "De Service is being restarted Due to low disk Activity" message about every 30 seconds and it keeps saying Drive 2 88% done

I've had this running now for about 24 hours with the same messages over and over... there is disk activity being reported in the Disk Management plugin. It just does not seem to be going past Drive 2 and 88% done.

Any ideas?

FloridaGuy

[rmassey]

I'm getting pretty much the same result here as others, I run the app, do not include D, clear the LZ, quit, restart, do not include D and then pick equal percent (#1) and it just runs forever (20+ hours) and keeps restarting DE every 30 seconds or so.

I'd really like to get this working as my drives are way out of balance. I tried to run this twice over night with the same results. I cleared off 150 GB yesterday to an external USB drive to see if this would make a difference. Trying run #3 today.

Here's my system.
Disc 0- 1 (300, 500) are on the onboard IDE controler.
Disc 2-5 (4 x 500) are on a Promise Raid controler as JBOD.
Disc 6 (1.5T) is on a PCI eSata controler and is in an ext. Antec case (just added and unused by WHS so far)
Disc 7 (1.5T) is another eSata used for backups, not part of the storage pool.

Disc 0 is full now due to the fake files. Disc 2 & 5 are really that full.
I am not using duplication on any folders.

Pleases let me know of any suggestions I can try to get the app working

[rmassey]

Just for fun, I removed Disc 1 and 6 from the storage pool, leaving the 300 boot drive and 4-500s on the Raid controler, and I'm running Drive Balancer again. Not much change, DE still endlessly restarting. Is the problem perhaps related to my use of the Promise Raid controler with my 4x500 drives?

[Beefcake]

The part I'm confused about is you say you cleared the (d:) landing zone on the first run, but in your pics, the d drive is 92% used still. Something does add up there. Perhaps I needs to tweak the landing zone clearing algorithm to be more like the "crunch time" algorithm.

Could you comment on why D: shows as 92% in your images....are those images after or before the Clearing of the D drive was done?

For now, what I would try is to enter in a longer wait time between checking disk activity. try something like 30 seconds. Then, it would take like 5*30 = 150 seconds of continuous low disk throughput to restart DE on your system. Then, watch your disk throughput as DB is running as waiting. Are files actually getting moved (i.e. - what is the disk throughput?).

-Brian

Just for fun, I removed Disc 1 and 6 from the storage pool, leaving the 300 boot drive and 4-500s on the Raid controler, and I'm running Drive Balancer again. Not much change, DE still endlessly restarting. Is the problem perhaps related to my use of the Promise Raid controler with my 4x500 drives?

[rmassey]

The part I'm confused about is you say you cleared the (d:) landing zone on the first run, but in your pics, the d drive is 92% used still. Something does add up there. Perhaps I needs to tweak the landing zone clearing algorithm to be more like the "crunch time" algorithm.

Could you comment on why D: shows as 92% in your images....are those images after or before the Clearing of the D drive was done?

For now, what I would try is to enter in a longer wait time between checking disk activity. try something like 30 seconds. Then, it would take like 5*30 = 150 seconds of continuous low disk throughput to restart DE on your system. Then, watch your disk throughput as DB is running as waiting. Are files actually getting moved (i.e. - what is the disk throughput?).

-Brian

The 92% disc 0 is from the fake files, I did not remove them prior to the second run.
Thi pic is from after the clearing and during the second run, answering no - do not include D.

I'll have to check the throughput numbers, but it does not appear that anything is moving files.

[Beefcake]

Ok, this at least makes more sense.

I would follow my second recommendation of using a larger timeout between checking disk activity as well as a lower disk throughput threshold. I'd try 30 seconds and 2MB/sec for the values and see what happens.

-Brian

The 92% disc 0 is from the fake files, I did not remove them prior to the second run.
Thi pic is from after the clearing and during the second run, answering no - do not include D.

I'll have to check the throughput numbers, but it does not appear that anything is moving files.

[rmassey]

OK, I tried what you suggest last night (30 sec & 2) with the same results. I forget the exact numbers, but it just keeps thrashing away, constantly restarting DE, and reporting the same disc % results for hours. No files are moving from what I see.

The problem is this completely ties up my WHS as I shutdown SqueezeCenter and try to keep server activity to RDP only. I've run this app 5-6 times now with zero results. Unless I can get specific suggestions for my WHS, I think I may be done.

edit - one more idea.. I have pyTivo service (serves video to my HDtivos) running. Could this causing issues, even though I am not xfering any files? I will stop this service and try again before bedtime 2nite.

Ok, this at least makes more sense.

I would follow my second recommendation of using a larger timeout between checking disk activity as well as a lower disk throughput threshold. I'd try 30 seconds and 2MB/sec for the values and see what happens.

-Brian