396 replies to this topic

[bassfiend]

*sigh* In the hope that someone who can make a decision at MS about this is reading can I add another plea to do something about "demigrator.exe"?

I've just worked out that my current streaming issues are down to demigrator.exe deciding to strut its stuff on my server every so often - its definitely not CPU bound but what it is is I/O bound. If demigrator.exe starts doing its stuff on a drive which I'm streaming from then it completely screws my hi-def streaming for maybe a couple of minutes whilst it moves on to another drive.

I know people have suggested not using pooled drives in this case but that isn't any good for me - my entire WHS machine is one *HUGE* media server (currently 12 x 1Tb Western Digital Green Power drives) and having a share per drive just wouldn't work very well. (This is why I moved from a server based around a Windows XP install and over to WHS.)

Phil

Edit (by Drashna):
If you are experiencing this issue also, please visit this Connect Bug report and vote and confirm the issue. Add your CAB number and Hardware config if possible to the confirmation.

Remember, this will *only* get fixed if it's brought to their attention, and as there seems to be no prior bug report for this, it won't get fixed without a lot of votes and confirmations.

Bug has been marked as "Resolved [Won't Fixed]".

Checklist before blaming Drive Extender Migrator Service
aka, a list of possible culprits:

Quote

Here's a good checklist:

For the Network Adapter (in device management or "Configure" on the properties page):
Flow control: on (client and server)
"checksum" or "Offload": Off (client and server)
jumbo frames: off or default (client or server)
Green Ethernet: off (this decreases the power used, and for me seems to cause issues, Client and Server)
Interrupt Moderation: off (client and server)

run "netsh interface tcp set global autotuninglevel=highlyrestricted" in an elevated console on the client machine (if Vista or Windows 7). (client)
Under network connection, where "Internet Protocol Version 4" is, make sure "QoS Packet Scheduling" is turned off or removed on all clients. (Client and Server)

Services/programs:
Make sure nothing is using d:\shares, if possible. If something is, disable it for a while to make sure it's not the culprit (like WMP11's WMPNSS, utorrent, or or the like)
Windows Media Connect: disable/manual (at least temporarily)
*note: WHS will alert you about this. If you want to permanently disable that, set "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Home Server\Storage Manager", "UseThirdPartyMediaSharingProvider" (dword) to 1
Windows Media Player Network Sharing Service: Disabled/manual (if you have WMP11 installed, and again temp)
Windows Search Service: Disabled (and probably a good idea to move it to another drive anyhow, or at least d:\)
DEMigrator: Disabled and stopped.

Misc:
Run chkdsk on all the disks. Full is best, but just to make sure
Defrag.
*both DisKeeper and Perfect Disk have trial versions for WHS that can be used at least to clean up the system once. If Defragging helps/solves the problem, I'd say it's a sound investment.

Workaround
Create these two files, and add them to the task schedular at appropriate times for you to stop and start DEMigrator, respectively.

DemigratorOff.bat

@echo off
sc stop "DriveExtenderMigrator"
sc config "DriveExtenderMigrator" start= disabled
exit

DemigratorOn.bat

@echo off
sc config "DriveExtenderMigrator" start= demand
sc start "DriveExtenderMigrator"
exit

Also, Damian over at MediaSmartServer.net was kind enough to put together a blog post on how exactly to schedule starting and stopping Drive Extender. Check it out here.
Edited title to be more search friendly.

[Jeff Flowerday]

 bassfiend, on May 25 2009, 05:50 PM, said:

*sigh* In the hope that someone who can make a decision at MS about this is reading can I add another plea to do something about "demigrator.exe"?

I've just worked out that my current streaming issues are down to demigrator.exe deciding to strut its stuff on my server every so often - its definitely not CPU bound but what it is is I/O bound. If demigrator.exe starts doing its stuff on a drive which I'm streaming from then it completely screws my hi-def streaming for maybe a couple of minutes whilst it moves on to another drive.

I know people have suggested not using pooled drives in this case but that isn't any good for me - my entire WHS machine is one *HUGE* media server (currently 12 x 1Tb Western Digital Green Power drives) and having a share per drive just wouldn't work very well. (This is why I moved from a server based around a Windows XP install and over to WHS.)

Phil

Are your drives all hooked up sata or do you have some USB in the mix?

[bassfiend]

 Jeff Flowerday, on May 26 2009, 03:06 PM, said:

Are your drives all hooked up sata or do you have some USB in the mix?

Everything's hooked up SATA - have 12 data drives in the box itself (10 x 1Tb WD Green Power + 2 x 1Tb Samsung F1 - I forgot about the two Samsung F1's!) + 250Gb boot drive. There's another 15 drive bays going onto the machine later this week as three 5-bay SATA backplanes in an external enclosure with three 5 drive eSATA port multiplier boards and a PCI-X 4 x eSATA card that supports port multipliers giving me the capacity for 27 drives. (Should be enough for another year I reckon.)

No USB or firewire drives in the mix at all.

Phil

[Jeff Flowerday]

Well I haven't had any of my HD stuff stutter at all because of demigrator. I'm running 6 seagate 1.5TB drives all via sata and eSata. 2x1TB Hitachi in a Raid 1 for the system drive.

Knock on wood!

I assume you've recently added some new files to the server thus demigrator being busy? If you haven't done so in a few days there is no way demigrator should be causing you issues, IMO.

[Drashna Jaelre (WGS)]

Try installing Windows Desktop Search 4.0. During the public beta of PP1, it would help this issue a lot.

[bassfiend]

 Jeff Flowerday, on May 26 2009, 05:02 PM, said:

I assume you've recently added some new files to the server thus demigrator being busy? If you haven't done so in a few days there is no way demigrator should be causing you issues, IMO.

Hi Jeff, Thanks for the input...

I usually add new movies to my server every few days as I buy them - however I don't think that is the cause of it as Demigrator seems to run at regular intervals no matter what and each time it does it spends about 20-30 seconds on each of the drives in turn (you can see demigrator.exe start to take up 4 or 5% of CPU usage in task manager and watch the drive activity lights cycling through).

The thing is, no matter whether demigrator is running to actually move files around or whether it's just running to do housekeeping, whatever it is doing is causing the server to thrash its drives and, when it hits a drive I'm streaming hi-def from, this causes it to stutter. Demigrator.exe should not be coming in and doing that when the drive is busy - simple.

Phil

[Jeff Flowerday]

Not sure this is going to help you but, i'm running all my drives in AHCI mode. In this situation Command Queuing might be helping me out, thus why I don't see stuttering when demigrator kicks in.

The other thing is my drives are all seagate 7200 RPM seagate drives those Green drives rotate at about 5400 I do believe which may make them more susceptable to bottle necking when multiple things are hitting them.

Or I might be lucky and it never hit a drive while I was watching a movie.

[bassfiend]

 Drashna (WGS), on May 30 2009, 03:33 AM, said:

Try installing Windows Desktop Search 4.0. During the public beta of PP1, it would help this issue a lot.

OK - I've just installed that and rebooted the server ... it's currently hammering the hell out of the drives so I assume that's it busy indexing. Let's give it a couple of days to settle down and see if it makes a difference.

Cheers

Phil

[niw_uk1964]

 bassfiend, on Jun 21 2009, 11:57 AM, said:

OK - I've just installed that and rebooted the server ... it's currently hammering the hell out of the drives so I assume that's it busy indexing. Let's give it a couple of days to settle down and see if it makes a difference.

Cheers

Phil

FWIW when demigrator runs on my box it hits the ability to stream...as you would expect because i/o gets hammered by disk writes/reads performed by the process. I don't know a way around it myself other than to keep the drives balances as much as possible.

[marwerno]

Just to add to the conversation another plead to Microsoft with the same problem:
I only have 1 Hard disk for the server (Plus 1 as Backup disk)
<off topic>
I even disabled the WHS Backup using a 3rd Party Backup which is more reliable and Desktop search to remove some more system hoggers (Desktop search is SO useless when you have ever changing extensions that are not in the Search index and I am not going to manually add all the fancy extensions that are popping up)
</off topic>.
AND YET: DEMIGRATOR STILL USES UP TO 50%!
Why does it run? I have no clue, it should basically be out of work since there is nothing to distribute, is there?
And it also clamps me down right at the times when I am watching a movie/ streaming from the server, very annoying....

[parker]

I have the same problem especially streaming HD recorded TV back to Media Center. Setup is 5 500GB Sata 2 drives and an IDE 320 system drive. You can see demigrator start up in the Task Viewer and then the disk activity spikes (watching this on a laptop while also watching recorded tv on the main media center computer). When the disk activity spikes, the HD starts stuttering. Haven't seen it with ripped DVD's though, I don't think they require the same level of disk activity. Network is all a wired 1GB network and Task Viewer only shows peak of around 20 MB/sec on network activity (unless you fast forward which can peak up to 100MB).

[benogil]

I have seen a post,elsewhere, which said a possible cure is 7200 rpm drives. The green's are 5400. This topic is seen in many places.

Ben

[marwerno]

Well, of course, the more "Hi-tech" you trow at a computer (Speed) the better it will get.
The thing is: It shouldn't be required to work with such workarounds.
It should work out of the box:
Meaning: Demigrator should stop doing whatever it is doing when there is other activity happening. Or it should be possible to "Schedule" it, just set a time during nighttime when you are most likely not using the Server, than I am happy for Demigrator to use 100% CPU. Or the other way round, set "forbidden" times when Demigrator should NOT run.
I am especially puzzled why it still kicks in at my machine where I only have 1 drive with shares on it.
I used to have 2 Drives as shares, but after finding out all the problems (cannot record with the server TV programs, Demigrator constantly running, streaming pretty much worthless, since the TV recordings would happen at the same time I am at home and want to stream something else., Backup is not working properly. Backup can get damaged, real problem with "always open" files (PST file issue), etc.
Also: If Demigrator is NOT running I have no problem recoding 2 TV shows (SD) AND watch a HD TV stream (all of 1 Hard disk!), which means Demigrator just puts too much stress on the drive.

I am currently seriously thinking of "downgrading" (which currently, in performance terms would actually be an Upgrade) myself to Windows XP and set that up as Server! Additionally (I know it is not Microsoft fault) there is still no real value for money (or better free) good quality virus scanner out there that runs on the server....

[benogil]

I agree with you. I was only trying to relate some viable fixes. The experiences of others suggest four ways to go - faster drives, turn off duplication, use diskeeper, or flexraid. The standard build, for many, will not function as a media streamer reliably.
If you read the MS forum, you can find much dissembling on this topic.

Ben

[shadymg]

count me in on those being affected by this, it's pretty terrible. It seems like every night i try to stream anything in HD and the demigrator kicks in. Of course i have all 4 drives (EX487) pooled together, but that's the whole point...dissapointing. Hopefully a fix comes out sooner than later

[autodrivel]

 bassfiend, on May 26 2009, 12:50 AM, said:

I know people have suggested not using pooled drives in this case but that isn't any good for me - my entire WHS machine is one *HUGE* media server (currently 12 x 1Tb Western Digital Green Power drives) and having a share per drive just wouldn't work very well. (This is why I moved from a server based around a Windows XP install and over to WHS.)

Phil

I handle the "Share-per-Drive" problem of non-pooled drives by using NTFS junctions - That is; I have a single shared directory which just contains a collection of NTFS junctions pointing to each of my DVD-rips on the other drives. My library is currently about 1000 DVDs stored on 6 x 1TB Green Power Drives (just waiting for the 2TB Greens to drop to around £160 and I will start replacing the 1TB units ).

The junctions are easy enough to setup by hand, but I have written a Powershell script to handle this, so I can easily recreate the entire collection just by running the script.

[marwerno]

@autodrivel

You are still using WHS? Or another OS?
Do I get it right (IF you do use WHS): You are having only 1 drive that is visible by WHS and then have more drives where you create NTFS junctions inside the shared folders?
Do you still have the Demigrator issue?

Otherwise, how do you do this? NTFS junctions is the same as mounting a new drive as Folder?

[benogil]

There was another post on this on the MS WHS forum, sorry, it received no attention. Smallnetworkbuilder had an article, but no help. Autodrivel's model of front end/back end may be the only reliable work around.

Ben

[Drashna Jaelre (WGS)]

To be completely honest/blunt, as I've come to understand it, the DE uses these same type of junction points, but uses a file system filter to handle everything other than migration.

Personally, I've *never* be affected by this issue. I don't have a lot of HD content, and my videos are not duplicated (not enough storage space to do so). The only time I've had issues is with hardware/software stuff not configured properly (including write caching on the HDDs, and settings on my NIC). I get an average of 30-40MB/s reliably with it rarely, if ever dropping below that. And I run my server as a DC DHCP/DNS/WDS/WINS server, run VMs, subsonic, a public blog, as a media server, a file server, as well as test various other things on it. So at best, I can only theorize that it is a config issue without knowing every configuration, settings, et cetera.

Though, anyone having *any* issues should submit a bug report on Connect anyways. The *only* way it will get resolved is if the WHS Team knows about it.

I know it sucks for everyone experiencing it. And I do hope for a good solution, rather than trying to track down possible "weak points" in each and every system.

[bassfiend]

 bassfiend, on Jun 21 2009, 11:57 AM, said:

OK - I've just installed that and rebooted the server ... it's currently hammering the hell out of the drives so I assume that's it busy indexing. Let's give it a couple of days to settle down and see if it makes a difference.

Cheers

Phil

This didn't help by the way - I also reformatted all my drives with 64k blocks over the last few weeks (Jesus that was a chore!) but this evening whilst streaming a BluRay ISO it started stuttering again. I quickly popped up a Remote Desktop window to the WHS machine and sure enough Demigrator had kicked in and was running using up 15-25% CPU.

I stopped the Demigrator process just to see what happened and *IMMEDIATELY* the stuttering in my streaming stopped ... a bit later it began stuttering again, I checked the task manager on the WHS box and Demigrator was once again running and taking up CPU time. It's not the CPU usage which is the issue but the fact that it's trying to defragment / shuffle files on a drive that's being streamed off at the same time. :-(

Phil