Forum Search:
Forum.Brain-Cluster.com: Brain Cluster Technical Forum
Ultimate forum for Technical Discussions

Home » Microsoft » Windows Server » Server Clustering » Hyper-v cluster guests loosing network access and or hosts getting blue screen :-(
Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #422325] Tue, 30 March 2010 06:56 Go to next message
Tiago Lock Martins  is currently offline Tiago Lock Martins
Messages: 19
Registered: July 2009
Junior Member
Hello,

I`m having some issues on my Hyper-V environment.

My environment:

Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710 with
32GB RAM.
3 node cluster set up with quorum disk, have default type cluster disks
and CSV disks.
1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached to
virtual switch 2, 2 NICs as link aggregation attached to virtual switch 1
2 fiber channel HBAs attached to Brocade switches and Dell EMC CX300
storage.
Around 52 LUNs associated to the cluster
Some LUNs contains VHDs and others are RAW disks
All guests NICs are synthetics

When I had a 2 node cluster, some guests just loose network connection,
even removing virtual switch connection with the guest NIC and attaching it
back, doesn`t resolves the problem, need to restart the guest to get back to
network. Sometimes the guests just hang.

After adding a 3rd node to the cluster, the hosts started to restart
after blue screens ( got Overlapped I/O message on 1 of them ), sometimes
the host don`t restart but it simply restarts the guests running on it.

I`ve been searching for 1 week already and didn`t found nothing that
really helps.

Anyone have any clue?

If need more info, please tell me. Will be happy to help you to help me
:-D

Thanks for your time
Re: Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #423343 is a reply to message #422325] Wed, 31 March 2010 15:02 Go to previous messageGo to next message
RCan  is currently offline RCan
Messages: 248
Registered: January 2010
Senior Member
Hi Tiago,

first of all did you had changed the quorum model when you added the 3rd
node afterwards ? To verify, in your case the quorum model should be now
"node majority".
http://technet.microsoft.com/en-us/library/cc770620(WS.10).aspx#BKMK_RequirementsNandFS

can you also please share more details around your eventlogs and more
important the cluster log which should be generated (cluster /log gen)
shortly after this happens.

Regards
Ramazan

"Tiago Lock Martins" <TLock@community.nospam> wrote in message
news:68ADEA5E-7F0C-4E0D-8BB8-4108079E3654@microsoft.com...
> Hello,
>
> I`m having some issues on my Hyper-V environment.
>
> My environment:
>
> Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710 with
> 32GB RAM.
> 3 node cluster set up with quorum disk, have default type cluster disks
> and CSV disks.
> 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached to
> virtual switch 2, 2 NICs as link aggregation attached to virtual switch 1
> 2 fiber channel HBAs attached to Brocade switches and Dell EMC CX300
> storage.
> Around 52 LUNs associated to the cluster
> Some LUNs contains VHDs and others are RAW disks
> All guests NICs are synthetics
>
> When I had a 2 node cluster, some guests just loose network connection,
> even removing virtual switch connection with the guest NIC and attaching
> it
> back, doesn`t resolves the problem, need to restart the guest to get back
> to
> network. Sometimes the guests just hang.
>
> After adding a 3rd node to the cluster, the hosts started to restart
> after blue screens ( got Overlapped I/O message on 1 of them ), sometimes
> the host don`t restart but it simply restarts the guests running on it.
>
> I`ve been searching for 1 week already and didn`t found nothing that
> really helps.
>
> Anyone have any clue?
>
> If need more info, please tell me. Will be happy to help you to help me
> :-D
>
> Thanks for your time
>
>
RE: Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #423815 is a reply to message #422325] Wed, 31 March 2010 06:09 Go to previous messageGo to next message
v-mervzh  is currently offline v-mervzh
Messages: 39
Registered: July 2009
Member
Hello,

The managed support service of the newsgroup Hyper-V is now available
instead on:

Hyper-V
http://social.technet.microsoft.com/Forums/en-US/winserverhy perv/threads

Would you please repost the question in the forum with the Windows Live ID
used to access your Subscription benefits? Our engineers will assist you in
the new platform. In the future, please post Windows Server related
questions directly to the forums. If you have any questions or concerns,
please feel free to contact us: tngfb@microsoft.com.

Regards,
Mervyn Zhang
Microsoft Online Community Support

==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
Re: Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #423975 is a reply to message #422325] Thu, 01 April 2010 11:46 Go to previous messageGo to next message
Brad  is currently offline Brad  Canada
Messages: 46
Registered: July 2009
Member
Hello Tiago,

I am curious to know if the suggestions from RCan helped you resolve this.

This sounds suspiciously like an issue I had at University of Ottawa. To
band-aid the issue, I would console to the VM and (disable/enable) the NIC
in the guest to reset the IP stack. This was on IBM servers.

At the time, we thought this was due to Broadcom firmware from IBM information
but the problem was never completely solved and since I don't work there
anymore, I don't know to this day if it was...

I realize your scenario is not the same nor do you have the same hardware.
I hope the band-aid helps save you time if it resolves your issue faster
than a reboot...

Do you have any network stats being monitored? This is where I would start
looking...

> Hello,
>
> I`m having some issues on my Hyper-V environment.
>
> My environment:
>
> Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710
> with
> 32GB RAM.
> 3 node cluster set up with quorum disk, have default type cluster
> disks
> and CSV disks.
> 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached
> to
> virtual switch 2, 2 NICs as link aggregation attached to virtual
> switch 1
> 2 fiber channel HBAs attached to Brocade switches and Dell EMC
> CX300
> storage.
> Around 52 LUNs associated to the cluster
> Some LUNs contains VHDs and others are RAW disks
> All guests NICs are synthetics
> When I had a 2 node cluster, some guests just loose network
> connection, even removing virtual switch connection with the guest NIC
> and attaching it back, doesn`t resolves the problem, need to restart
> the guest to get back to network. Sometimes the guests just hang.
>
> After adding a 3rd node to the cluster, the hosts started to
> restart after blue screens ( got Overlapped I/O message on 1 of them
> ), sometimes the host don`t restart but it simply restarts the guests
> running on it.
>
> I`ve been searching for 1 week already and didn`t found nothing
> that really helps.
>
> Anyone have any clue?
>
> If need more info, please tell me. Will be happy to help you to
> help me :-D
>
> Thanks for your time
>
Re: Hyper-v cluster guests loosing network access and or hosts get [message #429951 is a reply to message #423975] Mon, 12 April 2010 03:59 Go to previous messageGo to next message
AndyS  is currently offline AndyS
Messages: 3
Registered: April 2010
Junior Member
Hi folks

This sounds like a problem that we're having and we are using Broadcom NICs.
We are running Hyper-v on Win 2008R2 on a dell R610 and have Windows Server
2003 SP2 guests. The symptoms of the problem are virtual server guests loose
network connectivity randomly (once every week or so) and some perform so
poorly after loosing network connectivity that they have to be forced to shut
down rather than rebooted properly.

From what information I can find there seems to be a link with Broadcom
adapters. Some suggest disabling 'IPv4 Large Send Offload' on the physical
adapters on the host which we have done, however we still get servers falling
over. Another suggestion was to disable 'IPv4 Large Send Offload' on the
guest virtual adapters (inside the guest Win server 2003 OS) but this caused
servers to fall over every few hours. The only errors I can find before the
guests loos network connectivity is 'Event ID 5 - The miniport 'Microsoft
Virtual Machine Bus Network Adapter' hung.' followed by 'Event ID 4 - The
miniport 'Microsoft Virtual Machine Bus Network Adapter' reset.'

We have a call open with Microsoft regarding this issue but we haven't got
very far.

Cheers

Andy

"Brad Bird (MVP)" wrote:

> Hello Tiago,
>
> I am curious to know if the suggestions from RCan helped you resolve this.
>
> This sounds suspiciously like an issue I had at University of Ottawa. To
> band-aid the issue, I would console to the VM and (disable/enable) the NIC
> in the guest to reset the IP stack. This was on IBM servers.
>
> At the time, we thought this was due to Broadcom firmware from IBM information
> but the problem was never completely solved and since I don't work there
> anymore, I don't know to this day if it was...
>
> I realize your scenario is not the same nor do you have the same hardware.
> I hope the band-aid helps save you time if it resolves your issue faster
> than a reboot...
>
> Do you have any network stats being monitored? This is where I would start
> looking...
>
> > Hello,
> >
> > I`m having some issues on my Hyper-V environment.
> >
> > My environment:
> >
> > Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710
> > with
> > 32GB RAM.
> > 3 node cluster set up with quorum disk, have default type cluster
> > disks
> > and CSV disks.
> > 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached
> > to
> > virtual switch 2, 2 NICs as link aggregation attached to virtual
> > switch 1
> > 2 fiber channel HBAs attached to Brocade switches and Dell EMC
> > CX300
> > storage.
> > Around 52 LUNs associated to the cluster
> > Some LUNs contains VHDs and others are RAW disks
> > All guests NICs are synthetics
> > When I had a 2 node cluster, some guests just loose network
> > connection, even removing virtual switch connection with the guest NIC
> > and attaching it back, doesn`t resolves the problem, need to restart
> > the guest to get back to network. Sometimes the guests just hang.
> >
> > After adding a 3rd node to the cluster, the hosts started to
> > restart after blue screens ( got Overlapped I/O message on 1 of them
> > ), sometimes the host don`t restart but it simply restarts the guests
> > running on it.
> >
> > I`ve been searching for 1 week already and didn`t found nothing
> > that really helps.
> >
> > Anyone have any clue?
> >
> > If need more info, please tell me. Will be happy to help you to
> > help me :-D
> >
> > Thanks for your time
> >
>
>
> .
>
Re: Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #434286 is a reply to message #423343] Tue, 20 April 2010 08:55 Go to previous messageGo to next message
Tiago Lock Martins  is currently offline Tiago Lock Martins
Messages: 19
Registered: July 2009
Junior Member
Thx for the reply RCan,

Yes, I changed the quorum do node majority.

In fact, now it is set to disk and node majority, as I have 4 nodes now
:-)...... omw to 5th :-D

Will see the cluster /log gen results as soon as it happens again.

Will keep group informed.


"RCan" <noospam@arcor.de> escreveu na notícia da
mensagem:C8D2696C-E32A-41D9-9B2D-63E857F3066E@microsoft.com...
> Hi Tiago,
>
> first of all did you had changed the quorum model when you added the 3rd
> node afterwards ? To verify, in your case the quorum model should be now
> "node majority".
> http://technet.microsoft.com/en-us/library/cc770620(WS.10).aspx#BKMK_RequirementsNandFS
>
> can you also please share more details around your eventlogs and more
> important the cluster log which should be generated (cluster /log gen)
> shortly after this happens.
>
> Regards
> Ramazan
>
> "Tiago Lock Martins" <TLock@community.nospam> wrote in message
> news:68ADEA5E-7F0C-4E0D-8BB8-4108079E3654@microsoft.com...
>> Hello,
>>
>> I`m having some issues on my Hyper-V environment.
>>
>> My environment:
>>
>> Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710 with
>> 32GB RAM.
>> 3 node cluster set up with quorum disk, have default type cluster
>> disks
>> and CSV disks.
>> 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached to
>> virtual switch 2, 2 NICs as link aggregation attached to virtual switch 1
>> 2 fiber channel HBAs attached to Brocade switches and Dell EMC CX300
>> storage.
>> Around 52 LUNs associated to the cluster
>> Some LUNs contains VHDs and others are RAW disks
>> All guests NICs are synthetics
>>
>> When I had a 2 node cluster, some guests just loose network
>> connection,
>> even removing virtual switch connection with the guest NIC and attaching
>> it
>> back, doesn`t resolves the problem, need to restart the guest to get back
>> to
>> network. Sometimes the guests just hang.
>>
>> After adding a 3rd node to the cluster, the hosts started to
>> restart
>> after blue screens ( got Overlapped I/O message on 1 of them ), sometimes
>> the host don`t restart but it simply restarts the guests running on it.
>>
>> I`ve been searching for 1 week already and didn`t found nothing that
>> really helps.
>>
>> Anyone have any clue?
>>
>> If need more info, please tell me. Will be happy to help you to help
>> me
>> :-D
>>
>> Thanks for your time
>>
>>
Re: Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #434292 is a reply to message #423975] Tue, 20 April 2010 09:05 Go to previous messageGo to next message
Tiago Lock Martins  is currently offline Tiago Lock Martins
Messages: 19
Registered: July 2009
Junior Member
Hi Brad,

Will see que results of cluster /log gen when the issue happens again.

Regarding disabling/enabling the guest NIC doesn`t repair the conectivity
:-(

Glad to see other thoughs.

"Brad Bird (MVP)" <brad@bradbird.ca> escreveu na notícia da
mensagem:c9743e005f0f68cc9fa820ccac1b@news.microsoft.com...
> Hello Tiago,
>
> I am curious to know if the suggestions from RCan helped you resolve this.
>
> This sounds suspiciously like an issue I had at University of Ottawa. To
> band-aid the issue, I would console to the VM and (disable/enable) the NIC
> in the guest to reset the IP stack. This was on IBM servers.
>
> At the time, we thought this was due to Broadcom firmware from IBM
> information but the problem was never completely solved and since I don't
> work there anymore, I don't know to this day if it was...
>
> I realize your scenario is not the same nor do you have the same hardware.
> I hope the band-aid helps save you time if it resolves your issue faster
> than a reboot...
>
> Do you have any network stats being monitored? This is where I would
> start looking...
>
>> Hello,
>>
>> I`m having some issues on my Hyper-V environment.
>>
>> My environment:
>>
>> Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710
>> with
>> 32GB RAM.
>> 3 node cluster set up with quorum disk, have default type cluster
>> disks
>> and CSV disks.
>> 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached
>> to
>> virtual switch 2, 2 NICs as link aggregation attached to virtual
>> switch 1
>> 2 fiber channel HBAs attached to Brocade switches and Dell EMC
>> CX300
>> storage.
>> Around 52 LUNs associated to the cluster
>> Some LUNs contains VHDs and others are RAW disks
>> All guests NICs are synthetics
>> When I had a 2 node cluster, some guests just loose network
>> connection, even removing virtual switch connection with the guest NIC
>> and attaching it back, doesn`t resolves the problem, need to restart
>> the guest to get back to network. Sometimes the guests just hang.
>>
>> After adding a 3rd node to the cluster, the hosts started to
>> restart after blue screens ( got Overlapped I/O message on 1 of them
>> ), sometimes the host don`t restart but it simply restarts the guests
>> running on it.
>>
>> I`ve been searching for 1 week already and didn`t found nothing
>> that really helps.
>>
>> Anyone have any clue?
>>
>> If need more info, please tell me. Will be happy to help you to
>> help me :-D
>>
>> Thanks for your time
>>
>
>
Re: Hyper-v cluster guests loosing network access and or hosts get [message #434305 is a reply to message #429951] Tue, 20 April 2010 09:07 Go to previous messageGo to next message
Tiago Lock Martins  is currently offline Tiago Lock Martins
Messages: 19
Registered: July 2009
Junior Member
Hi Andys,

Quote : " The only errors I can find before the
> guests loos network connectivity is 'Event ID 5 - The miniport 'Microsoft
> Virtual Machine Bus Network Adapter' hung.' followed by 'Event ID 4 - The
> miniport 'Microsoft Virtual Machine Bus Network Adapter' reset.' "

Same thing this side :-(

Anyone knows why this happen ( or may cause this behavior ) ?

Thanks for your time so far ppl.

"AndyS" <AndyS@discussions.microsoft.com> escreveu na notícia da
mensagem:082BBE80-B544-42B4-B7DF-0A1CA8C071B3@microsoft.com...
> Hi folks
>
> This sounds like a problem that we're having and we are using Broadcom
> NICs.
> We are running Hyper-v on Win 2008R2 on a dell R610 and have Windows
> Server
> 2003 SP2 guests. The symptoms of the problem are virtual server guests
> loose
> network connectivity randomly (once every week or so) and some perform so
> poorly after loosing network connectivity that they have to be forced to
> shut
> down rather than rebooted properly.
>
> From what information I can find there seems to be a link with Broadcom
> adapters. Some suggest disabling 'IPv4 Large Send Offload' on the
> physical
> adapters on the host which we have done, however we still get servers
> falling
> over. Another suggestion was to disable 'IPv4 Large Send Offload' on the
> guest virtual adapters (inside the guest Win server 2003 OS) but this
> caused
> servers to fall over every few hours. The only errors I can find before
> the
> guests loos network connectivity is 'Event ID 5 - The miniport 'Microsoft
> Virtual Machine Bus Network Adapter' hung.' followed by 'Event ID 4 - The
> miniport 'Microsoft Virtual Machine Bus Network Adapter' reset.'
>
> We have a call open with Microsoft regarding this issue but we haven't got
> very far.
>
> Cheers
>
> Andy
>
> "Brad Bird (MVP)" wrote:
>
>> Hello Tiago,
>>
>> I am curious to know if the suggestions from RCan helped you resolve
>> this.
>>
>> This sounds suspiciously like an issue I had at University of Ottawa. To
>> band-aid the issue, I would console to the VM and (disable/enable) the
>> NIC
>> in the guest to reset the IP stack. This was on IBM servers.
>>
>> At the time, we thought this was due to Broadcom firmware from IBM
>> information
>> but the problem was never completely solved and since I don't work there
>> anymore, I don't know to this day if it was...
>>
>> I realize your scenario is not the same nor do you have the same
>> hardware.
>> I hope the band-aid helps save you time if it resolves your issue faster
>> than a reboot...
>>
>> Do you have any network stats being monitored? This is where I would
>> start
>> looking...
>>
>> > Hello,
>> >
>> > I`m having some issues on my Hyper-V environment.
>> >
>> > My environment:
>> >
>> > Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710
>> > with
>> > 32GB RAM.
>> > 3 node cluster set up with quorum disk, have default type cluster
>> > disks
>> > and CSV disks.
>> > 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached
>> > to
>> > virtual switch 2, 2 NICs as link aggregation attached to virtual
>> > switch 1
>> > 2 fiber channel HBAs attached to Brocade switches and Dell EMC
>> > CX300
>> > storage.
>> > Around 52 LUNs associated to the cluster
>> > Some LUNs contains VHDs and others are RAW disks
>> > All guests NICs are synthetics
>> > When I had a 2 node cluster, some guests just loose network
>> > connection, even removing virtual switch connection with the guest NIC
>> > and attaching it back, doesn`t resolves the problem, need to restart
>> > the guest to get back to network. Sometimes the guests just hang.
>> >
>> > After adding a 3rd node to the cluster, the hosts started to
>> > restart after blue screens ( got Overlapped I/O message on 1 of them
>> > ), sometimes the host don`t restart but it simply restarts the guests
>> > running on it.
>> >
>> > I`ve been searching for 1 week already and didn`t found nothing
>> > that really helps.
>> >
>> > Anyone have any clue?
>> >
>> > If need more info, please tell me. Will be happy to help you to
>> > help me :-D
>> >
>> > Thanks for your time
>> >
>>
>>
>> .
>>
Re: Hyper-v cluster guests loosing network access and or hosts getting blue screen :-( [message #434541 is a reply to message #434292] Tue, 20 April 2010 15:25 Go to previous messageGo to next message
RCan  is currently offline RCan  Germany
Messages: 248
Registered: January 2010
Senior Member
Hi Tiago,

let me know about the results of your cluster logs. Mainly this is related
to network communication issues between nodes. Cross-check your network
config from MS best practice perspective.

Regards
Ramazan

"Tiago Lock Martins" <TLock@community.nospam> wrote in message
news:6B64CCD1-5C0F-4172-9BF0-21A2D0F4BD16@microsoft.com...
> Hi Brad,
>
> Will see que results of cluster /log gen when the issue happens again.
>
> Regarding disabling/enabling the guest NIC doesn`t repair the conectivity
> :-(
>
> Glad to see other thoughs.
>
> "Brad Bird (MVP)" <brad@bradbird.ca> escreveu na notícia da
> mensagem:c9743e005f0f68cc9fa820ccac1b@news.microsoft.com...
>> Hello Tiago,
>>
>> I am curious to know if the suggestions from RCan helped you resolve
>> this.
>>
>> This sounds suspiciously like an issue I had at University of Ottawa. To
>> band-aid the issue, I would console to the VM and (disable/enable) the
>> NIC in the guest to reset the IP stack. This was on IBM servers.
>>
>> At the time, we thought this was due to Broadcom firmware from IBM
>> information but the problem was never completely solved and since I don't
>> work there anymore, I don't know to this day if it was...
>>
>> I realize your scenario is not the same nor do you have the same
>> hardware. I hope the band-aid helps save you time if it resolves your
>> issue faster than a reboot...
>>
>> Do you have any network stats being monitored? This is where I would
>> start looking...
>>
>>> Hello,
>>>
>>> I`m having some issues on my Hyper-V environment.
>>>
>>> My environment:
>>>
>>> Windows 2008 R2 Enterprise Edition running on Dell PowerEdge R710
>>> with
>>> 32GB RAM.
>>> 3 node cluster set up with quorum disk, have default type cluster
>>> disks
>>> and CSV disks.
>>> 1 NIC host dedicated, 1 NIC set to trunk with two VLANs attached
>>> to
>>> virtual switch 2, 2 NICs as link aggregation attached to virtual
>>> switch 1
>>> 2 fiber channel HBAs attached to Brocade switches and Dell EMC
>>> CX300
>>> storage.
>>> Around 52 LUNs associated to the cluster
>>> Some LUNs contains VHDs and others are RAW disks
>>> All guests NICs are synthetics
>>> When I had a 2 node cluster, some guests just loose network
>>> connection, even removing virtual switch connection with the guest NIC
>>> and attaching it back, doesn`t resolves the problem, need to restart
>>> the guest to get back to network. Sometimes the guests just hang.
>>>
>>> After adding a 3rd node to the cluster, the hosts started to
>>> restart after blue screens ( got Overlapped I/O message on 1 of them
>>> ), sometimes the host don`t restart but it simply restarts the guests
>>> running on it.
>>>
>>> I`ve been searching for 1 week already and didn`t found nothing
>>> that really helps.
>>>
>>> Anyone have any clue?
>>>
>>> If need more info, please tell me. Will be happy to help you to
>>> help me :-D
>>>
>>> Thanks for your time
>>>
>>
>>
Re: Hyper-v cluster guests loosing network access and or hosts get [message #440873 is a reply to message #434305] Sun, 02 May 2010 22:55 Go to previous message
v-mervzh  is currently offline v-mervzh
Messages: 39
Registered: July 2009
Member
Hello,

The managed support service of the newsgroup Windows Server Clustering is
now available instead on:

Clustering
http://social.technet.microsoft.com/Forums/en-US/winserverCl ustering/threads

Would you please repost the question in the forum with the Windows Live ID
used to access your Subscription benefits? Our engineers will assist you in
the new platform. In the future, please post Windows Server related
questions directly to the forums. If you have any questions or concerns,
please feel free to contact us: tngfb@microsoft.com.

Regards,
Mervyn Zhang
Microsoft Online Community Support

==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
Goto Forum:
  


Current Time: Sun Sep 24 15:14:51 EDT 2017

Total time taken to generate the page: 0.05712 seconds
.:: Contact :: Home ::Sitemap::.

Powered by: FUDforum 3.0.0RC2.
Copyright ©2001-2009 FUDforum Bulletin Board Software