I got called up about an hour ago by my monitoring service that I had a server down. Sure enough, it was down.
I opened a ticket with the data center explaining which server was down (I have four servers in the same rack) and other important details.
One thing about the techs there at the data center, they are FAST. Within a couple minutes, I had a tech reply with: "I'm rebooting it now".
Shortly after, another of my other servers went down.
I replied to the ticket that he rebooted the wrong one.
Meanwhile, I added that I thought there was something else weird at the switch becuase my other servers were really slow in the login process. It could be that the down server was locked in a loop and hogging all the available switch bandwidth, but that was unlikely. It seemed more than just a server that puked.
While I was posting that, the tech replied again "Sorry about doing the wrong one, I'm rebooting the right one now"
Well, he missed again . . . yet another running server went off line while it was rebooting.
I told him he rebooted yet a different working server, and he asked me if it was the one on top of all the others.
Well, you're in Phoenix. I've never been there. I don't know what order they are in in the rack since you guys put them in. I can't see you or my servers from here, so I don't know which one you put on top.
Server number 3 is in a distinctly different case than the others so I told him it is NOT that one, but the only other one left that he hadn't rebooted.
Meanwhile, I posted again that it seems like a problem with the switch just the way things were acting from what I could see.
He then got the down server rebooted, but it was still unaccessible. They replaced the network cable between the server and the switch and everything came back up like it should. Yay!
I STILL think that the switch went flakey and unplugging the cable let it come back to normal. They think it was the cable (and it could be) but time will tell if it happens again. Just in case, I think I'll send them a new switch just to have on hand.
After we got all done, the tech took a few minutes to label the servers correctly so next time, they will know which one is which . . . . good idea, guys.
Now, I'll see if I can calm down and get back to sleep.
Ahoy, maties! Avast it be another day of totin' barges and liftin' bales aboard the good ship "Sinkin' Feelin'"
Moving slowly yet this morning. I went to see Rev. Peyton's Big D@mn Band last night. If you have never heard of them, I strongly encourage for you to check out their music if you get the chance. I never in my wildest thought I would say to another living being, "Ma'am... you can really ROCK a washboard!"... but I did. Amazingly well worth the listen.
Not much going on today, so I wasn't in a big ol' hurry to get to the office. We'll see how the day progresses.
__________________
MM
That which does not kill me postpones the inevitable.