(Lviv community of .NET developers)

Notify me about server failure

May 23, 2009 21:32 by alexk
Hi everyone, sorry for keeping your waiting for my next post so long time. I was busy on Android OS investigation. But now I'm back.

Introduction


Do you know how to check is your server is still running and give your customers something? No. Then you should go to administrator room and ask technical stuff there. I think after hearing the answer you will be a little disappointed. On market still does not exist simple and easy solution.
Microsoft suggest to use there servers that cost a lot; several vendors suggest to buy there services (InternetSeer for example), others simply waiting customer feedback and re-act on it.
I think this is not right. It's time to change those things. I have to feel my servers on fingers, and re-act faster then any customer catch the problem.

Concept


In our company we have several servers each one specially installed for particular task. For administrator is critical to check stability and availability of those servers. But administrator can not sit 24/7 and watching servers. So we need small automation here.

Several questions from the begging catch my attention:
1) if all servers in domain failed, how I can receive notification about that? What happens if internet channel become broken physically?
2) How I can detect that several services on server stop working properly?
3) Can I provide some "sugar" for solution?!

So I start thinking and designing. Phone call interrupt me for a minute, but also give me a nice idea: GSM Modem can become a backup channel that can notify me about server problems.
WOW! Its was very nice idea and easy to implement. 10 minutes and GSM modem was ordered over internet. My choice is: NOVACOM GNS-30 CRA. It cost a little, it done in industrial standard metal case, easy to mount, easy to program.
NOVACOM GNS-30 CRA
Second question was easy to answer. I will use old PING ICMP protocol technology for that. If server respond to PING then it's working. But for checking services required something more interesting and solution as always was so easy - we are checking network ports. In case of services that does not communicate with network should be used some other monitoring things, like: event logs, custom logs, MSMQ messages and etc...
Third question is my favorite. And my answer is: check e-mails on server for administrative group and in case of e-mail with high priority notify about it by SMS.

Implementation


Design of library was a simple task and done by me in several hours.
Library design
Two general solution points are:
- support GSM modems pool;
- support attachable and enhanceable protocols;

Abstraction used in implementation give other developers great advantage: easy implementation of own protocols, that can access any known to us location; and easy attaching of other GSM Modems/devices.
I implement SMTP, POP3. IMAP4, HTTP and PING protocols which cover 99% of servers functionality. Also can be create protocols for checking SQL Server connection, Network Port or MSMQ, but I leave that for future.

Sending of SMS is well described in article: Developers Home

Key elements of the solution:
#1 - GSM Modems pool;
#2 - Special exceptions used by GSM modems providers;
#3 - Background Analyzer (ROOT object of the solution). User should instantiate it for accessing library functionality;
#4 - Access Protocol abstraction. Protocol can have several states, like: Unknown, Ok, Error, Pending, etc.
#5 - Message accessing protocols;
Rules classes
Special rules that now how to process "messages" extracted by "access protocols". RulesHelper static class that keeps all rules together and allows fast rules processing.
Actions
Actions is a part of the rule and defines how we should re-act on rule match.

Configuration


Configuration that describing our local domain settings will looks like simple and easy XML:

xml version ="1.0"?>

<configuration

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <servers>

    <server name="microsoft.com">

      <protocols>

        <protocol xsi:type="PingProtocol" host="microsoft.com"

protocol-id="813b11d4-5a10-43c3-8507-b108df589fcc" />

      protocols>

      <accounts />

    server>

    <server name="sbs.artfulbits.com">

      <protocols>

        <protocol xsi:type="PingProtocol" host="sbs.artfulbits.com"

protocol-id="eb4d8bb7-d312-49d9-b90d-c2619ab366b7" />

        <protocol xsi:type="Pop3Protocol" host="sbs.artfulbits.com"

protocol-id="99b0b839-2845-49f6-b832-1fb974f202c9"

port="110" SPA="false" SSL="false" />

        <protocol xsi:type="SmtpProtocol" host="sbs.artfulbits.com"

protocol-id="55b3666e-8d21-445a-b59c-5083949a9ab2"

port="25" SPA="false" SSL="false" />

      protocols>

      <accounts>

        <client protocol-ref="99b0b839-2845-49f6-b832-1fb974f202c9">

          <login>alexk@artfulbits.comlogin>

          <password>PASSWORD_PASSWORDpassword>

        client>

      accounts>

    server>

    <server name="fsrv">

      <protocols>

        <protocol xsi:type="PingProtocol" host="fsrv"

protocol-id="4181a782-777e-4edb-8563-0f29a7832dd0" />

      protocols>

      <accounts />

    server>

  servers>

  <modems>

    <modem xsi:type="GenericModem">

      <port>COM4,9600,8,None,One,None,False,False,-1,-1port>

    modem>

  modems>

  <rules>

    <rule xsi:type="HighPriorityRule">

      <Action xsi:type="SendSmsAction">

        <PhoneNumbers>

          <phone number="+380676729814" />

        PhoneNumbers>

      Action>

    rule>

  rules>

configuration>

Prototyping


For testing the library I create console application, that in few hours can be transformed into windows service. And for people that like nice GUI I design simple forms:
GUI desing

Smoke test


Let's code speak for itself.
/// private static void TestNotifyGsm() { // ---> CREATE BACKGROUND ANALYZER <--- s_analyzer.Started += analyzer_Started; s_analyzer.ProgressChanged += analyzer_ProgressChanged; s_analyzer.Error += analyzer_Error; s_analyzer.Canceled += analyzer_Canceled; s_analyzer.Completed += analyzer_Completed; s_analyzer.RunWorkerCompleted += analyzer_RunWorkerCompleted; // ---> REGISTER MODEMS <--- // ---> 1) initialize com port, change it settings // ---> 2) initialize connect to the modem SerialPort com1 = new SerialPort( "COM4" ); GenericModem provider = new GenericModem( com1 ); s_analyzer.Context.ModemsPool.RegisterModemProvider( provider ); // ---> REGISTER RULES <--- // ---> 3) Create Rules s_analyzer.Context.Rules.Add( new HighPriorityRule( "+380676729814" ) ); // ---> SERVERS CONFIGURATION <--- // ---> 4) Create Servers List ServerComputer serverOutside = new ServerComputer( "microsoft.com" ); // gateway ServerComputer serverSBS = new ServerComputer( "sbs.artfulbits.com" ); // SBS ServerComputer serverFSRV = new ServerComputer( "fsrv" ); // FSRV // 4.1) Register ICMP-Ping protocols serverOutside.RegisterProtocol( new PingProtocol( serverOutside.Name ) ); serverSBS.RegisterProtocol( new PingProtocol( serverSBS.Name ) ); serverFSRV.RegisterProtocol( new PingProtocol( serverFSRV.Name ) ); // 4.2) Register e-mail protocols EmailProtocol protoPop = new Pop3Protocol( serverSBS.Name ); EmailProtocol protoSmtp = new SmtpProtocol( serverSBS.Name ); serverSBS.RegisterProtocol( protoPop ); serverSBS.RegisterProtocol( protoSmtp ); // 4.3) Register e-mail accounts EmailAccount account = new EmailAccount( protoPop ); account.UserName = "alexk@artfulbits.com"; // SECURITY ISSUE: do not enter password here!!! file stored in SVN. account.Password = "password-password"; serverSBS.RegisterAccount( account ); // 4.4) attach configured servers to the context s_analyzer.Context.Servers.Add( serverOutside ); s_analyzer.Context.Servers.Add( serverSBS ); s_analyzer.Context.Servers.Add( serverFSRV ); #if SERIALIZATION_TEST // ---> SERIALIZATION <--- // ---> 5) Save settings into configuration file using( Stream stream = File.OpenWrite( "c:\\gsm_notification.config" ) ) { WorkerContext.Serialize( stream, s_analyzer.Context ); } // ---> DESERIALIZATION <--- using( Stream stream = File.OpenRead( "c:\\gsm_notification.config" ) ) { WorkerContext context = WorkerContext.DeSerialize( stream ); Dump( context ); } #endif // ---> CHECK CYCLE <--- // --> 5) Run processing s_analyzer.RunWorkerAsync(); s_sync.WaitOne(); System.Console.WriteLine( "Press ENTER to exit" ); System.Console.ReadLine(); }

Logic of work


Algorithm of work for application is not so trivial as it can be.
For each registered server we should proceed following steps:
1) Check protocols of server for availability (connectivity). Failure of one protocol does not mean that server is broken;
1.1) On protocol failure start to count attempts from our side for recovering connectivity;
1.2) If connectivity restored then continue without any admins disturbing;
1.3) If we reach critical point then create message for SMS and put it into rules processing queue;
1.4) If after failure connectivity recovered then create message and also put it into rules processing queue;
1.5) Rules are the same as used in e-mail processing. So this step simply redirected to 2.2 point in this list;
2) Check e-mail accounts of the server:
2.1) If protocol OK then extract e-mails from server;
2.2) Check each e-mail on Rules match;
2.3) If we have match at least on one rule then required SMS sending.

Do you need a commercial solution?


This project has a commercial support now. If you are interesting in similar solution with support and customization from our side please contact ARTFULBITS sales. Thanks.

Currently rated 5.0 by 4 people

  • Currently 5/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Comments

July 4. 2009 14:13

Poor web server can cause a lot of unpleasant moments, so a good server is very important for all.

symptoms

August 5. 2009 17:17

Great article. Where else could anyone get that kind of information in such a perfect way of presentation.

Denver dating

October 26. 2009 12:13

>>1) if all servers in domain failed, how I can receive >>notification about that? What happens if internet >>channel become broken physically? имхо гсм модем не самое лучшее решение. самый простой пример - проблемы с электрикой - внешний канал из-за них пропадает, а сота находится рядом и тоже вырубается или сильно понижается мощность сигнала (пример не надуманый, лайф окого моего дома себя ведет подобный образом, мобилка остается в сети, но уровень сигнала не позволяет позвонить при пропаже электрики больше чем на 10 минут), еще можно придумать кучу форс-мажуров когда капец наступает всему что рядом с сервером. Идеологически более правильно будет развести watch-dog и наблюдаемую систему, например поставить его в другом офисе, или на отдельном хостинге.

cencio

October 28. 2009 10:02

What will happen to my data if the server's hard disk or drive got cracked?. Thanks for the helpful post!

Matchmaking Sites

December 6. 2009 19:38

私は、上から下には、読書の後、すぐに私は自分のブログのための新しいアイデアを得たのは、検索エンジンからこの記事を見つけた。おかげで、私のインスピレーションになります。

Sears Parts

January 19. 2010 09:14

Your blog is a helpful one.Thanks a lot for sharing.

Mx motocross

February 5. 2010 14:05

Good work!

Bob

February 19. 2010 09:12

This is the full information post regarding the server failure. I am fan of you. Hats off to you.

Car hire in Cairns

March 16. 2010 13:59

It's the best solution you have shared for the server failure.

Gourmet Hampers Melbourne

April 19. 2010 14:51

Keep posting materiaal als dit bevalt me uitstekend

Czeslawa

June 1. 2010 13:46

Very informative and helpful post. You have good command on the topic and have explained in a very nice way. Thanks for sharing.

Attache

June 6. 2010 17:21

The post is written in very a good manner and it contains many useful information for me. You have a very impressive writing style. Thanks for sharing.

Background Check

June 13. 2010 13:42

Hello, i must say this is a nice blog. Please continue this awesome work. Sincerely...

Jordan Hydro

July 4. 2010 01:33

A weed is no more than a flower in disguise. http://www.clicknpayday.com

personal loans

July 6. 2010 23:33

When i will face this type of server problem i will surely inform you and hope to get the result very quick.

Go Karting In Birmingham

July 7. 2010 03:17

I read about your system several months ago. I've made something of the kind at home with some innovations and I must say it makes my life a bit easier. )))

VIP slots online casino

July 14. 2010 18:41

I like the concept in your company. I want to share some ideas on how to recover data after server failure. In my experience Windows Home Server as the backend of a home network is not immune to hardware defects, bad drivers, misconfigurations, incompatible software installations or user errors. While it offers possibilities to backup the content of shared folders and even the client backups, these processes have to be done manually and need a lot of additional external storage space. So these will often not be performed regulary.

Sydney Wedding Photographer

July 14. 2010 21:11

This is the full information post regarding the server failure. I am fan of you. Hats off to you.

vibram fivefingers

July 20. 2010 19:24

An error notification system operated independently of a connected computer receives signals indicating an error condition in operation of the computer, matches the signal with a prerecorded message, and dials phone numbers from a priority list of responsible parties. When a call is answered, the notification system announces the message, informing the answering agent of the error condition, and requests a response requiring a person. If the requested response is not given, the system dials the phone number next in priority, and so on, until a person answers and provides the human response. The system is applied to networked computers, such as file servers, and in one embodiment, the answering party can initiate corrective action.

high heel wine holder

July 21. 2010 13:25

Following looking all-around for a legitimate hotmail password hack strategy, I identified this application. Be mindful with it though, I'm not positive no matter whether its legal to fit it to implement to hack someones hotmail or yahoo account password, its only legal to place it to make use of on your own in case you neglect the password. http://www.hotmailpasswordhack.net

hotmail password crack

July 21. 2010 23:41

I received my first <a href="http://bestfinance-blog.com/topics/mortgage-loans";>mortgage loans</a> when I was 20 and it aided me a lot. However, I require the consolidation loans also.

CassieSimpson19

July 24. 2010 09:16

When I boot my computer, there is a message that says,Failure to Connect to Windows Server. What does this mean and how do I fix it?Please help.

Puerto Banus Beach Club

July 27. 2010 08:39

Wow that's interesting. Good news though, they make easy to use bikini hair removal creams. They work just as good as a wax for women who still want to be clean down there.
http://www.guccionlineoutlet.com/gucci-lady's-handbags-8/

Gucci handbags

July 27. 2010 09:34

I wonder why they haven't invented professional tools of the kind. It's very convenient. I had to invent a similar system as well.

vegas casino online

July 30. 2010 00:55

Hi, I can' figure it out, but when I enter your RSS link into my aggregator, it's not working. Can you give me the RSS URL to ensure I'm using the right one? Thank you. :)

Coach Sabrina

July 30. 2010 14:42

Great work, man! Is it only your invention? Cool if so.

play online slots

Add comment


(Will show your Gravatar icon)