How often have you been in the need to get some log statistics? I have seen many packages around, in particular those monitoring some specific internet applications (web, ftp, mail, …), everybody knows stats packages like analog and webalizer, if we are talking about web you can also use Google Analytics.
But more often than not, you always face one of the following problems:
- The log format is not the standard web log
- You need realtime info.
In the first case you end up writing your own parser of the log file (unless you have written the application, but then we would come on which are the requirements of the application, and why was monitoring not included). We usually do that in perl, although I have seen parsers in many languages including shell, and even Java…
But in the second case, what you would need is an online parser which gets you statistics by parsing incrementally the log files.
Who has not used MRTG, Cacti, Cricket, or any of those RRD applications that allow you to see how well your Internet line is doing?
In our case we have come up with a little solution, which might seem quite hard at the beginning but that does the job quite well. We use the combination of 4 elements:
- Net-SNMP. This provides us with the tools to speak the SNMP protocol.
- SNMP MibProxy. This provides us a neat clean interface to the snmpd daemon.
- SNMP Logparser. This is our online and incremental log parser build template.
- Cacti. We use cacti to display graphs, although we have used others like opennms, cricket, mrtg, (or other non opensource software).
Ok, now we have these 4 different tools and the question is how we build them together? I will not get into the specific details, please use the man page of snmpd, the documentation of cacti and the POD of the specific perl modules for that, but I will try to highlight which are the main steps of getting this right.
- Create the MIB for the counters and stats that you want to monitor. I will not get into the details of this here. My quick recomendation is that you copy one of the existing mibs and change them to suit your own needs (in my computer you can find it in /usr/share/snmp/mibs/IF-MIB.txt). See among RFC1212 and others (http://www.ietf.org/rfc/rfc1902.txt).
- Create the interface in snmpd with MibProxy. This is accomplished by adding the following line to snmpd.conf (obviously you need to change the OID):
pass_persist .1.3.6.1.4.1.17171.1.8 /usr/local/bin/mibProxy
- Test the interface with snmpget. For this you need to add a static value to the mib counters in /var/lib/mibProxy/logparser.properties
myDescr.0=test2
myCounter.0=21
If you get your 21 from the command line “snmpget localhost myCounter.0″ then you are in the right track, if this doesn’t work, please check the man page for mibProxy.
- Create a SNMP::LogparserDriver subclass which should implement the following methods: evalBegin, evalIterate, evalEnd. These methods will allow you to easily parse the info that you need and leave them in the former /var/lib/mibProxy/logparser.properties
- Run the logparser script, which will invoke your specific class, from cron, every 5 minutes for example.
- Graph the results in cacti via SNMP
Here are some of the things that we have used this for:
- Show the rate of new subscribers. Very useful for new campaign launches in marketing.
- Show the number of purchase transactions per seconds (including failed transactions)
- Get this nicely displayed on a handset, so that you always know how things are going
- plug this info to an NMS system, to alert whenever the number of transactions were lower than expected.
- Create dashboards which reflected the business as a whole. Just to note that some of them were used as much by the Technology guys as well as the marketing guys, providing a real good team building, and common objectives…
I hope you enjoyed this one as well.
This is one of this Telco proposition that Qindel offers to their customers. A lot of people use Asterisk as a PBX, and it does its job quite well. In fact, there is a lot of competition in this area, other players have taken the Open Source Asterisk and have put some nice graphical interfaces around them such as Trixbox or Switchbox. Here comes another proposition using Asterisk as a Ringback Tones solution.
So what is a Ringback Tone solution? It is that music that you hear when you dial someone on your mobile phone. Obviously this tone can depend on who is calling and of group of callers.
Well asterisk has that play tones functionality, and deciding what tone to play depending on the callerId is simply a database lookup.
So how has Qindel implemented this:
- Using Asterisk as the IVR (this could be connected via SS7, QSIG, SIP, H.323, …)
- Using as the backend an LDAP database. The LDAP databases have natively support for replication, and are optimized for reads, with penalization on writes, although Asterisk supports LDAP from 1.6, there is a patch from 1.4 onwards and we currently use the lookup inside the AGI script.
- Using an AGI script to do the database lookup (Do you know the Net::Server and the Asterisk::FastAGI perl modules?)
The challenge in all the real operators is then the provisioning and the monitoring. Provisioning, is really operator dependent and I won’t get into it today (this is a discussion for another day), but monitoring can be easily achieved with Nagios and Cacti (another one which should have its own discussion, in particular with Qindel’s CMDB), and the native support of Asterisk for SNMP (from version 1.4.x)
I hope you enjoyed this entry.
This little post is about trying to substitute the Sun Messaging server MTA with a sendmail MTA with as little disruption as possible.
Imagine the following situation:
- You have several Sun MTAs
- You have several Sun Messaging Stores
- All the mail routing and delivery information is a replicated LDAP infrastructure
The first question that would arise is why would you like to change, and coming later to the results you will see that in terms of LDAP lookups Sun Messaging Server is more efficient than sendmail.
The reasoning behind this, in our case, was that we were using sendmail as a filtering platform for antivirus and spam. You know, spamassassin, mailscanner, and the like… And what would be your choice:
1) Putting another set of MTAs in front of the existing Sun MTAs. This would be the preferred choice as an external managed service, but could also be done in the customer premises.
2) Replacing the existing Sun Messaging MTAs with sendmail (just to reduce the hardware costs).
In the second case the sendmail MTA has to mimic the behaviour of the Sun Messaging MTA. This particularly means, that we are going to avoid another SMTP hop, and obviously reduce hardware as well.
Let’s get a bit into the technical details, and revise the LDAP main features that we have to mimic (we will not get into the full detail, but rather, get into the two important bits):
- Message store selection for incoming mail. See the mailHost attribute from the inetLocalMailRecipient Class. This can be easily reproduced with the sendmail M4 feature ldap_routing:
FEATURE(`ldap_routing’, `ldap -1 -T -v mailHost -k”(&(objectClass=inetLocalMailRecipient)(|(mail=%0)(mailalternateaddress=%0)))”‘, `ldap -1 -T -v mailRoutingAddress -k”(&(objectClass=inetLocalMailRecipient)(uid=%s))”‘ , `bounce’ ) dnl
LDAPROUTE_DOMAIN(…)
- Email expansion for each delivery channel. This has to do with the mailDeliveryOption and the two main values that we are going to see here mailbox and forward. The first one delivers mail to the message store and the second one uses the mailForwarding address to duplicate the mail and deliver it to a different recipient.
The mail expansion to different recipients is one of those challenges, the only way we know of expanding an email address to several other is by using email expansion in the sendmail mailer, this can be done either by using the ALIAS feature or by using the USERDB feature. In our case we decided to use the ALIAS expansion (the USERDB does not currently work with LDAP)
define(`ALIAS_FILE’, `ldap: -T -v”mailforwardingaddress,mail”
-k”(&(objectClass=inetLocalMailRecipient)(maildeliveryoption=forward)(maildeliveryoption=mailbox)(|(mail=%0)(mailalternateaddress=%0)))”‘, />
`ldap: -T -v”mail”
-k”(&(objectClass=inetLocalMailRecipient)(maildeliveryoption=mailbox)(|(mail=%0)(mailalternateaddress=%0)))”‘,
`ldap: -T -v”mailforwardingaddress”
-k”(&(objectClass=inetLocalMailRecipient)(maildeliveryoption=forward)(|(mail=%0)(mailalternateaddress=%0)))”‘,
)dnl
This setting sets up the A flag in the local mailer indicating that the it should expand the addresses set up in the ALIAS_FILE. Also the behaviour of the ALIAS_FILE is that it tries to expand an alias with the first alias definition, that is if maildeliveryoption has both attributes forward and mailbox, if that doesn’t match it searches for entries with maildeliveryoption mailbox, and if that fails it tries to expand aliases with maildeliveryoption forward.
As you can see this is very inefficient in terms of LDAP searches. If you compare that to the Sun MTA, this one would just do one lookup to get all the attributes, but in sendmail we are doing several. That means, be warned, if you the sendmail route you will have more ldap lookups. The good news, is that your answers will be cached…
One last think, the delivery of the email to the message stores will be done with LMTP. The setting that we use is:
Mlmtp, P=[IPC], F=AmDFMuXa0z, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\n, L=990,
m=4, T=DNS/RFC822/SMTP,
A=TCP $h 225
Well this is just a glimpse of the whole settings, but as you know each case is different. If you are bored and read this part let me know and I can post more details about the config.
Why would a guy like me start writing a blog?
Well the first reason is probably that the blogs get read. Secondly it seems like a good way to share information about things that one does, which sometimes seem not too important to put in a formal HOWTO, FAQ or even in some forums, so I will write them down here.
Another good reason might be that Qindel tends to do too many different things and it is very hard to show them all on our corporate web page and not overwhelm with information perhaps too detailed… (is too much detail a problem?).
Well, enough for today, I hope you enjoy it.
, although I have to admit that the architecture of Postix outweights the one of sendmail…).
