The “Spot the log entry” contest

February 6, 2007

As a programmer, most of us would have been taught to write comments, externalize constants, indent code and of course log the flow of control. Credit goes to OpenSource for having provided such wonderful frameworks like Log4j to enable logging.

The framework authors did foresee the problem of growing log files and were smart to provide means to control the level of logging. While, this does help a lot to categorize log entries, it doesnot solve the most commonly occurring problem for a developer:

That of “Spotting the log entry” that is most relevant to the situation.

Let me explain. Picture this: a large application has been deployed in production for a while. The log level has been set to INFO because the application support team argues that ERROR doesnot give them enough detail for trouble shooting. In the application code, a few developers have gotten carried away and have logged entry and exit of each method call (re-inventing the “around” advice in AOP if you may call it that way). The result – log files that have rolled many times over and totalling to around 100 MB across just as many files.

These files are not always easily available to the developer debugging a problem that has been reported in production. The developer needs to quickly find the log entry that helps to diagnoze the issue and here the “Spot the log entry” contest begins.

Obstacles in this course are(and not limited to) :

  • Having to sift through potentially hundreds of thousands of log entries
  • Log files located on remote machines accessible via protocols like SSH/SFTP.
  • Noise in log files – entries that log entry and exit from methods
  • Absence of tools to analyze the structured data contained in log entries. Text editors are an often used, but poor choice.
  • Inability to analyze contiguous log events. For e.g log entries made from one thread donot appear together in a log file in medium to heavy use systems.

There are a few ways in which this problem can be addressed:

  • Avoid AOP like log statements in code. Use AOP at run time to instrument byte-code on the fly if required.
  • Clear logging strategy
    • Log errors when encountered the first time and not when it is handled each time, say as it progresses up the call stack.
    • Use of appropriate log levels to differentiate between debug information, messages, warnings and errors.
    • Use of log patterns that provide sufficient information to analyze the flow. Logging timestamp, thread, category and priority for e.g

What if your log files are still huge after all this? Its time to invest in tools that help you spot your log entry.

Some of us at MindTree(http://www.mindtree.com) looked around for OpenSource tools for log analysis when we had to inspect logs from aorund a dozen servers. Chainsaw (http://logging.apache.org/log4j/docs/chainsaw.html) was a decent implementation but not good enough. Commercial tools were not satisfactory either.

Thats when we decided to implement Insight – an application analysis tool.

Insight

To start with, it was conceived to do comprehensive log analysis. In brief it provides the following:

  1. Provide visual analysis of any pattern based log files
  2. Analyze logs from remote servers over (S)Ftp and Http.
  3. Supports tailing of local files and a plug-in for Eclipse
  4. Provides summary and detailed view of the log event
  5. Supports “no-mutating” analysis of the data set – such as search, sort.
  6. Supports “mutating” analysis of data set – via progressive filtering
  7. Helps to locate the “context” of an event i.e snap shot of log entries around a specific log entry.
  8. Optimized for performance and footprint size
    1. Loads 1000 entries in around 375 ms
    2. VM size between 45 to 60MB even after loading 110 000 entries

See attached presentation for details on Insight and testimonials : Insight features

Our developers are now front runners in the “Spot the log entry” contest :)

———————————————————————————–

MindTree Insight is now an OpenSource project on SourceForge and is available at :
http://sourceforge.net/projects/mindtreeinsight

The download of the latest release is available at:

http://sourceforge.net/project/showfiles.php?group_id=212019&package_id=254922
——————————————————————————–

About these ads

22 Responses to “The “Spot the log entry” contest”

  1. sohel Says:

    did you ever finish developing it? is it available to download or did you commercialize it?


  2. Sohel,

    Yes we did complete development. Insight has seen 5 major releases till date. MindTree has a enterprise-wide licensing model for Insight. There is also talks of making it open-source. It could take a while though as the legal department is working on the license.
    In the meanwhile, the commercial licensing model will continue. Let me know if you require details.

  3. naisioxerloro Says:

    Hi.
    Good design, who make it?


  4. [...] Collegues ( idea by Regu ) created a tool they named insight last year and have just made it open source. Here is the [...]


  5. MindTree Insight is now an OpenSource project on SourceForge and is available at :

    http://sourceforge.net/projects/mindtreeinsight

    The download of the latest release is available at:

    http://sourceforge.net/project/showfiles.php?group_id=212019&package_id=254922

  6. Nicolas Gonzalez Says:

    Hi,
    It seems to be a nice tool.
    I’ve been having some problems when using it.

    I’m setting the primary and secondary patterns, which are the same that I have configured in my log4j configuration file, and when I try to open a log file the tool would say:
    “specified log pattern doesnot match data in: …”

    Could you tell me what I’m doing wrong?

    Nico


  7. Nicolas,

    Its got most likely to do with your log file and the pattern you have used. Please pass on your log4j.properties file and a sample log file and I can figure out the pattern for you.
    Note that the pattern is case and space sensitive and has to be an exact match.

  8. Nicolas Gonzalez Says:

    Hi again!
    As you said, it was a problem with the pattern layout.

    Thx for everything!

    Nico
    Buenos Aires – Argentina

  9. Eugene Lucash Says:

    Note after looking at screenshot:
    Actually log events has `Severity` not `Priority`. Althought they can be prioritized by severity, but it seems odd to call it `Priority` in the first place


  10. Eugene,

    I guess we simply borrowed what Log4j said. See

    http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html

    where “p” in the pattern string is defined as “Used to output the priority of the logging event.”

  11. Alejandro Pimentel Says:

    Grats!

    You did a useful and easy-to-use tool.

    I’d like to collaborate, if that is posible.

    berserkpi@gmail.com


  12. Alejandro,

    Glad to know you liked MindTree Insight. You are of course welcome to participate!
    To start, please take a look at the bugs and feature requests on the project site and let us know if you would like to take up any of those.

  13. Alejandro Pimentel Says:

    OK

    I’ll do that.

    Thx.

  14. Aken Says:

    Ive downloaded your application hoping I could use RegEx to match it to my cutom log file. It’s based on a log4j format. I’ve looked at your preference.xml and I think I can’t customize it for my needs. Any chance you could lead me to some docs I could use to adapt your tool to read my logfiles?

    Thanks!


  15. Aken,

    We have exported and viewed Windows events using Insight. This goes to say that you can view any pattern based log file so long as the fields in the pattern are ones supported by log4j and Insight. Of course, you may have some issues with priorities other than debug,info,warn,error and fatal.
    We unfortunately don’t have documentation on the entire design. The classes are well documented though. You may want to look at the following:

    Log4JPatternInterpeter – Creates a ReceiverFormat object from the specified pattern. You may want to customize this class to recognize RegEx patterns.

    LogInterpreter – Parses and creates the LogEvent instances from the log files. Internally uses multiple Apache Oro Regex pattern matching classes.

    The LogEvent is the data object recognized by rest of Insight and contains all the information pertaining to a n event logged by the logging framework.

    I hope this helps.

  16. Craig Says:

    Hi,

    Thanks for the post and this log viewer.

    I’d like to extend the functionality to filter and color on duration where duration is either the difference between the current and next log statement timestamp.

    Are there any plans to add this?

    Thanks,

    Craig


    • Hi,

      Thanks for your interest in MindTree Insight and glad to know that you liked it.
      Commits to Insight have not happened for a while due to various reasons. You may add your requirement to the feature requests on SourceForge and it will get addressed.
      However, adding the feature you have requested is quite easy. Let me know if you are interested and I can help you make the changes and commit it.

      Regu

  17. Balaram Says:

    Hi Regu,

    Can I use this tool without the UI? I am looking to have this tool as a library for me which i can call from my program and display the result on my UI.

    Thanks
    Balaram.


    • Balaram,

      The UI is sufficiently decoupled from the parser, preferences, pagination, filters and other components. You will need to look at the packages to cull out just the parts that you need. The Javadoc will help you understand the classes.

      However please note that this is licensed under GPLv2. This means you will need to open source any derivative works that you create on the Insight code base.

  18. Ravisankar Says:

    Hi Mr.Ragunath, does your tool supports pattern used by Log4Php. My application has been written using PHP.
    In my application I used log4php for logging.


    • Ravisankar,

      I have not tried it with log4php. However looking at the pattern that is identical to log4j and the sample log files, I dont see a reason why MindTree Insight cannot be used for analyzing those log files.
      Go ahead and give it a try – I am reasonably sure it works.

  19. Krzysztof Says:

    Hello.
    I have implemented similar tool called OtrosLogViewer http://code.google.com/p/otroslogviewer/.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: