![\"\"](\"https:\/\/adlumin.wpenginepowered.com\/wp-content\/uploads\/Picture1-2-1.png\")
<\/figure>\nBefore applying the parser tree, logs are preprocessed. Logs are tokenized according to common delimiters. Then, they are preprocessed by applying pre-defined regex patterns that replace well-known parameters (e.g., r\u2019 [0-9a-fA-F]{12}\u2019 for a 12-digit hexadecimal value) with specialized parameter tokens. Typically, any token that contains a number is also treated as a parameter and replaced with a parameter token. Once preprocessed, the parser tree can determine log associations.<\/p>\n Thus, the intuition is that logs said to be similar in the content will have similar sequences of messages and parameters that can be represented as tokens for easy comparison. The first layer of the parser tree is assigned based on the total number of tokens in the preprocessed log. This step is utilized because parameters are typically a fixed number of tokens, meaning this step quickly groups associated logs. Subsequent layers of the parser tree are fit according to the token in the nth position of the preprocessed log, where n comes from a pre-defined set of token positions. To prevent parser tree explosion, if the number of branches at any given layer exceeds a threshold, then a wild card token is used for all new tokens.<\/p>\n Once the bottom of the parser tree is reached, if there are no other logs already at this position of the parser tree, then the incoming log is used as a new log format. Otherwise, the incoming log is compared to the log formats at this position using token edit distance.<\/p>\n If the incoming log is sufficiently similar to an existing log format, then the log is associated with that format. In addition, the log format is updated such that any dissimilar tokens between the log format and the incoming log are replaced with wildcard tokens. If the incoming log is insufficiently similar to all existing log formats, then the incoming log is used as a new log format.<\/p>\n Once all training logs have been processed, all the log formats generated by the parser tree are returned. Particularly rare log formats are dropped. For each log format, we generate a log template which includes: a regular expression pattern capable of matching logs associated with the template, a set of the tokens contained in the log template, and a log template score based on the frequency that the log template appeared in the training set.<\/p>\n Another set of recent log data is used to generate a score threshold. Logs in this dataset are scored using the log templates. To score, an incoming log is preprocessed and checked to see if it matches any log templates\u2019 regular expression pattern. If the incoming log matches a regular expression pattern, then it is assigned the score associated with the log template that it matches.<\/p>\n Otherwise, the incoming log\u2019s tokens are compared to each of the log templates\u2019 token sets using Jaccard similarity. The incoming log is associated with the log template that has the most similar token set and assigned a score based on the log template\u2019s score and the similarity between the token sets. This matching process allows previously unseen log formats to be assigned appropriate rarity scores.<\/p>\n Once all the score threshold dataset has been processed, all scores assigned to logs are considered. Based on these scores, a global score threshold is determined using the percentile score and standard deviation of the set of all scores.<\/p>\n In contrast to the training process, inference occurs in a streaming environment and finds anomalous records in near real-time.<\/p>\n As in the score threshold process, incoming logs are preprocessed and either matched to log templates using regular expression patterns or token set similarity. The incoming log is assigned a score of the matched log template or the matched log template and a similarity measure, respectively. If the incoming log\u2019s score is greater than the score threshold, then the log is considered anomalous.<\/p>\n To facilitate real-time analysis, we utilize AWS Lambda for conducting inference and a DynamoDB table for template storage. We\u2019re able to spin up additional Lambda instances automatically and adjust DynamoDB read capacity as needs demand them.<\/p>\n Classifying logs and assigning rarity scores opens a wide variety of analysis opportunities: keyword lists that would otherwise generate numerous false positives are more effective when applied to the anomalous dataset. Log template associations enable time series analysis of specific log messages. By determining the location of parameters, specific information can be extracted and analyzed, even in unseen log formats. The Adlumin platform leverages a variety of these techniques to provide preventative alerts of specific threats, malfunctions, and IT operations failures.<\/p>\n","protected":false},"excerpt":{"rendered":" Enhance log anomaly detection with real-time scoring, template matching, and AWS Lambda. Discover better threat detection and IT failure alerts with Adlumin.<\/p>\n","protected":false},"author":24,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"class_list":["post-70314","post","type-post","status-publish","format-standard","hentry","topic-cyber-resilience","topic-security"],"acf":[],"yoast_head":"\n![\"\"](\"https:\/\/adlumin.wpenginepowered.com\/wp-content\/uploads\/Picture2-2.png\")
<\/figure>\n![\"\"](\"https:\/\/adlumin.wpenginepowered.com\/wp-content\/uploads\/Picture3-2.png\")
<\/figure>\n![\"\"](\"https:\/\/adlumin.wpenginepowered.com\/wp-content\/uploads\/Picture4-2.png\")
<\/figure>\nInference<\/h2>\n