A common business requirement of many enterprises is the forwarding, collection, and analysis of syslog messages. In practical terms, this means central event log storage and processing for most all network gear except Microsoft Windows Server and applications which have the Event Log service.

Using a syslog server as a collection point for logging activities allows all network logs to be stored in one place so they can be searched easily. A syslog collection facility is a must for network security because without a syslog server, logs remain on scattered devices and can never be reviewed or archived.

All Linux and UNIX servers have syslog capability and most networking gear such as Cisco routers, Check Point and Juniper firewalls, and F5 “BIGIP” load balancers can be configured to generate and forward syslog messages as well. And while a central syslog server is a good idea, a management burden emerges as the enterprise grows since a single server destination can quickly have capacity and performance issues.

Azure Log Analytics: Purpose built for high volume event processing

The Log Analytics service in Azure ingests and processes high volume event and security log information from Windows and Linux computers running OMS Agents. Syslog is the cross-platform equivalent of Windows Event Log, and by leveraging Azure Log Analytics you can collect and surface your Windows and non-Windows event data in a holistic fashion. That includes messages from network devices that can forward syslog via Linux computers.

Microsoft’s Azure public cloud service includes a Log Analytics service which combined with the Microsoft Operations Management Suite (OMS) repository and OMS alerting features is a valid enterprise approach to syslog management. Figure 1 illustrates how one or more Linux VMs with OMS Agents forward the syslog messages of network devices to OMS Log Analytics, where the data is searchable by using the OMS portal, the OMS mobile app, or other programmatic access.

Figure 1 – A five-server Linux VM syslog forwarder farm in a large-enterprise attach model

What are syslogs and what use can they be?

A syslog message is an event, a log entry either routine or urgent, from a device or computer. Syslog technology is a standard for message logging that you can think of as having three components:

  1. The software or firmware that generates messages, running on a device or computer,
  2. The system that stores them, and
  3. The software that reports and analyzes them.

In the Microsoft Log Analytics syslog solution, similar solution components consist of:

  1. Syslog-generating devices on your network(s)
  2. One or more syslog-forwarding Linux computers with OMS agents
  3. The Azure Log Analytics portal, mobile app, and alerting features to report and analyze syslog messages.

For example, most Cisco devices use the syslog protocol to manage system logs and alerts. But unlike their PC and server counterparts, Cisco devices lack internal storage space for these logs. Cisco devices use the UNIX-style syslog protocol to send messages to an external device for storing. This option is not enabled by default. (Read more: Configuring Cisco Devices to Use a Syslog Server.)

A great tip from Cisco is to make sure network devices are configured with the right date, time, and time zone. Consider configuring all network devices to use Network Time Protocol (NTP). Setting the devices with the accurate time using a common NTP server is helpful for event correlation, which is one of the invaluable benefits of a syslog analysis solution. Find an NTP Server in the US.

Inside a syslog message

Since each syslog message is labeled with a facility code, indicating the software type generating the message, and assigned a severity label such as “info” or “warn”, an infinite variety of log analytics queries can be constructed. Queries can be run in two useful ways:

  1. On-demand to generate historical reporting data for trend and anomaly detection (Execute a query in OMS “Log Search”)
  2. Scheduled queries can be run automatically and generate alerting actions when specific syslog messages are seen or syslog thresholds are exceeded (“OMS Alerts”)

There is a lot of value in just knowing and confirming the normal pattern of syslog volume and distribution. Knowing the baseline is the first step to detecting departures from expected or desired syslog message traffic.

Figure 2 shows what a typical syslog message looks like. Notice the attribute labels on the left, those in blue are automatically indexed by Log Analytics. That is, clicking on these labels dynamically pivot the grouping selections in the query result view, essentially letting you ‘fly around’ through your data to spot trends and anomalies.

Figure 2 – Typical syslog message with OMS-indexed elements in blue

Using a Cloud Service for syslog processing

There are on-premisess syslog processing solutions that can scale to avoid message queues and provide the fastest query result time. A market leader for syslog processing is Splunk. However even a medium size enterprise can generate over 1 billion syslog events per week, larger enterprises can generate that many per day. Management of these data quantities and the resources for the storage and the analysis of the data add complexity to the business requirement, not to mention the cost of enterprise syslog management software.

The use of cloud services for central syslog processing has appeal for many reasons. Among these are:

  1. Reduction in potential bottlenecks and single points of failure
  2. The ability to leverage hyper-scale cloud computing technology to perform faster syslog queries
  3. Having broader access options to view and work with the collected data
  4. Eliminating on-premisess burdens for syslog processing, storage, and data backup
  5. Higher standard of evidence for security sensitive logging
  6. Integration of syslog messages into a holistic cloud-based security solution

Splunk has an offering Splunk Cloud that naturally developed as an extension of their on-premisess datacenter technology. Many enterprises are using Splunk or Splunk Cloud and may also be Azure customers. For Microsoft customers with an Azure roadmap, serious consideration should be made to leveraging Log Analytics.

The limiting factor on cloud-based syslog solutions is the processing capacity of the on-premisess syslog collector, not the Internet bandwidth or the cloud service capacity. As your syslog throughput increases, additional collectors are added in a load-balanced fashion. Splunk ‘Heavy Forwarders’ and OMS Linux Agents are analogous as scale units, and Splunk queries can port over 1:1 as OMS Alerts triggers by Azure Log Analytics queries. Microsoft even makes a handy Splunk to Azure Log Analytics query conversion “Cheat Sheet” available.

Getting value from the syslog solution

In the More Resources section of this article are links to get an OMS Workspace created, deploy a Linux OMS Agent, and configure the OMS Agent to forward syslogs to the OMS Repository. Once you have done these things, you can configure your network devices to use the Linux server(s) with the OMS Agent as their syslog collector. The forwarded syslog data will arrive in OMS and be ready to be acted on by recurring Alert-generating scripts or ad-hoc during interactive Log Analytics query searches.

To help you get value from the syslog solution faster, consider adding this OMS View to your OMS workspace:  Syslog Analysis Solution for Microsoft OMS. This pre-built solution gets you started with syslog-focused queries condensed into a high-value dashboard. Figure 3 shows the solution after it’s installed in an OMS workspace with syslog-generating devices. All views refer to the last 24 hours of data.

Figure 3 – The Syslog Analysis solution in the OMS Portal

  • The first display in the dashboard shows syslog data type distribution by Process Name, Facility, and Hostname. Spikes in traffic will be easy to spot across metrics.
  • The second pane with the pie chart helps spot above-normal error or warning events.
  • The third pane confirms timely data insertion into Azure, as well as actual data volumes.

Just glancing at this dashboard each day will help you keep tabs on your syslog and device population! Working with the dashboard will also quickly give you comfort with the Log Analytics indexes, and you will come up with ideas for more custom views that work for your business or organization.

After downloading the Syslog-Analysis.omsview file at Syslog Analysis Solution for Microsoft OMS, import the file into your OMS Subscription as shown in Figure 4. In the View Designer, click Import.

Figure 4 – Importing a custom solution to OMS

The syslog solution dashboard is fully customizable once you import it into your OMS environment–Add more panes or make your own solution and views based on your specific syslog device profiles. Remember OMS Alerting can key on any aspect of syslog messages such as warning, error, critical, and emergency syslog messages.

Figure 5 shows the syslog solution top-level dashboard among the other OMS solutions that have been acquired for the OMS workspace.  The Agent Health and Alert Management solutions are naturals to accompany the syslog solution.

Figure 5 – The syslog solution surfaces among other installed OMS solutions in the OMS Home view

Many sysadmins will appreciate the immediate usefulness of the OMS Mobile App. Like all OMS solutions, the dashboard views and data tools are available right away in the mobile client. Figure 6 shows the syslog solution dashboard panes in the OMS Mobile App for Android.

Figure 6 – syslog solution blades available on the OMS Mobile Client (Android shown)

Azure Log Analytics Customers

Organizations with Azure roadmaps that are customers of Splunk Cloud should investigate financial or feature incentives to migrating some or all syslog management to OMS.

Another place where the Azure Log Analytics syslog solution is a great fit is for in-Azure Linux computers and virtual appliances generating syslog messages. Rather than pipe those messages back on-premisesss for syslog processing, keep the traffic in Azure and avoid cloud egress charges!

For all organizations looking for those early cloud business enablers just as DR/backup, include Azure Log Analytics for event log and syslog enterprise correlation. Consider the ability of the Azure Security Center to leverage syslog message streams and you could gain tighter and perhaps less expensive SIEM integration.

More resources:

Create a Log Analytics workspace in the Azure portal

Collect data from Linux computers hosted in your environment

Collect and analyze Syslog messages in OMS Log Analytics

Create and share dashboards of Log Analytics data