Archive

Posts Tagged ‘RedHat’

Extending Microsoft OMS to monitor Squid Proxy running in Linux with a plugin – part 1/3 #MSOMS

November 24, 2016 1 comment

Since Microsoft released OMS, I have been an early adopter and evangelist for the solution. Not only it is simple to deploy but it gives you a full spectrum of many of the workloads you have either on-premises or in the cloud and it does not matter which cloud. Be it Azure, AWS, Google and many others.

So, as I was advising on OMS for a customer, I found that they were running Squid Proxy servers. The Squid proxy server is one of the most famous proxy servers in the world and it has been utilised for years in many organisations. For that reason I then I decided to look at how OMS could leverage the monitoring for Squid.

squi3

As you can see here: https://github.com/Microsoft/OMS-Agent-for-Linux/tree/master/installer/conf/omsagent.d there are already many plugins for OMS to  monitor Windows and many Linux OS as well, DNS, Network, SQL, MySQL, Postgree, VMware, MongoDB, Security, Audit, Change Tracking and so on.

But, there was no Squid plugin and that’s where I brought back my past years of experience as a developer and although that was a long, long time go, I was able to developer in ruby a Squid plugin for Microsoft OMS.

How I developed it?

PART 1 : LOG Files

  1. I started but investigating the squid log on /var/log/squid/access.log and then I research REGEX expressions to extract information out of it. Below is a extract of it

1479696836.902    134 10.1.1.4 TCP_MISS/301 488 open http://cnn.com/ – HIER_DIRECT/151.101.0.73 –
1479696848.110    242 10.1.1.4 TCP_MISS/400 486 open http://www.sydney.com/ – HIER_DIRECT/54.253.253.77 text/html
1479696860.004    407 10.1.1.4 TCP_MISS/301 636 open http://www.7news.com.au/ – HIER_DIRECT/203.84.217.229 text/html

The initial difficult part for me was of to decouple the date/time to get it on a human readable format. So, after long hours of research and playing along, I decided for the following REGEX :

 REGEX =/(?<eventtime>(\d+))\.\d+\s+(?<duration>(\d+))\s+(?<sourceip>(\d+\.\d+\.\d+\.\d+))\s+(?<cache>(\w+))\/(?<status>(\d+))\s+(?<bytes>(\d+)\s+)(?<response>(\w+)\s+)(?<url>([^\s]+))\s+(?<user>(\w+|\-))\s+(?<method>(\S+.\S+))/
(If you have a better one, please feel free to shot me)

 

  1. I then wrote a squidparserlog.rb in ruby to parse the Squid access.log file and turn it into a OMS format
class SquidLogParserLib
require ‘date’
require ‘etc’
require_relative ‘oms_common’
require ‘fluent/parser’
    def initialize(error_handler)
@error_handler = error_handler
end
    REGEX =/(?<eventtime>(\d+))\.\d+\s+(?<duration>(\d+))\s+(?<sourceip>(\d+\.\d+\.\d+\.\d+))\s+(?<cache>(\w+))\/(?<status>(\d+))\s+(?<bytes>(\d+)\s+)(?<response>(\w+)\s+)(?<url>([^\s]+))\s+(?<user>(\w+|\-))\s+(?<method>(\S+.\S+))/
    def parse(line)
      data = {}
time = Time.now.to_f
      begin
REGEX.match(line) { |match|
data[‘Host’] = OMS::Common.get_hostname
          timestamp = Time.at( match[‘eventtime’].to_i() )
data[‘EventTime’] = OMS::Common.format_time(timestamp)
data[‘EventDate’] = timestamp.strftime( ‘%Y-%m-%d’ )
data[‘Duration’] = match[‘duration’].to_i()
data[‘SourceIP’] = match[‘sourceip’]
data[‘cache’] = match[‘cache’]
data[‘status’] = match[‘status’]
data[‘bytes’] = match[‘bytes’].to_i()
data[‘httpresponse’] = match[‘response’]
data[‘bytes’] = match[‘bytes’].to_i()
data[‘url’] = match[‘url’]
data[‘user’] = match[‘user’]
data[‘method’] = match[‘method’]}
rescue => e
@error_handler.logerror(“Unable to parse the line #{e}”)
end
      return time, data
end   #def
   end   #class
3. Finally, I wrote the squid.conf for OMS
# enhanced parse log with date format , which pass the path for the log to the SquidLogParser and tag it as oms.api.Squid. By doing this, you will end up with 11 custom fields in OMS for the LOG TYPE Squid_CL
<source>
type tail
format SquidLogParser
path /var/log/squid/access.log
pos_file /var/opt/microsoft/omsagent/state/var_log_squid_access.pos
tag oms.api.Squid
log_level error
</source>
squid-fields

 

On my next article I will go through the next part, which is getting Squid Proxy Statistics in OMS, along with the full code.

squid2.png

 

Hybrid Cloud Computing with Microsoft and Red Hat

November 5, 2015 1 comment

Microsoft and Red Hat announced a partnership that will help customers embrace hybrid cloud computing by providing greater choice and flexibility deploying Red Hat solutions on Microsoft Azure.

redhat-msRed Hat Enterprise Linux will be the preferred option for enterprise Linux workloads on Microsoft Azure.

Microsoft Azure will become a Red Hat Certified Cloud and Service Provider, enabling customers to run their Red Hat Enterprise Linux applications and workloads on Microsoft Azure. Red Hat Cloud Access subscribers will be able to bring their own virtual machine images to run in Microsoft Azure.

Customers will be offered cross-platform, cross-company support spanning the Microsoft and Red Hat offerings in an integrated way, unlike any previous partnership in the public cloud. By co-locating support teams on the same premises, the experience will be simple and seamless, at cloud speed.

Red Hat CloudForms will interoperate with Microsoft Azure and Microsoft System Center Virtual Machine Manager, offering Red Hat CloudForms customers the ability to manage Red Hat Enterprise Linux on both Hyper-V and Microsoft Azure. Support for managing Azure workloads from Red Hat CloudForms is expected to be added in the next few months, extending the existing System Center capabilities for managing Red Hat Enterprise Linux

In addition, Expanding on the preview of .NET on Linux announced by Microsoft in April, developers will have access to .NET technologies across Red Hat offerings, including Red Hat OpenShift and Red Hat Enterprise Linux, jointly backed by Microsoft and Red Hat. Red Hat Enterprise Linux will be the primary development and reference operating system for .NET Core on Linux.

Source: http://www.redhat.com/en/about/press-releases/microsoft-and-red-hat-deliver-new-standard-enterprise-cloud-experiences

Workaround for Pluggable Time Source support for RHEL 5.x 64-bit Hyper-V guests

When installed in a supported Linux virtual machine running on Hyper-V, the Linux Integration Components provide:

• Driver support for synthetic devices: The Linux integration components include support for both the synthetic network controller and synthetic storage controller that have been developed specifically for Hyper-V. These components take advantage of the new high-speed bus, VMBus, which was developed for Hyper-V.

• Fastpath Boot Support: Boot devices now take advantage of the storage VSC to provide enhanced performance.

• Timesync: The clock inside the virtual machine will remain synchronized with the clock on the host.

• Integrated Shutdown: Virtual machines running Linux can be shut down from either Hyper-V Manager or System Center Virtual Machine Manager, using the “Shut Down” command.

• Symmetric Multi-Processing (SMP) Support: Supported Linux distributions can use up to 4 virtual processors (VP) per virtual machine.

• Heartbeat: Allows the host to detect whether the guest is running and responsive.

• Pluggable Time Source: A pluggable clock source module is included to provide a more accurate time source to the guest.

Although this version of the integration services for Hyper-V supports Novell SUSE Linux Enterprise Server 10 SP3, SUSE Linux Enterprise Server 11, and Red Hat Enterprise Linux 5.2 / 5.3 / 5.4 / 5.5, Red Hat Enterprise Linux 5 x64 editions do not support using the Pluggable Time Source component.

However, you can use the existing timesync component with the adjtimex RPM package, to compensate for time drift:

If using the x64 version of Red Hat Enterprise Linux, follow these additional instructions:

 1.   Attach the Red Hat Enterprise Linux ISO that was used for installation in the virtual machine, and mount it using the following commands at a shell prompt:

# mkdir /mnt/cdrom

# mount /dev/cdrom /mnt/cdrom

2.   Install the adjtimex RPM for more accurate time keeping in the virtual machine:

# rpm –ivh /mnt/cdrom/Server/adjtimex-1.20-2.1.x86_64.rpm

 

To install the Linux IC.

After the ISO is “available”.
Firstly, install the Required Components via Command Line

Red Hat updates

# yum install kernel-devel

# yum groupinstall “development tools”

# yum update 

Restart the VM and then, insert the LinuxIC iso into the DVD drive and copy the contents to /opt/linuxic directory.
Run the commands below, to install the IC:  

 # mount /dev/cdrom /mnt

# mkdir /opt/linux_is
 

# cp –r /mnt/* /opt/linux_is

# cd /opt/linux_is

# make

# make install 

 
To verify that the Linux ICs are working.

# /sbin/lsmod | grep vsc

# modinfo vmbus

To verify the Synthetic Ethernet adapter is working

# ifconfig seth0
 
Note:  Article also published by myself at http://social.technet.microsoft.com/wiki/contents/articles/workaround-for-pluggable-time-source-support-for-rhel-5-x-64-bit-hyper-v-guests.aspx
My thanks to Sumesh P - MSFT, who helped with the tip.