Archive

Posts Tagged ‘Ubuntu’

Extending Microsoft OMS to monitor Squid Proxy running in Linux with a plugin – part 1/3 #MSOMS

November 24, 2016 1 comment

Since Microsoft released OMS, I have been an early adopter and evangelist for the solution. Not only it is simple to deploy but it gives you a full spectrum of many of the workloads you have either on-premises or in the cloud and it does not matter which cloud. Be it Azure, AWS, Google and many others.

So, as I was advising on OMS for a customer, I found that they were running Squid Proxy servers. The Squid proxy server is one of the most famous proxy servers in the world and it has been utilised for years in many organisations. For that reason I then I decided to look at how OMS could leverage the monitoring for Squid.

squi3

As you can see here: https://github.com/Microsoft/OMS-Agent-for-Linux/tree/master/installer/conf/omsagent.d there are already many plugins for OMS to  monitor Windows and many Linux OS as well, DNS, Network, SQL, MySQL, Postgree, VMware, MongoDB, Security, Audit, Change Tracking and so on.

But, there was no Squid plugin and that’s where I brought back my past years of experience as a developer and although that was a long, long time go, I was able to developer in ruby a Squid plugin for Microsoft OMS.

How I developed it?

PART 1 : LOG Files

  1. I started but investigating the squid log on /var/log/squid/access.log and then I research REGEX expressions to extract information out of it. Below is a extract of it

1479696836.902    134 10.1.1.4 TCP_MISS/301 488 open http://cnn.com/ – HIER_DIRECT/151.101.0.73 –
1479696848.110    242 10.1.1.4 TCP_MISS/400 486 open http://www.sydney.com/ – HIER_DIRECT/54.253.253.77 text/html
1479696860.004    407 10.1.1.4 TCP_MISS/301 636 open http://www.7news.com.au/ – HIER_DIRECT/203.84.217.229 text/html

The initial difficult part for me was of to decouple the date/time to get it on a human readable format. So, after long hours of research and playing along, I decided for the following REGEX :

 REGEX =/(?<eventtime>(\d+))\.\d+\s+(?<duration>(\d+))\s+(?<sourceip>(\d+\.\d+\.\d+\.\d+))\s+(?<cache>(\w+))\/(?<status>(\d+))\s+(?<bytes>(\d+)\s+)(?<response>(\w+)\s+)(?<url>([^\s]+))\s+(?<user>(\w+|\-))\s+(?<method>(\S+.\S+))/
(If you have a better one, please feel free to shot me)

 

  1. I then wrote a squidparserlog.rb in ruby to parse the Squid access.log file and turn it into a OMS format
class SquidLogParserLib
require ‘date’
require ‘etc’
require_relative ‘oms_common’
require ‘fluent/parser’
    def initialize(error_handler)
@error_handler = error_handler
end
    REGEX =/(?<eventtime>(\d+))\.\d+\s+(?<duration>(\d+))\s+(?<sourceip>(\d+\.\d+\.\d+\.\d+))\s+(?<cache>(\w+))\/(?<status>(\d+))\s+(?<bytes>(\d+)\s+)(?<response>(\w+)\s+)(?<url>([^\s]+))\s+(?<user>(\w+|\-))\s+(?<method>(\S+.\S+))/
    def parse(line)
      data = {}
time = Time.now.to_f
      begin
REGEX.match(line) { |match|
data[‘Host’] = OMS::Common.get_hostname
          timestamp = Time.at( match[‘eventtime’].to_i() )
data[‘EventTime’] = OMS::Common.format_time(timestamp)
data[‘EventDate’] = timestamp.strftime( ‘%Y-%m-%d’ )
data[‘Duration’] = match[‘duration’].to_i()
data[‘SourceIP’] = match[‘sourceip’]
data[‘cache’] = match[‘cache’]
data[‘status’] = match[‘status’]
data[‘bytes’] = match[‘bytes’].to_i()
data[‘httpresponse’] = match[‘response’]
data[‘bytes’] = match[‘bytes’].to_i()
data[‘url’] = match[‘url’]
data[‘user’] = match[‘user’]
data[‘method’] = match[‘method’]}
rescue => e
@error_handler.logerror(“Unable to parse the line #{e}”)
end
      return time, data
end   #def
   end   #class
3. Finally, I wrote the squid.conf for OMS
# enhanced parse log with date format , which pass the path for the log to the SquidLogParser and tag it as oms.api.Squid. By doing this, you will end up with 11 custom fields in OMS for the LOG TYPE Squid_CL
<source>
type tail
format SquidLogParser
path /var/log/squid/access.log
pos_file /var/opt/microsoft/omsagent/state/var_log_squid_access.pos
tag oms.api.Squid
log_level error
</source>
squid-fields

 

On my next article I will go through the next part, which is getting Squid Proxy Statistics in OMS, along with the full code.

squid2.png

 

Linux Bash Command Line natively on Windows 10

 

If you a Linux fan or like to play around with Linux or are experienced Linux, Mac OS X or Unix developer, this feature is perfect for you: the new Bash feature will give you a Linux root shell, which means that you won’t even need to use sudo to become “Administrator”. As root user you will have full system access, like you have in Linux and Unix systems.

 

6

 

Before we start, is important to understand that an application that you install in the Bash shell is restricted to Bash shell. You won’t be able to access the application from PowerShell, Cmd or Explorer in Windows.

 

The solution

 

To offer a full Bash shell based on Ubuntu, Microsoft worked with Canonical to provide a shell that runs atop the subsystem allowing you to run the Bash shell and the exact same binaries you’d normally run on Ubuntu Linux.

There are some limitations as you won’t be able to install Linux server applications or to start Linux graphical software. The target for the feature are developers who want to run Linux command-line utilities on Windows.

 

Requirements

 

  • Windows 10 Insider Preview Build (minimum build: 14316)
  • Developer Mode activated

 

Getting there in simple 10 steps

 

1. Click on Windows Start, click on Settings and then click on Update & Security

2. On the left menu, click on Windows Update, then click on Advance Options

3. On the Advanced Options window, if you are not already an Windows Insider, click on Get Started and follow the instructions to become an Windows Insider. You will be requested to restart your computer and after that, return to this same windows and select the Fast mode to have at least the 14316 build installed.

0

4. Once the computer have the required minimum build, go back to Update & Security and on the left menu select For Developers.

1

  1. On Windows Start, type Program and Features

2

 

6. On the left panel, click on Turn Windows Feature on or off

3

 

7. Select Windows Subsystem for Linux (Beta) from the list and click on OK.  You’ll be requested to restart your computer,  for the feature to get installed.

4

8. When you computer gets back, click on the Windows Start button and type bash then select the bash command or press enter.

5v2

9. You will be requested to accept the terms of license and after accepting it a bash window will open. Press Y to accept the download of the Bash Ubuntu on Windows application from the Windows Store.

6.png

10. That’s it. You now have a full command-line bash shell based on Linux Ubuntu. You now have access to all the Linux command line software.

Note: As we installed the bash for Ubuntu and because they’re the same binaries as you would have on a normal Linux Ubuntu installation, you can use the same apt-get command you used to run on Ubuntu, to install software from Ubuntu’s repositories.

Notes:

  • In Linux the Bash shell is case-sensitive and the
  • Windows file system is located at /mnt/c in the Bash shell environment.

 

If you find that Microsoft is helping you and your business  as I as do, please help me out by recommending it on Recomazing a new tech platform where socially connected networks store and share trusted recommendations. Please click  here to help our community.

Have Fun!!!

 

 


Categories: Cloud, Microsoft Tags: , , , , , ,