Extending Microsoft OMS to monitor Squid Proxy running in Linux with a plugin – part 1/3 #MSOMS

November 24, 2016 1 comment

Since Microsoft released OMS, I have been an early adopter and evangelist for the solution. Not only it is simple to deploy but it gives you a full spectrum of many of the workloads you have either on-premises or in the cloud and it does not matter which cloud. Be it Azure, AWS, Google and many others.

So, as I was advising on OMS for a customer, I found that they were running Squid Proxy servers. The Squid proxy server is one of the most famous proxy servers in the world and it has been utilised for years in many organisations. For that reason I then I decided to look at how OMS could leverage the monitoring for Squid.

squi3

As you can see here: https://github.com/Microsoft/OMS-Agent-for-Linux/tree/master/installer/conf/omsagent.d there are already many plugins for OMS to  monitor Windows and many Linux OS as well, DNS, Network, SQL, MySQL, Postgree, VMware, MongoDB, Security, Audit, Change Tracking and so on.

But, there was no Squid plugin and that’s where I brought back my past years of experience as a developer and although that was a long, long time go, I was able to developer in ruby a Squid plugin for Microsoft OMS.

How I developed it?

PART 1 : LOG Files

  1. I started but investigating the squid log on /var/log/squid/access.log and then I research REGEX expressions to extract information out of it. Below is a extract of it

1479696836.902    134 10.1.1.4 TCP_MISS/301 488 open http://cnn.com/ – HIER_DIRECT/151.101.0.73 –
1479696848.110    242 10.1.1.4 TCP_MISS/400 486 open http://www.sydney.com/ – HIER_DIRECT/54.253.253.77 text/html
1479696860.004    407 10.1.1.4 TCP_MISS/301 636 open http://www.7news.com.au/ – HIER_DIRECT/203.84.217.229 text/html

The initial difficult part for me was of to decouple the date/time to get it on a human readable format. So, after long hours of research and playing along, I decided for the following REGEX :

 REGEX =/(?<eventtime>(\d+))\.\d+\s+(?<duration>(\d+))\s+(?<sourceip>(\d+\.\d+\.\d+\.\d+))\s+(?<cache>(\w+))\/(?<status>(\d+))\s+(?<bytes>(\d+)\s+)(?<response>(\w+)\s+)(?<url>([^\s]+))\s+(?<user>(\w+|\-))\s+(?<method>(\S+.\S+))/
(If you have a better one, please feel free to shot me)

 

  1. I then wrote a squidparserlog.rb in ruby to parse the Squid access.log file and turn it into a OMS format
class SquidLogParserLib
require ‘date’
require ‘etc’
require_relative ‘oms_common’
require ‘fluent/parser’
    def initialize(error_handler)
@error_handler = error_handler
end
    REGEX =/(?<eventtime>(\d+))\.\d+\s+(?<duration>(\d+))\s+(?<sourceip>(\d+\.\d+\.\d+\.\d+))\s+(?<cache>(\w+))\/(?<status>(\d+))\s+(?<bytes>(\d+)\s+)(?<response>(\w+)\s+)(?<url>([^\s]+))\s+(?<user>(\w+|\-))\s+(?<method>(\S+.\S+))/
    def parse(line)
      data = {}
time = Time.now.to_f
      begin
REGEX.match(line) { |match|
data[‘Host’] = OMS::Common.get_hostname
          timestamp = Time.at( match[‘eventtime’].to_i() )
data[‘EventTime’] = OMS::Common.format_time(timestamp)
data[‘EventDate’] = timestamp.strftime( ‘%Y-%m-%d’ )
data[‘Duration’] = match[‘duration’].to_i()
data[‘SourceIP’] = match[‘sourceip’]
data[‘cache’] = match[‘cache’]
data[‘status’] = match[‘status’]
data[‘bytes’] = match[‘bytes’].to_i()
data[‘httpresponse’] = match[‘response’]
data[‘bytes’] = match[‘bytes’].to_i()
data[‘url’] = match[‘url’]
data[‘user’] = match[‘user’]
data[‘method’] = match[‘method’]}
rescue => e
@error_handler.logerror(“Unable to parse the line #{e}”)
end
      return time, data
end   #def
   end   #class
3. Finally, I wrote the squid.conf for OMS
# enhanced parse log with date format , which pass the path for the log to the SquidLogParser and tag it as oms.api.Squid. By doing this, you will end up with 11 custom fields in OMS for the LOG TYPE Squid_CL
<source>
type tail
format SquidLogParser
path /var/log/squid/access.log
pos_file /var/opt/microsoft/omsagent/state/var_log_squid_access.pos
tag oms.api.Squid
log_level error
</source>
squid-fields

 

On my next article I will go through the next part, which is getting Squid Proxy Statistics in OMS, along with the full code.

squid2.png

 

Innovation Days: Event Report and Feedback

September 30, 2016 Leave a comment

Innovation Days was held from 9am to 5pm on Saturday 17th September at NSI TAFE NSW Campus. Over 130 attendees took part in a ground-breaking new event. The content focused on the following topics: Datacenter and Cloud (Private, Hybrid and Public), Identity and Security, Linux, Collaboration, IoT, Database, Business Analytics and Integration and Application Development.

1

2

Information gathered during registration showed that 73% of attendees were interested in Cloud and Datacenter management, 64% in Data Analytics and 50% in Modern Apps.

 

3

It also showed that 53.77% are aware that their company has a cloud strategy, 27.36% who don’t know and 18.87% whose company does not have a cloud strategy yet.

 

4

 

It showed also some insights about Linux utilisation/deployment at about 42% and what are their planning public clod adoption, with Azure as preferred by 44.81% followed by AWS at 21.23% and Unknown/None at 35.38%.

 

5.png
The event was sponsored by NSI Tafe NSW, Microsoft, RXP and RedHat. Significant in-kind sponsorship was received from NSI Tafe NSW, and I am very thankful for that. The event ran at a non-profit and it was free to the attendees, thanks the sponsorship.

The feedback from participants gathered during informal conversations at the networking time, indicated a very high level of satisfaction with the event. Quoting one attendee: “The event was inspiring. The IoT session was great. I am going to buy a Raspberry PI and start coding.  It’s amazing what we can achieve with that.”

The speakers were amongst industry experts and Australian Most Valuable Professionals (MVP) who presented great content about innovative technologies and based on their real world experience in short action packed sessions. I would like to thank you all the speakers for taking time to prepare and present great content sessions.6

The event started 9am with an introduction from NSI Tafe NSW Derik Pola, Faculty Director of Information Technology, Media and Business , followed by a keynote session “Go Mobile, stay in Control: Enterprise Mobility + Security” from Microsoft Evangelist Jeff Alexander, followed by another keynote on Modern Cloud by Alessandro Cardoso and then 3 sessions: Nano Server:  Minimize reboots and improve security with next-gen server deployment with Jeff Alexander, Introducing Microsoft Pimg_5140owerApps with Bill Chesnut and .Net Core with Jordan Knight.

After lunch, we had another keynote Session: IMG_5149.JPGContainers Anywhere with OpenShift with Stefano Picozzi from RedHat, who give away an Openshift book and I saw a many interested in getting a copy, which showed that the subject is at much interest.

Then we had 9 great sessions:

  • Azure IoT End-to-End with Martin Abbott, MVP
  • Business Case for Upgrading to SQL Server 2016 with Victor Isakov
  • API Management and Hybrid Integration with Bill Chesnut, MVP
  • “Best mates” Power BI and Machine Learning with Grant Paisley, MVP
  • Using Azure Active Directory B2C in your next consumer App with Simon Waight, MVP
  • Hyper-V, Nested Virtualisation and Linux with Alessandro Cardoso, MVP
  • Windows 10 Integration with Organisational Identities in The Cloud with Mark O’Shea, MVP
  • Collaborate beyond the boundaries of an enterprise: Your Enterprise bot is here with Amr Fouad, MVP
  • Understanding Rights Management with Robert Crane, MVP

You can find more about our speakers here: http://innovationdays.com.au/speakers

As stories goes, one of the speakers came out the hospital where his wife just had a baby. I would like to congratulate Victor Isakov for the new born and for taking time away from his family to present a great session.

img_5085The motto of the event: Delivering an engaging, informing and technically focussed event – “Whether you know your way around the cloud and are trying to stay current, or are just getting started and figuring out the best path forward – having access to information, best practices and training is more critical than ever”, was fully achieved. There were several objectives which Innovation Days aimed to fulfil: bringing together the ICT community, discovering new solutions and new ways to implement it, connecting with industry experts, IT professionals and technology partners.

The range and depth of presentations was incredibly encouraging and networking with other professional colleagues was cited as one of the mimg_5145ost important reasons to attend.

But the conference was not just talks and workshops. We had a good networking time at lunch time featuring Microsoft Cognitive Services as well as a social gathering after event with drinks.

At the end, thanks to the sponsors, we had many prizes and a lucky winner of a Lenovo Laptop sponsored by RXP Services.

wp_20160917_17_20_37_proThis event could not be great success as it was without the support from John Barnet from NSI Tafe, Michael O’Keeffe from Microsoft and our sponsors and speakers: Martin Abbott, Victor Isakov, Bill Chesnut, Grant Paisley, Simon Waight, Mark O’Shea, Amr Fouad, Robert Crane, Jeff Alexander, Stefano Picozzi and myself: Alessandro Cardoso.

 

Several of the presentations can be found on the event website http://innovationdays.com.au/session-slides, alongside with some photos from the event at http://innovationdays.com.au/photos

sponsors

I Looking forward to see you again in 2017!

Alessandro Cardoso Microsoft MVP | RXP Services Technology Strategist @cloudtidings

 

 

Categories: Cloud

Windows 2016 released and with it Hyper-V and System Center

September 27, 2016 Leave a comment

Microsoft released today at the Microsoft Ignite conference in Atlanta the newest release of Windows Server 2016!

Windows Server 2016 is jam-packed with innovation and customer response has been overwhelming, with more than half a million devices running the final Technical Preview. These customers range from large global enterprises to private cloud hosters to organizations of every size from every corner of the globe – Erin Chapple, General Manager, Windows Server

 

Windows Server 2016 delivers powerful innovation across three areas:

  • Advanced Multi-layer Security: Use Shielded Virtual Machines to help protect your virtual machines from a compromised fabric as well as improve your compliance. Shielded Virtual Machines are encrypted using BitLocker and will run on healthy hosts. To help prevent attacks and detect suspicious activity with new features to control privileged access, protect virtual machines and harden the platform against emerging threats.Watch an introduction to Shielded Virtual Machines
  • Software-defined Datacenter with Hyper-V: Run your datacenter with the utmost confidence with an automated, resilient server operating system. Azure utilises Windows Server and Hyper-V at a massive scale. Windows Servers delivers a more flexible and cost-efficient operating system for any datacenter, using software-defined compute, storage and network features inspired by Azure. Explore server virtualization with Hyper-V
  • Cloud-ready Application Platform: Run your existing apps on Windows Server 2016 without modifying them. Take advantage of enhanced security and efficiency features in the fabric. Applications are at the heart of every organization and its ability to serve customers and compete effectively for their loyalty.  Windows Server 2016 delivers new ways to deploy and run both existing and cloud-native applications – whether on-premises or in Microsoft Azure – using new capabilities such as Windows Server Containers and the lightweight Nano Server deployment option.  Learn more about containers  and Learn more about Azure Service Fabric on Windows Server 2016

 

Availability: Windows Server 2016 is available for evaluation starting today

Note: Volume licensing customers will be able to download fully licensed software at General Availability in mid-October.

Innovation Days. 2nd batch tickets released. Register Now!

I am not surprised to see so many registrations for Innovation Days : we have great speakers and amazing sessions organised for the attendees. Plus a laptop will be draw at end of the event! So make sure you secure your ticket before is gone!

innovationdays-site

Join the Australians Most Valuable Professionals (MVP) and Industry Speakers, for a one day of accelerated learning about innovative technologies. Expert speakers will present based on their real world experience in short action packed sessions. Content will focus on the following topics:

  • Cloud (Private, Hybrid and Public)
  • Identity and Security
  • Collaboration
  • Development
  • Analytics
  • Management and Monitoring

Register NOW

sponsors

http://innovationdays.com.au/register

Categories: Cloud

Why your business need a strategic cloud roadmap

Adoption of the “public cloud” IT model is growing faster, not doubt about it. Now, when I talk with customers about cloud, instead of “if”, they now ask “how” and “where to”.

According to Cloud Industry Forum, satisfaction with cloud services is high at 94% and cloud adoption is happening successfully across all types and sizes of organisation across every industry sector.

It is also becoming evident that the short answer for a successful adoption, starts with a roadmap.

The roadmap should contain:

  • The list of the strategic business priorities for the next years.
  • A prioritised list of Initiatives/projects/opportunities with expected timeline
  • Ownership, business sponsor
  • Costs and
  • High-level justification

More and more companies are looking forward to the cloud experience, as a way to drive the arrival of entirely new classes of automation, business innovation and competitive differentiation and of course, cost reduction.

Customer are realising the cloud is elastic, reliable, secure and can work on a OPEX model and no investment is needed in expensive hardware and better the IT don’t need to worry about server/storage maintenance and depending on the cloud model (PaaS/SaaS): patches, software upgrades, backups and other administrative tasks as the cloud provider would be able to offer/handle these entirety.

Cloud is affecting both IT and the LBDM as a whole and this new ethos requires their close alignment.

With Cloud, the steadiness between maintenance and innovation shifts and the new capabilities in the cloud make new kinds of powerful applications highly potential. Take the Microsoft PowerApps as an example: “Build apps that use device capabilities, including cameras, GPS, and pen control—without code. Connect to existing data sources and services—Excel files, SharePoint lists, CRM records, your custom APIs—to automatically generate a meaningful business app”.

A roadmap would help the IT Leadership to act more in line with the business strategy. It benefits both technology leaders and LBDM and encourages teamwork that results in true executive alignment on existing and new investments

Also, following the roadmap, an assessment for cloud migration at scale would involve looking at silos, current IT environment, applications and determining how it should be thought of in this new environment. Is further investment in certain technologies or applications reasonable? Should they be replaced/retired?

Many enterprises have held their technologies and applications for far too long without assigning to them a maintenance or retirement schedule. Therefore, for fear of complexity, lack of documentation, resources and many other reasons, some technologies and applications remain untouched. Even for technologies and applications that remain on-premises, modernization can save time and money.

Want to learn more about Cloud? Checkout my session at Innovation Days

Keynote Session: IaaS x PaaS : If you have been holding off on leveraging PaaS services, now is the time to revisit that decision. The modern cloud era is not IaaS-centric but modern app-centric. The latest Total Economic Impact Study by Forrester Consulting shows that migrating to PaaS from IaaS result in a 466% return on investment. For customers migrating from on-premises environments to PaaS, the return on investment can be even greater. Time to market also improved by as much as fifty percent, because of the efficiency and speed of deploying applications with PaaS services.
Are you interested in learn how you can drive application innovation and reduce costs, then this session is for you!

Note: A series of White Papers based on the Research are available free of charge from the Cloud Industry Forum website (www.cloudindustryforum.org).

Categories: Cloud

Innovation Days Australia

innovationdays-site

This year, I decided to organise a 1 day conference in Sydney and invited fellow MVP’s and other industry leaders to present great sessions and the result is Innovation Days.

Today, due to overwhelming number of registrations, the tickets sold out, almost 1 month in advance of the event which is happening on September 17th. To go on the waiting list, see site registration.

Thanks to our sponsors, a 2nd batch of tickets have been released. Don’t wait, register now! http://innovationdays.com.au/register

Although it is not the first time I organise events, I couldn’t be more proud to organised a great event covering topics like Cloud and Datacenter Management, Data Analytics and Integration, Identity.

I would like to thank the conference speakers and sponsors: Northern Sydney Institute Tafe NSW, RXP Services, Microsoft and RedHat. Thank you very much for your support.

For more information about the event, check out http://innovationdays.com.au/

Join Australian Most Valuable Professionals (MVP) and Industry Experts for a one day of accelerated learning about innovative technologies and real world experience in short action packed sessions. http://innovationdays.com.au

If you are asking yourself about other cities: stay tunned.

Categories: Cloud

Get the big picture of your IT environment

 

With today’s digital transformation how many experts does it take to cover the hardware and software that the company needs to be productive? And how do you get the big picture if they are all disconnected with teams using different tools to monitor and manage their own divided departments?

Savision will be hosting two live online sessions that will guide you step by step and show you how to obtain a holistic view of your IT environment by connecting your ITSM & Monitoring systems. They will show you how to aggregate and analyze dispersed data from systems like SCOM, Azure, SolarWinds, Nagios, ServiceNow, with more to come.

If you are having that problem within your organization, you cannot miss the upcoming live online sessions that Savision will be hosting. The sessions will walk you through the process of improving your IT operations and show you how to obtain a holistic view of your IT environment by connecting your ITSM & Monitoring systems. You can choose to join one of two sessions where you will find out more about: – How you can find out your organization’s IT Maturity level and how to improve it

  • How easy it is to connect the IT, the helpdesk and the business departments
  • How to aggregate and analyze dispersed data from your existing systems and obtain actionable information

The sessions will be hosted by Savision’s CTO, Rob Doucette and co-founder & VP of Product Management, Dennis Rietvink. You can choose one of the following dates to register, depending on your region.

US Session: Tuesday, June 21st – 2PM EDT | 8PM EST

EU Session: Thursday, June 23rd – 8AM EDT | 2PM EST

Register here.

Categories: Cloud