Thursday, December 18, 2014

sys diag kill command does not kill processes on a Fortigate

I recently had a Fortigate 1500D become bogged down due to the reporting daemon (reportd) utilizing 100% CPU.  I will not go into a lot of detail about diagnosing performance issues, as that is not the topic of this post.  Suffice to say that you’ll need to run the following commands to determine which process is misbehaving:

#get sys perf status
#diag sys top
#diag hard sys mem

Typically one would kill and respawn the offending process with the following command, where process_id is obtained via the diag sys top command.

#diag sys kill 11 process_id

Unfortunately in this case the kill command did not actually kill the process, and a reboot was not an option.  Fortunately I once had a remote session with Fortinet TAC where I saw them using some hitherto unknown (to me) commands.  So what follows is an unsupported way to absolutely kill processes dead.

The command we use is fnsysctl.  This command allows us access to a subset of Linux utilities, like cat, ls, kill etc.  In this case we’re interested in the kill command:

#fnsysctl kill –9 process_id

I have never had this command fail to kill a process, although I would recommend only using it as a last resort.

Happy Hunting!

Monday, December 15, 2014

Fortigate SSL VPN on multiple interfaces

There is an potential issue when setting up SSL VPN to listen on multiple interfaces on FortiOS 5.2.2 (have not tested on earlier versions).  When you create the first SSL VPN listener the Fortigate will automatically create a policy to allow SSL VPN traffic.  Due to the way the information is presented one can be excused for thinking the unit will do it for additional listeners as well.  The screenshot below shows how it looks when SSL VPN is enabled on multiple interfaces via the GUI:


These are the relevant bits taken from the configuration file:

config vpn ssl settings
    set idle-timeout 900
    set tunnel-ip-pools "SSLVPN_TUNNEL_ADDR1"
    set tunnel-ipv6-pools "SSLVPN_TUNNEL_IPv6_ADDR1"
    set source-interface "MTN" "MTC"
    set source-address "all"
    set source-address6 "all"
    set default-portal "tunnel-access"
        config authentication-rule
            edit 1
                set source-interface "MTN"
                set source-address "all"
                set groups "VPN"
                set portal "full-access"

From the above it is clear that the listener has been added, but the authentication rule has not been updated.  The fix is simple, just add an additional authentication rule via the CLI:

config vpn ssl settings
config authentication-rule
            edit 2
                set source-interface "MTC"
                set source-address "all"
                set groups "VPN"
                set portal "full-access"

Your SSL VPN clients will now be able to connect on all interfaces specified in the GUI.

Wednesday, September 3, 2014

Symantec Data Loss Prevention Implementation Strategy

Data Loss Prevention (DLP) is fast becoming an key element in protecting an organization's data.  Access to information is critical to an organizations success, conversely the consequence of a data loss incident has never been higher than now. 

Despite the hefty price tag, a DLP solution often proves to be a relatively easy sell (we protect *all* your data against electronic leakage!).  Whilst a successful implementation can achieve those lofty goals, the devil is in the details. 

The most important requirement is to have complete buy-in from the business, in fact achieving a successful implementation will be absolutely impossible without input from all the stakeholders.  Secondly we need to realize that any DLP solution is an iterative process.  Lastly we need to have a proper implementation plan in place and this will be the focus of this post.

High-Level Process Overview

Identify Confidential Data and Data at Risk (Phase 1)

Create Policies (Phase 2)

Discover Data-at-Rest (Phase 3)

Protect the Endpoints (Phases 4 - 6)

Protect Data-at-Rest and Data-in-Motion (Phases 7 - 8)

Remediation and risk reduction (Phase 9)

Breakdown of Phases

Phase 1 - Identify Confidential Data

During this phase we will train DLP to indentify data that is critical and confidential to the business.

  1. Involve relevant relevant stakeholders (compliance and risk officers, Heads of Departments, IT) to assist in identifying confidential data.

  2. Obtain sample data based on the above

  3. Introduce the sample data into the DLP Solution

    1. Sample data population for Exact Data Matching techniques (EDM) based policies

    2. Sample data population for Described Content Matching (DCM) based policies

    3. Sample data population for Indexed Data Matching techniques (IDM) based policies

    4. Submission of of positive and negative samples for Vector Machine Learning (VML)

Phase 2 - Policy Creation

  1. Identify reporting an alerting audience

  2. Create policies to alert based on the datasets submitted in Phase 1

Phase 3 Enable Data-at-Rest / Network Discover

During this phase we will enable the Network Discover component of Symantec DLP.  This allows us to scan data repositories on the LAN (File Servers / NAS's etc.) for policy violations.  We will only alert the DLP team during this phase.

  1. Deploy Symantec Data Insight (if purchased as part of the DLP Suite) to assist with identification of data owners, as well as to monitor and log permissions and data accesses by end-users.

  2. Configure Data Insight / DLP integration which will enable DLP to infer data owners based on intelligence gained from Symantec Data Insight.

  3. Consult with stakeholders to identify all relevant data repositories on the network

  4. Initiate a Network Discover scan of all the data repositories based on the previously created EDM, DCM, IDM and VML policies.

  5. Re-iterate the process to fine-tune the policies and EDM, DCM, IDM and VML data-sets

  6. Enable alerting to the relevant stakeholders and re-iterate based on feedback

Phase 4 - Pilot Deployment - Endpoints

During this phase we will only be monitoring.  Alerts will only be sent to the DLP team - no end user alerting or notification.

  1. EDM Policy Implementation for pilot group

  2. DCM based Policy Implementation on for pilot group

  3. IDM based Policy Implementation for pilot group

  4. Re-iterative review of DLP policies, EDM, DCM, IDM and VML data based on alerting and reporting

Phase 5 - Enable Prevent Mode for Pilot Group

During this phase we will actively start blocking and alerting on the endpoints.  Both the end-users and the DLP team will receive alerts and notification.  We will depend heavily on end-user feedback to further streamline our DLP policies.

  1. Enable end-user notification

  2. Fine tune DLP policies based on end-user feedback as well as alerts

  3. Fine tune EDM, DCM, IDM and VML data based on end-user feedback

  4. Re-iterate the process to reduce false-positives

  5. Enable alerting to relevant stakeholders (Data owners, Heads of Departments etc.)

Phase 6 - Roll out the Endpoint Agent to all users

During this phase we will roll-out the DLP endpoint agent to all users.  Our policies and confidential data has been thoroughly fine-tuned so we can safely enable end-user alerting as well.

  1. Ideally we should provide the end-users with DLP / Security awareness training during this phase

  2. Deploy the DLP endpoint agent via the organization's preferred deployment mechanism (GPO / SCCM / Altiris etc.)

  3. Further fine-tuning of policies and EDM, DCM, IDM and VML data-sets based on alerting and end-user feedback

Phase 7 - Enable Data-at-Rest / Network Prevent

During this phase we will activate the blocking mode of the Network Prevent component.  This will involve quarantining, copying, encrypting and/or applying DRM to sensitive and confidential data existing on your data repositories.  Alerting will be to the DLP team as well as the relevant stakeholders.

  1. Verify all Data repositories configured within DLP during Phase 5

  2. Create the necessary FlexResponse rules to automatically respond to incidents.

  3. Actions include quarantining, copying, encrypting and/or applying DRM

  4. Add the FlexResponse rule to your policies

  5. Scan your data repositories

Phase 8 - Deploy Network Monitor and Network Prevent for Email and Web

During this phase we deploy the Network Monitor component, which operates via a mirror / SPAN port to capture Data in Transit incidents.  We will also deploy the Web and Email DLP components.  Alerting will be done to the the DLP team as well as the relevant stakeholders.

  1. Install the physical DLP Network Monitor Server (This server cannot be virtualised as per Symantec)
  2. Integrate the DLP Network Prevent for Email component with your existing MTA (DLP can operate in either Reflecting or Forwarding mode)
  3. Integrate the DLP Network Prevent for Web component with your existing proxy server.  (The proxy server needs to act as an ICAP client to DLP.  Symantec DLP supports both the REQMOD and RESPMOD modes of ICAP)

  4. Further fine-tuning of policies and EDM, DCM, IDM and VML data-sets based on alerting

Phase 9 - Ongoing remediation and reporting on risk reduction

This phase will be perpetual.  As incidents occur they will be re-mediated, and reporting to the relevant stakeholders will be set up and adjusted as necessary.  Your DLP solution is now fully deployed, and your policies and EDM, DCM, IDM and VML fingerprints will be continuously updated by the DLP team as business data changes over time.

Monday, September 1, 2014

How to enable Windows Server 2008 R2 to issue SAN certificates

The Certificate Authority in Windows 2008 R2 cannot issue Subject Alternative Name certificates in it’s default configuration.  Therefore if you include a SAN entry (like for Exchange) the CA will issue your certificate, but omit all the SAN entries

To allow your CA to issue SAN certificates you need to run the following from an administrative command prompt:

certutil -setreg policy\EditFlags +EDITF_ATTRIBUTESUBJECTALTNAME2
net stop certsvc
net start certsvc

This is not necessary on a Windows 2012 and above.


Wednesday, August 27, 2014

Symantec Data Insight 4.5 Overview

Symantec recently released version 4.5 of their Data Insight product.  In a nutshell, Symantec Data Insight allows an organization to:

  • Identify who works with and owns their data
  • Understand what unstructured data they have
  • Maintain regulatory compliance for information access, use and retention
  • Ensure information is protected from exposure to unauthorized individuals
  • Review permissions on data and suggest changes

What’s New?

  • Self-service portal allows data owners to remediate incidents directly without necessarily involving IT.
  • Reporting enhancements
  • Deep integration with Symantec Enterprise Vault and Symantec Data Loss Prevention
    • Data Insight can infer ownership of a file and then add that intelligence to Symantec DLP
    • Allows you to utilise Symantec EV’s file archiving and retention management capabilities when remediating incidents

To sum it all up, if you need to reduce the effort you spend on securing your data, enhance your data protection levels as well as achieve / prove compliance then Symantec Data Insight will go a long way to assist you in achieving those goals.

Thursday, August 14, 2014

My Customer Service Rules

I've spent the last decade of my career dealing with customers, both directly and indirectly.  I've made my fair share of mistakes, and I also did a lot of things right.  Whilst I still have a lot to learn about customer service (I am technical, not sales after all) I also have some rules that I believe one should unflinchingly abide by.  These are, in no particular order:

  1. Never lie.  Not to your customers, not to your colleagues, not to anyone.
  2. If you make a mistake, be accountable.  Everyone makes mistakes and most people of worth respect and appreciate honesty.
  3. The quickest way to lose the respect of your peers and customers is to point fingers and trying to apportion blame.
  4. Get to know your customer and their business.
  5. Listen.  Understand the problem from the customer's point of view, not just a technical one.
  6. You expect to be paid like a professional, so act like one.  Do not become too "friendly" with the customer and do not swear in front of them. Ever.
  7. "It's not my problem" does not exist in your vocabulary.  If it's not within your domain then help the customer resolve it by roping in someone who can.
  8. Always notify the customer when you are about to make changes.  E-Mail is only sufficient if the customer acknowledges the mail, otherwise follow up with a phone call.
  9. Only stick to the communicated changes.  If you tell the customer you're rebooting the mail server, do not reboot the file server as well.
  10. In that vein, always make sure your customer does not get caught by surprise by always communicating updates.
  11. Stick to your appointments - if you say will show up at 9 then you show up at 9.  If you can't then you let the customer know in advance via a phone call - e-mail is not acceptable.
  12. Share your knowledge with the customer.  This will not make you redundant, it will allow you to provide value further up the chain.
  13. You exist to serve the customer and their needs (within reason).
  14. Speak to your customers often, no less than once a month.  In the services industry "Out of sight, out of mind" holds very true.
  15. Always protect yourself, the customer and your relationship by agreeing on things up front.  This includes costs, scope of work and handover / success criteria.  Do not negotiate these after the fact.
  16. Whenever possible work for and with people you respect. When you do, even the stressful times are easier to deal with.
  17. Pre-sales is an art, and the paint brush is 'why.'  Keep asking until you get to the root.
  18. Never, ever, bad-mouth another customer or vendor in front of a customer.  Always focus on and sell your strengths.
  19. Lastly, no-one who ever bought a drill needed a drill...they needed to make a hole.  I cannot stress enough how important this concept is.

H323 traffic failing to traverse a Fortigate firewall

Had a scenario recently where a Polycom video conferencing device just wouldn’t work when sat behind a Fortigate firewall.  This was despite all the necessary TCP ports being forwarded to the device, as verified by Polycom support.

What we were seeing is that one could dial the VC but it would just ring and never make the connection.  Time to debug the traffic on the Fortigate – this is what I saw:

id=13 trace_id=74 msg="vd-root received a packet(proto=6, x.x.x.x:1720->x.x.x.x:63665) from lan."
id=13 trace_id=74 msg="Find an existing session, id-02237475, reply direction"
id=13 trace_id=74 msg="SNAT x.x.x.x->x.x.x.x:1720"
id=13 trace_id=74 msg="run helper-h323(dir=reply)"

The “run helper” sequence kicked in as soon as one attempted to pick up the call on the VC.  In Polycom’s case they suggest explicitly disabling any h323 helpers, so that is exactly what I did.  I did it like so:

  1. From the cli, execute “config system session-helper”.  This will give you the following output (below is redacted)
    edit 2
            set name h323
            set port 1720
            set protocol 6
        edit 13
            set name sip
            set port 5060
            set protocol 17
  2. Now delete these helpers by executing
    config system session-helper
    delete 2
    delete 13
  3. Enter the following commands:
    config system settings
    set sip-helper disable
    set sip-nat-trace disable
  4. Lastly we disable RTP processing:
    config voip profile
    edit default
    config sip
    set rtp disable

Your h323 and SIP traffic should now traverse your Fortigate without issue.  In my experience this has only happened with Polycom devices, Microsoft Lync works fine without modification.

Thursday, July 24, 2014

Fortigate SSL VPN Client cannot resolve FQDNs

Recently had a customer complain that he cannot access documents on his file server when connected via SSL VPN. Closer inspection showed that the customer was trying to access the fileserver by hostname “\\fileserver” as opposed to “\\fileserver.corp.local”.

The fix seemed to be simple, implement a DNS search suffix. Unfortunately there is no such option in the GUI, so I had to set it via command-line.

Set DNS search suffix using CLI

config vpn ssl settings
set dns-suffix corp.local

Set Client DNS Server in the GUI

Navigate to VPN –> SSL –> Settings –> Tunnel Mode Client Settings.  Specify the DNS Server setting and enter the IP addresses of your corporate DNS servers.

Your Fortigate will now append the “corp.local” suffix to all non-qualified hostnames.  This was tested on FortiOS 5.06, 5.07 and 5.2

Tuesday, July 22, 2014

SCCM 2012 R2 Collection Queries

After spending a considerable amount outside of the Microsoft ecosystem I recently found myself running a project where I had to deploy SCCM 2012 R2 for a customer.  I immediately caught myself wishing I made notes of my basic collection queries during my prior deployments.  Consequently I decided to document them in one place now:-)

All Dell Systems

SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_COMPUTER_SYSTEM on SMS_G_System_COMPUTER_SYSTEM.ResourceID = SMS_R_System.ResourceID where SMS_G_System_COMPUTER_SYSTEM.Manufacturer like "Dell%"

All Server Systems

select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_SYSTEM on SMS_G_System_SYSTEM.ResourceId = SMS_R_System.ResourceId where SMS_G_System_SYSTEM.SystemRole = "Server"

All Workstations

select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_SYSTEM on SMS_G_System_SYSTEM.ResourceId = SMS_R_System.ResourceId where SMS_G_System_SYSTEM.SystemRole = "Workstation"

All Domain Controllers

select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System where SMS_R_System.PrimaryGroupID = "516"

Removing persistent proxy settings

This is one of those things that I find I constantly have to google.  Oftentimes app traffic will just not flow correctly, more often than not the issue is with an outdated proxy setting.  What trips one up here is the fact that this proxy is not listed in your IE proxy settings.  The issue is that the winhttp proxy component has been set – below is how we can manipulate it.

View Current Proxy Settings

  • Windows Vista / 7 / 8 command line: netsh winhttp show proxy
  • Windows XP command line: proxycfg

Reset Proxy Settings

  • Windows Vista / 7 / 8 command line: netsh winhttp reset proxy
  • Windows XP command line: proxycfg /d

Simple fix to a potentially frustrating problem!

Wednesday, April 16, 2014

Unable to set HA mode on FortiGate

I recently had to configure a FortiGate Active – Passive HA cluster.  I did the configuration through the GUI, but no matter what I did it always reverted back to Standalone mode.

I then dropped into jedi (aka the CLI) mode and tried to configure the cluster from there, like so:


FORTIGATE # config system ha
FORTIGATE (ha) # set mode

standalone    Standalone mode.

The system may run in HA A-A or HA A-P mode only when all interfaces are NOT using DHCP/PPPoE as an addressing mode.


And there is our problem – no interfaces are allowed to be set to DHCP if you want to enable HA.  I corrected this and then proceeded to configure HA.

FORTIGATE (ha) # set mode a-p

I tested and was also able to set this via the GUI now.  First prize would of course be if FortiNet properly handles the error in the GUI, but nice to know that the proper error message is given the CLI.

Thursday, March 20, 2014

Effort vs. Recognition

Today I'm writing only the second ever non-technical piece on my blog.  The first can be found here, and touches mainly on ethics in dealing with customers and putting the customer first, always.  Today's post is a bit more introspective, focusing on why we do what we do, and the rewards, if any.

Me, I am a techie, I want to know and understand how things work and then I want to put that know-how to use so that my customers can obtain the maximum benefit out of their investment.  My boss also comes from a technical background, which is brilliant because he “gets” it, even though he has long since left the technical side of things behind.  In fact one might say he went over to the dark side, since he is much more involved in sales nowadays.

That said, a lot of what I do is impossible for my superiors to grasp nor does it immediately reflect on a financial balance sheet.  A big part of that is also undoubtedly the fact that oftentimes I simply do not know how to elucidate or explain it in terms that anyone than a peer would understand.  I do know however, that what I do will save someone a lot of trouble and effort down the road.  It might not be today or tomorrow, but it *will* prove to valuable.

Every manager will tell you that something that can't be quantified can't be measured, and what can't be measured can not be recognized.  It follows that what cannot be recognized can not be rewarded.  This brings us to an important question - do we do something the right, proper and ethical way because it is right, or because we want to be rewarded?  

I feel that we are defined by our actions, thus we become what we do.  So put in the effort.  Chances are you won't get credit or recognition for it, but that is OK.  Even if no-one sees it, *you* know.  You do it because that is who you are.  And if it's not who you are now, it is who you want to become.

I really believe that integrity and the excellence that goes with it will get rewarded in the long run.  You might find yourself thinking "what did I do to deserve this opportunity", or how a seemingly random encounter turns into an amazing and rewarding professional journey.  But it's not just random luck – this is the reward.

I've been very fortunate in that I have been able to choose my bosses and managers very carefully and I can honestly say that my interaction with each and every one of them has left me a better person, both professionally and personally.  That said, bosses come and go, but you remain.  So do the right thing, not for their recognition (though it is certainly welcome!), but for yourself.

The biggest reward of all is being able to hold your head up high, and be proud of what you have created.  Be proud of who you are and what you do.

Wednesday, March 5, 2014

Disabling the Symantec DLP Agent notifications

I’ve been branching out from the normal infrastructure stuff I’ve been doing into more security oriented fields.  Part of what I now do includes Data Loss Prevention, and I’m proud to say that I’ve recently completed my first Symantec DLP deployment.  It also happened to be the first deployment in Africa, outside of South Africa.

By default the Symantec DLP endpoint agent displays a notification when it scans for sensitive content, like so:

In this case the customer did not want to let the end-user see what was going on so we had to disable.  Unfortunately this seemingly simple UI option is not so simple – here is what you have to do

Log into the DLP Console.  Go to System -> Agent Configuration

-> Edit config -> Advanced Agent Settings

Set the to any value other than 0 and the scan dialog will not be displayed.

All Fortigate FSSO users are placed in the FSSO_guest_users group

Getting AD authentication going on a Fortigate is a slightly finicky, but well documented process and once you get it working it works well.  If it’s something you battle with, leave a comment and I’ll do a HowTo.

That said I recently had an issue with a Fortigate unit that absolutely insisted on putting all FSSO users in the FSSO_guest_users group, which means none of my Policies using authentication was working.  This is what the Fortigate saw my logged on users as:


The fix in the end was fairly simple, turns out that on the Fortigate I had the groups configured in Advanced mode, like so: CN=Internet Access,OU=Security Groups,OU=Head Office,DC=corp,DC=root. 

The Collector Agent, however, was configured to use Standard mode.  The fix was to switch the FSSO Collector Agent Directory Access Mode to Advanced Mode.

Once the change was made I refreshed the FSSO groups on the Fortigate via the “execute fsso refresh” command and all was well again.

For those of you keeping notes, the Fortigate was running FortiOS v5 Patch 6 and the FSSO agent was v4.0 MR3 B0151

Thursday, January 23, 2014

“Unable to launch RDP native” using Fortigate SSL VPN Portal

Recently we started receiving calls from our Fortigate customers that they cannot launch RDP sessions from within their web portal.  After a bit of troubleshooting I discovered that this was caused by a recent Java update (Java 7 Update 51).  More information can be found here, here and here.

The exact error looks like this:



The workaround in this case is very simple, just add your Fortigate Portal address to the Exception Site List in your Java configuration, like so:


More information about the exception list can be found here.