Thursday, December 18, 2014

sys diag kill command does not kill processes on a Fortigate

I recently had a Fortigate 1500D become bogged down due to the reporting daemon (reportd) utilizing 100% CPU.  I will not go into a lot of detail about diagnosing performance issues, as that is not the topic of this post.  Suffice to say that you’ll need to run the following commands to determine which process is misbehaving:

#get sys perf status
#diag sys top
#diag hard sys mem

Typically one would kill and respawn the offending process with the following command, where process_id is obtained via the diag sys top command.

#diag sys kill 11 process_id

Unfortunately in this case the kill command did not actually kill the process, and a reboot was not an option.  Fortunately I once had a remote session with Fortinet TAC where I saw them using some hitherto unknown (to me) commands.  So what follows is an unsupported way to absolutely kill processes dead.

The command we use is fnsysctl.  This command allows us access to a subset of Linux utilities, like cat, ls, kill etc.  In this case we’re interested in the kill command:

#fnsysctl kill –9 process_id

I have never had this command fail to kill a process, although I would recommend only using it as a last resort.

Happy Hunting!