Linux commands to monitor a FCSDK system

Linux commands that can be used by sysadmins to monitor a FCSDK deployment on their network.

1. Confirm the Web Gateway is running on a given host:

netstat -anp | grep 5060 (to confirm the WG process is up & has bound to the port)

e.g. success return value:

tcp        0      0 192.168.9.34:5060           0.0.0.0:*                   LISTEN      24624/java
udp        0      0 192.168.9.34:5060           0.0.0.0:*                               24624/java

2. Confirm the Media Broker is running on a given host:

netstat -anp | grep 16000 (to confirm the MB process is up & has bound to the port.)

e.g. success return value:

udp        0      0 ::ffff:192.168.9.34:16000   :::*                                    11134/java

3. There should be 6 java processes for WG & MB running on same box.

ps -ef | grep java (to confirm desired number of java processes are running)

e.g. success return value:

[root@cs-fcsdk21 ~]# ps -fC java
UID        PID  PPID  C STIME TTY          TIME CMD
root     11134 11123  3 Oct24 ?        15:50:49 /opt/java/bin/java -DTRACE=true -Djava.awt.headless=true -Xmx512M -Xms512M -cp . -jar rtp-proxy.jar
root     24537 24488  0 Oct28 ?        00:13:48 /opt/java/bin/java -D[Process Controller] -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi
root     24553 24537  0 Oct28 ?        00:53:29 /opt/java/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/opt/cafex/FAS-2.1.17/domain/log/host-controller.log -Dcom.aliceca
root     24618 24537  1 Oct28 ?        04:03:09 /opt/java/bin/java -D[Server:appserver-cs-fcsdk21] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms256m -Xmx512m -server -XX:+UseG1GC -X
root     24624 24537  0 Oct28 ?        01:59:23 /opt/java/bin/java -D[Server:loadbalancer-cs-fcsdk21] -XX:PermSize=128m -XX:MaxPermSize=128m -Xms256m -Xmx512m -server -XX:+UseG1GC
root     24679 24537  0 Oct28 ?        00:46:50 /opt/java/bin/java -D[Server:management] -Xms64m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3

4. Monitoring logs for Exceptions **OPTIONAL - Some exceptions will be expected**

Only run this script if you think you have a problem that requires active monitoring.

Your server will need to be able to send outbound email.

Create a bash script to check for exceptions such as:

cat /root/check-for-exceptions.sh:
#!/bin/bash
# Script to check for exceptions and send email to user
grep "Exception" /opt/cafex/FAS-/domain/servers/appserver-cs-/log/server.log | mail -s "Exceptions in server.log" sparfitt@cafex.com

Then add to crontab:

*/30 * * * * /root/check-for-exceptions.sh

5. Checking CPU & Memory usage

Use the TOP command to monitor CPU & memory usage, if a CPU is at 100% it can lead to unexpected behaviour. Ensure the server has plenty of free CPU and memory. E.g

root@cs-fcsdk21 ~]# top 
top - 10:34:10 up 55 days, 23:13, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.0%id, 0.3%wa, 0.0%hi, 0.0%si, 0.4%st#
Mem: 3922700k total, 3805288k used, 117412k free, 167968k buffers
Swap: 8184k total, 8184k used, 0k free, 1334000k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20884 root 20 0 3067m 302m 14m S 10.0 7.9 1637:21 java
30045 root 20 0 3599m 651m 14m S 2.0 17.0 383:53.64 java
28920 root 20 0 3218m 405m 10m S 0.3 10.6 106:11.47 java
28977 root 20 0 2884m 396m 10m S 0.3 10.3 44:50.85 java
1 root 20 0 19360 1220 984 S 0.0 0.0 0:01.06 init<
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:13.13 migration
4 root 20 0 0 0 0 S 0.0 0.0 6:10.56 ksoftirqd

The above command shows us the CPU is at 0.7% so there is plenty available for FCSDK. It shows the server has 4GB of memory available of which 1GB is free, so again plenty of resources available to the FCSDK