Troubleshoot network problem using tshark

How tshark works ?

When a packet arrives at the network card, the MAC destination address is checked to see if it matches yours, in which case an interrupt service routine will be generated and handled by the network driver. 

Subsequently, the received data is copied to a memory block defined in the kernel and from there it will be processed by the corresponding protocol stack to be delivered to the appropriate application in user space. Parallel to this process, when Tshark is capturing traffic, the network driver sends a copy of the packets to a kernel subsystem called Packet Filter, which will filter and store in a buffer the desired packets. These packets will be received by Dumpcap (in user space) whose main goal will be to write them into a libpcap file format to be subsequently read by Tshark. As new packets arrive, Dumpcap will add them to the same capture file and it will notify Tshark about their arrival so that they can be processed.

My objective would be to give you brief tutorial on how to find problems related to performance of network, could be due to bandwidth etc.. so we could use tshark to try and find out which hosts are generating more traffic and what type of data are they sending..

List all the network interfaces - tshark -D

Capture traffic from network interface and write to file -

#tshark -i <interface> -w traffic.pacap

How to capture and analyze traffic using tshark ? 

1. Determine which IPs in your VLAN(IPADDRES/NETMASK) could be misusing the network would be able to get IP list. list by dfault would be sorted according to total number of frames, so it could give an idea of heavy talkers.

#tshark -r traffic.pcap -q -z "conv,ip,ip.addr==74.125.130.0/24"

================================================================================

IPv4 Conversations

Filter:ip.addr==74.125.130.0/24

                                               |       <-      | |       ->      | |     Total     |   Rel. Start   |   Duration   |

                                               | Frames  Bytes | | Frames  Bytes | | Frames  Bytes |                |              |

74.125.130.102       <-> 10.0.2.15                105     27191     129     21393     234     48584   112.306444555       260.0255

74.125.130.95        <-> 10.0.2.15                 34      3395      36     11639      70     15034   263.618378290       108.6899

74.125.130.93        <-> 10.0.2.15                 32      3601      37     11601      69     15202   109.882120656       177.0934

================================================================================

2. With above inforamtion we know that IP 74.125.130.102 represents one of the host which is generating more traffic to communicate with other machines on the network 74.125.130.0/24. 

You could create another pcap file just with the traffic generated by that machine(74.125.130.102)

#tshark -r traffic.pcap -R "ip.addr==74.125.130.102" -w ip.pcap

# capinfos ip.pcap | grep "Number\|time:"

Number of packets:   234

Start time:          Fri Jul 29 20:37:12 2016

End time:            Fri Jul 29 20:41:32 2016

3. Check that your host is not breaking any of your policies of your network, only HTTP & HTTPS is allowed. Below commands will tells us outbound connections to ports other than any (HTTP or HTTPS)

#tshark -o column.format:'" Source ","%s","Destination","%d", "dstport", "%uD","Protocol", "%p"' -r ip.pcap -R "ip.src == 74.125.130.102 && ! dns && tcp.dstport != 80 && tcp.dstport != 443"  | sort -u

74.125.130.102 -> 10.0.2.15    43536 TCP

74.125.130.102 -> 10.0.2.15    43536 TLSv1.2

74.125.130.102 -> 10.0.2.15    43540 TCP

74.125.130.102 -> 10.0.2.15    43540 TLSv1.2

4. I don't have any traffic violating my policies, anyway lets suppose we say if that do exists, then we would have those machines IP address and the port on which they are connected. so to make sure that the traffic is not from other service using the FTP port, lauch tcp stream of that session.

#tshark -o column.format:'"Source","%s","srcport", "%uS","Destination","%d", "dstport", "%uD","Protocol", "%p"' -r ip.pcap -R "tcp.dstport == 43536" | head -1

74.125.130.102 443 10.0.2.15    43536 TCP

#tshark -r ip.pcap -q -z  "follow,tcp,ascii,74.125.130.102:443,10.0.2.15:43536,1"

===================================================================

Follow: tcp,ascii

Filter: ((ip.src eq 74.125.130.102 and tcp.srcport eq 443) and (ip.dst eq 10.0.2.15 and tcp.dstport eq 43536)) or ((ip.src eq 10.0.2.15 and tcp.srcport eq 43536) and (ip.dst eq 74.125.130.102 and tcp.dstport eq 443))

Node 0: 10.0.2.15:43536

Node 1: 74.125.130.102:443

195

............sZ..@G"......!s.....?W...$..5......+./.....

.......3.9./.5.

s.youtube.com..........

...............................h2.spdy/3.1.http/1.1..........

===================================================================

5. Now you could observe that it was "youtube.com" was actullay consuming more bandwidth responsible for slowdown in network.

If you do come across any FTP sessions, troubleshoot the above way, also additionally you will check all the files downloaded by the client.

#tshark -r ip.pcap -q -z  "follow,tcp,ascii,74.125.130.102:443,<Destination machine>:21,1" | grep RETR

6. tshark also allows us to break down each of the protocols captured. Thus we can see hierarchically the number of frames and bytes associated with each protocol. Using capture file, let's see for example the distribution of HTTP and HTTPS traffic used by the IP 74.125.130.102:

#tshark -r traffic.pcap -q -z io,phs,"ip.addr==74.125.130.102 && ssl || http"

===================================================================

Protocol Hierarchy Statistics

Filter: ip.addr==74.125.130.102 && ssl || http

eth                                      frames:122 bytes:40644

  ip                                     frames:122 bytes:40644

    tcp                                  frames:122 bytes:40644

      ssl                                frames:122 bytes:40644

        tcp.segments                     frames:2 bytes:2589

          ssl                            frames:2 bytes:2589

===================================================================

7. It would practically tells us that SSL represents all traffic, let's see the IP's associated with that communication.

#tshark -o column.format:'"destination","%d"' -r  traffic.pcap -R "ip.src ==74.125.130.102 && ssl"| sort -u

10.0.2.15

#whois 10.0.2.15 | grep -i "netname\|netrange"

NetRange:       10.0.0.0 - 10.255.255.255

NetName:        PRIVATE-ADDRESS-ABLK-RFC1918-IANA-RESERVED

With whatever application or information your would get for the IP address/ports, you can create ACLs or IPtables rules to deny certain types of traffic, do a shutdown of a specific port, limit the bandwidth of some protocols so on ...

More references : 

https://www.wireshark.org/docs/wsug_html_chunked/AppToolstshark.html

https://www.youtube.com/watch?v=F2HAPAUqitc

Thanks for re-sharing !

NFS common errors and troubleshooting - Linux/Unix

I have seen some of the most common NFS Error/Issues which occurs in very common now and then to most of Linux/Unix based system admins. So I decided to put at one palace. Hope this helps most of them.

Environment: Linux/Unix

Error: "Server Not Responding"

Check your NFS server and the client using RPC message and they must be functional/online. 

use ping, traceroute to check are they reaching each other, if not check your NIC using ethtool to verify IP address.

sometimes due to heavy server or network loads causes the RPC message response to time out causing error message. try to increase timeout option.

Error: "rpc mount export: RPC: Timed out " 

NFS server or client was unable to resolve DNS. check forward/reverse DNS name resolution works. 

Check your DNS servers or /etc/hosts

 Error: "Access Denied" or "Permission Denied"

check export permission for the NFS file systems.

#showmount -e nfsserver  ==> client 

#exportfs -a ==> server

check you dont have any syntax issues in file /etc/exports(e.g  space, permissions, typos..etc) 

Error: "RPC: Port mapper failure - RPC: Unable to receive"

NFS requires both NFS service and portmapper service running on both client and the server

#rpcinfo -p

       or

#/etc/init.d/portmap status

if not, start the portmap service

Error: "NFS Stale File Handle"

system call 'open' calls to access NFS file in the same way application uses local file they by returns a file descriptor or handle which programs useses I/O commands to identify the file manipulations

When an NFS file share is either unshared or NFS server changes the file handler, and any NFS client which attempts to do further I/O on the share will receive the 'NFS Stale File Handler'.

on the client :

umount -f /nfsmount or if it is unable to inmount and remount 

kill the processes which uses that /nfsmount

or 

incase if above options didn't work, you can reboot the client to clear the stale NFS.

Error: "No route to host"

this could be reported when client attempts to mount the NFS file system, even when the client can ping them successfully.

This can be due to RPC messages being filtered by either host firewall, client firewall or network switch. verify firewall rules. 

stop suing iptables and try to check the port 2049 

Hope this helps all who might use NFS most of the times. I have figured out these commonly in my experience.

Thanks for sharing !

zfs cheat sheet - Creation of Storagepools & Filesystems using zpool & zfs #Solaris 11

The ZFS file system is a file system that fundamentally changes the way file systems are administered, with features and benefits not found in other file systems available today. ZFS is robust, scalable, and easy to administer.ZFS uses the concept of storage pools to manage physical storage, ZFS eliminates volume management altogether. Instead of forcing you to create virtualized volumes, ZFS aggregates devices into a storage pool.File systems are no longer constrained to individual devices, allowing them to share disk space with all file systems in the pool. You no longer need to predetermine the size of a file system, as file systems grow automatically within the disk space allocated to the storage pool. When new storage is added, all file systems within the pool can immediately use the additional disk space without additional work

zpool commands	Description
zpool create testpool c0t0d0	Create simple pool named testpool with single disk creating default mount point as poolname(/testpool) OPTIONAL: -n do a dry run on pool creation -f force creation of the pool
zpool create testpool mirror c0t0d0 c0t0d1	Create testpool mirroring c0t0d0 with c0t0d1 creating default mount point as poolname(/testpool)
zpool create -m /mypool testpool c0t0d0	Create pool with different mount point than default
zpool create testpool raidz c2t1d0 c2t2d0 c2t3d0	Create RAID-Z testpool
zpool add testpool raidz c2t4d0 c2t5d0 c2t6d0	Add RAID-Z disks to testpool
zpool create testpool raidz1 c2t1d0 c2t2d0 c2t3d0 c2t4d0 c2t5d0 c2t6d0	Create RAIDZ-1 testpool
zpool create testpool raidz2 c2t1d0 c2t2d0 c2t3d0 c2t4d0 c2t5d0 c2t6d0	Create RAIDZ-2 testpool
zpool add testpool spare c2t6d0	Add spare device to the testpool
zpool create testpool mirror c2t1d0 c2t2d0 mirror c2t3d0 c2t4d0	Disk c2t1d0 mirrored with c2t2d0 & c2t3d0 mirrored with c2t4d0
zpool remove testpool c2t6d0	Removes hot spares and cache disks
zpool detach testpool c2t4d0	Detach the mirror from the pool
zpool clear testpool c2t4d0	Clears specific disk fault
zpool replace testpool c3t4d0	Replace disk like disk
zpool replace testpool c3t4d0 c3t5d0	Replace one disk with another disk
zpool export testpool	Export the pool from the system
zpool import testpool	Imports specific pool
zpool import -f -D -d /testpool testpool	Import destroyed testpool
zpool import testpool newtestpool	Import a pool originally named testpool under new name newtestpool
zpool import 88746667466648	Import pool using ID
zpool offline testpool c2t4d0	Offline the disk in the pool Note: zpool offline testpool -t c2t4d0 will offline temporary
zpool upgrade -a	upgrade all pools
zpool upgrade testpool	Upgrade specific pool
zpool status -x	Health status of all pools
zpool status testpool	Status of pool in verbose mode
zpool get all testpool	Lists all the properties of the storage pool
zpool set autoexpand=on testpool	Set the parameter value on the storage pool Note: zpool get all testpool gives you all the properties on which it could be used to set value
zpool list	Lists all pools
zpool list -o name,size,altroot	show properties of the pool
zpool history	Displays history of the pool Note: Once the pool is removed, history is removed.
zpool iostat 2 2	Display ZFS I/O stastics
zpool destroy testpool	Removes the storage pool

zfs commands	Description
zfs list	Lists the ZFS file system's zfs list -t filesystem zfs list -t snapshot zfs list -t volume
zfs create testpool/filesystem1	Creates ZFS filesystem on testpool storage
zfs create -o mountpoint=/filesystem1 testpool/filesystem1	Different mountpoint created after ZFS creation
zfs rename testpool/filesystem1 testpool/filesystem2	Renames the ZFS filesystem
zfs unmount testpool	unmount the storagepool
zfs mount testpool	mounts the storagepool
NFS exports in ZFS	zfs share testpool - shares the file system for export zfs set share.nfs=on testpool - make the share persistent across reboot svcs -a nfs/server - NFS server should be online cat /etc/dfs/dfstab - Exported entry in the file showmount -e - storage pool has been exported
zfs unshare testpool	Remove NFS exports
zfs destroy -r testpool	Destroy storage pool and all datasets under it
zfs set quota=1G testpool/filesystem1	set quota of 1GB on the filesystem1
zfs set reservations=1G testpool/filesystem1	set reservation of 1GB on the filesystem1
zfs set mountpoint=legacy testpool/filesystem1	Disable ZFS auto mounting and enable through /etc/vsftab
zfs unmount testpool/filesystem1	unmounts ZFS filesystem1 in testpool
zfs mount testpool/filesystem1	mounts ZFS filestystem1 in testpool
zfs mount -a	mounts all the ZFS filesystems
zfs snapshot testpool/filesystem1@friday	Creates a snapshot of the filesystem1
zfs hold keep testpool/filesystem1@friday	Holds existing snapshot & attempts to destroy using zfs destroy will fail
zfs rename testpool/filesystem1@friday FRIDAY	Rename the snapshots Note: snapshots must exists in the same pools
zfs diff testpool/filesystem1@friday testpool/filesystem1@friday1	Identify the difference between two snapshots
zfs holds testpool/filesystem1@friday	Displays the list of snapshots help
zfs rollback -r testpool/fileystem1@friday	Roll back yesterday snapshot recursively
zfs destroy testpool/fileystem1@thursday	Destroy snapshot created yesterday
zfs clone testpool/filesystem1@friday testpool/clones/friday	Clone was created on the same snapshot Note: Cannot create clone of a filesystem in a pool that is different from where original snapshot resides.
zfs destroy testpool/clones/Friday	Destroy the clone

Thanks,

Create Local Repository - Solaris 11.2

Image packaging system(IPS) and important concept from Solaris 11 onwards. I would like to create local repository on the local system by downloading the files from the Oracle website using oracle's automated script.

Download below files and copy to your local Solaris server.

Below are the files copied on my local Solaris server. 

root@solnode1:/var/share/pkg# pwd

/var/share/pkg

root@solnode1:/var/share/pkg# ls -l

total 14373953

-rwx------   1 root     root        5594 Aug 29 20:52 install-repo.ksh

drwxr-xr-x   3 pkg5srv  pkg5srv        7 Aug 30 08:42 repositories

-rw-r--r--   1 root     root     1771800121 Aug 29 15:32 sol-11_2-repo-1of4.zip

-rw-r--r--   1 root     root     1889867782 Aug 29 15:35 sol-11_2-repo-2of4.zip

-rw-r--r--   1 root     root     1902167161 Aug 29 16:46 sol-11_2-repo-3of4.zip

-rw-r--r--   1 root     root     1790358735 Aug 29 16:44 sol-11_2-repo-4of4.zip

-rw-r--r--   1 root     root         227 Aug 29 16:14 sol-11_2-repo-md5sums.txt

root@solnode1:/var/share/pkg#

root@solnode1:/var/share/pkg#./install-repo.ksh -d /var/share/pkg/repositories/ -v -c

The script would compare the checksums of the downloaded files, uncompress and would initiate the repository creation. 

your current publisher would be pointing to "pkg.oracle.com" and it needs to be changed to your local repository. 

root@solnode1:~# pkg publisher

PUBLISHER                   TYPE     STATUS P LOCATION

solaris                     origin   online F http://pkg.oracle.com/solaris/release/

root@solnode1:~#

root@solnode1:~# pkg set-publisher -G '*' -M '*' -g file:///var/share/pkg/repositories solaris

root@solnode1:~#

root@solnode1:~# pkg publisher

PUBLISHER                   TYPE     STATUS P LOCATION

solaris                     origin   online F file:///var/share/pkg/repositories/

root@solnode1:~#

To enable clients to access the local repository via HTTP, enable the application/pkg/server Service Management Facility (SMF) service.

root@solnode1:~# svccfg -s application/pkg/server setprop pkg/inst_root=/var/share/pkg/repositories

root@solnode1:~# 

check does repos work

root@solnode1:~# svcprop -p pkg/inst_root application/pkg/server

/var/share/pkg/repositories

root@solnode1:~#

Reload the pkg.depotd repository service.

root@solnode1:~# svcadm refresh application/pkg/server

root@solnode1:~#

we had successfully created solaris 11.2 local repository.

Configure log rotation - Solaris 10

In order to have an easy administration of systems which generates large number of log files, you can configure your log files according by an utility called logroate, which allows automatic rotation, compression, removal and also mailing of log files which can be handled daily, weekly or when it grows too large.

Objective: compress and rotate logs after certain threshold on the file size.

Environment: Solaris 10 32-bit

The system log rotation is defined in the /etc/logadm.conf file. This file includes log rotation entries for processes such as syslogd. For example, one entry in the /etc/logadm.conf file specifies that the /var/log/ciscofirewall.log file is rotated weekly unless the file is empty. The most recent ciscofirewall.log file becomes ciscofirewall.log.0, the next most recent becomes ciscofirewall.log.1, and so on. Eight previous ciscofirewall log files are kept.

The /etc/logadm.conf file also contains time stamps of when the last log rotation occurred.

# vi /etc/logadm.conf 

/var/log/ciscofirewall.log -C 9 -s 10240k -z 4 -N -a 'kill -HUP `cat /var/run/syslog.pid`'

#

where, 

     -C = expire old logs until count remain.( 9 log files created and rotated )

     -N = not an error if log file nonexistent.

     -s = only rotate if given size or greater.

     -a = execute command after taking actions.

     -z = gzip old logs except most recent count ( last 5 log files would be compressed )

- Restart the syslogd to take changes effectively.

The command is often run on a cron job, which has the effect of fully automatic log rotation.

# crontab -l 

10 3 * * * /usr/sbin/logadm

# ls -l /var/log/ciscofirewall*.log.* | wc -l

       9

#

# ls -l /var/log/ciscofirewall*.log.*.gz | wc -l

       5

# 

# ls -ltr /var/log/ciscofirewall.log*

-rw-r--r--   1 root     root       41048 Apr  6 14:38 /var/log/ciscofirewall.log.8.gz

-rw-r--r--   1 root     root       42076 Apr  6 14:39 /var/log/ciscofirewall.log.7.gz

-rw-r--r--   1 root     root       41621 Apr  6 14:40 /var/log/ciscofirewall.log.6.gz

-rw-r--r--   1 root     root       41524 Apr  6 14:41 /var/log/ciscofirewall.log.5.gz

-rw-r--r--   1 root     root       41410 Apr  6 14:42 /var/log/ciscofirewall.log.4.gz

-rw-r--r--   1 root     root     21510944 Apr  6 14:43 /var/log/ciscofirewall.log.3

-rw-r--r--   1 root     root     21139079 Apr  6 14:44 /var/log/ciscofirewall.log.2

-rw-r--r--   1 root     root     21536814 Apr  6 14:45 /var/log/ciscofirewall.log.1

-rw-r--r--   1 root     root     21399755 Apr  6 14:46 /var/log/ciscofirewall.log.0

-rw-r--r--   1 root     root     16434041 Apr  6 14:46 /var/log/ciscofirewall.log

# 

all your logs has been rotated in a discipline manner, which would be easy to troubleshoot in-case of any errors.

OpenSSH security/hardening tips #Linux/Unix

Objective: Strengthen security on SSH

Environment: CentOS-6.3 (32-bit)

OpenSSH version : OpenSSH_5.3p1

SSH Protocol version : 2

I had observed more lines which were commented in sshd_config, hence I had enabled few of the options which could possibly strengthen ssh.

Any changed made to the /etc/ssh/sshd_config would require its corresponding daemon(sshd) to be restarted. Hence I would assume the reader would probably be aware and continuing explaining below options.

1. Change the port number on which sshd listens or specify the local address along with the port number on which sshd listens.

Port XXXX

or

ListenAddress 0.0.0.0:XXXX

2. The server disconnects if the user has not successfully logged within 120 seconds.

LoginGraceTime 2m

You may receive below snap if you would continue to be without logging in.

3. Once after logging into the server, and if there is no data communication for around 5 Minutes then we could force auto-logout option.

# grep -i tmout /etc/bashrc 

export TMOUT=300

 # 

You may receive the below error incase if the session is idle for 5 Minutes.

# timed out waiting for input: auto-logout

Connection to XXX.XXX.XXX.XXX closed.

4. Use the authentication methods(RSA, DSA) to grant access to the users, i..e password less logis to the users. How to create RSA/DSA to authenticate

PermitRootLogin without-password

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile      .ssh/authorized_keys

AuthorizedKeysCommand yes

# To disable tunneled clear text passwords, change to no here!

PermitEmptyPasswords no

PasswordAuthentication no

5. sshd should check file modes and ownership of the user's file and the home directories before accepting login.

StrictModes yes

6. Password attempts to be prompted would be ideally two, incase of any failures would be disconnected.

MaxAuthTries 3

7. Restricted the maximum concurrent un-authenticated connections to the SSH daemon.

MaxStartups 2

8. You can use allow/deny directives for users/groups to allow/disallow based on the primary and the supplementary groups lists matches one of the pattern.

Allowgroups sshgroup

9. Finally, you can configure chrootDirectory which specifies the pathname of a directory to jail the user after authentication. All the components of the pathname must be root owned.

If this interests you on how to jailusers Click here 

I, hereby conclude that most of the options related in securing communication between server & client by OpenSSH had been discussed.

Build Solaris 11 repository without network connection

Few days back had installed Oracle Solaris 11, and wanted to upgrade it to 11.1.

I came across IPS in Solaris 11 and I had to configure on the system. This system is not networked.

I hereby wanted to share as what is IPS and how to configure on the Solaris.

• The Image Packaging System (IPS) is a new network-eccentric software packaging and delivery system in Oracle Solaris 11. IPS allows efficient, observable, and controllable transitions between known configurations of software content providing administrators with safe system upgrade, environments and better control over planned system downtime schedules.

• The ZFS file system is integral to IPS, providing administrators the ability to perform updates on a file system clones on live production systems.

If you wish to download the ISO image, burn and insert into your CD/DVD and execute below as root.

#pkg set-publisher -G '*' -g file:///cdrom/sol11repo_full/repo solaris

If you like something more permanent while installing packages then you could configure as below

- After installing Solaris 11, download (on another system perhaps) the two files that make up the Solaris 11 repository from Oracle Solaris 11 Downloads

- Copy your files into the system

- Concatenate two files.

     # cat sol-11_1-repo-full.iso-a sol-11_1-repo-full.iso-b > sol-11_1-repo-full.iso 

- Mount ISO file to a location

    #mount -F hsfs sol-11-11-repo-full.iso /mnt

- Set the publisher to point to the /mnt/repo location, copy the repository from the mounted ISO image to a permanent, on disk location.

    #zfs create -o atime=off -o compression=on rpool/export/repoSolaris11

    #rsync -aP /mnt/repo /export/repoSolaris11

    #pkgrepo -s /export/repoSolaris11/repo refresh

    #pkg set-publisher -G '*' -g /export/repoSolaris11/repo solaris

- Check your publisher pointing to.

# pkg publisher

PUBLISHER                             TYPE     STATUS   URI

solaris                               origin   online   file:///export/repoSolaris11/repo/

#

- Now you can upgrade your system online without any internet connections.

IPS commands can be found here - click here