iostat service time (svctm) rule of thumb

iostat service time is a very useful metric when analyzing disk performance and finding bottlenecks

service time is essentially the inverse of IOPs

so if an operation takes 1ms to service, then your IOPs are 1000 (you can complete 1000 operations in a second if that operation took you 1 ms to complete)

the formula for this is as follows, just put this into google and it will do the math for you:

(S)^-1 = ? hz

S is the service time in milliseconds. ignore the hz word, that just to convert the output to IOPs instead of kiloIOPs or etc..

in your calculator its the equivalent of this:


for 1 ms service time we have 1000 IOPs

for 2 ms service time we have 500 IOPs

for 10 ms service time we have 100 IOPs

Generally, for SSDs I like to see service times between 0 and 1ms (it can jump every now and then above 1ms, but if it does that often look into speeding up your SSDs; perhaps you need to disable or enable disk caching)

For HDDs services times between 1 and 10ms are good. Between 10 and 15 is ok. Anything above and your disks are pretty busy.

Ex: running iostat -x 3 on my NAS we see HDD service times between 1 and 10 so we are good and dont have any HDD bottlenecks

If I saw these numbers consistently on an all SSD NAS then I would be worried.

  • svctm is how long a request takes to process outside of the OS.
  • where as await is how long a requests take to process in all (within the OS and outside of the OS).

I am doing a big write operation on my NAS so you see my w_awaits (write) are reading numbers but not my r_awaits (read). if I was reading then my r_awaits would have values. if I was doing both then I would have values in r_await and w_awaits.

notice that awaits will always be bigger then svctm, this is because its where the time measurements are taken, the await will always be bigger as it adds time it took to process within the OS as well.

Note: this was just a look at awaits + svctm. based on my other metrics, my queue size looks entirely too big, so if my NAS shutdown right now, I would have many operations not written. this might result in a corrupted filesystem.


Favorite Watch of 2019+2020

I am not a fan of bulky watches, I like them sleek and gorgeous. The winner goes to my friend’s company:


The Durden watch is a sleek and sexy watch. I have had the privilege to own both types. Personally I like the black background watch the most, but the white one was also very beautiful. In the end, both look amazing and last a long time. Mine lasted a year until I lost both of them, I like to imagine they are still running where ever they might be. The white one has better versatility to scratches – you don’t notice them. The only downside is that you will notice scratches on the glass surface easier on the black watch.

Why is it called Durden? The name comes from the Fight Club movie. It is a reference to the main character “Tyler Durden”. My friend has always enjoyed that movie.

So if you are a Fight Club fan, then this watch is a must have.


Coronavirus Dashboard – covid19.py

Here is my take on a coronavirus dashboard that uses daily updated json data from countries.

Source code (nothing fancy; it uses plotly for the charts): https://github.com/bhbmaster/covid19 .

This dashboard gets its json data from https://pomber.github.io/covid19/timeseries.json

For more info on this data go to the github link: https://pomber.github.io/covid19/

The charts are updated every 6 hours starting at midnight PST. However, the values to the data source are updated daily, so don’t expect new values until after midnight.


Suggested Robocopy Switches

Note: This article is not mine. It was written on http://www.rainingforks.com/blog/2015/suggested-robocopy-switches-explained.html. I am simply excerpting the whole article. This way if that site ever gets shutdown, at least there will be a copy here. I am not taking any credit for the material on this post. This article was written by Steve Schuler.

Suggested Robocopy Switches Explained

Windows’ robocopy.exe is a great command line program to quickly copy or fully backup your files, but there’s a lot of confusion out there about how to use its (not very well-documented) switches.  Here’s just what you need to know:

First of all, you probably already know that typing “robocopy /?” will give you a long list of switches to choose from.  Start there if you’re confused.  But since there are a LOT of choices, and they’re not well-explained, here’s a run-down of what I typically use, as I go about my day as an IT guy:

The basic format is: robocopy <source path> <destination path> <switches>

NOTE: I’ve found that using robocopy to copy across a network doesn’t always work using mapped drives!  Instead, use the full path (especially important when running as a Task in Windows Task Manager). For example, instead of “robocopy C:\Foo S:\Foo” do this: “robocopy C:\Foo \\SERVER\Foo

Another tip is if you’re using file paths that contain spaces, then you need to enclose each path in quotes. If no spaces, then quotes are optional.

/FFT is necessary to copy between file systems, such as Windows’ NTFS and Linux’s EXT4. If you don’t use this you can get wierdness like files looking like they’re newer than they really are, etc., since the two file systems keep time differently.

/COPYALL copies ALL aspects of the file/directory, including ownership and permissions info. Required if you’re backing up a server or something that you want to maintain group/user permissions, etc. for. (NOTE: Don’t use this switch when copying files from Linux to Windows if you aren’t logged in as the same user with admin rights on both machines! If you do, you’ll get a lot of errors like “A required privilege is not held by the client” and “The revision level is unknown” as it creates a bunch of empty folders at your Windows destination, but skips copying all your files! Instead you can use the /COPY:DT mentioned below, and if you really need to backup your Linux ownership & permissions info, save all the files in a tarball and then just backup that single file containing the directories & files with their attributes intact to Windows.)

/COPY:DT to just copy files & date/time stamps. This is good if you’re just copying some files to give to a friend, and don’t need permissions, etc. copied. Also good for copying files from Linux to Windows (see “note” in /COPYALL above).

/FP outputs the full path so you can more easily see where it is while it’s running.

/MIR exactly mirrors the files & directories, so things at the destination will be deleted if they’re not at the source. (This is the same as using /PURGE (which deletes stuff at the destination that doesn’t match) with /E (which includes Empty subdirectories)

/ZB tells robocopy to use restartable mode (which you want for large files, especially over WAN/unstable connections, since it’ll try to pick up where it left off if the connection gets dropped or there’s corruption mid-copy), and if access is denied, then it’ll use Backup mode, which allows you to copy files you might otherwise not have access to, assuming it’s being run under an account with sufficient privileges (e.g., member of Backup Operators, Administrators, etc.). (NOTE: the /Z switch sometimes slows down the copy speed, so if you don’t need it, don’t use it, especially if you feel like your Robocopy job is taking longer than it should. Sometimes there’s no speed difference, and sometimes it can be dramatic.)

/MT stands for Multi-Threaded, and tells robocopy to copy multiple files at once. The default number of threads is 8 (max is 128), but be careful, as running this over a network can really saturate your bandwidth, leaving none for anyone else. As a result, you may want to skip this one or try specifying less threads by doing something like /MT:2 which will just run two threads (instead of 1, which is what you get if you omit /MT entirely). (NOTE: This is only available in newer versions of Robocopy (Win7/2008R2 and later).  If you’re running older versions (or just don’t feel like bothering with this switch), you can simply open multiple command prompt windows and run it in multiple instances – I often will run two or three Robocopy batch files simultaneously. Also, this switch will make file copy progress numbers confusing, so it’s best to use the /NP switch mentioned below to disable outputting the copy’s progress. Some people speculate that running multiple threads can increase fragmentation, but I haven’t seen any hard evidence of this, and with increasing adoption of solid state drives, it may not matter for much longer anyway.)

/R:1 /W:3 are two switches you probably want to use together to tell robocopy how many times to retry accessing a file (1 in this example), and how long to wait between retries (3 seconds in this example).  If you leave this out, it’ll retry 1 million times with a 30 second wait between each one when it encounters a file it can’t access!!!

/XD is what you use when you want to tell robocopy to skip (i.e., exclude) a directory. Just follow /XD with a space and then the path to what you want excluded. If there are multiple directories you’d like to skip, separate them with a space. For example: /XD “C:\Foo\private stuff” “C:\Foo\plans for world domination”

/LOG:C:\LogFileName.txt /TEE /NP are three switches you’ll want to use together if you want to write the results of the copy to a log file (called “C:\LogFIleName.txt” in this example). If you want it to write what’s happening to the screen as well as to the log file, then you’ll also want to include /TEE.  And, possibly most importantly, you want to include /NP in there so that it does NOT show the progress as each file copies. If you leave this out, then your log file will be filled with every single percentage complete it displays! So you’ll have something like this: “0.0% 0.1% 0.2%” and so on, to 100% FOR EACH FILE, which is nuts.

Finally, if you want to append log file output to the end of an existing file, rather than creating a new file every time, you can modify the above line to include a plus sign, like so: /LOG+:C:\LogFileName.txt

That’s pretty much all you need to know! I figured most of this out from trial & error, as well as some internet searches. Unfortunately, I’ve read so many incorrect, or confused posts about robocopy on the internet over the years, that I wanted to write this to set things straight.  Hopefully you find this helpful – if nothing else, at least I’ll now have a handy reference for the next time I need to put together a robocopy command… 😉


How to RDP over an SSH tunnel

You can use this trick to access your home PC from a remote PC (like your work PC). Instead of using Teamviewer or other similar software. You can setup up your own encrypted and secure tunnel to work thru. The requirements are basically to have an SSH accessible server in the same network as the PC you want to access.

Server Side Requirements:

  1. Windows PC that you want to connect to (example local IP:
  2. Linux server with SSH in the same network as the Windows PC (example local IP: over port 22)
    1. This can be a virtual machine running off the Windows PC (just as long as it is accessible from the router; so make sure you use a Bridged Network Adapter)
  3. Internet access to the Linux server SSH (example WAN IP:
  4. This can be achieved by setting up a port forward on your router to send traffic destined to port 22 (or any port) from your Router to the Linux server’s port 22.
    1. Example1: route traffic hitting on TCP port 22 to internal port TCP 22 on (example: we port forwarded port TCP 22 from the router to 22 on the linux server)
    2. Example2: route traffic hitting on TCP port 12345 to internal port TCP 22 on
  5. Enable RDP on your Windows PC:
    1. Control Panel -> System and Security -> System -> Change Settings -> Remote -> allow RDP connections && uncheck the box “Allow Connections only from computers running Remote Desktop with Network Level Authentication (recommended) -> Select Users and add the Windows User[s] that will be connecting to the RDP

Client requirements

  1. For Windows machines connecting to the rdp tunnel: Make sure your Windows client has Cygwin installed with ssh program (The windows client is the one used to connect)


  • Verify the setup works by SSHing to your Linux server from a remote location.
  • Also if you can try to connect to your RDP from another PC in your home network. Windows+R then type “mstsc /v:

How to connect from a Window PC:

For the sake of the example I will use the IPs highlighted as examples.

Open cygwin and run “./sshrdp_cygwin.sh root 22” then put in your SSH password. Then the RDP window opens and put in your Windows Login credentials

You can then make an alias in your ~/.bashrc script to alway connect to your homepc (assuming you put sshrdp.sh into your /usr/bin directory).

alias homepc=’/usr/bin/sshrdp_cygwin.sh root 22′

From then you can just type homepc on your cygwin and it will launch up

Connect from a Mac

Follow the same steps as Windows but use this script instead. Also since MACs don’t have mstsc. You will need to install and open up RDP software manually each time a tunnel is setup. The sshrdp script will prompt for your SSH server address & give you instructions like

“Open RDP to localhost:10000”

Then you will need to login with your Windows credentials

Here is the MAC version of the same script:

Similarly, you can setup an alias to use on your MAC terminal, except you will need to put it in your ~/.bash_profile instead of your ~/.bash_rc (if I recall correctly, that is how it is done with MACs)

alias homepc=’/usr/bin/sshrdp_mac.sh root 22′

Then you can access your home pc by simply typing homepc.

Connect from a Linux Server

You probably just use the MAC steps – although I am not sure and have not tested it. The line of code with the netstat command might need a change / edit.


Bash convert seconds to human readable

function displaytime {
local T=$1
local D=$((T/60/60/24))
local H=$((T/60/60%24))
local M=$((T/60%60))
local S=$((T%60))
(( $D > 0 )) && printf ‘%d days ‘ $D
(( $H > 0 )) && printf ‘%d hours ‘ $H
(( $M > 0 )) && printf ‘%d minutes ‘ $M
(( $S > 0 )) && { (( $D > 0 || $H > 0 || $M > 0 )) && printf ‘and ‘
printf ‘%d seconds\n’ $S; } || printf ‘\n’

-bash-4.1$ displaytime 604800
7 days
-bash-4.1$ displaytime 604801
7 days and 1 seconds
-bash-4.1$ displaytime 800
13 minutes and 20 seconds
-bash-4.1$ displaytime 780
13 minutes


Wrong Port Connected Check With ifconfig

We need to run ifconfig and look for the status line.

Remember this about ifconfig output:

* if “status: active“, the port is linked up and sending traffic (now you just need to make sure it has an IP for it to be able to communicate on a network). You can see it is sending traffic with tcpdump -i <port>
(hit Control-C after a few seconds; even ports that are linked up w/ a missing IP should still be sending out and receiving some L2 switch traffic: STP, ARP, LLDP)

* if “status: no carrier“, the port is not linked up and therefore not sending traffic . If you run tcpdump on this port, you will see 0 packets.

* if “status:” line is missing, then the port is disabled. Enable it w/ “ifconfig <port> up” to see if it has a link (you can optionally disable the port afterwards if you see it has no carrier using “ifconfig <port> down“)


Grepping Thru All Rotated Logs

If your system gzips and rotates your logs.

You can use this trick to search thru all of them.

# zgrep -i — “SEARCHTERM” $(ls -1Str /var/log/messages*)

Another way

# zgrep -i — . $(ls -1Str /var/log/messages*) | grep “SEARCHTERM”

Sidenote: if you get an error try to replace “zgrep” with “zegrep”


A good free syslog server

Personally, I just tried “Syslog Watcher” version 5.0.4 and it does a good job. I can start it by hitting “Start Server” then configured syslog clients to point at it. You can click on a message, then it will show up in the message view with every detail. Additionally you can hold Control and Mouse Wheel Down or Up to zoom in or out (just like in Notepad++)

Of course everyone is aware of Kiwi Syslog Server. I find Syslog Watcher more interesting, but that is just me.