![]() ![]() GAVG–this is the full time (ms) the guest sees from the I/O it sends.This is usually zero, if it is not, you are overwhelming the device queue and then queuing inside of ESXi. ![]() KAVG–this is the time (ms) the I/O spends in the ESXi kernel.So this includes SAN transit and storage array processing. DAVG–this is the time (ms) seen from the ESXi host once it sends the I/O out of the HBA until it is acknowledged back.What about latency? There are a few other important numbers here that are important to understand: The throughput and IOPS reported here (MBREAD/s and CMD/s respectively) are the same as the FlashArray. This shows my physical device statistics (you can configure what it shows). I have one instance of esxtop running in this screenshot. More on what we report as our latency in a bit.Īlso, let’s assume memory and CPU is not in contention. The FlashArray doesn’t know about how long an original request takes inside of ESXi, like queueing in the guest or kernel, so if you see good latency on the FlashArray and bad latency in the VM, there must be a bottleneck in the ESXi host. Whoa! The FlashArray is reporting sub-millisecond latency! It MUST be lying! Not so, actually. But the workload is actually configured to push 130,000 IOPS, but for some reason it cannot. So, well, it must be the storage! Let’s look at the FlashArray: Also from the VDBench printout, the active queue depth in the VM for that virtual disk is 96, which makes sense as my VDBench workload is configured to use 96 threads. One is, sure enough the latency is relatively high: 1.4 ms (which is high for an AFA), especially just 4 KB reads. First let’s look at the virtual machine portion of the screen shot. The workload is configured to run 130,000 4K Read IOPS with 96 outstanding I/Os (threads). Workload virtual disk is on a different VMFS than the boot virtual disk of the VM.Paravirtual SCSI Adapter, default settings.Is your array lying? Why is my VM latency high? What can I do (if anything) to fix it?įor those curious, my virtual machine is configured as such: Furthermore, the latency on the FlashArray is reported as low. It was much better on my equivalent physical server. ![]() Customer and the latency in my virtual machine is high and my IOPS is not as high as I want it to be. There is A LOT of information here and it pretty much tells you everything you need to know to solve the problem. So nothing is interfering, so while this is not realistic, I think it is still valuable to explain how these things work. One thing to note here, this is one virtual machine running workload to a virtual disk on one VMFS. If you prefer a video, here is my 1 hr VMworld session that goes into depth on what I write below: And frankly not without direct guidance from VMware support. PLEASE do not make changes in your environment at least until you read my conclusion at the end.This workload is targeted specifically to make relationships easier to understand. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |