???????????????????????????? ??????????????/????????????????? ............/............... .........................../... ????????????????????????????? >>>>>>>>>>>>>/>>>>>>>>>>>>>>>>> ÿØÿà JFIF    ÿÛ „  ( %!1!%)+//.383,7(-.+  -%%-////---/-.+/--+------/------/--0+--/-/-----.-----ÿÀ  ¥2" ÿÄ     ÿÄ J    ! 1AQ"aq2‘#BR‚¡ÁÑ3br’¢±Âð$CSƒ²á4c“%DsÓñÿÄ   ÿÄ *  !1AQa‘"2q3±ð#b¡ÿÚ   ? ¼QxJQaÍuò¸Zö Úü8,ÐÚú "SSn<rçù–´âE—^ªBÖ9À\†¸ÔÁT­ÃÛ5 ëd´³Í#Ý;Þ38œî ¶H£M:wÎ3…³…âpÔF&‚FK¸9„â4àGEõªfÿ ‘ñ(ßw­pŽF|È¥ù®häðÍѶ¹‘[ÒinÙW¶ùñY˜Q{›K"išÒ[Ú8žë\F¹@-?v"ÔU”,ìöžkÿ {I‡£šÍ?e ríV ?> ......................................... ............................................................................. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>/>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ???????????????????????????? ??????????????/????????????????? ............/............... .........................../... ????????????????????????????? >>>>>>>>>>>>>/>>>>>>>>>>>>>>>>> ÿØÿà JFIF    ÿÛ „  ( %!1!%)+//.383,7(-.+  -%%-////---/-.+/--+------/------/--0+--/-/-----.-----ÿÀ  ¥2" ÿÄ     ÿÄ J    ! 1AQ"aq2‘#BR‚¡ÁÑ3br’¢±Âð$CSƒ²á4c“%DsÓñÿÄ   ÿÄ *  !1AQa‘"2q3±ð#b¡ÿÚ   ? ¼QxJQaÍuò¸Zö Úü8,ÐÚú "SSn<rçù–´âE—^ªBÖ9À\†¸ÔÁT­ÃÛ5 ëd´³Í#Ý;Þ38œî ¶H£M:wÎ3…³…âpÔF&‚FK¸9„â4àGEõªfÿ ‘ñ(ßw­pŽF|È¥ù®häðÍѶ¹‘[ÒinÙW¶ùñY˜Q{›K"išÒ[Ú8žë\F¹@-?v"ÔU”,ìöžkÿ {I‡£šÍ?e ríV ?> ......................................... ............................................................................. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>/>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Demonstrations of biosnoop, the Linux BPF/bpftrace version. This traces block I/O, and shows the issuing process (at least, the process that was on-CPU at the time of queue insert) and the latency of the I/O: # ./biosnoop.bt Attaching 4 probes... TIME(ms) DISK COMM PID LAT(ms) 611 nvme0n1 bash 4179 10 611 nvme0n1 cksum 4179 0 627 nvme0n1 cksum 4179 15 641 nvme0n1 cksum 4179 13 644 nvme0n1 cksum 4179 3 658 nvme0n1 cksum 4179 13 673 nvme0n1 cksum 4179 14 686 nvme0n1 cksum 4179 13 701 nvme0n1 cksum 4179 14 710 nvme0n1 cksum 4179 8 717 nvme0n1 cksum 4179 6 728 nvme0n1 cksum 4179 10 735 nvme0n1 cksum 4179 6 751 nvme0n1 cksum 4179 10 758 nvme0n1 cksum 4179 17 783 nvme0n1 cksum 4179 12 796 nvme0n1 cksum 4179 25 802 nvme0n1 cksum 4179 32 [...] This output shows the cksum process was issuing block I/O, which were completing with around 12 milliseconds of latency. Each block I/O event is printed out, with a completion time as the first column, measured from program start. An example of some background flushing: # ./biosnoop.bt Attaching 4 probes... TIME(ms) DISK COMM PID LAT(ms) 2966 nvme0n1 jbd2/nvme0n1-8 615 0 2967 nvme0n1 jbd2/nvme0n1-8 615 0 [...] There is another version of this tool in bcc: https://github.com/iovisor/bcc The bcc version provides more fields.