You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-30 - 15:42
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sat Aug 30, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
895021236,16sleep10-21swapper/106:59:451
93309910360,36cyclictest24339-21latency_hist07:47:411
93309910360,36cyclictest24339-21latency_hist07:47:411
77432978,60sleep27812-21latency_hist06:57:422
9330999646,43cyclictest24456-21fschecks_count10:18:351
9330999646,43cyclictest24456-21fschecks_count10:18:351
933799953,65cyclictest24047-21apt-get10:17:502
933799953,65cyclictest24047-21apt-get10:17:502
9330999351,34cyclictest0-21swapper/109:48:031
9330998959,21cyclictest23626-21apt-key10:17:391
9330998914,29cyclictest10805-21seq07:07:221
9330998914,29cyclictest10805-21seq07:07:221
9330998847,33cyclictest0-21swapper/109:43:161
933099884,36cyclictest12870-21cron07:12:361
933099883,39cyclictest27966-21systemd-journal07:32:371
87412888,55sleep08825-21sed06:58:250
9330998774,8cyclictest0-21swapper/112:12:561
9330998638,41cyclictest0-21swapper/109:58:021
933099863,74cyclictest13003-21apt-get08:52:501
933099863,37cyclictest31098-21fschecks_count11:32:591
933099863,36cyclictest9455-21taskset11:53:021
933099863,36cyclictest9455-21taskset11:53:021
933099863,36cyclictest6368-21/usr/sbin/munin11:48:011
9326998670,7cyclictest0-21swapper/010:18:020
9326998670,7cyclictest0-21swapper/010:18:020
933099853,35cyclictest375-21snmpd07:18:031
933099853,35cyclictest375-21snmpd07:18:031
933099852,36cyclictest2568-21/usr/sbin/munin09:38:271
9326998550,10cyclictest181rcu_preempt07:12:530
933099844,33cyclictest26922-21idleruntime-cro09:22:361
933099843,9cyclictest0-21swapper/112:02:511
933099843,36cyclictest9774-21cron08:42:361
933099843,34cyclictest18461-21/usr/sbin/munin10:08:021
933099843,32cyclictest384-21sshd10:35:101
933099843,32cyclictest384-21sshd10:35:101
9326998459,18cyclictest21730-21sh12:13:500
9326998413,11cyclictest363-21apt-get08:12:510
9326998413,11cyclictest363-21apt-get08:12:510
933099834,35cyclictest2108-21/usr/sbin/munin10:38:091
9330998337,39cyclictest0-21swapper/109:32:531
933099833,37cyclictest23667-21timerandwakeup11:18:131
933099833,37cyclictest23667-21timerandwakeup11:18:131
933099833,36cyclictest32553-21basename08:12:371
933099833,25cyclictest30428-21cat09:28:071
933099833,23cyclictest17462-21seq09:03:091
933099833,23cyclictest17462-21seq09:03:091
9330998326,52cyclictest0-21swapper/107:13:131
933099832,24cyclictest26880-21ssh10:22:581
9337998260,16cyclictest10114-21sh09:52:372
933099823,35cyclictest24595-21unixbench_singl12:18:191
933099823,34cyclictest6926-21cstates08:32:551
933099823,34cyclictest17821-21latency_hist07:27:381
933099823,34cyclictest17821-21latency_hist07:27:381
933099823,33cyclictest31615-21latency_hist10:32:391
933099823,33cyclictest31615-21latency_hist10:32:391
933099823,31cyclictest356-21runrttasks08:18:301
933099823,31cyclictest356-21runrttasks08:18:301
933099823,21cyclictest11862-21apt-get11:57:491
933099822,21cyclictest29568-21apt-get08:02:491
9326998264,11cyclictest4442-21rm11:43:070
9326998252,13cyclictest181rcu_preempt09:27:510
932699823,44cyclictest356-21runrttasks11:08:430
934399812,67cyclictest20718-21chrt12:13:063
9337998157,17cyclictest27616-21sshd07:57:392
933799814,61cyclictest28133-21kworker/u18:0+ext4-rsv-conversion07:58:282
9330998159,15cyclictest9316-21cyclictest09:52:471
933099814,34cyclictest9740-21/usr/sbin/munin10:53:181
933099813,39cyclictest18078-21ssh12:08:091
933099813,38cyclictest22742-21mailstats09:13:161
933099813,38cyclictest22742-21mailstats09:13:161
933099813,36cyclictest28092-21timerandwakeup09:23:151
933099813,35cyclictest4634-21/usr/sbin/munin10:43:031
933099813,35cyclictest20855-21proc_pri11:13:101
933099813,34cyclictest19961-21grep09:08:061
9330998133,40cyclictest0-21swapper/112:22:511
933099813,31cyclictest26556-21/usr/sbin/munin07:53:131
933099813,23cyclictest28831-21sendmail_mailqu11:28:111
933099812,38cyclictest1863-21latency_hist08:17:391
933099812,38cyclictest1863-21latency_hist08:17:391
933099812,36cyclictest17466-21ssh10:05:161
933099812,36cyclictest17466-21ssh10:05:161
9337998053,19cyclictest15659-21latency_hist10:02:402
933099803,36cyclictest10245-21fschecks_count08:42:581
933099803,35cyclictest25626-21cat11:22:551
933099803,35cyclictest25626-21cat11:22:551
933099803,33cyclictest3876-21idleruntime08:23:011
933099803,29cyclictest30028-21sh12:28:091
9330998032,40cyclictest0-21swapper/108:52:371
933099802,31cyclictest21182-21cat07:37:401
9326998064,10cyclictest21420-21ssh10:13:000
9326998059,15cyclictest6910-21latency_hist10:47:390
9326998049,9cyclictest181rcu_preempt08:27:530
934399793,29cyclictest27632-21chrt07:57:393
9337997961,12cyclictest27966-21systemd-journal08:32:362
9330997938,34cyclictest0-21swapper/110:58:161
933099793,34cyclictest29522-21latency_hist08:02:401
933099793,33cyclictest21232-21/usr/sbin/munin07:38:011
933099793,33cyclictest21232-21/usr/sbin/munin07:38:011
9330997932,40cyclictest0-21swapper/111:37:511
9330997932,40cyclictest0-21swapper/111:37:511
933099793,21cyclictest4377-21/usr/sbin/munin11:43:051
933099792,36cyclictest5806-21cut08:28:111
933099792,24cyclictest8267-21ssh10:48:501
933099792,24cyclictest8267-21ssh10:48:501
9330997915,27cyclictest15694-21sh08:58:081
9330997914,55cyclictest0-21swapper/111:12:391
9326997951,22cyclictest22095-21ssh09:12:530
9326997951,22cyclictest22095-21ssh09:12:530
933099784,33cyclictest24766-21fschecks_time07:48:181
933099772,34cyclictest15005-21/usr/sbin/munin11:03:081
933099772,34cyclictest15005-21/usr/sbin/munin11:03:081
9326997762,9cyclictest16505-21diskmemload12:07:550
9326997759,11cyclictest12119-21diskmemload11:58:000
9326997759,10cyclictest1325-21ssh11:37:550
9326997759,10cyclictest1325-21ssh11:37:550
9326997748,14cyclictest181rcu_preempt09:45:530
9326997735,17cyclictest181rcu_preempt09:33:140
9326997729,32cyclictest9950-21wc08:42:390
934399763,32cyclictest10393-21chrt07:03:163
934399763,32cyclictest10393-21chrt07:03:163
9326997656,13cyclictest24939-21idleruntime09:18:010
932699764,37cyclictest23538-21apt-get12:17:530
9337997554,14cyclictest375-21snmpd08:03:222
9337997525,39cyclictest14604-21cat07:17:392
932699756,57cyclictest28113-21cstates07:58:370
932699756,37cyclictest0-21swapper/012:23:000
9326997552,16cyclictest378-21rs:main0
9326997544,12cyclictest181rcu_preempt09:53:020
932699754,40cyclictest30824-21/usr/sbin/munin11:33:050
9326997538,14cyclictest181rcu_preempt07:03:160
9326997538,14cyclictest181rcu_preempt07:03:160
9337997458,11cyclictest9316-21cyclictest10:15:592
9337997455,13cyclictest365-21systemd-logind07:42:372
9337997455,13cyclictest365-21systemd-logind07:42:372
932699745,51cyclictest15-21ksoftirqd/010:27:530
932699745,51cyclictest15-21ksoftirqd/010:27:530
9326997455,11cyclictest16505-21diskmemload10:39:310
932699745,33cyclictest305-21switchtime10:33:120
932699745,33cyclictest305-21switchtime10:33:120
9326997449,9cyclictest181rcu_preempt07:52:520
9326997443,9cyclictest181rcu_preempt08:02:530
9326997437,32cyclictest9316-21cyclictest10:53:060
932699743,56cyclictest8623-21ssh09:48:160
9326997415,39cyclictest12213-21systemctl07:08:210
9343997360,7cyclictest0-21swapper/310:18:293
9343997360,7cyclictest0-21swapper/310:18:293
934399733,31cyclictest2595-21chrt11:39:273
934399733,31cyclictest2595-21chrt11:39:273
9337997356,11cyclictest356-21runrttasks12:14:482
9337997350,13cyclictest0-21swapper/210:26:102
9337997326,28cyclictest1-21systemd11:02:372
932699735,33cyclictest14655-21/usr/sbin/munin12:03:080
9326997341,14cyclictest181rcu_preempt08:07:510
932699733,34cyclictest25190-21modprobe11:22:360
932699733,34cyclictest25190-21modprobe11:22:360
932699730,26cyclictest0-21swapper/009:07:530
934399723,32cyclictest15995-21chrt10:03:003
934399723,32cyclictest15995-21chrt10:03:003
934399723,30cyclictest11966-21chrt08:48:023
9337997254,12cyclictest29631-21latency_hist09:27:402
9337997250,16cyclictest379-21cron07:22:372
9337997250,16cyclictest379-21cron07:22:372
9337997247,17cyclictest0-21swapper/207:52:402
933799724,48cyclictest29931-21cpuspeed09:27:552
9337997234,18cyclictest3309-21kworker/u17:3-rpciod11:46:122
9326997255,10cyclictest29901-21ssh11:30:300
9326997253,12cyclictest16505-21diskmemload11:12:570
9326997252,13cyclictest18200-21systemd-run10:07:370
9326997252,13cyclictest18200-21systemd-run10:07:370
934399713,35cyclictest16720-21chrt09:02:403
9337997153,12cyclictest27966-21systemd-journal09:18:362
9337997148,17cyclictest16505-21diskmemload11:53:222
9337997148,17cyclictest16505-21diskmemload11:53:222
933799713,46cyclictest22466-21munin-plugin-st11:17:362
9326997142,21cyclictest15634-21ntp_kernel_err08:58:050
9326997125,10cyclictest181rcu_preempt11:50:360
934399702,30cyclictest25075-21taskset12:19:343
9337997056,10cyclictest27858-21kworker/u17:0-rpciod10:51:102
9337997056,10cyclictest27858-21kworker/u17:0-rpciod10:51:102
9337997054,9cyclictest0-21swapper/209:43:162
9337997052,11cyclictest0-21swapper/211:07:392
9337997052,11cyclictest0-21swapper/211:07:392
9337997047,16cyclictest20184-21apt-key12:12:382
9337997032,29cyclictest8162-21modprobe08:37:362
9337997026,9cyclictest381ktimers/210:08:012
9337997021,14cyclictest181rcu_preempt10:33:032
9337997021,14cyclictest181rcu_preempt10:33:032
932699704,43cyclictest30419-21tail12:28:140
932699703,36cyclictest11548-21munin-run11:57:360
932699703,36cyclictest11548-21munin-run11:57:360
932699703,35cyclictest356-21runrttasks07:52:020
9326997031,11cyclictest181rcu_preempt07:18:090
9326997031,11cyclictest181rcu_preempt07:18:090
934399695,32cyclictest4306-21chrt11:43:033
9337996948,15cyclictest378-21rs:main2
933799693,47cyclictest16771-21tail10:03:152
933799693,47cyclictest16771-21tail10:03:152
933799693,45cyclictest13531-21apt-get08:52:532
9337996933,28cyclictest11911-21kernelversion07:08:042
9326996949,14cyclictest9517-21latency_hist10:52:380
9326996949,14cyclictest9517-21latency_hist10:52:380
932699694,54cyclictest7431-21processes08:33:220
932699694,32cyclictest356-21runrttasks11:04:170
932699694,32cyclictest356-21runrttasks11:04:170
9326996938,16cyclictest181rcu_preempt10:26:160
932699693,48cyclictest10154-21cstates08:42:540
932699693,36cyclictest12065-21latency_hist10:57:390
9326996932,14cyclictest181rcu_preempt10:10:560
9326996932,13cyclictest181rcu_preempt09:41:350
934399683,27cyclictest20851-21taskset10:12:363
9337996853,9cyclictest31169-21sshd10:31:092
9337996853,9cyclictest31169-21sshd10:31:092
9337996845,15cyclictest9316-21cyclictest08:58:062
933799684,41cyclictest32531-21munin-run08:12:372
9337996837,22cyclictest19703-21ssh09:08:012
933799682,32cyclictest16505-21diskmemload12:23:182
932699685,38cyclictest19494-21cat07:32:390
9326996850,10cyclictest13177-21latency_hist08:52:380
932699683,27cyclictest17144-21fschecks_time09:03:000
932699683,27cyclictest17144-21fschecks_time09:03:000
932699682,23cyclictest2223-21fschecks_time08:18:000
932699682,23cyclictest2223-21fschecks_time08:18:000
9337996746,13cyclictest27858-21kworker/u17:0-rpciod11:29:092
933799674,46cyclictest356-21runrttasks11:33:372
933799672,47cyclictest1722-21idleruntime-cro08:17:362
933799672,47cyclictest1722-21idleruntime-cro08:17:362
9326996740,9cyclictest181rcu_preempt08:52:500
9326996738,11cyclictest181rcu_preempt11:22:590
9326996738,11cyclictest181rcu_preempt11:22:590
9326996737,9cyclictest181rcu_preempt07:43:070
9326996737,9cyclictest181rcu_preempt07:43:070
933799665,38cyclictest10682-21grep10:53:232
933799664,45cyclictest356-21runrttasks09:39:442
933799664,42cyclictest19347-21latency_hist07:32:392
9337996644,14cyclictest0-21swapper/211:22:382
9337996644,14cyclictest0-21swapper/211:22:382
9337996642,17cyclictest19818-21fschecks_count07:32:592
933799662,43cyclictest32707-21fschecks_count09:32:592
9337996623,32cyclictest18106-21ntp_kernel_pll_11:08:062
9337996622,8cyclictest381ktimers/207:43:082
9337996622,8cyclictest381ktimers/207:43:082
9337996619,26cyclictest39-21ksoftirqd/209:56:022
9326996629,11cyclictest181rcu_preempt09:59:470
932699662,35cyclictest27320-21ssh09:22:530
916126612,32sleep30-21swapper/307:02:363
9337996546,12cyclictest26259-21latency_hist12:22:382
9337996545,13cyclictest0-21swapper/212:28:132
933799654,42cyclictest3783-21latency_hist11:42:402
933799654,42cyclictest3783-21latency_hist11:42:402
9337996544,15cyclictest5316-21processes10:43:132
9337996544,12cyclictest0-21swapper/207:22:542
9337996544,12cyclictest0-21swapper/207:22:542
9326996546,11cyclictest181rcu_preempt08:23:000
9337996450,8cyclictest0-21swapper/210:37:592
9337996443,14cyclictest8722-21latency_hist11:52:392
933799643,34cyclictest356-21runrttasks08:43:252
9337996420,8cyclictest381ktimers/212:00:502
201262640,5sleep30-21swapper/307:33:053
934399635,41cyclictest0-21swapper/308:53:123
933799634,37cyclictest17374-21ntp_kernel_err09:03:052
933799634,37cyclictest17374-21ntp_kernel_err09:03:052
933799633,43cyclictest3372-21latency_hist08:22:382
933799633,43cyclictest3372-21latency_hist08:22:382
933799632,43cyclictest26031-21ntp_kernel_err11:23:052
933799632,43cyclictest26031-21ntp_kernel_err11:23:052
9337996322,31cyclictest9207-21timerandwakeup08:38:142
932699633,40cyclictest22587-21apt07:42:360
932699633,40cyclictest22587-21apt07:42:360
9326996329,13cyclictest181rcu_preempt07:22:410
9326996329,13cyclictest181rcu_preempt07:22:410
9337996243,11cyclictest0-21swapper/212:02:582
933799622,42cyclictest9732-21sort07:03:022
933799622,42cyclictest9732-21sort07:03:022
9337996145,9cyclictest0-21swapper/208:23:152
933799614,28cyclictest22730-21mailstats09:13:162
933799614,28cyclictest22730-21mailstats09:13:162
78632600,8sleep30-21swapper/309:47:573
934399592,42cyclictest24063-21taskset09:16:503
934399592,42cyclictest24063-21taskset09:16:503
934399584,37cyclictest31252-21taskset11:33:033
934399583,39cyclictest17278-21chrt07:23:133
934399583,39cyclictest17278-21chrt07:23:133
934399583,37cyclictest29970-21taskset12:28:063
934399582,40cyclictest954-21taskset10:34:333
934399582,40cyclictest954-21taskset10:34:333
934399573,41cyclictest29415-21chrt09:27:153
934399563,36cyclictest13919-21taskset11:00:053
934399563,35cyclictest859-21taskset08:13:063
934399563,35cyclictest859-21taskset08:13:063
934399563,34cyclictest8993-21taskset11:52:393
934399563,33cyclictest10769-21chrt10:53:243
934399553,40cyclictest10783-21chrt11:55:233
934399553,40cyclictest10783-21chrt11:55:233
934399553,35cyclictest30193-21chrt08:03:103
934399552,41cyclictest2075-21taskset10:37:403
934399552,39cyclictest23380-21chrt11:18:083
934399552,39cyclictest23380-21chrt11:18:083
934399542,36cyclictest23721-21taskset07:43:263
934399542,36cyclictest23721-21taskset07:43:263
934399542,35cyclictest23697-21chrt10:17:393
934399542,33cyclictest29145-21chrt10:27:403
934399533,33cyclictest13331-21chrt11:59:283
934399533,32cyclictest26064-21taskset11:23:063
934399533,32cyclictest26064-21taskset11:23:063
934399532,38cyclictest27766-21taskset12:23:523
934399532,35cyclictest11885-21chrt09:54:263
934399524,32cyclictest17687-21chrt11:07:553
934399523,33cyclictest7374-21chrt09:47:203
934399523,33cyclictest3284-21taskset09:38:253
934399523,31cyclictest19387-21chrt07:32:363
934399522,33cyclictest13729-21chrt09:58:063
934399513,36cyclictest5957-21chrt08:28:133
934399513,36cyclictest24994-21taskset07:48:243
934399513,34cyclictest30650-21chrt11:32:383
934399513,32cyclictest17612-21taskset12:07:553
934399513,31cyclictest31910-21chrt08:08:123
934399512,35cyclictest16549-21taskset11:05:193
934399512,35cyclictest16549-21taskset11:05:193
934399503,35cyclictest29796-21taskset10:28:083
934399503,35cyclictest29796-21taskset10:28:083
934399502,34cyclictest6584-21taskset10:46:113
934399502,33cyclictest7199-21chrt08:33:033
934399502,33cyclictest14166-21chrt07:13:163
934399502,32cyclictest30593-21chrt09:28:133
934399493,31cyclictest14740-21taskset12:02:573
934399493,30cyclictest22203-21chrt07:38:153
934399493,30cyclictest22203-21chrt07:38:153
934399493,29cyclictest20207-21chrt11:12:493
934399492,33cyclictest3405-21chrt08:22:383
934399492,33cyclictest3405-21chrt08:22:383
934399482,29cyclictest9880-21chrt08:42:393
934399473,30cyclictest20718-21chrt09:09:003
934399472,29cyclictest12285-21chrt07:08:223
934399463,29cyclictest25799-21taskset09:18:363
934399453,29cyclictest9628-21chrt10:52:393
934399453,29cyclictest9628-21chrt10:52:393
934399453,28cyclictest18830-21chrt09:06:453
934399453,28cyclictest18830-21chrt09:06:453
9343994522,11cyclictest0-21swapper/307:58:373
934399442,29cyclictest1179-21taskset09:33:233
934399422,25cyclictest3675-21taskset08:22:543
934399413,27cyclictest15001-21taskset07:17:583
934399413,27cyclictest15001-21taskset07:17:583
934399413,26cyclictest10062-21taskset08:42:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional