You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-19 - 20:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Jul 19, 2026 12:45:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
909564999187,2cyclictest0-21swapper/007:40:020
909573997740,25cyclictest30-21ksoftirqd/211:40:022
909573997740,25cyclictest30-21ksoftirqd/211:40:012
90956499731,7cyclictest1-21systemd11:50:010
90956499731,7cyclictest1-21systemd11:50:000
909564997232,20cyclictest974122-21latency_hist11:15:030
909564996817,25cyclictest955275-21kworker/0:110:45:030
909564996811,14cyclictest12-21ksoftirqd/009:40:020
909564996810,8cyclictest131rcu_preempt10:14:120
90956499664,7cyclictest121ksoftirqd/009:20:270
90957899637,40cyclictest364-21snmpd12:00:023
90957899637,40cyclictest364-21snmpd12:00:023
909564996336,20cyclictest987961-21ssh11:45:000
909564996321,8cyclictest12-21ksoftirqd/011:56:020
909564996321,8cyclictest12-21ksoftirqd/011:56:010
909564996319,13cyclictest12-21ksoftirqd/012:10:300
90956499631,6cyclictest12-21ksoftirqd/010:29:130
90956499624,53cyclictest915516-21latency_hist09:05:020
90956499622,41cyclictest983311-21wc11:35:000
909573996135,6cyclictest914970-21kworker/2:210:26:202
90956499616,9cyclictest12-21ksoftirqd/010:40:010
90956499614,7cyclictest12-21ksoftirqd/010:21:500
90956499604,7cyclictest121ksoftirqd/012:26:320
909564996027,29cyclictest101086450rm12:33:170
90956499599,12cyclictest12-21ksoftirqd/010:20:010
909564995917,10cyclictest12-21ksoftirqd/008:15:020
909564995917,10cyclictest12-21ksoftirqd/008:15:010
909564995915,10cyclictest12-21ksoftirqd/012:35:490
90956499591,36cyclictest0-21swapper/012:00:470
909564995913,23cyclictest0-21swapper/009:33:360
909564995913,23cyclictest0-21swapper/009:33:350
90956499587,5cyclictest12-21ksoftirqd/011:08:320
90956499585,9cyclictest12-21ksoftirqd/009:20:000
909564995828,14cyclictest991445-21ssh11:52:150
90956499581,26cyclictest3003354-1kworker/0:1H12:20:010
90956499581,26cyclictest3003354-1kworker/0:1H12:20:010
909564995811,12cyclictest12-21ksoftirqd/010:05:010
90956499579,22cyclictest0-21swapper/009:28:320
90956499574,7cyclictest12-21ksoftirqd/011:30:000
909564995740,11cyclictest914163-21cron08:40:010
909564995740,11cyclictest914163-21cron08:40:000
909564995730,12cyclictest933401-21ssh09:47:550
909564995711,28cyclictest131rcu_preempt09:11:050
909578995648,5cyclictest0-21swapper/307:10:013
90956499567,7cyclictest121ksoftirqd/011:03:280
90956499567,6cyclictest12-21ksoftirqd/009:52:000
909564995620,31cyclictest12-21ksoftirqd/012:25:020
909564995620,29cyclictest16850irq/134-800000011:16:310
909564995620,29cyclictest16850irq/134-800000011:16:310
909564995615,14cyclictest12-21ksoftirqd/007:10:020
909564995612,10cyclictest12-21ksoftirqd/007:40:000
909573995547,5cyclictest925019-21kworker/2:311:53:272
90956499554,6cyclictest12-21ksoftirqd/010:10:030
90956499551,6cyclictest0-21swapper/009:57:400
909564995513,9cyclictest12-21ksoftirqd/008:10:020
909573995446,6cyclictest30-21ksoftirqd/211:34:532
90956499546,6cyclictest121ksoftirqd/010:31:030
90956499546,6cyclictest121ksoftirqd/010:31:020
90956499545,11cyclictest12-21ksoftirqd/008:55:030
90956499545,11cyclictest12-21ksoftirqd/008:55:030
909564995426,13cyclictest965703-21ssh10:57:150
90956499541,29cyclictest0-21swapper/009:45:000
90956499537,8cyclictest12-21ksoftirqd/011:36:020
90956499537,8cyclictest12-21ksoftirqd/011:36:010
90956499536,6cyclictest12-21ksoftirqd/010:55:010
909564995327,11cyclictest958567-21ssh10:41:530
909578995237,11cyclictest1-21systemd09:45:013
909573995244,5cyclictest914970-21kworker/2:212:25:012
909573995244,5cyclictest914970-21kworker/2:212:15:022
909573995244,5cyclictest914970-21kworker/2:210:23:182
909573995244,5cyclictest914970-21kworker/2:209:30:392
909573995244,5cyclictest914970-21kworker/2:209:30:382
909573995243,6cyclictest925019-21kworker/2:310:44:572
909573995243,6cyclictest925019-21kworker/2:310:36:392
909573995243,6cyclictest914970-21kworker/2:210:57:422
909573995229,19cyclictest3467602-21systemd-journal09:00:002
90956499525,4cyclictest131rcu_preempt11:23:090
90956499524,12cyclictest12-21ksoftirqd/008:05:010
90956499523,7cyclictest12-21ksoftirqd/012:05:590
90956499523,7cyclictest12-21ksoftirqd/012:05:580
90956499522,33cyclictest19350irq/34-mmc107:25:020
90957899514,4cyclictest131rcu_preempt09:10:013
909573995143,5cyclictest925019-21kworker/2:310:50:582
909573995142,5cyclictest925019-21kworker/2:310:09:592
909569995131,16cyclictest912107-21systemctl08:00:011
909569995131,16cyclictest912107-21systemctl08:00:001
909569995124,6cyclictest23-21ksoftirqd/108:35:001
90956999512,3cyclictest131rcu_preempt12:19:311
90956999512,3cyclictest131rcu_preempt12:19:311
909564995131,15cyclictest275-21dbus-daemon08:25:000
909564995131,15cyclictest1-21systemd07:15:000
909578995031,15cyclictest1-21systemd11:15:013
909573995042,5cyclictest925019-21kworker/2:312:00:422
909573995042,5cyclictest914970-21kworker/2:211:24:072
909573995041,6cyclictest925019-21kworker/2:310:45:392
909573995041,6cyclictest914970-21kworker/2:211:08:312
909573995041,6cyclictest901090-21kworker/2:107:24:042
909573995040,7cyclictest925019-21kworker/2:311:47:412
909573995040,7cyclictest925019-21kworker/2:311:47:402
909573995040,6cyclictest914970-21kworker/2:211:18:242
909573995040,6cyclictest914970-21kworker/2:211:18:242
909573995034,11cyclictest1-21systemd10:15:012
909569995022,5cyclictest23-21ksoftirqd/109:33:441
909569995022,5cyclictest23-21ksoftirqd/109:33:441
909569995021,6cyclictest23-21ksoftirqd/110:54:401
90956499506,13cyclictest12-21ksoftirqd/007:55:010
909564995046,2cyclictest0-21swapper/007:20:030
90956499502,11cyclictest912106-21modprobe08:00:010
90956499502,11cyclictest912106-21modprobe08:00:000
90957899495,3cyclictest131rcu_preempt08:20:023
909578994931,14cyclictest275-21dbus-daemon11:20:013
909578994931,14cyclictest275-21dbus-daemon11:20:003
909573994941,5cyclictest925019-21kworker/2:309:35:152
909573994941,5cyclictest914970-21kworker/2:209:40:482
909573994941,5cyclictest914970-21kworker/2:209:25:042
909573994941,5cyclictest901090-21kworker/2:108:09:202
909573994941,5cyclictest901090-21kworker/2:107:50:002
909573994940,5cyclictest925019-21kworker/2:312:28:542
909573994940,5cyclictest914970-21kworker/2:209:21:492
909569994921,6cyclictest23-21ksoftirqd/111:50:501
909569994910,7cyclictest131rcu_preempt10:10:031
90956499495,10cyclictest12-21ksoftirqd/009:10:010
90956499494,11cyclictest12-21ksoftirqd/007:30:010
90956499491,10cyclictest12-21ksoftirqd/008:59:590
909573994840,5cyclictest925019-21kworker/2:310:03:192
909573994840,5cyclictest925019-21kworker/2:309:52:472
909573994839,6cyclictest914970-21kworker/2:212:32:542
90956999487,6cyclictest23-21ksoftirqd/110:25:021
90956999482,41cyclictest150-21pr/ttyS210:36:581
909569994821,6cyclictest23-21ksoftirqd/110:46:321
909569994821,5cyclictest23-21ksoftirqd/109:46:141
90956999481,3cyclictest131rcu_preempt09:30:011
909569994812,5cyclictest131rcu_preempt09:17:211
909578994738,5cyclictest961140-21sshd10:47:303
90957899473,3cyclictest131rcu_preempt11:00:023
90957899471,3cyclictest131rcu_preempt08:00:013
90957899471,3cyclictest131rcu_preempt08:00:013
909573994739,5cyclictest925019-21kworker/2:311:59:222
909573994739,5cyclictest925019-21kworker/2:311:59:222
909573994739,5cyclictest914970-21kworker/2:212:36:052
909573994739,5cyclictest901090-21kworker/2:108:25:022
909573994738,6cyclictest914970-21kworker/2:209:13:292
909573994736,7cyclictest914970-21kworker/2:209:49:172
90956999475,7cyclictest23-21ksoftirqd/109:42:171
90957899464,4cyclictest131rcu_preempt12:21:573
90957899464,3cyclictest131rcu_preempt09:19:403
90957899463,3cyclictest131rcu_preempt09:37:283
909578994631,12cyclictest957580-21run-parts10:39:593
909573994638,5cyclictest925019-21kworker/2:312:06:572
909573994638,5cyclictest925019-21kworker/2:312:06:562
909573994638,5cyclictest925019-21kworker/2:310:31:142
909573994638,5cyclictest925019-21kworker/2:310:31:132
909573994638,5cyclictest925019-21kworker/2:309:56:362
909573994638,5cyclictest914970-21kworker/2:212:17:582
909573994638,5cyclictest914970-21kworker/2:212:17:572
909573994638,5cyclictest914970-21kworker/2:211:25:152
909573994638,5cyclictest914970-21kworker/2:210:17:582
909573994637,5cyclictest914970-21kworker/2:209:19:102
90956999465,4cyclictest131rcu_preempt10:15:011
909569994632,10cyclictest1-21systemd08:45:011
909569994629,12cyclictest275-21dbus-daemon11:10:001
909569994626,14cyclictest275-21dbus-daemon07:30:011
909569994623,4cyclictest23-21ksoftirqd/107:40:011
90956499465,7cyclictest12-21ksoftirqd/007:35:030
90956499465,7cyclictest12-21ksoftirqd/007:35:020
90956499465,10cyclictest12-21ksoftirqd/008:35:000
90956499464,9cyclictest12-21ksoftirqd/008:30:020
90956499462,14cyclictest913181-21munin-run08:20:010
909578994528,13cyclictest1-21systemd11:05:013
90957899452,4cyclictest131rcu_preempt10:31:193
90957899452,4cyclictest131rcu_preempt10:31:193
90957899451,3cyclictest131rcu_preempt11:35:323
90957899451,3cyclictest131rcu_preempt11:35:323
90957899451,3cyclictest131rcu_preempt11:10:033
90957899451,3cyclictest131rcu_preempt09:27:523
90956999455,3cyclictest131rcu_preempt08:12:311
90956999455,3cyclictest131rcu_preempt08:12:301
90956999453,8cyclictest23-21ksoftirqd/112:00:011
90956999453,8cyclictest23-21ksoftirqd/112:00:011
909569994535,6cyclictest978663-21cat11:25:011
909569994527,14cyclictest275-21dbus-daemon09:15:011
909569994527,13cyclictest275-21dbus-daemon09:25:011
909569994523,5cyclictest23-21ksoftirqd/112:21:151
90956999451,3cyclictest131rcu_preempt12:03:451
90956499453,6cyclictest12-21ksoftirqd/008:50:000
90957899444,3cyclictest131rcu_preempt12:10:203
90957899444,3cyclictest131rcu_preempt11:46:453
90957899444,3cyclictest131rcu_preempt11:46:453
90957899443,3cyclictest131rcu_preempt07:45:033
909578994428,11cyclictest3467602-21systemd-journal10:00:013
90957899442,3cyclictest131rcu_preempt12:35:213
90957899442,3cyclictest131rcu_preempt11:25:513
90957899442,3cyclictest131rcu_preempt11:24:123
90957899442,3cyclictest131rcu_preempt10:54:593
90957899441,4cyclictest131rcu_preempt09:52:183
90957899441,4cyclictest131rcu_preempt09:33:303
90957899441,4cyclictest131rcu_preempt09:33:293
909573994436,5cyclictest914970-21kworker/2:211:10:462
909573994435,6cyclictest901090-21kworker/2:108:14:152
909573994435,6cyclictest901090-21kworker/2:108:14:152
909573994428,12cyclictest275-21dbus-daemon07:20:012
90956999446,4cyclictest131rcu_preempt09:56:511
90956999445,4cyclictest131rcu_preempt12:35:011
909569994435,6cyclictest23-21ksoftirqd/111:40:011
909569994435,6cyclictest23-21ksoftirqd/111:40:001
909569994432,9cyclictest911821-21sshd07:53:291
909569994425,13cyclictest275-21dbus-daemon11:15:011
90956499445,10cyclictest12-21ksoftirqd/008:44:590
90957899434,3cyclictest131rcu_preempt11:30:203
90957899434,3cyclictest131rcu_preempt10:10:023
90957899434,3cyclictest131rcu_preempt10:05:013
90957899434,3cyclictest131rcu_preempt09:21:283
90957899433,3cyclictest131rcu_preempt09:45:583
909578994331,9cyclictest914426-21sshd08:40:303
909578994331,9cyclictest275-21dbus-daemon12:35:013
909578994328,11cyclictest275-21dbus-daemon08:55:013
909578994328,11cyclictest275-21dbus-daemon08:55:003
90957899432,3cyclictest131rcu_preempt09:11:043
90957899431,3cyclictest131rcu_preempt12:00:493
90957899431,3cyclictest131rcu_preempt11:52:103
90957899431,3cyclictest131rcu_preempt11:44:503
90957899431,3cyclictest131rcu_preempt10:21:123
90957899431,3cyclictest131rcu_preempt10:19:153
90957899431,3cyclictest131rcu_preempt10:13:583
90957899431,22cyclictest0-21swapper/310:29:293
909573994335,5cyclictest925019-21kworker/2:311:04:582
909573994330,9cyclictest3467602-21systemd-journal07:40:002
90956999435,6cyclictest131rcu_preempt10:42:031
90956999433,4cyclictest131rcu_preempt10:33:141
90956999433,4cyclictest131rcu_preempt10:33:131
909569994329,11cyclictest3467602-21systemd-journal09:10:011
909569994318,5cyclictest23-21ksoftirqd/112:10:531
909569994317,5cyclictest23-21ksoftirqd/109:39:591
90956499435,7cyclictest12-21ksoftirqd/007:50:010
90957899423,4cyclictest131rcu_preempt12:28:023
90957899423,3cyclictest131rcu_preempt12:08:143
90957899423,3cyclictest131rcu_preempt12:08:143
909578994231,9cyclictest37-21ksoftirqd/308:40:033
909578994230,6cyclictest915214-21sendmail08:59:593
909578994229,10cyclictest1-21systemd08:25:003
909578994228,11cyclictest911827-21sshd07:54:563
909578994228,10cyclictest309-21in:imuxsock12:20:013
909578994228,10cyclictest309-21in:imuxsock12:20:013
90957899421,3cyclictest131rcu_preempt09:05:003
90957899421,3cyclictest131rcu_preempt07:25:003
909573994234,5cyclictest914970-21kworker/2:211:41:112
90956999428,3cyclictest131rcu_preempt10:30:011
90956999423,9cyclictest23-21ksoftirqd/109:54:391
909569994230,8cyclictest3467602-21systemd-journal07:25:011
90956999422,6cyclictest23-21ksoftirqd/111:16:481
90956999422,6cyclictest23-21ksoftirqd/111:16:471
909569994226,12cyclictest275-21dbus-daemon08:30:011
909569994223,4cyclictest23-21ksoftirqd/109:05:001
909569994222,5cyclictest23-21ksoftirqd/108:40:021
909569994222,5cyclictest23-21ksoftirqd/108:40:011
90956999421,3cyclictest131rcu_preempt12:26:171
90956999421,3cyclictest131rcu_preempt10:58:351
90956999421,3cyclictest131rcu_preempt10:17:041
909578994127,10cyclictest1-21systemd07:35:023
909578994127,10cyclictest1-21systemd07:35:013
90957899412,3cyclictest131rcu_preempt07:26:303
90957899411,3cyclictest131rcu_preempt10:41:143
90957899411,3cyclictest131rcu_preempt08:05:013
909573994133,5cyclictest901090-21kworker/2:108:25:032
909573994132,6cyclictest908900-21kworker/2:307:53:312
90957399412,3cyclictest131rcu_preempt07:10:012
90956999417,4cyclictest131rcu_preempt11:33:381
909569994129,9cyclictest23-21ksoftirqd/111:01:421
909569994127,10cyclictest275-21dbus-daemon08:55:011
909569994127,10cyclictest275-21dbus-daemon08:55:001
90956999412,3cyclictest131rcu_preempt11:25:511
90956999411,6cyclictest0-21swapper/112:36:021
90956999411,3cyclictest131rcu_preempt08:05:011
909569994111,3cyclictest131rcu_preempt12:10:021
909569994111,3cyclictest131rcu_preempt12:10:021
909578994025,10cyclictest3467602-21systemd-journal07:14:593
90957899401,34cyclictest37-21ksoftirqd/308:50:013
909573994032,5cyclictest901090-21kworker/2:108:40:212
909573994031,7cyclictest30-21ksoftirqd/207:45:022
90956999404,3cyclictest131rcu_preempt11:48:311
90956999404,3cyclictest131rcu_preempt11:48:311
909569994019,5cyclictest23-21ksoftirqd/107:20:011
909569994018,6cyclictest23-21ksoftirqd/108:50:011
909569994018,5cyclictest23-21ksoftirqd/111:45:011
90957899393,3cyclictest131rcu_preempt07:39:593
90957899392,3cyclictest131rcu_preempt08:30:013
90957899392,3cyclictest131rcu_preempt08:15:023
90957899392,3cyclictest131rcu_preempt08:15:013
909573993926,9cyclictest1-21systemd09:10:012
90956999394,32cyclictest150-21pr/ttyS207:43:431
909569993920,14cyclictest275-21dbus-daemon07:35:021
909569993920,14cyclictest275-21dbus-daemon07:35:011
909569993917,4cyclictest23-21ksoftirqd/107:15:021
90956999391,3cyclictest131rcu_preempt10:03:181
90957899385,3cyclictest131rcu_preempt07:20:003
90957899383,3cyclictest131rcu_preempt08:10:013
909573993829,7cyclictest30-21ksoftirqd/208:35:052
909573993829,7cyclictest30-21ksoftirqd/208:35:052
909573993829,6cyclictest901090-21kworker/2:108:50:022
90957399381,6cyclictest0-21swapper/208:05:012
90956999387,9cyclictest131rcu_preempt08:55:031
909569993816,6cyclictest23-21ksoftirqd/108:20:011
909578993724,10cyclictest911670-21cat07:50:033
909569993720,4cyclictest23-21ksoftirqd/108:25:011
909569993716,5cyclictest23-21ksoftirqd/108:10:011
909573993623,9cyclictest1-21systemd08:20:002
90957399361,3cyclictest131rcu_preempt09:05:022
909569993623,9cyclictest909677-21sed07:10:011
90957899351,3cyclictest131rcu_preempt08:40:013
90957399356,7cyclictest30-21ksoftirqd/207:35:032
90957399356,7cyclictest30-21ksoftirqd/207:35:022
909573993526,7cyclictest30-21ksoftirqd/208:50:032
909573993526,7cyclictest30-21ksoftirqd/208:50:022
909573993518,13cyclictest1-21systemd07:15:002
909569993527,5cyclictest911563-21sshd07:47:311
909578993424,7cyclictest37-21ksoftirqd/308:35:013
909573993421,10cyclictest1-21systemd08:00:002
909573993421,10cyclictest1-21systemd08:00:002
90957399337,15cyclictest0-21swapper/208:34:162
90957399311,4cyclictest131rcu_preempt07:30:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional