You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 07:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Apr 05, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
786235991121,108cyclictest0-21swapper/021:00:030
7862469910467,29cyclictest30-21ksoftirqd/220:10:022
78624699902,11cyclictest131rcu_preempt20:30:022
786235998724,16cyclictest876953-21ssh00:25:200
786235998720,61cyclictest16850irq/134-800000023:13:180
78623599852,78cyclictest48-21kcompactd023:54:530
786235998432,33cyclictest864788-21ssh23:57:220
786235998427,39cyclictest806258-21ssh21:41:400
78623599832,77cyclictest849153-21ssh23:20:490
786235998326,40cyclictest866918-21ssh00:02:080
786235998326,40cyclictest866918-21ssh00:02:080
786235998227,51cyclictest817418-21ssh22:07:420
786235998224,13cyclictest812245-21ssh21:55:370
78623599821,63cyclictest841208-21ssh23:02:460
78623599821,63cyclictest841208-21ssh23:02:460
78623599811,76cyclictest881476-21ssh00:36:000
78623599809,6cyclictest16850irq/134-800000023:32:160
78623599809,6cyclictest16850irq/134-800000023:32:150
78623599802,70cyclictest0-21swapper/022:30:030
786235998025,8cyclictest833033-21ssh22:43:510
786235998018,52cyclictest16850irq/134-800000023:15:360
786235997919,51cyclictest16850irq/134-800000022:32:160
786235997819,48cyclictest16850irq/134-800000023:28:420
786235997818,56cyclictest16850irq/134-800000022:10:050
786235997818,56cyclictest16850irq/134-800000022:10:050
786235997728,33cyclictest798065-21ssh21:22:520
786235997717,49cyclictest16850irq/134-800000022:45:520
786235997628,44cyclictest856882-21ssh23:39:100
786235997626,16cyclictest836203-21ssh22:50:580
78623599761,58cyclictest880171-21ssh00:33:090
786235997530,28cyclictest808204-21ssh21:46:140
786235997523,4cyclictest0-21swapper/021:31:550
786235997518,52cyclictest16850irq/134-800000022:22:220
78623599751,27cyclictest0-21swapper/021:51:150
786235997511,54cyclictest16850irq/134-800000023:05:270
78623599742,9cyclictest792425-21munin-plugin-st21:10:000
786235997425,13cyclictest814285-21ssh22:00:090
786235997318,45cyclictest16850irq/134-800000000:07:560
786235997222,4cyclictest0-21swapper/021:15:390
786235997220,11cyclictest838315-21ssh22:55:560
78624699716,23cyclictest131rcu_preempt00:05:022
78624699716,23cyclictest131rcu_preempt00:05:022
786235997132,35cyclictest16950irq/224-800000021:36:010
786235997126,13cyclictest873311-21ssh00:17:080
78623599712,4cyclictest791893-21munin-run21:00:000
786235997062,4cyclictest12-21ksoftirqd/019:20:020
786235997026,40cyclictest787193-21sshd19:27:330
78623599701,25cyclictest0-21swapper/000:13:590
786235996936,30cyclictest16950irq/224-800000023:41:350
786235996816,15cyclictest793696-21ssh21:12:460
786235996720,13cyclictest799258-21ssh21:25:200
78623599671,62cyclictest860799-21ssh23:48:080
78624699663,4cyclictest131rcu_preempt19:30:022
786243996621,8cyclictest23-21ksoftirqd/100:00:011
78623599663,17cyclictest786229-21cyclictest19:10:010
786235996624,8cyclictest789825-21readlink20:20:000
786246996524,4cyclictest851403-21sh23:26:092
786243996513,11cyclictest23-21ksoftirqd/122:19:591
78623599653,6cyclictest16850irq/134-800000022:36:540
786246996420,5cyclictest829221-21latency_hist22:35:022
786235996436,24cyclictest1-21systemd19:15:010
78624699632,19cyclictest3467602-21systemd-journal19:15:012
786246996320,9cyclictest817580-1kworker/u9:222:14:342
786246996320,9cyclictest817580-1kworker/u9:222:14:342
786246996318,13cyclictest3467602-21systemd-journal00:35:022
786235996316,6cyclictest16850irq/134-800000000:20:190
786246996229,30cyclictest849669-21sh23:22:122
786246996222,4cyclictest856230-21sh23:37:282
786246996219,8cyclictest808861-1kworker/u9:121:58:222
786243996224,22cyclictest23-21ksoftirqd/121:45:031
786243996215,9cyclictest23-21ksoftirqd/123:38:251
786243996215,12cyclictest23-21ksoftirqd/123:43:001
786243996215,10cyclictest23-21ksoftirqd/122:00:001
786235996229,29cyclictest789041-21latency_hist20:05:020
786235996229,29cyclictest789041-21latency_hist20:05:010
786246996129,28cyclictest841193-21sh23:02:432
786246996129,28cyclictest841193-21sh23:02:432
786246996124,9cyclictest30-21ksoftirqd/221:15:012
786246996118,11cyclictest30-21ksoftirqd/222:17:202
786243996113,12cyclictest23-21ksoftirqd/122:45:121
786246996029,28cyclictest30-21ksoftirqd/223:56:152
78624699602,23cyclictest833510-21wc22:45:022
78624399604,20cyclictest841841-21ssh23:04:161
78624399604,20cyclictest841841-21ssh23:04:161
786243996013,10cyclictest23-21ksoftirqd/100:10:021
786235996028,28cyclictest788004-21latency_hist19:45:010
786235996028,28cyclictest16950irq/224-800000022:16:370
786235996020,13cyclictest12-21ksoftirqd/019:20:020
78625099597,6cyclictest131rcu_preempt23:50:263
786246995923,10cyclictest30-21ksoftirqd/221:51:072
786246995919,9cyclictest30-21ksoftirqd/222:09:182
786246995919,10cyclictest30-21ksoftirqd/223:35:002
786246995919,10cyclictest30-21ksoftirqd/223:35:002
786243995914,10cyclictest23-21ksoftirqd/100:00:151
786243995914,10cyclictest23-21ksoftirqd/100:00:151
786243995910,12cyclictest23-21ksoftirqd/123:07:371
786246995829,26cyclictest814615-21sh22:01:072
786246995824,8cyclictest30-21ksoftirqd/222:25:012
786246995822,5cyclictest309-21in:imuxsock21:42:502
78624699582,24cyclictest844286-21sh23:10:002
786243995816,9cyclictest23-21ksoftirqd/100:20:001
786243995815,8cyclictest23-21ksoftirqd/121:54:331
786246995725,5cyclictest829598-21sh22:35:402
786246995723,6cyclictest30-21ksoftirqd/220:20:012
786246995721,5cyclictest839950-21sendmail23:00:002
786246995720,8cyclictest30-21ksoftirqd/223:52:112
786246995718,9cyclictest309-21in:imuxsock00:35:012
78624399578,16cyclictest863205-21ssh23:53:481
78624399574,16cyclictest836296-21sh22:51:111
786243995733,14cyclictest3467602-21systemd-journal00:35:001
786243995714,9cyclictest23-21ksoftirqd/121:25:011
786243995712,10cyclictest23-21ksoftirqd/119:45:001
786235995717,13cyclictest275-21dbus-daemon19:35:010
786250995610,24cyclictest785774-21kworker/u8:019:15:013
786246995650,4cyclictest30-21ksoftirqd/219:50:012
786246995622,4cyclictest870695-21sh00:10:512
786246995621,8cyclictest30-21ksoftirqd/223:11:462
786246995621,8cyclictest30-21ksoftirqd/220:33:302
78624399569,16cyclictest882656-21ssh00:38:561
786243995634,15cyclictest3467602-21systemd-journal22:34:591
786243995619,7cyclictest23-21ksoftirqd/123:32:481
786243995619,7cyclictest23-21ksoftirqd/123:32:471
78623599562,49cyclictest284-21systemd-logind20:15:000
786246995525,7cyclictest30-21ksoftirqd/223:15:432
786246995523,5cyclictest797403-21sh21:21:062
78624699552,18cyclictest791894-21munin-plugin-st20:59:592
786246995518,8cyclictest30-21ksoftirqd/222:48:212
78624399559,9cyclictest23-21ksoftirqd/123:16:181
786243995514,10cyclictest23-21ksoftirqd/121:49:161
786243995511,7cyclictest23-21ksoftirqd/121:31:421
78625099545,7cyclictest37-21ksoftirqd/300:00:123
78625099545,7cyclictest37-21ksoftirqd/300:00:123
786250995439,11cyclictest275-21dbus-daemon22:50:013
78624699544,10cyclictest131rcu_preempt20:40:012
78624699544,10cyclictest131rcu_preempt20:40:012
786246995425,6cyclictest30-21ksoftirqd/221:29:272
786246995420,10cyclictest30-21ksoftirqd/200:22:062
786246995417,10cyclictest309-21in:imuxsock21:09:592
786246995416,10cyclictest30-21ksoftirqd/200:18:522
786246995415,10cyclictest30-21ksoftirqd/220:42:452
78624399549,10cyclictest23-21ksoftirqd/122:13:241
78624399549,10cyclictest23-21ksoftirqd/122:13:241
78624399546,15cyclictest793238-21ssh21:11:421
786243995410,7cyclictest23-21ksoftirqd/120:55:011
786235995437,13cyclictest1-21systemd20:45:040
78625099534,34cyclictest871567-21kworker/u8:100:35:023
786246995345,5cyclictest852953-21kworker/2:200:06:242
786246995322,7cyclictest30-21ksoftirqd/221:45:242
786246995321,7cyclictest30-21ksoftirqd/219:45:012
78624399535,16cyclictest790539-21cat20:30:031
786243995314,8cyclictest23-21ksoftirqd/100:25:171
78623599531,29cyclictest0-21swapper/020:26:050
786246995243,5cyclictest786019-21kworker/2:019:35:002
786246995232,14cyclictest275-21dbus-daemon21:20:012
786246995221,5cyclictest803844-21sh21:36:042
786243995214,7cyclictest23-21ksoftirqd/122:05:351
786243995212,8cyclictest23-21ksoftirqd/122:25:011
786235995225,6cyclictest12-21ksoftirqd/019:55:020
786250995137,11cyclictest37-21ksoftirqd/300:00:023
786246995143,5cyclictest852953-21kworker/2:223:45:482
786246995121,6cyclictest837872-21cat22:55:012
786246995121,4cyclictest789229-21latency_hist20:05:032
786246995121,4cyclictest789229-21latency_hist20:05:032
786246995119,8cyclictest30-21ksoftirqd/222:26:342
786246995117,9cyclictest30-21ksoftirqd/219:57:362
78624399506,10cyclictest23-21ksoftirqd/122:37:201
78624399505,15cyclictest850893-21sed23:25:021
786235995042,5cyclictest16950irq/224-800000019:40:010
78624699495,17cyclictest3467602-21systemd-journal19:19:592
786246994925,6cyclictest30-21ksoftirqd/200:28:362
786243994913,7cyclictest23-21ksoftirqd/123:13:221
786250994825,16cyclictest275-21dbus-daemon19:45:013
78624399487,9cyclictest23-21ksoftirqd/123:29:281
78624399487,9cyclictest23-21ksoftirqd/100:12:281
786243994815,7cyclictest23-21ksoftirqd/122:02:511
786235994819,8cyclictest12-21ksoftirqd/020:38:460
786235994819,8cyclictest12-21ksoftirqd/020:38:450
786246994739,5cyclictest780258-21kworker/2:321:34:132
78624699471,4cyclictest131rcu_preempt23:42:312
78624399477,9cyclictest23-21ksoftirqd/122:40:051
786243994711,8cyclictest0-21swapper/123:46:591
786250994620,21cyclictest37-21ksoftirqd/321:23:203
786246994636,6cyclictest789055-21kworker/2:221:05:022
78624399467,6cyclictest23-21ksoftirqd/121:16:111
78624399465,7cyclictest23-21ksoftirqd/100:21:031
786250994529,13cyclictest37-21ksoftirqd/323:32:223
786250994529,13cyclictest37-21ksoftirqd/323:32:223
786250994512,23cyclictest792412-21modprobe21:10:003
786243994531,10cyclictest786700-21systemctl19:20:001
786235994525,14cyclictest3467602-21systemd-journal19:50:010
78623599451,40cyclictest0-21swapper/020:32:440
78625099443,30cyclictest37-21ksoftirqd/300:30:023
786250994431,9cyclictest37-21ksoftirqd/323:15:283
786250994431,10cyclictest876511-21sshd00:24:493
78625099441,5cyclictest0-21swapper/322:19:533
786246994430,10cyclictest3467602-21systemd-journal20:50:012
786246994430,10cyclictest3467602-21systemd-journal20:50:002
786243994412,6cyclictest0-21swapper/119:50:011
786243994411,9cyclictest0-21swapper/121:27:071
786250994332,8cyclictest3467602-21systemd-journal23:26:453
786246994335,5cyclictest789055-21kworker/2:220:20:082
786243994326,13cyclictest838192-21ssh22:55:391
786243994323,16cyclictest804896-21sshd21:38:451
78623599431,4cyclictest0-21swapper/020:21:240
78623599431,25cyclictest0-21swapper/020:50:240
786250994232,7cyclictest37-21ksoftirqd/322:32:413
786250994224,14cyclictest275-21dbus-daemon23:45:013
786250994224,12cyclictest3467602-21systemd-journal22:25:013
786250994219,9cyclictest309-21in:imuxsock22:56:443
78625099421,7cyclictest0-21swapper/322:51:383
78625099421,6cyclictest0-21swapper/321:46:473
78625099421,5cyclictest0-21swapper/300:38:003
786250994212,9cyclictest131rcu_preempt22:15:013
786250994212,9cyclictest131rcu_preempt22:15:013
78624699423,3cyclictest131rcu_preempt20:10:522
786243994225,14cyclictest825579-21ssh22:26:271
786250994129,9cyclictest37-21ksoftirqd/323:10:193
786250994128,10cyclictest37-21ksoftirqd/321:37:143
78625099411,5cyclictest0-21swapper/300:06:463
78624699412,7cyclictest3467602-21systemd-journal19:40:012
786243994126,12cyclictest23-21ksoftirqd/120:50:011
786243994126,12cyclictest23-21ksoftirqd/120:50:011
786250994031,7cyclictest37-21ksoftirqd/323:50:013
786250994031,6cyclictest831327-21sh22:40:003
786250994030,7cyclictest37-21ksoftirqd/321:17:553
786250994029,8cyclictest813970-21ssh22:00:003
786250994025,11cyclictest275-21dbus-daemon22:05:013
786250994023,13cyclictest37-21ksoftirqd/321:53:513
786250994023,13cyclictest275-21dbus-daemon20:10:013
78625099401,5cyclictest0-21swapper/322:28:433
786250993929,6cyclictest842719-21sh23:06:013
786250993928,8cyclictest37-21ksoftirqd/322:43:113
786250993928,8cyclictest37-21ksoftirqd/321:30:003
786250993927,9cyclictest37-21ksoftirqd/321:42:593
786250993924,11cyclictest275-21dbus-daemon20:19:593
786250993920,13cyclictest275-21dbus-daemon20:40:003
786250993920,13cyclictest275-21dbus-daemon20:40:003
786250993917,5cyclictest37-21ksoftirqd/323:39:133
78624699392,3cyclictest131rcu_preempt20:55:012
786243993930,6cyclictest788510-21modprobe19:55:011
786243993926,9cyclictest786953-21sshd19:24:451
786243993925,10cyclictest790059-21sshd20:21:451
78623599392,5cyclictest791380-21munin-run20:50:010
78623599392,5cyclictest791380-21munin-run20:50:000
786250993827,8cyclictest37-21ksoftirqd/300:14:273
786250993824,11cyclictest37-21ksoftirqd/320:55:013
786250993820,8cyclictest874509-21ssh00:20:013
78625099381,5cyclictest131rcu_preempt21:14:593
786243993826,8cyclictest787706-21sshd19:38:461
786243993824,11cyclictest23-21ksoftirqd/119:25:231
786243993824,10cyclictest791932-21cat21:00:021
786243993818,15cyclictest1-21systemd21:10:011
786235993825,9cyclictest12-21ksoftirqd/019:59:200
786250993725,9cyclictest850791-21systemd-run23:25:013
786250993724,10cyclictest37-21ksoftirqd/321:00:003
786250993722,10cyclictest0-21swapper/319:20:033
786250993721,11cyclictest0-21swapper/320:05:003
786250993721,11cyclictest0-21swapper/320:04:593
78625099371,5cyclictest0-21swapper/323:01:323
78625099371,5cyclictest0-21swapper/323:01:323
786243993723,10cyclictest789520-21sshd20:12:391
786243993723,10cyclictest3467602-21systemd-journal20:09:151
786235993724,10cyclictest275-21dbus-daemon20:10:010
786250993627,6cyclictest37-21ksoftirqd/319:40:003
786250993621,11cyclictest37-21ksoftirqd/320:00:003
786246993625,8cyclictest30-21ksoftirqd/219:55:022
786243993623,9cyclictest48-21kcompactd020:16:101
786243993623,10cyclictest3467602-21systemd-journal20:40:491
786243993621,11cyclictest364-21snmpd21:00:161
786250993525,8cyclictest37-21ksoftirqd/322:05:163
786250993523,10cyclictest37-21ksoftirqd/320:25:013
786243993523,9cyclictest790809-21sshd20:35:451
786243993523,9cyclictest790809-21sshd20:35:451
786243993521,9cyclictest48-21kcompactd019:14:021
78625099344,4cyclictest131rcu_preempt21:31:003
786250993418,13cyclictest37-21ksoftirqd/320:50:013
786250993418,13cyclictest37-21ksoftirqd/320:50:013
78625099341,7cyclictest0-21swapper/320:12:393
786243993423,8cyclictest23-21ksoftirqd/119:35:021
786243993422,8cyclictest23-21ksoftirqd/120:34:591
786243993414,15cyclictest789057-21systemctl20:05:011
786243993414,15cyclictest789057-21systemctl20:05:011
786250993324,6cyclictest37-21ksoftirqd/320:30:043
786250993321,8cyclictest309-21in:imuxsock19:35:003
786243993318,11cyclictest364-21snmpd19:55:541
786250993223,5cyclictest787161-21latency_hist19:25:013
786250993222,6cyclictest309-21in:imuxsock19:50:023
786250993221,9cyclictest37-21ksoftirqd/321:04:453
786250993119,10cyclictest37-21ksoftirqd/320:45:003
786250993118,9cyclictest309-21in:imuxsock19:25:023
78625099308,17cyclictest275-21dbus-daemon20:30:013
78624699301,3cyclictest131rcu_preempt19:20:032
78625099291,5cyclictest131rcu_preempt19:50:003
78624399298,18cyclictest786249-21cut19:10:011
786246992518,4cyclictest786336-21cat19:10:012
786250992415,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional