You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-30 - 17:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 30, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31563499767,12cyclictest131rcu_preempt11:20:021
31563499767,12cyclictest131rcu_preempt11:20:011
315643997538,26cyclictest37-21ksoftirqd/308:20:023
315634997323,17cyclictest365560-21ssh10:39:011
315634997323,17cyclictest365560-21ssh10:39:001
315634997213,19cyclictest329021-21ssh09:22:281
315634997117,20cyclictest369336-21ssh10:46:391
315634997117,20cyclictest369336-21ssh10:46:381
315629997117,50cyclictest12-21ksoftirqd/011:01:220
31562999711,67cyclictest0-21swapper/010:40:020
315643997052,4cyclictest37-21ksoftirqd/311:50:033
315643997031,8cyclictest275-21dbus-daemon11:25:013
31562999709,40cyclictest12-21ksoftirqd/010:18:180
315629997026,37cyclictest3467602-21systemd-journal12:20:010
315629997026,37cyclictest3467602-21systemd-journal12:20:010
315634996914,18cyclictest417498-21ssh12:28:221
315634996914,15cyclictest336467-21ssh09:37:501
315634996812,21cyclictest332453-21ssh09:29:501
315629996816,47cyclictest16850irq/134-800000010:35:020
315629996813,51cyclictest12-21ksoftirqd/010:28:510
315634996720,14cyclictest358381-21ssh10:24:011
315634996720,11cyclictest408185-21ssh12:08:521
315634996720,11cyclictest408185-21ssh12:08:511
315634996719,14cyclictest353523-21ssh10:13:421
315629996723,9cyclictest3467602-21systemd-journal08:19:590
315629996718,44cyclictest12-21ksoftirqd/010:10:010
315629996718,44cyclictest12-21ksoftirqd/008:30:120
315634996614,14cyclictest348289-21ssh10:02:471
315629996658,5cyclictest12-21ksoftirqd/009:30:020
315634996421,7cyclictest349905-21ssh10:05:501
315634996419,10cyclictest321262-21systemd-xdg-aut08:57:091
315634996418,14cyclictest359687-21sh10:26:281
315634996413,14cyclictest373601-21rm10:55:381
315634996413,14cyclictest373601-21rm10:55:371
315629996443,18cyclictest12-21ksoftirqd/010:12:410
315634996319,7cyclictest400072-21ssh11:51:401
315634996310,19cyclictest392201-21ssh11:35:051
31562999633,55cyclictest335098-21grep09:35:020
315629996319,40cyclictest337442-21munin-plugin-st09:40:000
315634996210,19cyclictest354764-21ssh10:16:111
315634996210,19cyclictest319165-21modprobe08:17:011
315638996151,6cyclictest324593-21kworker/2:309:57:002
315638996150,7cyclictest324593-21kworker/2:310:19:182
315634996110,19cyclictest390076-21ssh11:30:361
315629996119,37cyclictest12-21ksoftirqd/009:14:590
315629996118,39cyclictest12-21ksoftirqd/012:25:120
31563899608,9cyclictest30-21ksoftirqd/211:35:002
31563499609,15cyclictest422751-21ssh12:39:241
315634996010,17cyclictest364-21snmpd10:33:141
315638995950,6cyclictest324593-21kworker/2:311:28:422
315634995919,14cyclictest380519-21ssh11:10:331
315634995910,15cyclictest325721-21ssh09:15:171
31563499588,14cyclictest343757-21ssh09:53:111
315634995815,13cyclictest333446-21sh09:31:281
315634995813,13cyclictest372526-21ssh10:53:431
315629995826,15cyclictest397646-21ssh11:46:330
315629995729,11cyclictest414797-21ssh12:22:370
315629995726,3cyclictest3657241ssh10:39:290
315629995726,3cyclictest3657241ssh10:39:290
315629995714,37cyclictest1-21systemd09:10:010
315629995714,37cyclictest1-21systemd09:10:000
315638995648,5cyclictest324593-21kworker/2:310:32:482
315638995648,5cyclictest321706-21kworker/2:009:23:182
315638995647,5cyclictest382461-21kworker/2:012:06:362
315638995647,5cyclictest382461-21kworker/2:012:06:352
31563499569,16cyclictest399050-21sh11:49:501
315634995643,10cyclictest316883-21munin-run07:35:001
315634995618,13cyclictest385300-21ssh11:20:261
315634995617,9cyclictest405441-21ssh12:03:021
315634995616,14cyclictest0-21swapper/112:30:311
315634995616,12cyclictest325362-21latency_hist09:15:021
315629995619,19cyclictest16850irq/134-800000011:17:160
315629995619,19cyclictest16850irq/134-800000011:17:160
315629995616,36cyclictest12-21ksoftirqd/009:21:070
31562999561,24cyclictest131rcu_preempt12:05:010
315638995547,5cyclictest339905-21kworker/2:210:09:582
315638995546,6cyclictest306965-21kworker/2:308:25:332
315638995546,6cyclictest306965-21kworker/2:308:25:322
315634995513,9cyclictest0-21swapper/109:55:081
315634995512,10cyclictest0-21swapper/111:57:221
315634995512,10cyclictest0-21swapper/111:57:221
31562999555,27cyclictest12-21ksoftirqd/010:20:020
315629995538,13cyclictest275-21dbus-daemon08:57:590
315629995528,11cyclictest420888-21ssh12:35:020
315629995528,11cyclictest343777-21ssh09:53:130
315629995527,12cyclictest402679-21ssh11:57:200
315629995527,12cyclictest402679-21ssh11:57:200
315629995519,18cyclictest16850irq/134-800000011:10:540
315629995518,6cyclictest16850irq/134-800000011:32:420
315638995446,5cyclictest324593-21kworker/2:310:44:302
315638995445,5cyclictest324593-21kworker/2:311:05:002
315638995445,5cyclictest324593-21kworker/2:309:54:372
315634995434,15cyclictest3467602-21systemd-journal09:05:031
315634995434,15cyclictest3467602-21systemd-journal09:05:021
315629995427,13cyclictest407586-21ssh12:07:270
315629995427,13cyclictest407586-21ssh12:07:270
315629995419,18cyclictest16850irq/134-800000009:49:270
315629995418,18cyclictest16850irq/134-800000011:37:100
315629995413,8cyclictest12-21ksoftirqd/011:25:000
315638995345,5cyclictest382461-21kworker/2:012:33:192
315638995345,5cyclictest324593-21kworker/2:310:36:502
315638995345,5cyclictest324593-21kworker/2:310:36:502
315638995345,5cyclictest306965-21kworker/2:308:20:012
315638995344,6cyclictest382461-21kworker/2:012:35:512
315638995344,6cyclictest321706-21kworker/2:009:29:022
315638995344,5cyclictest324593-21kworker/2:311:49:592
31563499539,15cyclictest411798-21ssh12:16:161
31563499539,15cyclictest411798-21ssh12:16:151
315634995335,14cyclictest388717-21ssh11:27:551
315634995328,21cyclictest320778-21munin-run08:50:011
315634995313,10cyclictest0-21swapper/112:12:371
315629995327,11cyclictest369817-21ssh10:47:510
315629995327,11cyclictest369817-21ssh10:47:510
31562999532,45cyclictest401616-21latency_hist11:55:020
315629995323,15cyclictest347375-21ssh10:00:300
315629995322,16cyclictest379971-21ssh11:09:450
315629995319,17cyclictest16850irq/134-800000011:28:260
315629995319,16cyclictest16850irq/134-800000011:41:130
315629995318,6cyclictest16850irq/134-800000010:57:490
315629995318,6cyclictest16850irq/134-800000010:57:480
315629995318,18cyclictest16850irq/134-800000010:51:130
315629995317,18cyclictest16850irq/134-800000009:55:100
315643995212,25cyclictest320735-21kworker/u8:009:05:023
315638995244,5cyclictest382461-21kworker/2:012:20:392
315638995244,5cyclictest382461-21kworker/2:012:11:032
315638995244,5cyclictest339905-21kworker/2:210:05:022
315638995244,5cyclictest324593-21kworker/2:310:46:552
315638995244,5cyclictest324593-21kworker/2:310:46:542
315638995243,6cyclictest324593-21kworker/2:310:53:082
315638995243,5cyclictest382461-21kworker/2:012:25:292
315638995243,5cyclictest324593-21kworker/2:311:16:432
315638995243,5cyclictest324593-21kworker/2:311:16:422
315638995243,5cyclictest324593-21kworker/2:309:40:132
315638995243,5cyclictest306965-21kworker/2:308:51:102
315638995242,6cyclictest361190-21kworker/2:111:13:362
315634995238,11cyclictest376431-21ssh11:02:121
315634995234,14cyclictest366900-21ssh10:41:351
315634995228,11cyclictest415979-21ssh12:25:021
315634995216,7cyclictest340781-21ssh09:46:361
315634995214,10cyclictest0-21swapper/111:09:281
315629995218,17cyclictest16850irq/134-800000012:14:450
315629995217,18cyclictest16850irq/134-800000009:41:460
315638995143,5cyclictest324593-21kworker/2:311:54:442
315638995143,5cyclictest324593-21kworker/2:310:24:172
315638995143,5cyclictest306965-21kworker/2:307:50:012
315638995142,5cyclictest306965-21kworker/2:308:40:002
315638995140,7cyclictest324593-21kworker/2:309:48:402
315638995134,8cyclictest339905-21kworker/2:210:12:362
315634995137,12cyclictest395184-21ssh11:41:231
315634995137,10cyclictest319714-21cron08:30:011
315634995137,10cyclictest319714-21cron08:30:001
315634995135,13cyclictest339612-21ssh09:44:301
315634995132,13cyclictest275-21dbus-daemon07:55:021
315634995130,17cyclictest318193-21munin-run08:00:001
315629995141,6cyclictest320235-21sendmail08:40:010
315629995135,13cyclictest12-21ksoftirqd/012:30:150
315638995043,4cyclictest382461-21kworker/2:012:00:142
315638995042,5cyclictest382461-21kworker/2:011:41:392
315638995042,5cyclictest361190-21kworker/2:111:05:312
315638995042,5cyclictest324593-21kworker/2:311:55:362
315638995042,5cyclictest324593-21kworker/2:311:55:362
315638995042,5cyclictest324593-21kworker/2:310:28:022
315638995042,5cyclictest321706-21kworker/2:009:14:082
315638995041,6cyclictest382461-21kworker/2:011:35:582
315638995041,6cyclictest321706-21kworker/2:009:17:272
315638995041,5cyclictest324593-21kworker/2:310:58:102
315638995041,5cyclictest324593-21kworker/2:310:58:092
315638995041,5cyclictest321706-21kworker/2:009:05:012
315629995016,18cyclictest16850irq/134-800000009:19:410
315643994932,13cyclictest3467602-21systemd-journal11:40:033
315643994931,14cyclictest3467602-21systemd-journal07:28:123
315638994941,5cyclictest382461-21kworker/2:012:16:072
315638994941,5cyclictest382461-21kworker/2:012:16:062
315638994941,5cyclictest321706-21kworker/2:009:35:142
315638994941,5cyclictest321706-21kworker/2:009:30:232
315638994932,11cyclictest275-21dbus-daemon07:25:012
315634994935,11cyclictest318901-21sshd08:10:101
315643994832,12cyclictest284-21systemd-logind08:57:373
315643994829,15cyclictest3467602-21systemd-journal09:40:013
315643994819,8cyclictest37-21ksoftirqd/309:43:173
315638994839,5cyclictest324593-21kworker/2:311:22:312
315638994832,12cyclictest3467602-21systemd-journal08:57:442
315634994833,11cyclictest318650-21sshd08:05:121
315629994840,4cyclictest16950irq/224-800000007:40:010
315629994840,4cyclictest16950irq/224-800000007:40:000
315643994731,12cyclictest1-21systemd08:15:003
315643994720,9cyclictest37-21ksoftirqd/312:38:243
315638994739,5cyclictest306965-21kworker/2:308:15:032
315638994739,5cyclictest306965-21kworker/2:308:05:002
315638994739,5cyclictest306965-21kworker/2:307:40:012
315638994739,5cyclictest306965-21kworker/2:307:40:002
315638994729,13cyclictest1-21systemd08:45:022
315638994729,13cyclictest1-21systemd08:45:012
315634994733,11cyclictest319947-21sshd08:30:121
315634994733,11cyclictest319454-21sshd08:23:121
315634994728,15cyclictest3467602-21systemd-journal07:15:431
31562999474,37cyclictest0-21swapper/007:20:000
315638994638,5cyclictest306965-21kworker/2:308:25:022
315638994638,5cyclictest306965-21kworker/2:307:30:022
31562999463,36cyclictest1-21systemd08:30:010
31562999463,36cyclictest1-21systemd08:30:000
315629994628,14cyclictest275-21dbus-daemon07:50:010
315629994627,14cyclictest3467602-21systemd-journal07:30:020
31564399452,4cyclictest131rcu_preempt11:36:243
315638994537,5cyclictest306965-21kworker/2:307:15:022
315634994532,10cyclictest320512-21munin-run08:45:011
315634994532,10cyclictest320512-21munin-run08:45:001
315634994530,12cyclictest316592-21sshd07:28:121
315634994526,15cyclictest1-21systemd07:40:011
315634994526,15cyclictest1-21systemd07:40:001
315629994538,4cyclictest16950irq/224-800000008:00:000
315643994432,9cyclictest1-21systemd10:20:013
315638994435,6cyclictest306965-21kworker/2:307:55:022
315634994432,9cyclictest315789-21sshd07:10:121
315634994431,9cyclictest48-21kcompactd008:39:581
315634994431,9cyclictest3467602-21systemd-journal08:54:121
31564399431,3cyclictest131rcu_preempt12:05:023
31564399431,36cyclictest309-21in:imuxsock07:25:003
315638994332,7cyclictest306965-21kworker/2:307:45:022
315638994328,12cyclictest318140-21cron08:00:012
315634994335,4cyclictest316330-21sshd07:20:111
315629994332,7cyclictest317400-21munin-plugin-st07:44:590
315643994231,8cyclictest37-21ksoftirqd/310:47:553
315643994231,8cyclictest37-21ksoftirqd/310:47:543
315643994230,9cyclictest329787-21sshd09:24:183
315643994229,11cyclictest37-21ksoftirqd/310:38:053
315643994229,11cyclictest37-21ksoftirqd/310:38:043
315643994228,10cyclictest309-21in:imuxsock11:02:103
315643994227,12cyclictest37-21ksoftirqd/311:08:023
315643994226,14cyclictest37-21ksoftirqd/312:26:503
315643994226,11cyclictest37-21ksoftirqd/309:53:073
315643994223,14cyclictest275-21dbus-daemon07:50:013
31564399421,3cyclictest131rcu_preempt10:32:193
315638994230,9cyclictest3467602-21systemd-journal08:33:122
315638994230,9cyclictest322890-21sshd09:08:122
315638994230,9cyclictest322890-21sshd09:08:122
315638994227,12cyclictest30-21ksoftirqd/208:50:002
315634994228,10cyclictest322638-21sshd09:03:111
315629994235,4cyclictest16950irq/224-800000008:02:120
315629994226,12cyclictest316602-21sshd07:29:570
315643994130,7cyclictest346896-21sendmail10:00:003
315643994129,8cyclictest37-21ksoftirqd/311:20:033
315643994129,8cyclictest37-21ksoftirqd/311:20:023
315643994123,14cyclictest275-21dbus-daemon12:15:013
315634994124,13cyclictest318586-21sed08:05:031
315643994026,5cyclictest318151-21sh07:59:593
315643994026,11cyclictest37-21ksoftirqd/311:28:193
315643994026,10cyclictest37-21ksoftirqd/311:11:443
315643994025,11cyclictest37-21ksoftirqd/310:04:403
315643994025,11cyclictest37-21ksoftirqd/309:15:323
31564399402,3cyclictest131rcu_preempt10:11:453
31564399401,6cyclictest0-21swapper/311:55:003
31564399401,10cyclictest37-21ksoftirqd/308:34:593
315638994032,6cyclictest306965-21kworker/2:307:35:012
315629994025,12cyclictest317876-21sshd07:52:110
315643993931,5cyclictest320761-21latency_hist08:50:003
315643993927,9cyclictest37-21ksoftirqd/312:25:013
315643993927,8cyclictest37-21ksoftirqd/310:45:013
315643993925,11cyclictest316883-21pluginstate07:35:013
315643993924,12cyclictest37-21ksoftirqd/312:10:023
315643993924,12cyclictest37-21ksoftirqd/312:10:013
315643993924,12cyclictest3467602-21systemd-journal09:30:003
315643993922,12cyclictest1-21systemd07:55:013
315638993931,5cyclictest306965-21kworker/2:307:20:002
315634993929,7cyclictest3467602-21systemd-journal07:45:281
31562999394,9cyclictest0-21swapper/007:24:110
31562999392,33cyclictest0-21swapper/009:05:020
31564399383,4cyclictest131rcu_preempt08:05:033
315643993826,9cyclictest340192-21rm09:45:233
315643993825,9cyclictest3467602-21systemd-journal11:31:113
315643993825,10cyclictest37-21ksoftirqd/310:08:523
315643993824,11cyclictest374296-21sh10:57:143
315643993824,11cyclictest374296-21sh10:57:143
315643993824,11cyclictest37-21ksoftirqd/309:10:163
315643993823,11cyclictest37-21ksoftirqd/310:25:033
315643993822,12cyclictest1-21systemd12:35:013
315643993817,16cyclictest275-21dbus-daemon08:30:013
315643993817,16cyclictest275-21dbus-daemon08:30:013
315634993824,10cyclictest364-21snmpd07:40:491
315634993821,13cyclictest315636-21latency_hist07:10:011
31564399376,6cyclictest131rcu_preempt08:25:003
315643993729,5cyclictest37-21ksoftirqd/308:50:153
315643993725,10cyclictest37-21ksoftirqd/310:51:223
315643993724,8cyclictest3467602-21systemd-journal12:15:023
315643993724,8cyclictest3467602-21systemd-journal12:15:013
31564399372,3cyclictest131rcu_preempt09:30:203
315629993721,12cyclictest320933-21sed08:50:020
315629993720,13cyclictest3467602-21systemd-journal07:15:010
315643993626,5cyclictest309-21in:imuxsock08:45:003
315643993626,5cyclictest309-21in:imuxsock08:45:003
315643993624,9cyclictest37-21ksoftirqd/310:20:353
315643993623,10cyclictest37-21ksoftirqd/311:57:363
315643993623,10cyclictest37-21ksoftirqd/311:57:353
315643993623,10cyclictest37-21ksoftirqd/307:20:003
315638993623,9cyclictest3467602-21systemd-journal08:10:002
31562999362,6cyclictest318647-21sshd08:05:060
315643993526,5cyclictest317139-21sendmail07:40:013
315643993526,5cyclictest317139-21sendmail07:40:003
315643993518,13cyclictest3467602-21systemd-journal07:40:113
315629993527,5cyclictest19350irq/34-mmc108:55:020
315629993526,6cyclictest12-21ksoftirqd/008:15:030
31562999351,30cyclictest0-21swapper/008:40:050
31562999351,30cyclictest0-21swapper/008:40:040
315643993425,7cyclictest37-21ksoftirqd/309:07:123
315643993425,7cyclictest37-21ksoftirqd/309:07:113
315643993421,10cyclictest37-21ksoftirqd/307:15:033
315638993424,8cyclictest30-21ksoftirqd/207:10:022
31562999334,23cyclictest315633-21latency_hist07:10:020
315629993322,9cyclictest12-21ksoftirqd/008:25:000
31564399324,3cyclictest131rcu_preempt08:40:013
315643993023,4cyclictest318679-21latency_hist08:10:013
315643992920,6cyclictest315570-21latency_hist07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional