You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-15 - 20:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Aug 15, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10569539910093,4cyclictest1077396-21sh09:40:030
105695699842,43cyclictest1157012-21ssh12:29:201
105695699832,47cyclictest1063669-21ssh09:10:391
105695699831,44cyclictest1094193-21rm10:15:371
105695399821,40cyclictest1153283-21ssh12:21:070
105695699811,45cyclictest1151319-21ssh12:17:091
1056953998131,12cyclictest1151042-21ssh12:16:270
1056953998127,12cyclictest1080095-21ssh09:45:470
105695399801,37cyclictest1099501-21ssh10:26:550
105695399801,37cyclictest1099501-21ssh10:26:550
1056953997828,13cyclictest1114161-21ssh10:58:200
1056953997828,13cyclictest1114161-21ssh10:58:190
1056953997827,12cyclictest1069584-21ssh09:23:330
1056956997726,28cyclictest23-21ksoftirqd/110:06:551
1056956997726,28cyclictest23-21ksoftirqd/110:06:541
105695399761,10cyclictest1157262-21sh12:30:000
105695699751,38cyclictest1146021-21ssh12:05:431
1056953997527,13cyclictest1094848-21ssh10:17:070
105695399751,39cyclictest1138027-21ssh11:48:560
1056953997425,14cyclictest1087808-21ssh10:02:120
105695399741,37cyclictest1106665-21ssh10:42:010
105695699732,38cyclictest1112509-21ssh10:54:511
1056953997327,13cyclictest1127867-21ssh11:27:190
105695399731,41cyclictest48-21kcompactd009:15:090
1056956997227,39cyclictest309-21in:imuxsock11:40:001
1056953997232,7cyclictest19350irq/34-mmc108:45:030
105695399721,68cyclictest0-21swapper/011:45:020
105695699712,40cyclictest1082756-21ssh09:51:271
105695699711,45cyclictest1103543-21ssh10:35:201
105695699711,45cyclictest1103543-21ssh10:35:191
105695699711,41cyclictest1098314-21ssh10:24:421
105695699711,39cyclictest1143975-21ssh12:01:241
1056953997128,12cyclictest1089646-21ssh10:05:520
1056953997128,12cyclictest1089646-21ssh10:05:510
1056953997125,32cyclictest12-21ksoftirqd/010:11:500
105695699701,43cyclictest1086065-21ssh09:58:361
105695699701,42cyclictest1128315-21ssh11:28:221
105695399701,40cyclictest1146718-21ssh12:07:240
105695699691,43cyclictest1070780-21ssh09:25:561
1056953996937,29cyclictest16950irq/224-800000008:00:000
105695699681,38cyclictest364-21snmpd07:37:421
105695699671,62cyclictest1069217-21ssh09:22:481
105695699671,41cyclictest1149538-21ssh12:13:351
1056953996734,8cyclictest1131384-21munin-plugin-st11:35:000
105695399672,14cyclictest1082338-21ssh09:50:240
105695699661,61cyclictest1141945-21ssh11:57:091
105695699661,61cyclictest1121340-21ssh11:13:331
105695699661,61cyclictest1080105-21ssh09:45:511
105695699661,61cyclictest1078702-21ssh09:42:591
105695699651,60cyclictest1138077-21ssh11:49:031
105695699651,60cyclictest1116623-21ssh11:03:341
105695699651,60cyclictest1099507-21ssh10:26:551
105695699651,60cyclictest1099507-21ssh10:26:551
105695699651,60cyclictest1073189-21ssh09:31:011
1056953996527,4cyclictest131rcu_preempt11:01:580
105695699641,59cyclictest3467602-21systemd-journal10:11:111
105695699641,59cyclictest3467602-21systemd-journal09:38:561
105695699641,59cyclictest1161439-21ssh12:38:421
105695699641,59cyclictest1088385-21ssh10:03:341
105695399643,40cyclictest19350irq/34-mmc108:05:020
105695699632,58cyclictest1124598-21ssh11:20:031
105695699632,57cyclictest1122703-21ssh11:16:181
105695699632,57cyclictest1057945-21munin-run07:30:001
105695699631,58cyclictest1129509-21ssh11:30:351
105695699631,58cyclictest1119087-21ssh11:08:461
105695699631,58cyclictest1114415-21ssh10:58:541
105695699631,58cyclictest1114415-21ssh10:58:531
105695699631,58cyclictest1108220-21ssh10:45:101
105695699631,58cyclictest1108220-21ssh10:45:091
105695699631,58cyclictest1102118-21ssh10:32:321
105695699631,58cyclictest1066418-21ssh09:16:431
105695699631,39cyclictest1105952-21ssh10:40:261
1056953996332,14cyclictest1117900-21ssh11:05:550
105695699622,57cyclictest1157847-21ssh12:30:491
105695699621,57cyclictest1153496-21ssh12:21:461
105695699621,57cyclictest1138822-21ssh11:50:191
105695699621,57cyclictest1134920-21ssh11:42:031
105695399621,39cyclictest1097600-21ssh10:23:020
105695699612,55cyclictest1060554-21munin-run08:20:021
105695699602,52cyclictest1061295-21munin-run08:35:011
1056960995951,5cyclictest1072517-21kworker/2:209:50:562
105695699592,52cyclictest1061840-21munin-run08:45:011
105695699591,54cyclictest1060214-21sshd08:10:101
105695699591,54cyclictest1059998-21munin-run08:10:011
105695699591,54cyclictest1059998-21munin-run08:10:001
1056953995941,15cyclictest12-21ksoftirqd/009:38:070
1056956995830,24cyclictest23-21ksoftirqd/108:50:001
1056956995830,24cyclictest23-21ksoftirqd/108:50:001
1056953995825,14cyclictest1121370-21ssh11:13:360
1056953995818,21cyclictest16850irq/134-800000012:33:540
105695699572,51cyclictest1058965-21munin-run07:50:011
1056964995639,13cyclictest275-21dbus-daemon09:55:003
105695699562,49cyclictest1062844-21sshd09:03:291
105695699561,50cyclictest1058196-21munin-run07:35:021
1056953995628,13cyclictest1141622-21ssh11:56:280
105696499554,38cyclictest1058485-21kworker/3:209:00:013
1056960995546,6cyclictest1070509-21kworker/2:012:09:342
1056960995546,6cyclictest1070509-21kworker/2:010:56:272
1056960995546,6cyclictest1070509-21kworker/2:010:56:262
1056960995545,5cyclictest1070509-21kworker/2:011:57:002
105695699551,50cyclictest1-21systemd07:55:011
105695699551,50cyclictest1-21systemd07:55:011
105695699551,49cyclictest364-21snmpd09:07:131
1056953995528,12cyclictest1126137-21ssh11:23:480
1056953995527,24cyclictest1070837-21ssh09:26:040
1056960995446,6cyclictest1119772-21kworker/2:112:29:172
1056960995446,5cyclictest1070509-21kworker/2:009:35:082
1056960995445,6cyclictest1072517-21kworker/2:209:55:052
1056960995445,6cyclictest1070509-21kworker/2:011:25:532
105695699541,49cyclictest1058697-21sshd07:44:261
1056953995426,13cyclictest1139846-21ssh11:52:400
105696499532,39cyclictest361rcuc/307:35:023
1056960995345,5cyclictest1070509-21kworker/2:011:15:152
1056960995345,5cyclictest1070509-21kworker/2:011:13:022
1056960995344,6cyclictest1070509-21kworker/2:012:02:002
1056960995344,6cyclictest1070509-21kworker/2:011:50:082
1056960995342,7cyclictest1070509-21kworker/2:009:27:592
1056960995342,7cyclictest1049144-21kworker/2:109:00:292
1056960995337,13cyclictest3467602-21systemd-journal08:20:002
105695699532,47cyclictest1062339-21systemd-run08:55:011
105695699532,47cyclictest1057383-21sshd07:19:271
105695699531,48cyclictest1060756-21sshd08:20:141
105695699531,3cyclictest48-21kcompactd007:55:141
1056960995244,5cyclictest1119772-21kworker/2:112:16:022
1056960995244,5cyclictest1072517-21kworker/2:210:00:112
1056960995244,5cyclictest1070509-21kworker/2:011:33:082
1056960995244,5cyclictest1070509-21kworker/2:010:36:062
1056960995244,5cyclictest1070509-21kworker/2:010:36:052
1056960995244,5cyclictest1070509-21kworker/2:010:31:092
1056960995243,6cyclictest1119772-21kworker/2:112:30:572
1056960995243,6cyclictest1072517-21kworker/2:210:18:472
1056960995243,6cyclictest1049144-21kworker/2:109:25:032
1056960995243,6cyclictest1049144-21kworker/2:108:40:022
1056960995242,6cyclictest1070509-21kworker/2:010:27:532
1056960995242,6cyclictest1070509-21kworker/2:010:27:522
105695699521,4cyclictest1057111-21sshd07:11:261
1056953995229,8cyclictest1123176-21ssh11:17:180
105695399521,36cyclictest1149781-21ssh12:14:060
1056960995143,5cyclictest1072517-21kworker/2:210:06:302
1056960995143,5cyclictest1072517-21kworker/2:210:06:292
1056960995143,5cyclictest1070509-21kworker/2:011:06:082
1056960995143,5cyclictest1070509-21kworker/2:011:01:352
1056960995143,5cyclictest1070509-21kworker/2:009:42:032
1056960995143,5cyclictest1070509-21kworker/2:009:32:122
1056960995142,6cyclictest1072517-21kworker/2:209:49:322
1056960995142,6cyclictest1070509-21kworker/2:011:35:472
1056960995142,5cyclictest1070509-21kworker/2:011:46:412
1056960995141,7cyclictest1070509-21kworker/2:012:13:302
1056953995120,7cyclictest12-21ksoftirqd/007:52:460
1056953995120,7cyclictest12-21ksoftirqd/007:52:450
105695399511,29cyclictest0-21swapper/010:30:430
1056964995033,13cyclictest275-21dbus-daemon07:20:023
1056960995042,5cyclictest1072517-21kworker/2:210:20:042
1056960995042,5cyclictest1072517-21kworker/2:210:11:112
1056960995042,5cyclictest1070509-21kworker/2:012:37:572
1056960995042,5cyclictest1070509-21kworker/2:011:20:472
1056960995042,5cyclictest1070509-21kworker/2:010:51:372
1056960995042,5cyclictest1070509-21kworker/2:010:45:482
1056960995042,5cyclictest1070509-21kworker/2:010:45:472
1056960995042,5cyclictest1060265-21kworker/2:309:16:422
1056960995041,6cyclictest1049144-21kworker/2:109:14:242
1056960995040,6cyclictest1070509-21kworker/2:011:41:202
1056960995033,11cyclictest3467602-21systemd-journal07:15:002
105695699501,6cyclictest120-21kswapd009:00:011
105695699501,45cyclictest1057649-21sendmail-mta07:20:501
1056953995025,3cyclictest0-21swapper/010:36:450
1056953995025,3cyclictest0-21swapper/010:36:450
1056953995023,11cyclictest1133189-21ssh11:38:410
1056953995020,5cyclictest12-21ksoftirqd/012:35:280
105695399501,34cyclictest1112480-21ssh10:54:470
1056960994941,5cyclictest1119772-21kworker/2:112:21:062
1056960994941,5cyclictest1049144-21kworker/2:108:00:012
1056960994941,5cyclictest1049144-21kworker/2:107:15:132
105695699492,43cyclictest1061133-21latency_hist08:30:011
105695699491,44cyclictest1061527-21modprobe08:38:291
105695699491,44cyclictest1059722-21sshd08:02:271
105695399492,9cyclictest1110250-21munin-plugin-st10:50:010
105695399492,9cyclictest1110250-21munin-plugin-st10:50:000
1056953994926,9cyclictest1084722-21ssh09:55:260
1056964994834,11cyclictest37-21ksoftirqd/311:25:023
1056960994840,5cyclictest1049144-21kworker/2:108:40:292
1056960994840,5cyclictest1049144-21kworker/2:108:30:002
1056960994840,5cyclictest1049144-21kworker/2:107:39:262
1056960994824,12cyclictest1106032-21rm10:40:362
105695399482,14cyclictest1145546-21munin-run12:05:000
1056953994818,14cyclictest1073552-21ssh09:31:570
1056953994818,14cyclictest1064095-21ssh09:11:480
1056964994728,14cyclictest275-21dbus-daemon12:05:003
1056960994739,5cyclictest1049144-21kworker/2:108:23:292
1056960994738,5cyclictest1049144-21kworker/2:109:00:012
1056960994737,6cyclictest1049144-21kworker/2:108:30:432
1056953994728,4cyclictest16850irq/134-800000008:20:270
1056960994638,5cyclictest1049144-21kworker/2:109:09:062
1056960994638,5cyclictest1049144-21kworker/2:107:47:272
1056953994634,9cyclictest1060516-21cron08:20:000
1056960994537,5cyclictest1049144-21kworker/2:107:31:462
1056960994537,5cyclictest1049144-21kworker/2:107:22:112
105695399454,35cyclictest0-21swapper/009:10:010
1056953994528,13cyclictest3467602-21systemd-journal08:40:020
1056953994514,26cyclictest0-21swapper/008:27:110
105696499444,3cyclictest131rcu_preempt10:19:353
1056964994433,8cyclictest3467602-21systemd-journal12:20:003
1056964994428,13cyclictest1105492-21cron10:40:013
1056964994428,13cyclictest1105492-21cron10:40:003
1056964994428,12cyclictest3467602-21systemd-journal12:30:013
1056960994436,5cyclictest1049144-21kworker/2:108:54:372
1056964994333,7cyclictest37-21ksoftirqd/307:14:593
1056960994335,5cyclictest1049144-21kworker/2:108:09:282
1056960994335,5cyclictest1049144-21kworker/2:108:09:282
105695699432,36cyclictest1056880-21latency_hist07:10:031
105695699432,36cyclictest1056880-21latency_hist07:10:021
105695399432,6cyclictest1058429-21modprobe07:40:000
1056964994228,10cyclictest1-21systemd12:00:023
1056964994222,14cyclictest3467602-21systemd-journal07:30:003
105695399426,19cyclictest16950irq/224-800000007:15:020
1056953994229,9cyclictest12-21ksoftirqd/008:15:020
1056953994222,14cyclictest275-21dbus-daemon08:35:000
105696499414,3cyclictest131rcu_preempt11:44:153
1056964994130,8cyclictest1069077-21sshd09:22:303
1056964994129,8cyclictest1-21systemd11:40:013
1056964994129,10cyclictest37-21ksoftirqd/310:07:523
1056964994129,10cyclictest37-21ksoftirqd/310:07:523
1056964994127,10cyclictest37-21ksoftirqd/309:57:403
105696499411,6cyclictest0-21swapper/310:02:383
1056960994134,5cyclictest1049144-21kworker/2:107:42:552
1056960994134,4cyclictest1049144-21kworker/2:108:47:502
1056960994134,4cyclictest1049144-21kworker/2:108:47:492
1056953994132,6cyclictest19350irq/34-mmc109:04:120
1056953994125,11cyclictest12-21ksoftirqd/008:55:000
105696499403,4cyclictest131rcu_preempt11:00:293
1056964994029,9cyclictest37-21ksoftirqd/312:11:063
1056964994025,11cyclictest37-21ksoftirqd/310:56:163
1056964994025,11cyclictest37-21ksoftirqd/310:56:153
1056964994025,11cyclictest37-21ksoftirqd/309:27:203
105696499401,35cyclictest37-21ksoftirqd/308:30:013
1056960994025,11cyclictest3467602-21systemd-journal08:00:272
1056953994028,9cyclictest12-21ksoftirqd/009:00:010
1056964993927,9cyclictest1118560-21sshd11:07:303
105696499391,3cyclictest131rcu_preempt07:40:003
1056960993931,5cyclictest1049144-21kworker/2:107:50:092
1056960993931,5cyclictest1049144-21kworker/2:107:50:082
1056960993929,7cyclictest1058069-21latency_hist07:30:012
1056953993924,12cyclictest1-21systemd07:25:010
1056953993922,12cyclictest309-21in:imuxsock07:35:000
105696499383,13cyclictest0-21swapper/308:05:003
1056964993828,8cyclictest37-21ksoftirqd/311:54:453
1056964993828,7cyclictest1108019-21sed10:45:013
1056964993826,9cyclictest37-21ksoftirqd/312:32:173
1056964993826,9cyclictest37-21ksoftirqd/310:11:163
1056964993826,8cyclictest37-21ksoftirqd/308:20:003
1056964993824,11cyclictest37-21ksoftirqd/309:14:513
1056964993823,10cyclictest275-21dbus-daemon11:20:003
1056964993821,13cyclictest3467602-21systemd-journal08:50:303
1056964993811,6cyclictest131rcu_preempt11:48:283
1056964993810,14cyclictest0-21swapper/311:15:023
1056960993824,10cyclictest309-21in:imuxsock08:10:032
105695399383,6cyclictest131rcu_preempt07:10:020
105695399383,6cyclictest131rcu_preempt07:10:020
1056953993822,12cyclictest3467718-21lldpd07:50:010
105695399381,33cyclictest1057121-21sshd07:13:270
1056964993728,6cyclictest309-21in:imuxsock12:25:023
1056964993728,6cyclictest1-21systemd09:50:003
1056964993726,7cyclictest37-21ksoftirqd/308:50:023
1056964993726,7cyclictest37-21ksoftirqd/308:50:013
1056964993723,11cyclictest37-21ksoftirqd/310:20:143
1056964993722,9cyclictest0-21swapper/312:40:013
1056964993712,8cyclictest1129009-21cron11:30:003
105696499364,4cyclictest131rcu_preempt09:17:143
1056964993626,8cyclictest37-21ksoftirqd/309:38:583
1056964993625,9cyclictest37-21ksoftirqd/310:48:183
1056964993625,9cyclictest37-21ksoftirqd/310:48:173
1056964993625,8cyclictest37-21ksoftirqd/307:25:013
1056964993624,10cyclictest37-21ksoftirqd/310:31:243
1056964993624,10cyclictest37-21ksoftirqd/309:44:513
1056964993618,14cyclictest275-21dbus-daemon10:55:003
1056953993630,4cyclictest19350irq/34-mmc107:40:030
1056964993524,8cyclictest1059695-21sed08:00:023
105696499352,30cyclictest37-21ksoftirqd/311:33:063
1056964993521,11cyclictest37-21ksoftirqd/308:10:033
1056964993521,11cyclictest37-21ksoftirqd/308:10:023
105696499351,31cyclictest48-21kcompactd012:06:093
105695399351,31cyclictest1-21systemd08:05:290
105695399351,31cyclictest1-21systemd08:05:280
105695399351,2cyclictest0-21swapper/008:50:030
105695399351,2cyclictest0-21swapper/008:50:020
1056964993426,5cyclictest1099578-21sh10:27:053
1056964993426,5cyclictest1099578-21sh10:27:043
1056964993424,7cyclictest1058685-21sshd07:41:263
105696499342,3cyclictest131rcu_preempt09:30:303
1056964993419,11cyclictest1062027-21cut08:45:043
105695399341,30cyclictest0-21swapper/007:28:440
105696499333,3cyclictest131rcu_preempt08:35:003
1056964993326,4cyclictest1056990-21latency_hist07:10:033
1056964993326,4cyclictest1056990-21latency_hist07:10:023
1056964993317,13cyclictest3467602-21systemd-journal07:45:273
105696499331,5cyclictest0-21swapper/309:05:033
105696499324,21cyclictest1060786-21modprobe08:25:003
1056964993226,4cyclictest1061552-21latency_hist08:40:023
105696499321,28cyclictest48-21kcompactd008:11:183
105696099323,22cyclictest1057032-21wc07:10:022
105696099323,22cyclictest1057032-21wc07:10:022
1056964993117,11cyclictest1-21systemd09:10:013
1056964993016,11cyclictest3467602-21systemd-journal07:55:013
1056964993016,11cyclictest3467602-21systemd-journal07:55:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional