You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-29 - 18:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri May 29, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1279182999026,51cyclictest1285356-21diskmemload11:00:022
127917399883,16cyclictest1290034-21latency_hist09:20:030
1279173998316,6cyclictest16850irq/134-800000010:09:020
1279173998316,6cyclictest16850irq/134-800000010:09:010
1279173998028,48cyclictest1338491-21ssh11:08:030
127917399792,30cyclictest1365580-21ssh12:05:400
1279173997827,34cyclictest1286985-21ssh09:13:240
127918799772,44cyclictest1336256-21ssh11:03:023
127918799752,50cyclictest1284032-21sshd08:41:223
127917399741,56cyclictest1298501-21ssh09:39:110
127917399741,56cyclictest1298501-21ssh09:39:100
127917399733,5cyclictest1341590-21cat11:15:030
127917399732,66cyclictest1284854-21systemd-run09:00:000
127917399731,68cyclictest0-21swapper/010:24:390
127917399731,13cyclictest1316396-21modprobe10:18:580
127917399731,13cyclictest1316396-21modprobe10:18:570
1279187997217,50cyclictest1280933-21cron07:45:013
1279173997232,35cyclictest16950irq/224-800000011:55:010
127917399721,67cyclictest1302657-21ssh09:48:220
127918799711,39cyclictest1374593-21ssh12:25:483
1279178997127,10cyclictest23-21ksoftirqd/111:55:011
1279173997117,48cyclictest16850irq/134-800000012:00:000
127917399711,53cyclictest1322351-21ssh10:32:000
1279182997060,7cyclictest30-21ksoftirqd/211:25:012
1279173997030,35cyclictest16850irq/134-800000010:45:010
1279173997023,31cyclictest1319604-21ssh10:25:420
1279173997023,31cyclictest1319604-21ssh10:25:420
1279187996941,19cyclictest37-21ksoftirqd/310:55:023
127918799692,41cyclictest1327128-21ssh10:42:453
127918799691,63cyclictest1338600-21ssh11:08:183
127917399693,6cyclictest1369609-21munin-run12:15:010
127917399692,17cyclictest1363217-21modprobe12:00:550
1279173996919,10cyclictest1285340-21cron09:09:590
127918799681,35cyclictest1371030-21ssh12:18:053
127918799681,35cyclictest1371030-21ssh12:18:043
127917399687,55cyclictest0-21swapper/009:44:590
1279173996832,33cyclictest16950irq/224-800000007:55:010
1279173996815,13cyclictest1308444-21ssh10:00:570
1279173996813,12cyclictest1324921-21ssh10:37:490
127918799671,62cyclictest1333556-21ssh10:57:013
127918799671,60cyclictest1316143-21ssh10:18:313
127918799671,60cyclictest1316143-21ssh10:18:303
127918799671,34cyclictest131rcu_preempt12:20:533
127917399673,6cyclictest1355030-21munin-run11:45:010
1279173996732,30cyclictest16950irq/224-800000011:37:530
1279173996730,33cyclictest275-21dbus-daemon08:30:010
1279173996730,33cyclictest16850irq/134-800000011:05:000
1279173996723,31cyclictest0-21swapper/011:30:230
1279173996723,31cyclictest0-21swapper/011:30:230
1279173996717,2cyclictest0-21swapper/009:35:030
127918799661,61cyclictest1364428-21ssh12:03:273
127918799661,61cyclictest1356171-21sshd11:47:213
127918799661,61cyclictest1329193-21ssh10:47:153
127918799661,61cyclictest1306208-21ssh09:56:013
127918799661,60cyclictest1319294-21ssh10:25:033
127918799661,60cyclictest1319294-21ssh10:25:023
127918799661,60cyclictest1291846-21ssh09:24:153
1279187996615,23cyclictest1285356-21diskmemload10:00:423
127917399664,20cyclictest1279630-21modprobe07:20:000
1279173996629,33cyclictest12-21ksoftirqd/010:55:020
1279173996615,13cyclictest1283159-21latency_hist08:25:020
1279173996615,13cyclictest1283159-21latency_hist08:25:020
1279173996614,13cyclictest3467602-21systemd-journal12:28:530
127918799652,59cyclictest1352053-21ssh11:38:173
127918799652,59cyclictest1310535-21ssh10:05:383
127918799652,59cyclictest1310535-21ssh10:05:373
127918799652,59cyclictest1289031-21ssh09:17:593
127918799651,60cyclictest1377866-21ssh12:33:173
127918799651,60cyclictest1354083-21ssh11:42:453
127918799651,60cyclictest1347476-21ssh11:28:003
127918799651,60cyclictest1347476-21ssh11:27:593
127918799651,60cyclictest1299258-21ssh09:40:293
127918799651,60cyclictest1293826-21ssh09:28:413
127918799651,59cyclictest1359865-21ssh11:55:243
1279173996527,35cyclictest0-21swapper/012:36:160
127917399651,4cyclictest0-21swapper/011:27:110
127917399651,4cyclictest0-21swapper/011:27:100
127918799643,57cyclictest1368484-21sshd12:12:233
127918799642,57cyclictest1324029-21ssh10:35:363
127918799641,59cyclictest1345314-21ssh11:23:143
127918799641,59cyclictest1302453-21ssh09:47:543
127918799641,59cyclictest1295690-21ssh09:32:413
127918799641,59cyclictest1286580-21ssh09:12:283
127918799641,58cyclictest1298165-21ssh09:38:223
127918799641,58cyclictest1298165-21ssh09:38:223
127918799641,32cyclictest1285329-21sshd09:08:223
127917399645,21cyclictest1294432-21munin-run09:30:000
127917399645,13cyclictest3467602-21systemd-journal07:50:010
1279173996429,32cyclictest12-21ksoftirqd/007:35:020
1279173996420,12cyclictest1344333-21rm11:20:510
127918799631,59cyclictest1323401-21ssh10:34:333
127918799631,58cyclictest1380488-21ssh12:39:043
127918799631,58cyclictest1365934-21ssh12:06:393
127918799631,58cyclictest1357940-21ssh11:51:083
127918799631,58cyclictest1343728-21ssh11:19:533
127918799631,58cyclictest1340972-21ssh11:13:413
127918799631,58cyclictest1317392-21ssh10:20:433
127918799631,58cyclictest1312627-21ssh10:10:193
127917399634,13cyclictest1330372-21grep10:50:010
127917399633,21cyclictest1284317-21munin-run08:50:000
127917399632,7cyclictest1-21systemd09:05:020
127917399632,7cyclictest1-21systemd09:05:010
127917399632,3cyclictest0-21swapper/011:50:010
127917399632,18cyclictest3467602-21systemd-journal11:00:000
1279173996318,5cyclictest1307935-21sh10:00:010
127917399631,58cyclictest0-21swapper/011:15:430
127918799624,54cyclictest1279883-21sshd07:22:213
1279187996234,24cyclictest3467602-21systemd-journal07:39:213
127918799622,56cyclictest1281198-21munin-run07:50:013
127918799621,57cyclictest1348684-21ssh11:30:353
127918799621,57cyclictest1348684-21ssh11:30:343
127918799621,57cyclictest1282206-21sshd08:07:223
1279173996225,32cyclictest1305693-21latency_hist09:55:020
127918799612,55cyclictest1280384-21sshd07:31:213
127918799611,56cyclictest1304322-21ssh09:51:553
127918799611,55cyclictest1280141-21sshd07:29:213
127917399615,21cyclictest1281975-21modprobe08:04:590
1279173996117,38cyclictest309-21in:imuxsock07:40:020
1279173996114,4cyclictest16950irq/224-800000012:20:010
1279173996114,4cyclictest16950irq/224-800000012:20:000
127918799601,55cyclictest1285063-21sshd09:00:213
127918799601,55cyclictest1285063-21sshd09:00:213
127918799601,54cyclictest1282701-21sshd08:15:203
127917899603,47cyclictest1314671-21ssh10:15:011
127917899602,47cyclictest1339498-21ssh11:10:021
127917399605,9cyclictest1280183-21cat07:30:010
1279173996017,4cyclictest19350irq/34-mmc108:13:350
127918799592,52cyclictest1284582-21/usr/sbin/munin08:55:013
127918799591,54cyclictest1283493-21sshd08:30:213
127918799591,54cyclictest1281744-21munin-run08:00:013
1279187995912,42cyclictest0-21swapper/307:53:113
1279182995910,9cyclictest0-21swapper/212:10:022
1279178995938,15cyclictest3467602-21systemd-journal09:15:001
127917399594,10cyclictest284-21systemd-logind08:45:010
1279173995928,28cyclictest1312725-21ssh10:10:310
1279173995923,32cyclictest12-21ksoftirqd/007:10:010
127917399591,23cyclictest0-21swapper/009:20:400
1279182995816,8cyclictest30-21ksoftirqd/210:34:272
127917399582,12cyclictest1283709-21latency_hist08:35:020
1279173995815,6cyclictest1281703-21sh08:00:000
127917399581,21cyclictest0-21swapper/012:30:190
127918799571,52cyclictest1279333-21sshd07:10:223
127918799571,51cyclictest48-21kcompactd008:26:113
127918799571,51cyclictest48-21kcompactd008:01:133
1279182995713,8cyclictest30-21ksoftirqd/212:24:572
1279178995739,12cyclictest3467602-21systemd-journal12:15:011
127917899572,8cyclictest23-21ksoftirqd/112:00:191
1279173995730,23cyclictest16950irq/224-800000008:40:000
1279173995714,10cyclictest1373314-21sh12:23:110
127918799562,50cyclictest1284862-21munin-run09:00:013
127918799561,49cyclictest364-21snmpd08:10:033
1279182995614,7cyclictest30-21ksoftirqd/211:34:272
1279182995614,7cyclictest30-21ksoftirqd/211:34:272
127917899565,37cyclictest1320897-21kworker/u8:211:30:031
127917899565,37cyclictest1320897-21kworker/u8:211:30:031
1279178995638,15cyclictest3467602-21systemd-journal10:18:591
1279178995638,15cyclictest3467602-21systemd-journal10:18:581
1279178995621,10cyclictest23-21ksoftirqd/110:54:401
127917399562,15cyclictest1280756-21grep07:40:020
127918799551,50cyclictest48-21kcompactd008:36:433
127918799551,49cyclictest364-21snmpd07:19:013
1279182995510,9cyclictest30-21ksoftirqd/212:03:122
1279182995510,9cyclictest30-21ksoftirqd/211:47:342
127917899551,12cyclictest23-21ksoftirqd/112:10:001
127917399554,9cyclictest1279391-21grep07:15:010
127917399553,10cyclictest1282755-21munin-plugin-st08:20:000
127918799542,48cyclictest364-21snmpd08:48:423
127918799541,49cyclictest48-21kcompactd008:20:053
127918799541,49cyclictest48-21kcompactd008:20:043
1279182995417,8cyclictest30-21ksoftirqd/209:59:002
1279182995414,8cyclictest30-21ksoftirqd/210:10:022
1279182995414,8cyclictest30-21ksoftirqd/210:10:022
127917899542,7cyclictest23-21ksoftirqd/111:55:141
127917899542,13cyclictest23-21ksoftirqd/111:46:221
1279182995311,9cyclictest30-21ksoftirqd/210:40:492
127917899539,6cyclictest131rcu_preempt11:44:521
127917899532,9cyclictest23-21ksoftirqd/109:45:021
1279178995312,8cyclictest131rcu_preempt09:23:291
127917399532,15cyclictest364-21snmpd08:10:000
1279182995233,15cyclictest275-21dbus-daemon12:15:012
1279182995215,8cyclictest30-21ksoftirqd/212:32:422
1279178995216,10cyclictest23-21ksoftirqd/110:37:551
1279178995212,5cyclictest131rcu_preempt10:32:051
127918299519,8cyclictest30-21ksoftirqd/211:02:562
127918299517,9cyclictest30-21ksoftirqd/209:23:142
127918299517,7cyclictest30-21ksoftirqd/209:50:402
127918299516,7cyclictest30-21ksoftirqd/211:57:212
1279182995113,8cyclictest30-21ksoftirqd/207:10:022
127917899518,10cyclictest23-21ksoftirqd/108:20:001
127917899516,8cyclictest23-21ksoftirqd/110:10:031
127917899516,8cyclictest23-21ksoftirqd/110:10:021
1279178995111,7cyclictest131rcu_preempt12:26:211
127917899511,11cyclictest23-21ksoftirqd/112:20:131
127918299508,8cyclictest30-21ksoftirqd/209:19:282
127918299507,10cyclictest30-21ksoftirqd/211:25:502
127918299507,10cyclictest30-21ksoftirqd/211:25:492
127918299507,10cyclictest30-21ksoftirqd/210:53:382
127918299505,10cyclictest30-21ksoftirqd/209:44:592
127918299504,6cyclictest291rcuc/212:19:462
127918299504,6cyclictest291rcuc/212:19:452
1279182995036,10cyclictest275-21dbus-daemon10:50:012
1279182995016,8cyclictest30-21ksoftirqd/212:30:022
1279182995011,8cyclictest0-21swapper/210:10:312
127917899508,7cyclictest23-21ksoftirqd/110:42:101
127917899506,8cyclictest23-21ksoftirqd/110:55:301
1279178995016,5cyclictest131rcu_preempt09:37:421
1279178995016,5cyclictest131rcu_preempt09:37:411
1279178995010,9cyclictest23-21ksoftirqd/110:23:531
127918299497,8cyclictest30-21ksoftirqd/211:15:002
127918299497,8cyclictest30-21ksoftirqd/209:11:262
127918299496,6cyclictest30-21ksoftirqd/210:22:302
1279182994940,4cyclictest1285098-21munin-plugin-st09:05:012
1279182994940,4cyclictest1285098-21munin-plugin-st09:05:002
1279182994914,6cyclictest131rcu_preempt08:20:002
127917899498,5cyclictest131rcu_preempt12:17:201
127917899498,5cyclictest131rcu_preempt12:17:201
127917899498,5cyclictest131rcu_preempt11:15:521
127917899495,5cyclictest131rcu_preempt12:38:271
1279178994941,6cyclictest221rcuc/107:40:001
1279178994939,6cyclictest23-21ksoftirqd/109:30:001
127917899491,7cyclictest23-21ksoftirqd/109:48:581
127917899491,39cyclictest1352768-21ssh11:40:011
127917399494,8cyclictest1279932-21latency_hist07:25:020
127918299485,7cyclictest30-21ksoftirqd/209:25:182
1279182994812,4cyclictest131rcu_preempt12:40:002
1279182994811,8cyclictest0-21swapper/210:20:002
1279182994811,8cyclictest0-21swapper/210:20:002
127918299481,11cyclictest30-21ksoftirqd/209:38:522
127918299481,11cyclictest30-21ksoftirqd/209:38:512
127917899488,6cyclictest131rcu_preempt11:28:031
127917899488,6cyclictest131rcu_preempt11:28:021
127917899486,8cyclictest23-21ksoftirqd/110:29:571
127917899486,8cyclictest23-21ksoftirqd/110:29:571
127917899485,7cyclictest23-21ksoftirqd/109:31:021
1279178994837,8cyclictest23-21ksoftirqd/109:20:001
1279178994812,6cyclictest131rcu_preempt11:03:461
127918299474,7cyclictest30-21ksoftirqd/211:51:542
127918299474,7cyclictest30-21ksoftirqd/209:50:032
127918299474,10cyclictest30-21ksoftirqd/210:02:242
127918299474,10cyclictest30-21ksoftirqd/208:45:002
1279182994735,9cyclictest3467602-21systemd-journal07:34:212
1279182994711,9cyclictest0-21swapper/208:14:592
127917899475,10cyclictest23-21ksoftirqd/111:23:211
127917899473,5cyclictest131rcu_preempt12:33:561
127917899472,11cyclictest23-21ksoftirqd/110:49:521
127917899471,10cyclictest23-21ksoftirqd/109:55:001
127918299467,10cyclictest30-21ksoftirqd/211:41:352
1279182994630,10cyclictest1283785-21cron08:40:012
127917899469,11cyclictest23-21ksoftirqd/110:02:451
127917899465,5cyclictest131rcu_preempt09:55:331
1279178994612,6cyclictest131rcu_preempt11:05:491
127918299454,7cyclictest30-21ksoftirqd/208:35:012
1279182994511,7cyclictest30-21ksoftirqd/210:25:582
1279182994511,7cyclictest30-21ksoftirqd/210:25:582
127918299446,9cyclictest30-21ksoftirqd/208:55:022
1279182994435,6cyclictest30-21ksoftirqd/207:30:012
1279182994417,8cyclictest30-21ksoftirqd/211:08:132
127917899448,10cyclictest23-21ksoftirqd/107:41:521
127917899445,5cyclictest131rcu_preempt08:40:001
127918299436,10cyclictest30-21ksoftirqd/208:10:012
127918299435,7cyclictest30-21ksoftirqd/211:39:062
127918299435,6cyclictest30-21ksoftirqd/210:37:542
127918299431,9cyclictest30-21ksoftirqd/211:17:502
127917899439,9cyclictest23-21ksoftirqd/109:07:291
1279178994330,10cyclictest1281162-21sshd07:45:211
1279178994327,11cyclictest3467602-21systemd-journal09:00:001
127917899431,5cyclictest150-21pr/ttyS208:49:231
127918799422,34cyclictest1279258-21latency_hist07:10:013
1279182994226,12cyclictest30-21ksoftirqd/208:29:592
127917899426,6cyclictest23-21ksoftirqd/108:31:291
127917899424,7cyclictest23-21ksoftirqd/107:10:011
1279178994229,11cyclictest23-21ksoftirqd/107:54:251
127918299414,8cyclictest30-21ksoftirqd/209:10:002
1279178994131,8cyclictest23-21ksoftirqd/107:55:031
127917899410,7cyclictest150-21pr/ttyS208:52:021
127918299402,8cyclictest30-21ksoftirqd/209:33:242
1279182994027,10cyclictest30-21ksoftirqd/207:40:012
127918299401,7cyclictest1281982-21munin-run08:05:002
127917899401,8cyclictest150-21pr/ttyS208:07:171
127917899401,6cyclictest0-21swapper/108:42:341
127918299393,6cyclictest30-21ksoftirqd/207:14:172
1279182993928,8cyclictest1279887-21sshd07:23:222
1279182993927,9cyclictest30-21ksoftirqd/207:55:002
127917899396,5cyclictest131rcu_preempt08:30:001
1279178993929,8cyclictest23-21ksoftirqd/107:15:021
1279178993927,8cyclictest3467602-21systemd-journal07:17:011
127917899392,12cyclictest23-21ksoftirqd/108:10:471
127918299387,5cyclictest131rcu_preempt09:00:012
1279182993823,10cyclictest30-21ksoftirqd/207:20:002
127917899389,6cyclictest131rcu_preempt07:33:211
127917899381,6cyclictest150-21pr/ttyS209:04:281
127917899381,6cyclictest150-21pr/ttyS209:04:281
127917899381,6cyclictest150-21pr/ttyS207:28:511
127917899381,10cyclictest23-21ksoftirqd/107:20:311
1279182993722,12cyclictest275-21dbus-daemon08:50:012
1279178993711,9cyclictest23-21ksoftirqd/108:04:101
1279178993710,9cyclictest23-21ksoftirqd/108:20:341
1279178993710,9cyclictest23-21ksoftirqd/108:20:341
1279182993626,7cyclictest1278940-21kworker/2:208:00:012
127917399361,31cyclictest0-21swapper/008:51:160
127918299355,6cyclictest30-21ksoftirqd/208:23:362
127918299355,6cyclictest30-21ksoftirqd/208:23:362
1279182993521,11cyclictest1281383-21sed07:50:032
127918299335,6cyclictest30-21ksoftirqd/207:44:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional