You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 15:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Jul 20, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1236523999624,19cyclictest1273706-21ssh10:14:290
1236523999228,13cyclictest1255144-21ssh09:35:290
1236523998841,43cyclictest16850irq/134-800000011:00:010
1236523998841,43cyclictest16850irq/134-800000011:00:010
123652399866,3cyclictest131rcu_preempt12:00:020
1236523998620,15cyclictest1316197-21ssh11:43:170
1236523998620,15cyclictest1316197-21ssh11:43:170
1236523998618,63cyclictest16850irq/134-800000011:08:540
1236523998116,5cyclictest16850irq/134-800000009:29:540
1236523998026,12cyclictest1312554-21ssh11:35:230
1236523998016,59cyclictest16850irq/134-800000009:17:030
123652399801,27cyclictest0-21swapper/009:54:270
1236523997924,13cyclictest1304127-21ssh11:18:020
1236523997923,14cyclictest1326395-21ssh12:04:530
1236523997923,14cyclictest1326395-21ssh12:04:530
1236523997919,55cyclictest16850irq/134-800000010:34:590
1236523997910,57cyclictest16850irq/134-800000010:01:340
1236523997729,31cyclictest1305425-21ssh11:20:230
1236523997722,11cyclictest1260406-21ssh09:46:260
1236523997720,5cyclictest16850irq/134-800000011:51:520
1236523997713,59cyclictest16850irq/134-800000011:46:180
1236538997625,21cyclictest1238760-21latency_hist07:50:033
1236523997627,8cyclictest1311077-21ssh11:32:260
1236523997627,15cyclictest1289082-21ssh10:46:230
1236523997514,56cyclictest16850irq/134-800000010:39:080
123652399743,9cyclictest3467602-21systemd-journal11:25:480
1236523997426,11cyclictest1276688-21ssh10:20:140
1236538997322,20cyclictest1283511-21latency_hist10:35:023
1236523997328,14cyclictest1297257-21ssh11:03:420
1236523997328,12cyclictest1330492-21ssh12:13:270
1236523997328,12cyclictest1330492-21ssh12:13:260
1236523997326,15cyclictest1341828-21ssh12:36:530
1236523997314,5cyclictest16850irq/134-800000010:05:350
1236534997232,5cyclictest1305075-21kworker/2:312:04:132
1236534997232,5cyclictest1305075-21kworker/2:312:04:132
1236523997213,6cyclictest16850irq/134-800000012:17:190
1236534997128,39cyclictest30-21ksoftirqd/211:14:022
1236523997116,45cyclictest16850irq/134-800000009:41:390
1236523997116,45cyclictest16850irq/134-800000009:41:380
1236523997114,52cyclictest16850irq/134-800000009:32:310
123652399703,61cyclictest1335975-21modprobe12:25:000
123652399703,61cyclictest1335975-21modprobe12:25:000
1236523997021,13cyclictest1339620-21ssh12:32:200
1236523997020,12cyclictest1279135-21ssh10:25:290
1236523997011,49cyclictest16850irq/134-800000012:28:040
1236538996910,50cyclictest1333771-21ssh12:20:023
1236526996927,38cyclictest1321906-21cat11:55:021
1236526996926,39cyclictest23-21ksoftirqd/112:01:221
1236526996926,39cyclictest23-21ksoftirqd/112:01:221
123652399694,17cyclictest16850irq/134-800000010:52:060
1236523996921,13cyclictest1274627-21ssh10:16:030
1236523996819,40cyclictest16850irq/134-800000010:42:350
1236523996818,45cyclictest16850irq/134-800000009:11:080
1236523996818,45cyclictest16850irq/134-800000009:11:080
123652399675,26cyclictest1240109-21systemctl08:19:590
1236534996625,37cyclictest3467602-21systemd-journal09:56:142
1236534996624,5cyclictest1330886-21sh12:14:152
1236534996624,5cyclictest1330886-21sh12:14:152
123653499662,22cyclictest1238560-21true07:50:002
123652699662,14cyclictest1335982-21munin-run12:25:001
123652699662,14cyclictest1335982-21munin-run12:25:001
123652399667,20cyclictest1241405-21modprobe08:45:000
123652399664,6cyclictest16850irq/134-800000012:09:350
1236523996623,4cyclictest0-21swapper/011:11:210
1236538996528,32cyclictest48-21kcompactd010:25:503
123653499653,20cyclictest1241665-21grep08:50:002
123653499652,22cyclictest1252340-21latency_hist09:30:012
1236534996519,13cyclictest1-21systemd10:55:012
1236526996523,37cyclictest1316271-21sh11:43:271
1236526996523,37cyclictest1316271-21sh11:43:271
123652399654,18cyclictest1237266-21munin-run07:25:000
123652399654,17cyclictest16850irq/134-800000009:21:240
123652399653,21cyclictest1241141-21systemctl08:40:000
123652699644,55cyclictest150-21pr/ttyS212:16:351
1236526996423,36cyclictest23-21ksoftirqd/107:41:191
123652699642,13cyclictest1260071-21ssh09:45:451
1236526996421,39cyclictest1293085-21sed10:55:021
123652699641,12cyclictest48-21kcompactd009:55:111
1236538996323,16cyclictest1257471-21rm09:40:113
1236538996323,16cyclictest1257471-21rm09:40:113
123653499634,11cyclictest1314526-21modprobe11:40:002
123653499633,24cyclictest1240106-21grep08:20:002
1236534996325,34cyclictest1249979-21cat09:25:022
1236534996324,35cyclictest1323002-21sh11:57:322
123652699632,13cyclictest1271257-21ssh10:09:241
123652699631,18cyclictest1254701-21munin-run09:35:001
123652399636,17cyclictest1-21systemd07:50:010
1236523996322,9cyclictest12-21ksoftirqd/009:10:020
123652399631,38cyclictest1264637-21ssh09:55:120
1236538996215,12cyclictest0-21swapper/310:56:213
1236538996215,12cyclictest0-21swapper/310:56:213
1236534996224,4cyclictest1286810-21sh10:41:402
123652699622,14cyclictest48-21kcompactd011:45:301
123652699622,10cyclictest1303020-21ssh11:15:261
1236526996220,38cyclictest1243055-21sh09:10:151
1236526996220,38cyclictest1243055-21sh09:10:151
1236526996219,38cyclictest1314170-21sh11:39:071
123652699621,16cyclictest1283489-21munin-run10:35:011
123652699621,14cyclictest1263350-21ssh09:52:421
123652399624,21cyclictest1241920-21cut08:55:000
123652399621,57cyclictest1237998-21sshd07:38:120
1236538996149,10cyclictest1250350-21ssh09:25:303
1236538996148,10cyclictest1248805-21ssh09:22:283
1236538996131,26cyclictest1288400-21sed10:45:023
1236534996128,27cyclictest1306628-21sh11:23:132
1236534996123,6cyclictest30-21ksoftirqd/212:16:072
1236534996118,8cyclictest1295044-21sh10:59:102
1236534996118,8cyclictest1295044-21sh10:59:102
123652699613,16cyclictest1248000-21ssh09:20:351
1236526996131,26cyclictest1290811-21sed10:50:021
1236526996123,33cyclictest23-21ksoftirqd/112:33:291
1236526996122,34cyclictest23-21ksoftirqd/109:25:071
123652699612,15cyclictest1240220-21date08:20:011
123652699611,14cyclictest1258965-21ssh09:43:381
123652699611,14cyclictest1258965-21ssh09:43:381
123652699611,13cyclictest364-21snmpd09:15:551
1236538996047,10cyclictest1236951-21cron07:17:003
1236538996047,10cyclictest1236951-21cron07:16:593
1236538996044,13cyclictest1304709-21ssh11:19:253
1236538996043,13cyclictest1274942-21ssh10:16:523
1236538996040,16cyclictest1255153-21ssh09:35:313
1236538996022,13cyclictest150-21pr/ttyS211:14:043
123653499603,21cyclictest1237267-21grep07:25:002
1236534996024,5cyclictest1260900-21sh09:47:362
123653499602,17cyclictest284-21systemd-logind10:35:012
123653499602,12cyclictest1257145-21munin-run09:40:002
1236534996019,4cyclictest1240800-21latency_hist08:30:022
1236534996019,4cyclictest1240800-21latency_hist08:30:022
1236534996011,15cyclictest275-21dbus-daemon07:40:002
123652699602,18cyclictest1329513-21ssh12:11:061
123652699602,18cyclictest1329513-21ssh12:11:061
1236526996012,4cyclictest0-21swapper/108:22:411
123652699601,15cyclictest364-21snmpd07:47:131
123652699601,11cyclictest1237268-21cut07:24:591
1236538995947,10cyclictest1263225-21ssh09:52:273
1236538995945,11cyclictest1326344-21ssh12:04:473
1236538995945,11cyclictest1326344-21ssh12:04:473
1236538995945,11cyclictest1270718-21ssh10:08:093
1236538995944,12cyclictest1289784-21ssh10:48:033
1236538995943,13cyclictest1245431-21ssh09:15:233
1236538995943,12cyclictest1341969-21ssh12:37:113
1236538995943,12cyclictest1244615-21ssh09:14:023
1236538995943,12cyclictest1244615-21ssh09:14:023
1236538995942,13cyclictest1296541-21ssh11:02:043
1236538995941,14cyclictest1318633-21ssh11:48:243
1236538995941,14cyclictest1273841-21ssh10:14:533
1236538995940,15cyclictest1320919-21ssh11:53:123
123653499593,22cyclictest1335983-21munin-plugin-st12:25:002
123653499593,22cyclictest1335983-21munin-plugin-st12:25:002
123653499593,12cyclictest1239837-21modprobe08:15:002
1236534995923,10cyclictest30-21ksoftirqd/209:40:592
1236534995923,10cyclictest30-21ksoftirqd/209:40:592
123653499592,28cyclictest1242685-21cron09:10:002
1236534995921,9cyclictest309-21in:imuxsock09:15:012
1236534995921,9cyclictest309-21in:imuxsock09:15:002
1236534995920,4cyclictest1285882-21sh10:40:002
1236534995912,11cyclictest3467602-21systemd-journal12:40:002
123652699592,15cyclictest1287963-21ssh10:44:131
123652699592,14cyclictest1337021-21ssh12:26:531
123652699591,14cyclictest364-21snmpd08:50:061
1236538995845,10cyclictest1336664-21ssh12:26:093
1236538995845,10cyclictest1239551-21sshd08:05:133
1236538995844,10cyclictest1310329-21ssh11:30:443
1236538995843,12cyclictest1284838-21ssh10:37:333
1236538995843,12cyclictest1259860-21ssh09:45:093
1236538995843,11cyclictest1330048-21ssh12:12:203
1236538995843,11cyclictest1330048-21ssh12:12:203
1236538995843,11cyclictest1242678-21sshd09:08:133
1236538995842,13cyclictest1328737-21ssh12:09:483
1236538995842,13cyclictest1315197-21ssh11:40:583
1236538995842,13cyclictest1315197-21ssh11:40:583
1236538995842,12cyclictest1323212-21ssh11:58:103
1236538995841,13cyclictest1306159-21ssh11:22:073
1236538995841,13cyclictest1252588-21ssh09:30:093
1236534995822,4cyclictest1336790-21sh12:26:252
123653499582,19cyclictest1340940-21grep12:35:022
1236534995821,5cyclictest1269991-21sh10:06:262
123653499582,11cyclictest1304989-21ssh11:20:002
1236534995819,6cyclictest1297752-21sh11:04:542
1236534995817,9cyclictest30-21ksoftirqd/210:22:202
1236534995817,4cyclictest1319341-21latency_hist11:50:012
1236526995836,13cyclictest1329031-21diskmemload12:10:021
123652699583,15cyclictest1276878-21ssh10:20:471
1236526995820,34cyclictest1242702-21diskmemload10:35:341
1236526995819,36cyclictest1318744-1kworker/u9:111:57:331
123652699581,15cyclictest1310187-21ssh11:30:251
123652699581,14cyclictest1301425-21ssh11:12:171
1236523995821,5cyclictest1237139-21latency_hist07:20:020
1236523995821,5cyclictest1237139-21latency_hist07:20:010
1236538995745,10cyclictest1264612-21ssh09:55:103
1236538995744,10cyclictest1267922-21ssh10:02:193
1236538995743,11cyclictest1307767-21ssh11:25:243
1236538995742,13cyclictest1334960-21ssh12:22:413
1236538995742,13cyclictest1334960-21ssh12:22:413
1236538995741,13cyclictest1277926-21ssh10:23:073
1236538995741,12cyclictest1339496-21ssh12:32:043
1236538995741,12cyclictest1298251-21ssh11:05:293
1236538995740,13cyclictest1291880-21ssh10:52:233
1236538995738,15cyclictest1240062-21cron08:17:013
123653499573,17cyclictest1236975-21cron07:20:002
123653499573,17cyclictest1236975-21cron07:19:592
123653499573,15cyclictest1276314-21cron10:19:592
1236534995721,5cyclictest1242171-21latency_hist09:00:032
123653499572,13cyclictest275-21dbus-daemon10:30:012
123652699572,9cyclictest1281092-21unin-run10:30:011
123652699572,23cyclictest1236974-21cron07:20:001
123652699572,23cyclictest1236974-21cron07:20:001
123652699572,14cyclictest1274785-21ssh10:16:321
1236526995713,11cyclictest3467602-21systemd-journal12:40:001
123652699571,13cyclictest48-21kcompactd007:29:411
123652399575,12cyclictest1242178-21munin-run09:00:000
1236538995644,9cyclictest1241392-21sshd08:43:133
1236538995643,10cyclictest1242243-21cat09:00:023
1236538995643,10cyclictest1240377-21munin-run08:25:003
1236538995643,10cyclictest1236688-21sshd07:10:143
1236538995642,12cyclictest1314481-21ssh11:39:573
1236538995641,11cyclictest1241664-21munin-run08:50:003
1236538995639,14cyclictest1240842-21sshd08:30:133
1236538995631,20cyclictest3467602-21systemd-journal08:15:013
1236534995621,11cyclictest3467602-21systemd-journal07:45:002
1236534995620,4cyclictest1242702-21diskmemload09:19:432
1236534995618,10cyclictest1-21systemd11:10:012
123653499561,22cyclictest1254686-21ssh09:35:002
123652699563,15cyclictest275-21dbus-daemon08:30:011
123652699563,15cyclictest275-21dbus-daemon08:30:011
1236526995629,23cyclictest23-21ksoftirqd/111:23:321
1236526995625,27cyclictest23-21ksoftirqd/107:35:021
1236526995625,27cyclictest23-21ksoftirqd/107:35:021
123652699562,18cyclictest1296919-21ssh11:02:591
123652699562,14cyclictest1299472-21ssh11:08:261
123652399564,20cyclictest1240630-21cut08:30:010
123652399564,20cyclictest1240630-21cut08:30:010
1236538995535,15cyclictest364-21snmpd08:52:363
123653499552,11cyclictest364-21snmpd11:35:022
1236534995520,4cyclictest1242702-21diskmemload11:43:412
1236534995520,4cyclictest1242702-21diskmemload11:43:412
123653499551,4cyclictest131rcu_preempt10:15:022
123652699553,8cyclictest275-21dbus-daemon08:40:011
123652699552,12cyclictest1268710-21ssh10:04:051
123652699551,15cyclictest1255272-21ssh09:35:451
123652699551,13cyclictest1242767-21latency_hist09:10:011
1236538995441,10cyclictest1237743-21sshd07:32:143
1236538995441,10cyclictest1237743-21sshd07:32:143
1236538995439,11cyclictest1237532-21munin-run07:30:013
123653499543,16cyclictest1241258-21latency_hist08:40:012
1236534995422,12cyclictest30-21ksoftirqd/207:15:022
123653499541,16cyclictest1290820-21cat10:50:022
123652699543,18cyclictest1241666-21cut08:49:591
123652699542,15cyclictest1273029-21ssh10:12:551
123652699542,13cyclictest1242450-21munin-run09:05:011
123652699542,13cyclictest1242450-21munin-run09:05:001
123652699541,13cyclictest1308198-21ssh11:26:161
123652699541,13cyclictest1295495-21latency_hist11:00:021
123652699541,13cyclictest1295495-21latency_hist11:00:021
123652399545,15cyclictest1240369-21modprobe08:25:000
123652399545,14cyclictest1241663-21modprobe08:50:000
1236538995343,5cyclictest1237239-21sshd07:22:133
1236534995333,15cyclictest3467602-21systemd-journal08:10:002
123652699531,13cyclictest1238984-21wc07:55:021
123652399534,17cyclictest1239838-21munin-plugin-st08:15:000
1236538995239,10cyclictest1242412-21sshd09:00:143
1236538995239,10cyclictest1242412-21sshd09:00:143
1236538995238,9cyclictest48-21kcompactd007:35:473
1236534995244,5cyclictest1238063-21kworker/2:108:20:022
1236534995226,23cyclictest30-21ksoftirqd/207:55:022
123653499522,12cyclictest1321707-21sh11:55:002
1236534995220,4cyclictest1308906-21sh11:27:542
123652699521,13cyclictest364-21snmpd07:37:541
123652399525,9cyclictest12-21ksoftirqd/008:00:010
123652399522,17cyclictest1-21systemd07:45:010
123652699517,6cyclictest0-21swapper/108:15:001
123652699513,13cyclictest1241404-21modprobe08:45:001
123652399512,4cyclictest131rcu_preempt08:35:010
1236538995039,8cyclictest48-21kcompactd008:35:423
1236538995038,9cyclictest48-21kcompactd008:26:163
1236538995038,9cyclictest48-21kcompactd008:26:163
1236538995038,9cyclictest48-21kcompactd007:44:063
1236534995041,5cyclictest1241947-21kworker/2:109:50:532
1236534995017,8cyclictest30-21ksoftirqd/209:02:212
1236534995017,8cyclictest30-21ksoftirqd/209:02:212
123652699501,17cyclictest1239077-21tr08:00:001
123652399503,21cyclictest1238808-21true07:55:010
1236523995021,4cyclictest1239330-21latency_hist08:05:020
123652399501,31cyclictest0-21swapper/009:02:300
123652399501,31cyclictest0-21swapper/009:02:300
1236538994936,9cyclictest364-21snmpd08:02:313
1236538994935,10cyclictest364-21snmpd07:59:343
1236538994932,13cyclictest3467602-21systemd-journal07:50:023
1236534994941,5cyclictest1241947-21kworker/2:110:03:002
123653499493,13cyclictest3467602-21systemd-journal08:45:002
123652699492,8cyclictest1242177-21modprobe09:00:001
1236538994723,19cyclictest1236578-21latency_hist07:10:013
1236534994719,5cyclictest1328196-21rm12:08:332
1236534994710,32cyclictest0-21swapper/208:00:002
123652699472,17cyclictest1236737-21systemctl07:15:001
123652699472,15cyclictest1-21systemd08:05:011
1236526994712,11cyclictest3467602-21systemd-journal08:34:591
123652699452,14cyclictest1239558-21sshd08:07:131
1236526994510,5cyclictest1236613-21wc07:10:011
1236534994436,5cyclictest1240903-21kworker/2:308:52:122
1236534994436,5cyclictest1238063-21kworker/2:108:03:002
1236534994418,5cyclictest30-21ksoftirqd/207:35:022
1236534994418,5cyclictest30-21ksoftirqd/207:35:022
1236534994335,5cyclictest1221592-21kworker/2:207:27:502
1236523994318,20cyclictest1236467-21latency_hist07:10:010
123652399431,27cyclictest0-21swapper/007:14:570
1236523994018,17cyclictest16850irq/134-800000007:26:130
123653499391,6cyclictest131rcu_preempt08:35:002
123652399371,27cyclictest0-21swapper/008:09:110
1236523993628,5cyclictest19350irq/34-mmc107:34:240
1236523993628,5cyclictest19350irq/34-mmc107:34:240
123653499261,21cyclictest1221592-21kworker/2:207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional