You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 23:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 11, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1171665997718,55cyclictest12-21ksoftirqd/009:48:520
117166599722,45cyclictest1244760-21ssh11:40:020
117166799705,28cyclictest1176889-21kworker/u8:109:20:022
117166699701,11cyclictest1222525-21ssh10:50:101
117166699701,11cyclictest1222525-21ssh10:50:091
117166699691,13cyclictest1255726-21ssh12:04:341
1171666996626,36cyclictest1197284-21sh09:52:571
1171666996625,36cyclictest1193038-21sh09:43:331
117166699662,15cyclictest1234566-21ssh11:17:091
117166699662,10cyclictest1177133-21sshd08:54:211
117166699661,13cyclictest1245873-21ssh11:42:211
1171666996523,37cyclictest23-21ksoftirqd/110:35:191
1171666996523,36cyclictest23-21ksoftirqd/112:09:241
117166699652,15cyclictest1178514-21ssh09:10:481
117166699651,14cyclictest1253192-21ssh11:58:541
117166699651,14cyclictest1219117-21ssh10:42:591
117166699651,14cyclictest1219117-21ssh10:42:581
117166699642,15cyclictest1187496-21ssh09:31:021
117166699642,14cyclictest1269320-21ssh12:34:481
117166699642,13cyclictest1236374-21ssh11:21:011
117166699642,13cyclictest1231650-21ssh11:10:291
117166699642,13cyclictest1224799-21ssh10:55:131
117166699641,13cyclictest1193922-21ssh09:45:121
117166699632,54cyclictest1200323-21ssh09:59:571
117166699632,14cyclictest1244091-21ssh11:38:341
117166699632,13cyclictest1247521-21ssh11:45:491
117166699632,13cyclictest1175289-21munin-run08:20:011
1171666996321,37cyclictest1213763-21sh10:30:451
1171666996321,37cyclictest1213763-21sh10:30:441
117166699632,12cyclictest1204693-21ssh10:09:381
117166699631,9cyclictest1173964-21sshd07:53:171
1171666996317,40cyclictest1175820-21cron08:30:001
117166699631,5cyclictest1230577-21ssh11:08:221
117166699631,14cyclictest1242184-21ssh11:34:171
117166699631,13cyclictest1258603-21ssh12:10:341
117166699631,13cyclictest1209981-21ssh10:21:011
117166699631,11cyclictest1185832-21ssh09:27:171
1171666996229,29cyclictest309-21in:imuxsock08:05:021
1171666996229,29cyclictest309-21in:imuxsock08:05:011
117166699622,15cyclictest1212975-21ssh10:29:171
117166699622,13cyclictest1183911-21ssh09:23:101
117166699622,12cyclictest1220274-21ssh10:45:111
117166699621,9cyclictest1266772-21ssh12:29:031
117166699621,9cyclictest1266772-21ssh12:29:021
117166699621,19cyclictest1271165-21ssh12:38:591
117166699621,19cyclictest1271165-21ssh12:38:581
117166699621,14cyclictest1264668-21ssh12:24:261
117166699621,13cyclictest1207312-21ssh10:15:071
117166699621,12cyclictest1250698-21ssh11:53:181
117166699621,12cyclictest1202088-21ssh10:03:391
117166699621,12cyclictest1202088-21ssh10:03:391
117166699621,12cyclictest1189471-21ssh09:35:161
117166699612,14cyclictest1227138-21ssh11:00:221
117166699612,14cyclictest1227138-21ssh11:00:221
117166699611,13cyclictest1261746-21ssh12:17:461
117166699611,12cyclictest1180832-21ssh09:16:041
1171667996016,6cyclictest131rcu_preempt09:30:322
117166699602,17cyclictest3467602-21systemd-journal11:26:101
117166699602,17cyclictest3467602-21systemd-journal11:26:101
117166699601,13cyclictest1206559-21ssh10:13:451
117166699601,13cyclictest1206559-21ssh10:13:451
1171666995925,31cyclictest23-21ksoftirqd/109:05:011
117166699591,6cyclictest1177409-21munin-run09:00:001
117166699585,44cyclictest284-21systemd-logind08:00:001
117166699582,5cyclictest364-21snmpd08:22:541
117166699582,16cyclictest275-21dbus-daemon08:50:001
117166699581,5cyclictest1174767-21munin-run08:10:001
1171666995812,41cyclictest1173200-21cron07:40:001
117166699581,12cyclictest364-21snmpd08:43:221
1171665995825,29cyclictest12-21ksoftirqd/009:50:110
1171666995716,38cyclictest1171731-21cat07:10:021
1171666995716,38cyclictest1171731-21cat07:10:021
1171665995728,13cyclictest1188714-21ssh09:34:050
1171665995726,25cyclictest1226802-21sendmail10:59:590
117166699562,6cyclictest1172699-21munin-run07:30:001
117166699562,4cyclictest1172145-21sshd07:18:181
117166699561,6cyclictest1176407-21munin-run08:40:021
1171665995652,2cyclictest0-21swapper/009:05:010
1171665995628,24cyclictest1254834-21ssh12:02:200
1171665995612,27cyclictest12-21ksoftirqd/009:56:510
1171667995534,16cyclictest3467602-21systemd-journal09:40:022
117166799552,7cyclictest30-21ksoftirqd/211:05:022
117166799552,7cyclictest30-21ksoftirqd/211:05:022
117166699552,5cyclictest364-21snmpd07:22:421
1171665995536,4cyclictest16850irq/134-800000010:55:030
1171665995536,4cyclictest16850irq/134-800000010:55:020
1171665995532,17cyclictest1262604-21ssh12:20:010
117166799546,10cyclictest30-21ksoftirqd/210:55:102
1171667995445,6cyclictest1165013-21kworker/2:010:04:572
1171667995445,6cyclictest1165013-21kworker/2:010:04:572
1171667995415,6cyclictest131rcu_preempt12:05:002
117166699542,5cyclictest364-21snmpd07:42:471
1171665995426,12cyclictest1228973-21ssh11:04:570
1171665995426,12cyclictest1228973-21ssh11:04:560
1171668995335,14cyclictest275-21dbus-daemon12:25:003
1171668995334,14cyclictest275-21dbus-daemon07:55:003
117166799536,7cyclictest30-21ksoftirqd/209:00:002
1171667995345,5cyclictest1235817-21kworker/2:311:44:442
1171667995342,7cyclictest1165013-21kworker/2:009:13:502
1171667995335,13cyclictest3467602-21systemd-journal07:40:002
117166699533,5cyclictest1175008-21modprobe08:15:011
117166699532,6cyclictest1172949-21munin-run07:35:001
117166699531,5cyclictest364-21snmpd07:47:481
117166699531,5cyclictest364-21snmpd07:47:471
117166599531,37cyclictest131rcu_preempt12:00:000
117166599531,37cyclictest1206051-21ssh10:12:270
117166599531,37cyclictest1206051-21ssh10:12:260
1171667995244,5cyclictest1262665-21kworker/2:212:37:172
1171667995244,5cyclictest1262665-21kworker/2:212:37:162
1171667995244,5cyclictest1262665-21kworker/2:212:25:302
1171667995244,5cyclictest1262665-21kworker/2:212:25:292
1171667995235,12cyclictest1175845-21systemd-run08:30:022
117166699521,5cyclictest364-21snmpd09:08:031
1171665995218,8cyclictest12-21ksoftirqd/009:10:140
117166799517,10cyclictest30-21ksoftirqd/210:06:322
1171667995143,5cyclictest1227241-21kworker/2:111:32:342
1171667995143,5cyclictest1165013-21kworker/2:010:23:132
1171667995142,6cyclictest1258145-21kworker/2:012:15:002
1171667995136,11cyclictest1264625-21sshd12:24:212
117166799511,8cyclictest1215458-21munin-run10:35:012
117166799511,8cyclictest1215458-21munin-run10:35:002
1171667995110,5cyclictest131rcu_preempt11:38:252
117166699512,6cyclictest48-21kcompactd007:11:421
1171665995125,12cyclictest1218997-21ssh10:42:330
1171665995125,12cyclictest1218997-21ssh10:42:320
1171665995125,11cyclictest1237926-21ssh11:24:460
117166599512,15cyclictest1177962-21munin-run09:10:010
1171665995120,27cyclictest0-21swapper/010:30:270
1171665995120,27cyclictest0-21swapper/010:30:260
117166799509,11cyclictest30-21ksoftirqd/211:05:422
117166799506,13cyclictest30-21ksoftirqd/211:56:322
1171667995042,5cyclictest1235817-21kworker/2:311:48:312
1171667995042,5cyclictest1235817-21kworker/2:311:25:222
1171667995042,5cyclictest1235817-21kworker/2:311:25:212
1171667995042,5cyclictest1177163-21kworker/2:210:47:002
1171667995042,5cyclictest1165013-21kworker/2:010:41:562
1171667995042,5cyclictest1165013-21kworker/2:010:41:552
1171667995042,5cyclictest1165013-21kworker/2:010:26:592
1171667995041,6cyclictest1177163-21kworker/2:209:26:272
1171667995040,6cyclictest1177163-21kworker/2:209:41:162
1171667995040,6cyclictest1165013-21kworker/2:011:17:382
117166599506,40cyclictest0-21swapper/009:20:000
1171665995026,9cyclictest1258564-21ssh12:10:300
117166899494,4cyclictest131rcu_preempt10:00:283
117166899494,4cyclictest131rcu_preempt10:00:273
1171668994932,12cyclictest37-21ksoftirqd/310:00:013
117166799496,5cyclictest131rcu_preempt09:25:022
1171667994941,5cyclictest1227241-21kworker/2:111:52:362
1171667994941,5cyclictest1171671-21kworker/2:108:32:142
1171667994941,5cyclictest1171671-21kworker/2:107:32:472
1171667994941,5cyclictest1165013-21kworker/2:011:12:042
117166799494,11cyclictest30-21ksoftirqd/210:55:032
117166799494,11cyclictest30-21ksoftirqd/210:55:022
1171667994940,6cyclictest1177163-21kworker/2:210:37:452
1171667994940,6cyclictest1165013-21kworker/2:009:08:032
1171667994940,5cyclictest1262665-21kworker/2:212:32:102
1171667994940,5cyclictest1235817-21kworker/2:311:20:292
1171667994940,5cyclictest1165013-21kworker/2:009:51:152
1171667994913,6cyclictest131rcu_preempt09:45:132
1171667994911,5cyclictest131rcu_preempt10:20:002
117166699492,4cyclictest364-21snmpd08:32:401
1171665994922,12cyclictest1270755-21ssh12:37:510
1171665994922,12cyclictest1270755-21ssh12:37:500
1171665994920,3cyclictest11917011ssh09:40:110
1171668994833,9cyclictest309-21in:imuxsock08:40:013
1171668994829,13cyclictest275-21dbus-daemon11:35:013
1171667994841,4cyclictest1171671-21kworker/2:107:58:022
1171667994840,5cyclictest1258145-21kworker/2:012:16:332
1171667994840,5cyclictest1227241-21kworker/2:112:06:062
1171667994840,5cyclictest1177163-21kworker/2:209:55:302
1171667994840,5cyclictest1177163-21kworker/2:209:01:102
1171667994840,5cyclictest1171671-21kworker/2:107:51:392
1171667994840,5cyclictest1171671-21kworker/2:107:24:462
1171667994840,5cyclictest1165013-21kworker/2:010:12:392
1171667994840,5cyclictest1165013-21kworker/2:010:12:382
1171667994840,5cyclictest1165013-21kworker/2:008:52:152
1171667994840,5cyclictest1165013-21kworker/2:008:02:322
1171667994840,5cyclictest1165013-21kworker/2:008:02:312
1171667994840,5cyclictest1165013-21kworker/2:007:17:042
1171667994839,6cyclictest1165013-21kworker/2:008:42:572
1171667994839,5cyclictest1171671-21kworker/2:108:14:052
117166599482,41cyclictest1264933-21grep12:24:590
1171665994819,13cyclictest1232212-21ssh11:11:550
1171668994729,14cyclictest37-21ksoftirqd/312:00:223
1171668994728,15cyclictest3467602-21systemd-journal08:15:023
1171667994740,4cyclictest1165013-21kworker/2:007:41:392
1171667994739,5cyclictest1171671-21kworker/2:107:25:262
1171667994739,5cyclictest1165013-21kworker/2:008:35:432
1171667994739,5cyclictest1165013-21kworker/2:008:20:112
1171667994739,5cyclictest1165013-21kworker/2:008:05:152
1171667994739,5cyclictest1165013-21kworker/2:007:10:202
1171667994738,5cyclictest1171671-21kworker/2:108:15:282
1171665994736,7cyclictest1-21systemd11:10:000
1171665994731,12cyclictest3467602-21systemd-journal08:45:030
1171665994731,11cyclictest275-21dbus-daemon07:55:000
1171665994729,14cyclictest1204613-21ssh10:09:250
1171665994727,13cyclictest3467602-21systemd-journal11:50:020
117166599472,40cyclictest1215802-21ssh10:35:180
1171665994721,22cyclictest1246150-21ssh11:43:060
1171665994718,26cyclictest1191097-21sh09:39:220
117166599471,17cyclictest364-21snmpd10:19:010
117166899469,33cyclictest1225838-21kworker/u8:111:05:033
117166899469,33cyclictest1225838-21kworker/u8:111:05:023
117166899464,4cyclictest131rcu_preempt09:16:363
1171668994630,11cyclictest275-21dbus-daemon07:35:003
1171668994628,13cyclictest275-21dbus-daemon07:45:003
1171667994638,5cyclictest1165013-21kworker/2:008:45:052
1171667994638,5cyclictest1165013-21kworker/2:007:45:412
1171667994638,5cyclictest1165013-21kworker/2:007:45:412
1171665994632,10cyclictest3467602-21systemd-journal09:22:200
1171665994628,14cyclictest1256954-21ssh12:07:050
117166599462,15cyclictest1221116-21ssh10:47:140
1171668994532,11cyclictest37-21ksoftirqd/307:20:003
117166899452,3cyclictest131rcu_preempt12:17:163
1171667994529,11cyclictest3467602-21systemd-journal07:10:022
1171667994529,11cyclictest3467602-21systemd-journal07:10:012
1171665994530,12cyclictest1268738-21ssh12:33:200
1171665994527,14cyclictest1265226-21ssh12:25:190
1171665994527,14cyclictest1265226-21ssh12:25:180
1171665994527,14cyclictest12383581ssh11:25:230
1171665994527,14cyclictest12383581ssh11:25:220
1171665994526,15cyclictest1247789-21rm11:46:340
117166599452,39cyclictest1241595-21ssh11:32:490
1171665994522,9cyclictest1183532-21kworker/u8:309:25:410
1171668994428,11cyclictest37-21ksoftirqd/311:53:263
1171668994427,13cyclictest3467602-21systemd-journal08:50:013
117166899441,4cyclictest131rcu_preempt09:53:393
117166899441,3cyclictest131rcu_preempt10:09:563
1171665994436,5cyclictest16950irq/224-800000007:40:000
1171665994423,4cyclictest12012461ssh10:01:370
1171665994423,4cyclictest12012461ssh10:01:360
1171665994412,28cyclictest12-21ksoftirqd/010:20:590
117166599441,12cyclictest1174259-21basename08:00:010
1171668994331,10cyclictest37-21ksoftirqd/310:59:333
1171668994329,11cyclictest37-21ksoftirqd/311:48:213
1171668994313,7cyclictest131rcu_preempt11:40:013
1171665994336,4cyclictest16950irq/224-800000008:05:000
1171665994336,4cyclictest16950irq/224-800000008:05:000
117166599432,7cyclictest1175292-21systemctl08:20:010
1171665994327,11cyclictest3467602-21systemd-journal08:58:210
1171668994229,11cyclictest37-21ksoftirqd/312:29:153
1171668994229,11cyclictest37-21ksoftirqd/312:29:143
1171668994229,10cyclictest37-21ksoftirqd/310:46:143
1171668994226,11cyclictest1-21systemd11:15:003
1171668994224,13cyclictest1215572-21wc10:35:023
1171668994224,13cyclictest1215572-21wc10:35:013
1171665994230,8cyclictest12-21ksoftirqd/007:35:000
1171665994228,11cyclictest1234727-21ssh11:17:320
1171665994228,11cyclictest1176331-21sshd08:35:310
1171665994228,11cyclictest1174993-21sshd08:12:190
1171665994226,12cyclictest1211947-21ssh10:26:490
1171668994129,9cyclictest37-21ksoftirqd/308:25:013
1171668994129,9cyclictest1252569-21sshd11:57:193
1171668994129,8cyclictest3467602-21systemd-journal07:30:003
1171668994128,9cyclictest1-21systemd08:10:013
1171668994126,12cyclictest37-21ksoftirqd/310:25:003
117166899412,4cyclictest131rcu_preempt10:13:533
117166899412,4cyclictest131rcu_preempt10:13:523
117166899412,4cyclictest131rcu_preempt08:59:593
1171665994123,12cyclictest275-21dbus-daemon08:10:010
117166899404,3cyclictest131rcu_preempt10:35:323
1171668994029,9cyclictest37-21ksoftirqd/312:38:083
1171668994029,9cyclictest37-21ksoftirqd/312:38:083
1171668994028,8cyclictest309-21in:imuxsock12:35:003
1171668994027,10cyclictest37-21ksoftirqd/308:00:033
1171668994027,10cyclictest37-21ksoftirqd/307:25:013
1171668994025,12cyclictest37-21ksoftirqd/312:10:313
1171668994016,19cyclictest275-21dbus-daemon09:35:003
1171665994033,4cyclictest16950irq/224-800000008:35:010
1171665994030,7cyclictest12-21ksoftirqd/008:25:010
117166599402,34cyclictest3467602-21systemd-journal07:23:430
1171668993927,9cyclictest37-21ksoftirqd/311:26:083
1171668993927,9cyclictest37-21ksoftirqd/311:26:073
1171668993927,9cyclictest1-21systemd10:45:003
1171668993927,9cyclictest1-21systemd10:45:003
1171668993926,10cyclictest37-21ksoftirqd/312:09:543
1171668993924,11cyclictest37-21ksoftirqd/309:24:083
1171668993910,8cyclictest37-21ksoftirqd/309:27:243
1171665993931,6cyclictest19350irq/34-mmc107:50:030
1171665993931,6cyclictest19350irq/34-mmc107:50:020
1171668993826,7cyclictest309-21in:imuxsock08:05:023
1171668993826,7cyclictest309-21in:imuxsock08:05:013
1171668993825,10cyclictest275-21dbus-daemon11:10:013
117166899382,3cyclictest131rcu_preempt09:46:083
1171668993823,11cyclictest275-21dbus-daemon08:15:003
1171668993822,11cyclictest37-21ksoftirqd/309:40:573
117166899381,8cyclictest0-21swapper/309:11:553
117166899381,4cyclictest131rcu_preempt08:45:003
1171665993831,4cyclictest0-21swapper/007:45:000
1171665993826,9cyclictest1172664-21sshd07:25:190
117166899374,5cyclictest131rcu_preempt10:51:043
117166899374,5cyclictest131rcu_preempt10:51:043
1171668993729,6cyclictest37-21ksoftirqd/309:40:003
1171668993726,8cyclictest37-21ksoftirqd/307:15:013
1171668993725,10cyclictest37-21ksoftirqd/311:23:153
1171668993724,10cyclictest37-21ksoftirqd/309:10:023
1171668993723,10cyclictest309-21in:imuxsock08:55:003
1171665993731,4cyclictest0-21swapper/007:15:000
1171665993728,6cyclictest12-21ksoftirqd/008:45:010
1171665993727,7cyclictest1175777-21sshd08:25:200
1171665993726,7cyclictest12-21ksoftirqd/007:20:000
1171665993711,6cyclictest1177182-21latency_hist08:55:010
1171668993628,5cyclictest1212600-21sh10:28:283
1171668993626,7cyclictest37-21ksoftirqd/307:40:013
1171668993623,11cyclictest37-21ksoftirqd/311:40:503
1171668993622,10cyclictest37-21ksoftirqd/311:16:043
1171668993622,10cyclictest1173762-21sed07:50:023
1171668993622,10cyclictest1173762-21sed07:50:013
117166899361,3cyclictest131rcu_preempt10:16:073
1171668993512,18cyclictest275-21dbus-daemon08:30:013
117166899342,3cyclictest131rcu_preempt09:05:023
1171665993312,18cyclictest16850irq/134-800000007:10:020
1171665993312,18cyclictest16850irq/134-800000007:10:010
117166899324,22cyclictest3467602-21systemd-journal08:34:593
1171668993019,8cyclictest1171716-21wc07:10:023
1171668993019,8cyclictest1171716-21wc07:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional