You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 15:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 07, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
116686899921,87cyclictest1245442-21ssh11:41:170
116686899883,8cyclictest1258927-21munin-run12:10:000
1166868998727,56cyclictest1227996-21ssh11:04:570
1166868998426,40cyclictest1232346-21ssh11:13:570
1166868998376,3cyclictest0-21swapper/007:25:020
1166868998328,51cyclictest1253840-21ssh11:59:170
1166868998328,51cyclictest1233316-21ssh11:15:460
1166868998328,37cyclictest1195640-21ssh09:56:560
1166868998128,36cyclictest1197748-21ssh10:01:120
1166868998029,33cyclictest1179579-21ssh09:23:210
116686899802,6cyclictest1237584-21munin-run11:25:010
1166868997930,44cyclictest16950irq/224-800000012:15:010
1166868997820,39cyclictest1204991-21ssh10:16:320
1166868997724,48cyclictest1199781-21ssh10:05:260
1166868997722,51cyclictest1216385-21ssh10:40:030
116686899751,56cyclictest1185418-21ssh09:35:170
116686899749,6cyclictest16850irq/134-800000010:54:480
1166868997436,34cyclictest16950irq/224-800000007:40:000
1166868997436,21cyclictest1247585-21ssh11:45:470
116686899742,6cyclictest3467602-21systemd-journal12:02:510
116686899741,68cyclictest1240033-21date11:30:010
1166868997332,37cyclictest16950irq/224-800000012:25:010
1166868997331,38cyclictest1193737-21ssh09:52:580
1166868997330,24cyclictest1219659-21ssh10:47:140
1166879997227,40cyclictest275-21dbus-daemon10:35:013
116686899721,52cyclictest1174845-21ssh09:13:350
1166868997127,26cyclictest1268304-21ssh12:29:260
116686899711,6cyclictest1213653-21ssh10:34:560
116686899711,2cyclictest0-21swapper/011:07:280
1166879996929,34cyclictest1-21systemd12:40:013
1166879996923,9cyclictest1255886-21sshd12:03:283
1166868996927,26cyclictest1191971-21ssh09:49:260
1166868996914,51cyclictest16850irq/134-800000010:23:030
116686899691,12cyclictest1204090-21ssh10:15:000
116687299685,11cyclictest23-21ksoftirqd/107:20:001
116686899682,16cyclictest1172015-21sshd08:46:280
1166868996715,47cyclictest16850irq/134-800000012:36:020
1166868996659,3cyclictest0-21swapper/007:45:020
1166868996659,3cyclictest0-21swapper/007:45:020
1166868996630,33cyclictest0-21swapper/012:35:010
1166868996629,33cyclictest1177442-21ssh09:19:040
1166868996629,33cyclictest1177442-21ssh09:19:040
1166868996615,46cyclictest16850irq/134-800000009:41:320
116686899661,4cyclictest1210526-21ssh10:28:030
1166872996529,12cyclictest23-21ksoftirqd/108:40:001
116686899651,58cyclictest1263807-21unin-run12:20:010
1166868996511,49cyclictest16850irq/134-800000010:55:370
1166879996322,5cyclictest1167413-21latency_hist07:20:023
1166879996322,10cyclictest1185233-21sed09:35:023
1166879996318,11cyclictest37-21ksoftirqd/311:24:033
1166868996328,30cyclictest48-21kcompactd011:33:300
116686899632,47cyclictest16950irq/224-800000010:40:020
116686899631,10cyclictest1167421-21latency_hist07:20:020
116687999623,19cyclictest1235329-21wc11:20:023
1166879996217,8cyclictest1259364-21sshd12:10:293
116686899629,12cyclictest1180883-21ssh09:25:510
116686899621,43cyclictest1159489-21kworker/u8:007:45:030
116686899621,12cyclictest48-21kcompactd009:30:250
116687999614,15cyclictest1242282-21modprobe11:35:003
116687999613,17cyclictest3467602-21systemd-journal07:50:003
116686899616,45cyclictest125067850ssh11:52:310
116686899612,6cyclictest1244671-21munin-run11:40:000
116686899612,5cyclictest1171025-21munin-run08:30:010
116686899612,55cyclictest1171492-21sshd08:38:280
1166879996040,15cyclictest1-21systemd09:40:013
1166879996024,5cyclictest1193664-21sh09:52:493
1166879996022,6cyclictest1206799-21latency_hist10:20:023
1166872996012,7cyclictest131rcu_preempt11:40:011
116686899601,8cyclictest1171772-21sshd08:43:280
116686899601,13cyclictest1172276-21sshd08:53:280
1166879995925,31cyclictest1243605-21sh11:37:283
116687999591,17cyclictest1180327-21cat09:25:013
1166877995951,5cyclictest1199478-21kworker/2:210:35:022
1166868995929,27cyclictest12-21ksoftirqd/007:15:010
116686899592,13cyclictest1172594-21munin-run09:00:010
116686899592,13cyclictest1172594-21munin-run09:00:010
1166868995910,44cyclictest1169979-21unin-run08:10:010
116687999585,18cyclictest3467602-21systemd-journal10:30:033
1166879995828,25cyclictest1263606-21sh12:19:423
1166879995825,30cyclictest1267320-21sh12:27:123
1166879995815,12cyclictest120-21kswapd010:45:003
1166879995813,7cyclictest1168386-21cron07:40:003
1166879995812,14cyclictest275-21dbus-daemon10:50:013
1166877995849,6cyclictest1159521-21kworker/2:309:13:432
116686899581,15cyclictest1169477-21munin-run08:00:010
116686899581,15cyclictest1169477-21munin-run08:00:010
116686899581,10cyclictest48-21kcompactd009:05:210
1166879995723,7cyclictest37-21ksoftirqd/311:09:293
116687999572,19cyclictest1173385-21ssh09:10:093
1166879995720,7cyclictest37-21ksoftirqd/309:59:493
116686899573,6cyclictest1170450-21sshd08:17:270
116686899573,6cyclictest1170450-21sshd08:17:270
116687999562,9cyclictest1170477-21modprobe08:20:003
116687999562,9cyclictest1170477-21modprobe08:20:003
1166879995626,26cyclictest1214586-21sh10:36:333
116687999562,20cyclictest1167241-21cat07:15:023
1166879995620,9cyclictest37-21ksoftirqd/312:00:003
1166879995615,9cyclictest1167600-21date07:20:033
1166877995647,5cyclictest1159521-21kworker/2:307:10:012
1166877995635,15cyclictest3467602-21systemd-journal10:44:592
116687299567,11cyclictest23-21ksoftirqd/112:40:001
116686899562,9cyclictest1168115-21sshd07:31:270
1166877995549,3cyclictest1242305-21kworker/2:012:18:312
1166877995546,6cyclictest1211380-21kworker/2:311:45:292
1166877995546,5cyclictest1242305-21kworker/2:012:27:312
1166877995546,5cyclictest1199478-21kworker/2:210:28:022
1166877995544,6cyclictest1211380-21kworker/2:311:27:552
1166877995539,6cyclictest1211380-21kworker/2:311:51:472
1166877995528,8cyclictest1182666-21kworker/2:009:53:312
116686899552,7cyclictest1167861-21sshd07:26:270
1166868995511,39cyclictest1170913-21latency_hist08:25:020
116687999542,20cyclictest1199528-21latency_hist10:05:023
116687999542,20cyclictest1169189-21true07:55:003
1166879995418,5cyclictest1189427-21sh09:44:033
1166879995416,5cyclictest1170356-21latency_hist08:15:043
1166877995446,5cyclictest1211380-21kworker/2:311:24:362
1166877995446,5cyclictest1182666-21kworker/2:009:59:532
1166877995445,5cyclictest1199478-21kworker/2:210:21:282
1166877995444,6cyclictest1159521-21kworker/2:309:21:032
1166877995436,9cyclictest1211380-21kworker/2:311:56:112
116687299545,6cyclictest131rcu_preempt10:00:301
1166872995440,8cyclictest23-21ksoftirqd/110:40:021
1166872995416,6cyclictest131rcu_preempt11:59:261
116686899541,5cyclictest48-21kcompactd008:00:260
116686899541,13cyclictest1169153-21sshd07:52:270
1166879995317,7cyclictest37-21ksoftirqd/310:10:463
1166879995316,9cyclictest37-21ksoftirqd/308:10:023
1166879995316,7cyclictest1227019-1kworker/u9:211:04:573
1166877995345,5cyclictest1242305-21kworker/2:012:31:442
1166877995345,5cyclictest1242305-21kworker/2:012:06:382
1166877995345,5cyclictest1211380-21kworker/2:311:38:362
1166877995345,5cyclictest1199478-21kworker/2:210:36:282
1166877995345,5cyclictest1182666-21kworker/2:010:06:022
1166877995345,5cyclictest1182666-21kworker/2:009:45:532
1166877995344,6cyclictest1242305-21kworker/2:012:38:122
1166877995344,5cyclictest1199478-21kworker/2:210:54:562
1166877995344,5cyclictest1182666-21kworker/2:010:12:362
1166877995343,6cyclictest1159521-21kworker/2:308:40:022
1166877995343,5cyclictest1242305-21kworker/2:012:00:142
1166877995342,6cyclictest1159521-21kworker/2:307:40:012
116686899538,40cyclictest16850irq/134-800000008:32:270
1166868995331,19cyclictest1170215-21munin-plugin-st08:15:000
1166879995223,5cyclictest1269874-21sh12:32:323
1166879995221,4cyclictest1181728-21sh09:27:483
1166879995218,7cyclictest37-21ksoftirqd/311:44:213
1166879995218,4cyclictest1264700-21sh12:21:353
1166879995215,9cyclictest37-21ksoftirqd/309:15:183
1166879995215,9cyclictest37-21ksoftirqd/309:15:183
1166877995244,5cyclictest1242305-21kworker/2:012:21:402
1166877995244,5cyclictest1211380-21kworker/2:311:16:422
1166877995243,6cyclictest1211380-21kworker/2:311:31:222
1166877995243,6cyclictest1199478-21kworker/2:210:19:542
1166877995243,6cyclictest1159521-21kworker/2:307:33:352
1166877995243,5cyclictest1242305-21kworker/2:012:12:242
1166877995243,5cyclictest1199478-21kworker/2:210:55:452
1166877995243,5cyclictest1159521-21kworker/2:308:00:012
1166877995243,5cyclictest1159521-21kworker/2:308:00:012
1166877995242,5cyclictest1182666-21kworker/2:009:41:092
1166877995241,7cyclictest1211380-21kworker/2:311:09:102
1166877995231,16cyclictest1-21systemd09:35:012
116687299522,12cyclictest23-21ksoftirqd/110:14:281
1166879995118,4cyclictest1168813-21latency_hist07:45:023
1166879995118,4cyclictest1168813-21latency_hist07:45:023
1166879995117,10cyclictest37-21ksoftirqd/309:45:393
1166877995143,5cyclictest1182666-21kworker/2:010:01:402
1166877995143,5cyclictest1159521-21kworker/2:308:58:522
1166877995143,5cyclictest1159521-21kworker/2:308:58:512
1166877995142,6cyclictest1159521-21kworker/2:309:19:072
1166877995142,6cyclictest1159521-21kworker/2:309:19:072
1166877995142,5cyclictest1159521-21kworker/2:308:50:012
1166877995142,5cyclictest1159521-21kworker/2:307:30:022
116687299516,6cyclictest131rcu_preempt09:39:001
116687999503,19cyclictest284-21systemd-logind08:30:013
1166879995021,7cyclictest37-21ksoftirqd/310:53:213
1166877995042,5cyclictest1159521-21kworker/2:309:26:412
1166877995042,4cyclictest1182666-21kworker/2:009:36:002
1166877995041,6cyclictest1199478-21kworker/2:210:49:202
1166877995041,5cyclictest1159521-21kworker/2:309:05:002
116687299507,5cyclictest131rcu_preempt10:56:561
1166872995028,17cyclictest275-21dbus-daemon12:35:001
116687299502,7cyclictest23-21ksoftirqd/110:24:291
116687299501,19cyclictest0-21swapper/110:29:061
116686899502,7cyclictest1173011-21latency_hist09:05:020
116687999499,6cyclictest131rcu_preempt10:58:183
1166879994935,11cyclictest1209065-21latency_hist10:25:013
1166879994921,9cyclictest37-21ksoftirqd/311:51:083
1166879994920,7cyclictest37-21ksoftirqd/311:26:033
1166879994919,7cyclictest37-21ksoftirqd/311:49:473
1166879994917,9cyclictest1183599-1kworker/u9:210:07:113
1166877994939,6cyclictest1159521-21kworker/2:307:50:002
1166872994917,6cyclictest131rcu_preempt11:19:361
1166872994915,6cyclictest131rcu_preempt11:10:011
1166872994913,9cyclictest131rcu_preempt09:29:021
1166872994911,6cyclictest131rcu_preempt09:10:001
1166877994840,5cyclictest1211380-21kworker/2:311:42:132
1166877994840,5cyclictest1211380-21kworker/2:311:11:522
1166877994840,5cyclictest1159521-21kworker/2:308:35:022
116687299488,5cyclictest131rcu_preempt11:33:281
116687299486,8cyclictest23-21ksoftirqd/110:45:001
116687299485,5cyclictest131rcu_preempt11:51:001
1166872994826,16cyclictest1-21systemd08:35:011
1166872994814,6cyclictest131rcu_preempt09:35:011
116687299481,15cyclictest0-21swapper/109:22:441
1166872994810,6cyclictest131rcu_preempt10:50:531
1166879994732,11cyclictest1173071-21cron09:10:003
1166879994717,6cyclictest1170800-21date08:25:013
1166877994738,5cyclictest1211380-21kworker/2:311:02:102
116687299479,5cyclictest131rcu_preempt11:13:411
1166872994738,7cyclictest23-21ksoftirqd/109:15:141
1166872994738,7cyclictest23-21ksoftirqd/109:15:141
1166872994738,6cyclictest23-21ksoftirqd/109:57:161
1166872994736,9cyclictest23-21ksoftirqd/107:35:271
1166872994736,7cyclictest23-21ksoftirqd/112:01:431
1166872994732,11cyclictest23-21ksoftirqd/111:50:001
1166872994725,13cyclictest23-21ksoftirqd/109:15:021
1166872994714,9cyclictest23-21ksoftirqd/110:09:181
1166872994712,8cyclictest131rcu_preempt12:25:001
116687299471,15cyclictest0-21swapper/112:18:091
116687299471,14cyclictest0-21swapper/111:23:181
1166877994638,5cyclictest1159521-21kworker/2:308:28:272
1166877994638,5cyclictest1159521-21kworker/2:308:15:012
1166877994638,5cyclictest1159521-21kworker/2:308:10:022
116687299468,5cyclictest131rcu_preempt11:30:021
116687299464,7cyclictest131rcu_preempt10:20:021
1166872994635,8cyclictest23-21ksoftirqd/109:55:011
1166872994613,5cyclictest131rcu_preempt12:27:001
116686899461,12cyclictest1167022-21latency_hist07:10:020
116687999453,8cyclictest131rcu_preempt08:40:003
1166879994528,13cyclictest1169423-21cron08:00:003
1166879994528,13cyclictest1169423-21cron08:00:003
1166877994537,5cyclictest1159521-21kworker/2:308:15:562
1166877994537,5cyclictest1159521-21kworker/2:308:15:562
1166872994537,6cyclictest23-21ksoftirqd/109:44:391
1166872994532,9cyclictest23-21ksoftirqd/110:34:381
1166872994511,6cyclictest131rcu_preempt08:20:001
1166872994511,6cyclictest131rcu_preempt08:20:001
1166877994436,5cyclictest1159521-21kworker/2:308:25:022
1166877994436,5cyclictest1159521-21kworker/2:307:55:012
1166872994434,7cyclictest23-21ksoftirqd/108:50:011
1166872994433,9cyclictest23-21ksoftirqd/111:41:171
1166872994430,11cyclictest23-21ksoftirqd/108:04:271
1166872994429,11cyclictest23-21ksoftirqd/111:04:371
1166872994426,13cyclictest275-21dbus-daemon09:50:001
1166872994413,7cyclictest23-21ksoftirqd/108:25:011
1166872994412,6cyclictest131rcu_preempt12:14:511
1166872994411,6cyclictest131rcu_preempt12:09:331
1166879994331,7cyclictest1171283-21latency_hist08:35:013
1166879994319,7cyclictest37-21ksoftirqd/311:12:453
1166877994335,5cyclictest1159521-21kworker/2:308:55:022
1166877994335,5cyclictest1159521-21kworker/2:308:45:002
1166877994335,5cyclictest1159521-21kworker/2:307:20:032
1166877994334,6cyclictest1159521-21kworker/2:307:23:272
116687299436,10cyclictest23-21ksoftirqd/110:46:181
1166872994332,8cyclictest23-21ksoftirqd/109:05:001
116687299431,9cyclictest0-21swapper/107:51:271
1166872994310,5cyclictest131rcu_preempt07:45:021
1166872994310,5cyclictest131rcu_preempt07:45:021
1166879994229,9cyclictest1-21systemd07:35:013
1166879994226,11cyclictest275-21dbus-daemon09:05:003
1166879994222,14cyclictest275-21dbus-daemon08:05:003
1166877994226,10cyclictest275-21dbus-daemon07:15:002
116687299423,33cyclictest1171506-21kworker/u8:108:45:021
1166872994231,8cyclictest23-21ksoftirqd/107:35:011
1166872994230,8cyclictest3467602-21systemd-journal09:00:001
1166872994230,8cyclictest3467602-21systemd-journal09:00:001
1166872994226,13cyclictest23-21ksoftirqd/107:30:001
1166877994123,13cyclictest30-21ksoftirqd/209:10:002
1166872994128,8cyclictest1169424-21cron08:00:001
1166872994128,8cyclictest1169424-21cron08:00:001
1166872994125,13cyclictest3467602-21systemd-journal07:25:001
1166872994124,12cyclictest3467602-21systemd-journal08:55:011
1166872994118,18cyclictest131rcu_preempt08:10:011
1166872994032,6cyclictest23-21ksoftirqd/107:10:261
1166872994030,7cyclictest221rcuc/108:15:021
1166879993927,10cyclictest37-21ksoftirqd/308:50:023
1166877993925,11cyclictest30-21ksoftirqd/208:05:022
1166879993826,6cyclictest309-21in:imuxsock09:00:003
1166879993826,6cyclictest309-21in:imuxsock09:00:003
116687299389,16cyclictest1166859-21cyclictest07:10:021
116687299383,9cyclictest23-21ksoftirqd/107:50:001
1166879993726,9cyclictest37-21ksoftirqd/312:05:043
1166879993723,11cyclictest1167055-21sed07:10:023
1166879993721,11cyclictest275-21dbus-daemon08:55:003
1166872993627,7cyclictest23-21ksoftirqd/108:30:011
1166877993423,8cyclictest30-21ksoftirqd/207:44:222
1166877993423,8cyclictest30-21ksoftirqd/207:44:222
1166879993124,4cyclictest1171793-21latency_hist08:45:023
1166879992821,4cyclictest1167910-21latency_hist07:30:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional