You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-09 - 23:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat May 09, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30941189910399,2cyclictest0-21swapper/011:50:020
30941189910399,2cyclictest0-21swapper/011:50:010
3094118998579,3cyclictest0-21swapper/007:25:020
309412399765,11cyclictest131rcu_preempt12:10:011
3094128997310,50cyclictest0-21swapper/309:10:023
3094128996961,6cyclictest37-21ksoftirqd/308:15:023
3094123996818,11cyclictest3134055-21cron10:19:591
3094123996721,7cyclictest23-21ksoftirqd/110:39:051
3094123996628,35cyclictest3159070-21sh11:11:531
3094123996628,35cyclictest3159070-21sh11:11:531
3094123996523,38cyclictest0-21swapper/111:45:011
309411899654,53cyclictest0-21swapper/009:55:030
3094128996318,9cyclictest37-21ksoftirqd/311:21:193
309412399635,6cyclictest131rcu_preempt10:55:021
3094118996327,32cyclictest3467602-21systemd-journal10:00:320
3094118996325,18cyclictest3201352-21ssh12:40:000
3094128996217,9cyclictest37-21ksoftirqd/309:46:363
3094128996215,10cyclictest37-21ksoftirqd/309:45:023
3094123996227,10cyclictest23-21ksoftirqd/111:15:231
309412399622,5cyclictest131rcu_preempt08:20:021
3094123996225,4cyclictest3200540-21sh12:38:121
3094123996214,11cyclictest275-21dbus-daemon10:50:011
3094123996123,7cyclictest23-21ksoftirqd/109:25:021
3094118996116,37cyclictest12-21ksoftirqd/011:47:400
3094128996019,9cyclictest37-21ksoftirqd/310:18:393
3094123996023,9cyclictest23-21ksoftirqd/111:51:071
3094123996023,9cyclictest23-21ksoftirqd/111:51:071
3094123996023,9cyclictest23-21ksoftirqd/107:15:011
3094123996018,8cyclictest309-21in:imuxsock12:10:251
309412899592,45cyclictest3110090-21sh09:30:023
3094128995923,8cyclictest37-21ksoftirqd/311:17:373
3094128995918,8cyclictest37-21ksoftirqd/308:40:003
3094123995924,10cyclictest23-21ksoftirqd/110:06:251
309412399592,23cyclictest3095884-21grep07:45:001
3094123995920,8cyclictest23-21ksoftirqd/112:02:181
3094123995920,8cyclictest23-21ksoftirqd/112:02:181
3094118995922,11cyclictest12-21ksoftirqd/007:30:000
3094118995913,42cyclictest12-21ksoftirqd/009:37:510
309412899585,18cyclictest3168473-21ssh11:31:303
3094128995814,9cyclictest37-21ksoftirqd/309:56:583
3094128995813,9cyclictest37-21ksoftirqd/312:12:403
3094128995813,10cyclictest37-21ksoftirqd/308:35:023
3094123995823,15cyclictest275-21dbus-daemon09:45:011
3094123995818,11cyclictest3165318-21sed11:25:021
3094118995841,13cyclictest275-21dbus-daemon10:25:010
3094118995827,15cyclictest3106288-21ssh09:22:010
309411899581,40cyclictest3142592-21ssh10:37:430
3094123995720,9cyclictest3172503-21latency_hist11:40:021
3094118995753,2cyclictest0-21swapper/010:35:020
3094118995729,13cyclictest3127093-21ssh10:05:120
3094118995728,12cyclictest3192973-21ssh12:22:230
3094118995716,7cyclictest12-21ksoftirqd/010:55:000
3094118995716,4cyclictest12-21ksoftirqd/009:29:340
3094128995637,14cyclictest275-21dbus-daemon12:05:013
3094128995637,14cyclictest275-21dbus-daemon12:05:013
3094128995617,8cyclictest37-21ksoftirqd/308:50:013
3094128995613,8cyclictest37-21ksoftirqd/310:09:153
3094128995613,8cyclictest37-21ksoftirqd/309:50:573
3094128995610,10cyclictest37-21ksoftirqd/311:42:083
3094125995648,5cyclictest3097987-21kworker/2:111:37:362
3094125995647,6cyclictest3099260-21kworker/2:209:11:152
3094123995627,9cyclictest23-21ksoftirqd/110:03:191
3094118995628,11cyclictest3196594-21ssh12:29:590
3094118995627,13cyclictest3115689-21ssh09:41:340
3094118995626,14cyclictest3122874-21ssh09:56:400
309412899557,6cyclictest37-21ksoftirqd/309:11:013
3094128995511,10cyclictest37-21ksoftirqd/312:27:283
3094128995510,8cyclictest37-21ksoftirqd/310:48:233
3094125995546,5cyclictest3131589-21kworker/2:311:01:542
3094125995546,5cyclictest3131589-21kworker/2:311:01:532
3094123995524,4cyclictest3095591-21cron07:40:001
3094123995523,4cyclictest3097960-21latency_hist08:25:011
3094123995518,10cyclictest23-21ksoftirqd/111:09:101
3094118995528,12cyclictest3157342-21ssh11:08:360
3094118995526,14cyclictest3166198-21ssh11:26:480
3094128995415,8cyclictest37-21ksoftirqd/312:30:053
3094128995411,13cyclictest37-21ksoftirqd/311:47:363
3094128995411,10cyclictest37-21ksoftirqd/309:31:183
3094128995410,10cyclictest37-21ksoftirqd/310:29:333
3094125995444,7cyclictest3097987-21kworker/2:110:31:072
3094125995435,15cyclictest275-21dbus-daemon10:20:012
3094123995419,11cyclictest1-21systemd07:20:011
309412899539,9cyclictest37-21ksoftirqd/309:17:073
3094128995311,7cyclictest37-21ksoftirqd/310:30:093
3094128995311,7cyclictest37-21ksoftirqd/309:23:513
3094128995310,9cyclictest37-21ksoftirqd/309:38:023
309412599535,19cyclictest3129882-21kworker/u8:312:15:032
3094125995344,5cyclictest3131589-21kworker/2:311:18:402
3094123995322,6cyclictest23-21ksoftirqd/110:58:211
3094123995322,10cyclictest23-21ksoftirqd/110:32:001
3094123995317,10cyclictest23-21ksoftirqd/111:49:491
3094118995349,2cyclictest0-21swapper/008:00:020
3094118995324,14cyclictest3191019-21ssh12:18:310
3094118995324,13cyclictest3133959-21ssh10:19:480
309411899531,9cyclictest12-21ksoftirqd/011:01:540
309411899531,9cyclictest12-21ksoftirqd/011:01:530
3094128995210,9cyclictest37-21ksoftirqd/311:27:373
3094125995244,5cyclictest3097987-21kworker/2:112:09:142
3094123995221,10cyclictest23-21ksoftirqd/111:57:481
3094118995231,16cyclictest1-21systemd11:40:010
3094118995221,15cyclictest3172987-21ssh11:40:480
309412899519,9cyclictest37-21ksoftirqd/310:05:003
309412899518,8cyclictest3154786-21ssh11:03:103
309412899518,8cyclictest3154786-21ssh11:03:103
3094128995113,6cyclictest37-21ksoftirqd/311:35:363
3094128995112,7cyclictest37-21ksoftirqd/312:07:163
3094125995143,5cyclictest3174830-21kworker/2:212:36:072
3094125995143,5cyclictest3079738-21kworker/2:307:11:452
3094125995142,6cyclictest3174830-21kworker/2:212:20:092
3094125995142,6cyclictest3097987-21kworker/2:109:57:332
3094123995123,9cyclictest23-21ksoftirqd/110:28:171
3094123995123,5cyclictest3095163-21latency_hist07:30:011
3094123995119,11cyclictest23-21ksoftirqd/109:37:171
3094123995119,11cyclictest23-21ksoftirqd/109:30:391
3094123995119,10cyclictest23-21ksoftirqd/112:28:581
3094118995124,12cyclictest3197513-21ssh12:31:400
3094118995124,12cyclictest3110693-21ssh09:31:060
3094118995117,30cyclictest12-21ksoftirqd/009:12:070
3094118995116,8cyclictest131rcu_preempt10:28:230
3094128995035,11cyclictest1-21systemd08:20:013
3094128995013,10cyclictest37-21ksoftirqd/311:50:213
3094128995013,10cyclictest37-21ksoftirqd/311:50:213
3094125995043,5cyclictest3174830-21kworker/2:212:03:262
3094125995043,5cyclictest3174830-21kworker/2:212:03:262
3094125995043,4cyclictest3131589-21kworker/2:311:20:232
3094125995041,5cyclictest3097987-21kworker/2:110:01:372
3094123995032,13cyclictest275-21dbus-daemon09:30:011
3094123995021,9cyclictest23-21ksoftirqd/111:27:351
3094123995017,11cyclictest23-21ksoftirqd/110:24:201
3094118995041,4cyclictest16950irq/224-800000007:15:010
3094118995023,12cyclictest3160283-21ssh11:14:440
3094118995023,12cyclictest3160283-21ssh11:14:430
309412899498,7cyclictest37-21ksoftirqd/312:16:133
309412899494,15cyclictest3180033-21ssh11:55:223
3094128994923,17cyclictest3467602-21systemd-journal07:20:013
3094125994941,6cyclictest3097987-21kworker/2:110:13:442
3094125994941,5cyclictest3174830-21kworker/2:211:54:262
3094125994941,5cyclictest3174830-21kworker/2:211:54:262
3094125994941,5cyclictest3097987-21kworker/2:110:05:142
3094125994941,5cyclictest3083832-21kworker/2:208:15:012
3094125994940,6cyclictest3099260-21kworker/2:209:18:182
3094125994940,6cyclictest3097987-21kworker/2:111:26:242
3094123994922,7cyclictest23-21ksoftirqd/109:55:001
3094123994921,8cyclictest23-21ksoftirqd/109:16:071
309411899492,14cyclictest3100332-21sort09:10:020
3094118994913,30cyclictest131rcu_preempt08:09:240
309412899486,9cyclictest37-21ksoftirqd/310:54:303
309412899484,16cyclictest3095882-21munin-run07:45:013
3094128994812,9cyclictest37-21ksoftirqd/310:39:443
309412599485,3cyclictest131rcu_preempt10:49:572
3094125994841,4cyclictest3099260-21kworker/2:209:45:102
3094125994841,4cyclictest3097987-21kworker/2:110:26:212
3094125994840,5cyclictest3174830-21kworker/2:212:26:092
3094125994840,5cyclictest3174830-21kworker/2:211:56:102
3094125994840,5cyclictest3131589-21kworker/2:311:05:412
3094125994840,5cyclictest3099260-21kworker/2:209:44:332
3094125994840,5cyclictest3097987-21kworker/2:111:44:362
3094125994840,5cyclictest3097987-21kworker/2:109:54:212
309411899484,11cyclictest12-21ksoftirqd/011:30:120
3094118994823,10cyclictest3180081-21ssh11:55:320
3094118994822,11cyclictest3151757-21ssh10:56:410
3094118994821,12cyclictest3184413-21ssh12:04:530
3094118994821,12cyclictest3184413-21ssh12:04:520
309412899475,8cyclictest3100142-21cat09:05:023
309412599474,39cyclictest291rcuc/207:10:022
3094125994730,12cyclictest30-21ksoftirqd/209:40:002
309412599471,3cyclictest131rcu_preempt10:44:342
309411899478,5cyclictest131rcu_preempt12:10:100
3094118994727,13cyclictest275-21dbus-daemon07:40:020
3094118994720,12cyclictest3104244-21ssh09:17:560
309411899471,8cyclictest309-21in:imuxsock09:04:590
309412899469,7cyclictest37-21ksoftirqd/312:22:333
309412899469,4cyclictest131rcu_preempt10:10:403
309412899468,8cyclictest37-21ksoftirqd/310:57:193
3094128994630,13cyclictest37-21ksoftirqd/311:07:263
3094125994638,5cyclictest3099260-21kworker/2:209:28:272
3094125994637,6cyclictest3131589-21kworker/2:311:13:072
3094125994637,6cyclictest3131589-21kworker/2:311:13:062
3094125994628,13cyclictest275-21dbus-daemon08:45:012
309411899468,12cyclictest12-21ksoftirqd/010:10:020
309411899467,5cyclictest131rcu_preempt12:05:190
3094118994632,10cyclictest3467602-21systemd-journal07:52:250
3094118994629,12cyclictest3467602-21systemd-journal08:35:010
3094118994623,8cyclictest3146189-21ssh10:45:040
3094118994620,11cyclictest3118347-21ssh09:47:100
3094125994535,7cyclictest3131589-21kworker/2:310:50:572
3094125994533,9cyclictest1-21systemd11:50:012
3094125994532,10cyclictest3095589-21cron07:40:012
3094125994532,10cyclictest30-21ksoftirqd/209:30:052
3094125994526,15cyclictest275-21dbus-daemon08:35:012
3094125994525,14cyclictest275-21dbus-daemon07:35:002
309412599452,3cyclictest131rcu_preempt12:31:332
3094123994532,9cyclictest1-21systemd07:35:011
3094123994531,9cyclictest275-21dbus-daemon08:10:001
3094118994538,5cyclictest0-21swapper/007:15:090
3094118994536,5cyclictest16950irq/224-800000008:20:010
3094118994528,13cyclictest3144522-21ssh10:41:460
3094118994527,14cyclictest3161487-21ssh11:16:580
309411899452,39cyclictest0-21swapper/008:40:000
3094128994431,10cyclictest3094867-21cat07:25:013
3094125994435,6cyclictest3097987-21kworker/2:110:36:542
309412599443,3cyclictest131rcu_preempt10:22:312
3094125994432,9cyclictest1-21systemd08:10:012
3094123994434,8cyclictest23-21ksoftirqd/109:15:031
3094118994428,13cyclictest3163906-21ssh11:21:560
3094128994332,9cyclictest3200871-21ssh12:38:533
3094128994327,12cyclictest1-21systemd08:30:013
3094128994327,12cyclictest1-21systemd08:30:003
3094128994326,12cyclictest3467602-21systemd-journal08:00:003
309412899431,16cyclictest0-21swapper/307:35:013
309412599434,3cyclictest131rcu_preempt09:22:532
3094125994335,5cyclictest3131589-21kworker/2:310:56:212
3094125994335,5cyclictest3097987-21kworker/2:112:12:292
3094125994327,11cyclictest30-21ksoftirqd/208:25:492
3094125994327,11cyclictest30-21ksoftirqd/208:25:482
3094125994320,17cyclictest3467602-21systemd-journal07:25:012
309412899423,9cyclictest37-21ksoftirqd/308:51:253
3094125994232,7cyclictest3083832-21kworker/2:207:52:222
3094118994232,7cyclictest12-21ksoftirqd/007:35:010
309411899421,8cyclictest131rcu_preempt07:10:010
309412899415,8cyclictest37-21ksoftirqd/311:10:083
309412899415,8cyclictest37-21ksoftirqd/311:10:073
3094128994127,11cyclictest3143996-21ssh10:40:303
3094128994126,12cyclictest3095567-21sshd07:35:113
3094128994125,13cyclictest3134400-21ssh10:20:193
3094125994134,4cyclictest3097182-21kworker/2:308:20:592
3094125994118,6cyclictest131rcu_preempt07:45:002
3094123994134,3cyclictest23-21ksoftirqd/108:00:031
3094123994132,7cyclictest23-21ksoftirqd/112:35:031
3094118994131,8cyclictest12-21ksoftirqd/008:48:290
3094118994130,8cyclictest3097962-21munin-plugin-st08:25:000
3094118994127,6cyclictest3099472-21kworker/u8:109:00:030
3094118994127,6cyclictest3099472-21kworker/u8:109:00:030
3094128994024,11cyclictest37-21ksoftirqd/308:00:283
3094125994032,5cyclictest3097987-21kworker/2:111:31:382
3094125994029,8cyclictest3100291-21/usr/sbin/munin09:10:022
3094123994028,9cyclictest3467602-21systemd-journal07:50:011
3094123994026,11cyclictest3096657-21tr08:00:011
309411899407,6cyclictest131rcu_preempt07:45:010
3094128993918,16cyclictest364-21snmpd07:14:483
309412399392,9cyclictest150-21pr/ttyS210:11:211
3094123993927,9cyclictest23-21ksoftirqd/111:35:031
3094123993925,10cyclictest23-21ksoftirqd/110:42:291
3094118993927,10cyclictest12-21ksoftirqd/008:55:000
3094128993827,8cyclictest3096353-21sshd07:50:243
3094128993824,10cyclictest37-21ksoftirqd/308:21:023
3094128993822,11cyclictest309-21in:imuxsock08:45:003
309412599384,4cyclictest131rcu_preempt08:00:002
309412599383,3cyclictest131rcu_preempt08:20:002
3094125993829,6cyclictest309-21in:imuxsock07:20:002
3094125993824,10cyclictest30-21ksoftirqd/208:02:492
3094125993823,11cyclictest3099308-21wc08:50:022
3094123993828,7cyclictest23-21ksoftirqd/111:05:021
3094123993828,7cyclictest23-21ksoftirqd/111:05:021
3094123993825,9cyclictest3194251-21cat12:25:011
3094123993825,8cyclictest23-21ksoftirqd/109:10:021
3094123993825,10cyclictest23-21ksoftirqd/112:15:171
3094118993826,9cyclictest12-21ksoftirqd/007:49:590
3094128993723,10cyclictest48-21kcompactd008:56:033
3094128993723,10cyclictest48-21kcompactd008:56:023
3094123993724,10cyclictest23-21ksoftirqd/108:15:011
309411899372,5cyclictest3467602-21systemd-journal08:04:410
309411899372,31cyclictest0-21swapper/008:15:000
3094128993618,12cyclictest3095087-21cron07:30:003
3094125993625,9cyclictest30-21ksoftirqd/208:54:232
3094125993624,9cyclictest30-21ksoftirqd/207:50:002
3094125993620,11cyclictest275-21dbus-daemon07:30:002
3094123993626,5cyclictest309-21in:imuxsock08:32:231
3094123993625,8cyclictest3119592-21cat09:50:021
3094128993527,4cyclictest3097134-21sshd08:05:243
3094125993527,4cyclictest3097182-21kworker/2:308:40:002
309412399353,5cyclictest131rcu_preempt08:40:001
3094123993524,9cyclictest23-21ksoftirqd/109:57:341
3094123993524,9cyclictest23-21ksoftirqd/107:55:001
3094123993518,13cyclictest275-21dbus-daemon08:55:011
3094118993529,4cyclictest0-21swapper/008:30:010
3094118993529,4cyclictest0-21swapper/008:30:000
3094128993426,5cyclictest3094049-21latency_hist07:10:023
309412399345,4cyclictest131rcu_preempt08:45:391
3094123993424,8cyclictest23-21ksoftirqd/108:55:031
3094123993424,8cyclictest23-21ksoftirqd/108:55:021
309411899341,29cyclictest0-21swapper/008:41:230
3094128993320,9cyclictest364-21snmpd07:45:393
3094125993321,10cyclictest30-21ksoftirqd/209:04:592
3094123993325,5cyclictest3098317-21latency_hist08:30:021
3094123993325,5cyclictest3098317-21latency_hist08:30:021
3094123993322,8cyclictest23-21ksoftirqd/109:05:011
3094123993322,7cyclictest0-21swapper/107:10:021
3094123993319,10cyclictest1-21systemd07:25:011
3094125993223,6cyclictest3099741-21cron09:00:012
3094125993223,6cyclictest3099741-21cron09:00:012
3094123993023,5cyclictest23-21ksoftirqd/108:45:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional