You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 15:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 17, 2026 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34942559910823,15cyclictest12-21ksoftirqd/008:35:030
349425599701,31cyclictest3533213-21rm10:23:070
349425599693,45cyclictest12-21ksoftirqd/010:41:300
349425599693,45cyclictest12-21ksoftirqd/010:41:290
349425599671,44cyclictest0-21swapper/011:41:120
3494260996616,8cyclictest23-21ksoftirqd/111:00:011
3494260996513,9cyclictest23-21ksoftirqd/110:15:341
3494255996538,8cyclictest12-21ksoftirqd/010:01:510
349425599635,6cyclictest121ksoftirqd/011:31:390
3494264996254,5cyclictest3482593-21kworker/2:109:14:382
349425599628,10cyclictest3585886-21ssh12:20:460
3494255996222,8cyclictest12-21ksoftirqd/011:20:000
3494255996219,8cyclictest12-21ksoftirqd/008:10:000
3494255996213,9cyclictest12-21ksoftirqd/009:25:010
3494260996115,7cyclictest23-21ksoftirqd/112:27:131
349426099611,21cyclictest3545910-21ssh10:51:171
3494255996127,8cyclictest12-21ksoftirqd/010:46:460
349426099605,15cyclictest3549890-21sh11:00:061
349426099603,15cyclictest3584300-21ssh12:17:181
3494260996016,9cyclictest23-21ksoftirqd/109:20:011
3494255996015,8cyclictest12-21ksoftirqd/011:02:510
349425599601,27cyclictest12-21ksoftirqd/010:51:040
349426099593,19cyclictest3580197-21ssh12:08:131
3494260995920,9cyclictest131rcu_preempt09:27:001
3494260995920,9cyclictest131rcu_preempt09:27:001
3494260995916,8cyclictest23-21ksoftirqd/109:21:231
349425599596,22cyclictest12-21ksoftirqd/012:35:000
3494264995850,5cyclictest3497346-21kworker/2:210:25:012
349426099585,18cyclictest3513536-21sh09:39:041
3494260995814,9cyclictest23-21ksoftirqd/109:50:021
3494260995812,9cyclictest23-21ksoftirqd/110:27:341
3494260995812,9cyclictest23-21ksoftirqd/110:27:341
3494255995828,14cyclictest3553903-21ssh11:09:330
3494264995749,5cyclictest3482593-21kworker/2:109:34:402
349426099573,21cyclictest3510542-21ssh09:32:171
3494260995720,8cyclictest131rcu_preempt12:24:211
3494260995715,8cyclictest23-21ksoftirqd/111:30:021
3494260995715,7cyclictest23-21ksoftirqd/112:32:131
349426099571,21cyclictest3565579-21ssh11:35:121
3494255995732,7cyclictest150-21pr/ttyS209:10:020
349425599571,29cyclictest12-21ksoftirqd/010:19:410
3494264995648,5cyclictest3482593-21kworker/2:109:43:492
349426099566,12cyclictest3543700-21sh10:46:161
3494260995615,9cyclictest23-21ksoftirqd/109:13:231
3494260995614,7cyclictest23-21ksoftirqd/110:13:531
349425599565,8cyclictest12-21ksoftirqd/009:36:410
3494255995636,16cyclictest1-21systemd07:30:000
3494255995622,7cyclictest12-21ksoftirqd/009:00:010
3494264995545,6cyclictest3534042-21kworker/2:111:26:272
349426099557,17cyclictest3556845-21sh11:15:431
3494260995516,9cyclictest23-21ksoftirqd/109:55:051
3494260995516,7cyclictest0-21swapper/110:25:011
3494255995542,8cyclictest3594236-21modprobe12:40:000
349425599554,15cyclictest3497815-21cron08:20:000
349425599554,11cyclictest12-21ksoftirqd/011:24:230
3494255995521,30cyclictest16850irq/134-800000009:57:030
3494255995520,31cyclictest16850irq/134-800000012:12:310
3494255995519,32cyclictest16850irq/134-800000012:18:470
3494255995519,32cyclictest16850irq/134-800000010:08:560
3494255995519,32cyclictest16850irq/134-800000010:08:560
3494255995518,32cyclictest16850irq/134-800000011:35:350
3494255995517,33cyclictest16850irq/134-800000012:06:520
3494264995446,5cyclictest3497346-21kworker/2:210:27:152
3494264995446,5cyclictest3497346-21kworker/2:210:27:142
3494264995445,6cyclictest3534042-21kworker/2:111:07:012
349426099549,12cyclictest3574669-21ssh11:55:341
349426099549,10cyclictest23-21ksoftirqd/110:41:281
349426099549,10cyclictest23-21ksoftirqd/110:41:281
349426099546,15cyclictest3560532-21ssh11:24:171
3494260995412,9cyclictest0-21swapper/112:03:581
3494260995410,7cyclictest3553463-21ssh11:08:301
349425599546,10cyclictest12-21ksoftirqd/010:56:120
3494255995445,6cyclictest151rcuc/010:35:000
3494255995445,6cyclictest151rcuc/009:50:200
3494255995426,12cyclictest3577845-21ssh12:02:580
3494255995420,31cyclictest16850irq/134-800000009:17:270
3494255995419,32cyclictest16850irq/134-800000011:59:140
3494255995418,32cyclictest16850irq/134-800000009:31:520
3494255995418,20cyclictest16850irq/134-800000010:26:560
3494255995418,20cyclictest16850irq/134-800000010:26:560
3494255995417,32cyclictest16850irq/134-800000011:10:380
3494255995416,33cyclictest16850irq/134-800000012:25:020
349425599541,43cyclictest0-21swapper/009:15:030
3494264995345,5cyclictest3534042-21kworker/2:112:35:002
3494264995345,5cyclictest3534042-21kworker/2:111:30:132
3494264995345,5cyclictest3534042-21kworker/2:111:15:522
3494264995345,5cyclictest3534042-21kworker/2:110:55:442
3494264995345,5cyclictest3534042-21kworker/2:110:52:462
3494264995345,5cyclictest3482593-21kworker/2:109:28:542
3494264995345,5cyclictest3482593-21kworker/2:109:28:542
3494264995345,5cyclictest3482593-21kworker/2:108:45:012
3494264995345,5cyclictest3482593-21kworker/2:108:36:352
3494264995344,6cyclictest3482593-21kworker/2:109:35:132
3494264995344,5cyclictest3534042-21kworker/2:111:55:042
349426099536,18cyclictest3519555-21sh09:52:241
349426099535,16cyclictest3516077-21sh09:44:491
3494255995319,31cyclictest16850irq/134-800000010:36:010
3494255995319,31cyclictest16850irq/134-800000009:41:090
3494255995318,31cyclictest16850irq/134-800000009:46:050
3494255995317,32cyclictest16850irq/134-800000011:27:470
3494255995317,21cyclictest16850irq/134-800000011:49:480
3494269995236,13cyclictest1-21systemd09:00:013
3494264995244,5cyclictest3554124-21kworker/2:312:20:492
3494264995244,5cyclictest3554124-21kworker/2:312:19:212
3494264995244,5cyclictest3534042-21kworker/2:112:01:322
3494264995244,5cyclictest3534042-21kworker/2:111:54:002
3494264995244,5cyclictest3497346-21kworker/2:210:11:342
3494264995244,5cyclictest3497346-21kworker/2:210:02:532
3494264995244,5cyclictest3497346-21kworker/2:209:58:282
3494264995244,5cyclictest3497346-21kworker/2:209:51:032
3494264995244,4cyclictest3485113-21kworker/2:007:35:022
3494264995243,6cyclictest3482593-21kworker/2:109:20:262
3494264995243,5cyclictest3534042-21kworker/2:111:04:132
3494264995243,5cyclictest3497346-21kworker/2:210:16:002
3494264995243,5cyclictest3482593-21kworker/2:109:49:472
349426099528,7cyclictest3540430-21ssh10:39:241
349426099526,10cyclictest23-21ksoftirqd/110:09:121
349426099526,10cyclictest23-21ksoftirqd/110:09:121
3494260995213,8cyclictest23-21ksoftirqd/111:49:181
3494260995212,11cyclictest23-21ksoftirqd/112:13:341
3494260995210,8cyclictest23-21ksoftirqd/108:39:591
3494255995218,31cyclictest16850irq/134-800000009:28:020
3494255995218,31cyclictest16850irq/134-800000009:28:020
3494255995217,31cyclictest16850irq/134-800000011:52:240
3494264995144,4cyclictest3534042-21kworker/2:112:35:202
3494264995144,4cyclictest3497346-21kworker/2:210:46:152
3494264995143,5cyclictest3534042-21kworker/2:112:07:582
3494264995143,5cyclictest3534042-21kworker/2:111:41:072
3494264995143,5cyclictest3534042-21kworker/2:111:37:202
3494264995143,5cyclictest3534042-21kworker/2:111:20:522
3494264995143,5cyclictest3497346-21kworker/2:210:41:142
3494264995143,5cyclictest3497346-21kworker/2:210:41:142
3494264995143,5cyclictest3482593-21kworker/2:107:52:352
3494264995142,6cyclictest3534042-21kworker/2:111:46:212
349426099519,12cyclictest3593683-21rm12:38:421
349426099518,9cyclictest23-21ksoftirqd/111:32:141
349426099516,9cyclictest23-21ksoftirqd/110:03:481
3494260995113,8cyclictest23-21ksoftirqd/110:33:251
349425599518,23cyclictest0-21swapper/008:25:020
3494255995111,8cyclictest12-21ksoftirqd/010:12:290
3494264995043,4cyclictest3497346-21kworker/2:209:05:012
3494264995042,5cyclictest3554124-21kworker/2:312:26:032
3494264995042,5cyclictest3482593-21kworker/2:109:16:252
3494264995042,4cyclictest3497346-21kworker/2:210:05:562
3494264995042,4cyclictest3497346-21kworker/2:210:05:562
3494264995032,4cyclictest30-21ksoftirqd/212:11:422
349426099509,8cyclictest3499935-21systemd-run09:00:011
349426099507,7cyclictest23-21ksoftirqd/111:12:441
3494264994941,5cyclictest3497346-21kworker/2:210:37:112
3494264994941,5cyclictest3482593-21kworker/2:107:55:342
3494264994940,6cyclictest3497346-21kworker/2:210:33:092
3494264994939,6cyclictest3482593-21kworker/2:108:17:342
349426099496,14cyclictest3568241-21sh11:41:161
349426099495,4cyclictest150-21pr/ttyS208:55:021
349426099490,14cyclictest3498636-21ls08:35:021
3494264994840,5cyclictest3534042-21kworker/2:111:12:082
3494264994832,11cyclictest3499910-21systemctl09:00:012
3494260994817,7cyclictest131rcu_preempt07:40:001
3494269994728,14cyclictest1-21systemd11:20:013
3494264994739,5cyclictest3482593-21kworker/2:108:10:002
3494264994739,5cyclictest3482593-21kworker/2:107:49:432
349426099479,8cyclictest23-21ksoftirqd/108:15:011
3494260994732,11cyclictest3467602-21systemd-journal07:42:341
349426099471,9cyclictest23-21ksoftirqd/111:52:421
3494255994739,5cyclictest16950irq/224-800000008:40:010
349425599471,42cyclictest3496141-21latency_hist07:45:020
3494264994636,7cyclictest3498732-21latency_hist08:35:032
349426099467,7cyclictest23-21ksoftirqd/107:17:001
349426099467,7cyclictest23-21ksoftirqd/107:16:591
349425599468,7cyclictest131rcu_preempt07:50:010
349425599466,30cyclictest3494257-21cat07:10:020
349425599466,30cyclictest3494257-21cat07:10:010
3494269994530,12cyclictest3529630-21sed10:15:013
349426999452,3cyclictest131rcu_preempt12:19:403
349426999452,3cyclictest131rcu_preempt09:55:023
349426999451,9cyclictest131rcu_preempt12:25:013
3494269994515,7cyclictest131rcu_preempt09:45:013
3494264994535,6cyclictest3485113-21kworker/2:007:20:012
3494264994535,6cyclictest3485113-21kworker/2:007:20:012
3494264994533,6cyclictest309-21in:imuxsock07:30:012
3494260994521,18cyclictest275-21dbus-daemon07:30:011
3494255994531,10cyclictest3467602-21systemd-journal08:55:010
349426999445,3cyclictest150-21pr/ttyS211:55:003
3494269994431,9cyclictest1-21systemd11:35:013
3494269994424,16cyclictest275-21dbus-daemon11:50:013
3494264994436,5cyclictest3485113-21kworker/2:007:24:312
3494264994436,5cyclictest3482593-21kworker/2:108:25:022
3494264994436,5cyclictest3482593-21kworker/2:107:43:072
3494264994426,14cyclictest275-21dbus-daemon08:30:002
3494264994426,14cyclictest275-21dbus-daemon08:05:012
3494264994426,14cyclictest275-21dbus-daemon08:05:012
3494260994431,8cyclictest1-21systemd09:05:001
3494255994430,11cyclictest3496761-21sshd07:58:340
349425599442,8cyclictest3467602-21systemd-journal08:25:020
3494255994410,5cyclictest131rcu_preempt07:40:000
349426999435,4cyclictest131rcu_preempt10:56:553
3494269994331,10cyclictest37-21ksoftirqd/311:25:013
3494269994326,9cyclictest3580700-21sshd12:09:323
3494269994326,14cyclictest3467602-21systemd-journal10:25:013
349426999431,18cyclictest131rcu_preempt07:40:013
3494264994335,5cyclictest3497346-21kworker/2:209:09:292
3494264994335,5cyclictest3485113-21kworker/2:007:11:262
3494264994334,6cyclictest0-21swapper/207:10:022
3494264994334,6cyclictest0-21swapper/207:10:022
349426099435,9cyclictest23-21ksoftirqd/107:15:021
349426099435,7cyclictest23-21ksoftirqd/107:50:011
3494255994311,5cyclictest0-21swapper/008:47:010
3494269994232,6cyclictest309-21in:imuxsock10:45:003
3494269994232,6cyclictest309-21in:imuxsock10:45:003
3494269994224,14cyclictest3545180-21munin-run10:50:013
349426999422,3cyclictest131rcu_preempt11:41:333
349426999421,3cyclictest131rcu_preempt12:00:003
3494264994223,15cyclictest1-21systemd08:50:012
3494260994230,9cyclictest3499376-21sshd08:49:351
3494255994228,11cyclictest3496502-21sshd07:51:340
3494255994225,11cyclictest275-21dbus-daemon09:05:000
349426999418,6cyclictest0-21swapper/309:21:473
3494269994130,8cyclictest37-21ksoftirqd/312:37:283
3494269994129,10cyclictest37-21ksoftirqd/310:35:003
3494269994126,12cyclictest37-21ksoftirqd/310:10:023
3494269994126,12cyclictest37-21ksoftirqd/310:10:023
349426999411,4cyclictest131rcu_preempt12:04:433
3494264994133,5cyclictest3482593-21kworker/2:108:51:202
349426099414,8cyclictest23-21ksoftirqd/108:21:331
3494260994123,11cyclictest275-21dbus-daemon08:10:001
349425599416,8cyclictest12-21ksoftirqd/007:16:350
349425599416,8cyclictest12-21ksoftirqd/007:16:350
349426999404,3cyclictest131rcu_preempt09:47:163
3494269994031,5cyclictest3531731-21munin-plugin-st10:20:003
3494269994028,8cyclictest364-21snmpd08:10:003
3494269994025,12cyclictest275-21dbus-daemon10:30:023
3494269994025,12cyclictest275-21dbus-daemon10:30:023
3494269994025,10cyclictest309-21in:imuxsock07:20:003
3494269994025,10cyclictest309-21in:imuxsock07:20:003
349426999401,5cyclictest131rcu_preempt07:40:013
349426999401,4cyclictest131rcu_preempt09:31:573
3494264994033,4cyclictest3482593-21kworker/2:107:39:002
349426099402,5cyclictest23-21ksoftirqd/107:24:231
349425599408,8cyclictest0-21swapper/007:24:350
349426999394,3cyclictest131rcu_preempt07:25:003
349426999393,5cyclictest131rcu_preempt08:49:593
349426999392,7cyclictest131rcu_preempt12:30:013
3494269993926,10cyclictest3498842-21sshd08:39:203
349426999392,3cyclictest131rcu_preempt09:40:003
349426999392,3cyclictest131rcu_preempt09:04:593
3494269993923,12cyclictest37-21ksoftirqd/308:20:013
3494269993923,12cyclictest1-21systemd11:30:013
3494269993923,12cyclictest0-21swapper/307:59:593
349426999391,7cyclictest131rcu_preempt10:55:013
349426999391,5cyclictest131rcu_preempt11:10:373
349426999391,4cyclictest131rcu_preempt07:15:023
349426999391,3cyclictest131rcu_preempt10:37:093
3494260993926,10cyclictest3497869-21systemctl08:20:011
3494255993926,10cyclictest12-21ksoftirqd/008:05:010
3494255993926,10cyclictest12-21ksoftirqd/008:05:000
3494255993923,11cyclictest3495473-21sshd07:33:340
349426999384,4cyclictest131rcu_preempt09:20:013
349426999384,3cyclictest131rcu_preempt10:04:593
3494269993830,6cyclictest37-21ksoftirqd/309:30:033
3494269993830,6cyclictest37-21ksoftirqd/309:30:023
349426999382,3cyclictest131rcu_preempt09:58:103
349426099389,5cyclictest131rcu_preempt08:40:031
349426099385,7cyclictest23-21ksoftirqd/107:58:261
349426099384,6cyclictest23-21ksoftirqd/107:35:011
349425599386,4cyclictest131rcu_preempt07:15:020
349425599384,7cyclictest12-21ksoftirqd/008:45:000
3494269993728,7cyclictest37-21ksoftirqd/308:45:013
3494269993726,8cyclictest37-21ksoftirqd/307:30:343
3494269993725,10cyclictest37-21ksoftirqd/311:02:103
3494269993725,10cyclictest37-21ksoftirqd/309:14:233
349426999371,6cyclictest0-21swapper/312:34:413
349426999371,4cyclictest131rcu_preempt12:12:533
3494269993713,8cyclictest37-21ksoftirqd/309:10:013
3494264993729,5cyclictest3482593-21kworker/2:108:14:092
349426099374,7cyclictest23-21ksoftirqd/107:52:351
3494260993726,8cyclictest23-21ksoftirqd/109:10:021
3494260993716,9cyclictest23-21ksoftirqd/108:01:351
3494260993716,9cyclictest23-21ksoftirqd/108:01:351
349425599374,8cyclictest12-21ksoftirqd/008:11:340
349426999363,5cyclictest131rcu_preempt11:38:033
349426999363,3cyclictest131rcu_preempt07:30:023
3494269993626,5cyclictest309-21in:imuxsock07:50:013
3494269993623,10cyclictest37-21ksoftirqd/308:26:353
349426999361,3cyclictest131rcu_preempt11:07:553
3494260993629,3cyclictest131rcu_preempt08:30:021
349426999351,30cyclictest3499728-21sed08:55:023
349426999341,5cyclictest131rcu_preempt08:03:353
349426999341,5cyclictest131rcu_preempt08:03:353
349426999341,5cyclictest131rcu_preempt07:10:023
349426999341,5cyclictest131rcu_preempt07:10:013
3494269993322,8cyclictest0-21swapper/308:35:023
349426999321,12cyclictest3496693-21cut07:55:023
3494269993123,5cyclictest3498175-21latency_hist08:25:023
349426999304,7cyclictest37-21ksoftirqd/308:10:523
349426099294,17cyclictest3494261-21latency_hist07:10:021
349426099294,17cyclictest3494261-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional