You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 10:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu Jun 11, 2026 00:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
279820996543,15cyclictest12650irq/32-eth023:15:260
279820995633,16cyclictest151ktimers/019:25:280
279820995536,13cyclictest151ktimers/019:45:270
27982299524,41cyclictest415269-21rm23:45:282
279820995232,5cyclictest171rcu_preempt23:55:300
27982099521,8cyclictest171rcu_preempt22:40:210
27982099514,41cyclictest348699-21ssh22:00:250
279820995127,18cyclictest12650irq/32-eth022:55:220
27982199504,41cyclictest440511-21ssh00:25:231
27982299492,40cyclictest361375-21ssh22:20:232
27982299492,40cyclictest361375-21ssh22:20:232
279820994927,15cyclictest12650irq/32-eth022:15:250
279820994927,15cyclictest12650irq/32-eth022:15:250
279820994927,14cyclictest151ktimers/019:55:290
279822994837,7cyclictest354029-21systemd-run22:10:022
279821994838,7cyclictest694-21cron00:20:011
279821994838,7cyclictest694-21cron00:20:011
279822994740,4cyclictest388869-21systemd-run23:05:012
279820994716,5cyclictest171rcu_preempt20:00:230
27982299462,36cyclictest355055-21ssh22:10:262
279821994636,6cyclictest433174-21systemd-run00:15:011
279822994536,5cyclictest306839-21systemd-run20:40:022
279822994534,4cyclictest0-21swapper/223:15:282
279820994532,9cyclictest12650irq/32-eth000:30:230
279820994524,13cyclictest431120-21ssh00:10:310
279822994435,6cyclictest421-21dbus-daemon20:10:012
27982099442,31cyclictest393197-21ssh23:10:280
27982199437,3cyclictest171rcu_preempt19:25:281
27982199434,29cyclictest326366-21ssh21:25:211
279821994335,5cyclictest423693-21tar00:00:011
279822994228,10cyclictest405715-21sh23:30:252
27982199422,33cyclictest389869-21ssh23:05:231
279820994226,11cyclictest12650irq/32-eth023:40:260
279822994132,4cyclictest0-21swapper/200:30:252
279822994121,12cyclictest351ktimers/222:35:242
27982199415,28cyclictest383615-21ssh22:55:281
279821994133,5cyclictest382530-21ssh22:55:021
279820994128,5cyclictest0-21swapper/021:40:260
279820994126,12cyclictest314227-21head21:00:370
279822994033,5cyclictest426870-21latency_hist00:05:012
27982299402,30cyclictest393042-21ssh23:10:242
279821994031,5cyclictest373003-21idleruntime-cro22:40:001
27982199401,36cyclictest374006-21ssh22:40:231
279820994033,5cyclictest1003-21mta-sts-daemon23:03:120
279820994032,6cyclictest12650irq/32-eth022:10:280
279822993932,4cyclictest299326-21systemd-run20:15:012
279822993932,4cyclictest299326-21systemd-run20:15:012
279822993927,6cyclictest0-21swapper/223:40:242
27982199394,30cyclictest408992-21ssh23:35:301
27982199392,13cyclictest28-21ksoftirqd/121:20:281
279822993831,3cyclictest0-21swapper/200:00:022
279822993830,5cyclictest281824-21fschecks_time19:15:202
279821993830,5cyclictest367297-21fschecks_time22:30:201
279821993830,5cyclictest367297-21fschecks_time22:30:201
279821993830,5cyclictest360524-21latency_hist22:20:021
279821993830,5cyclictest360524-21latency_hist22:20:021
279821993830,5cyclictest331786-21idleruntime-cro21:35:011
279820993835,2cyclictest0-21swapper/020:05:280
27982099382,33cyclictest319965-21ssh21:15:220
27982099381,22cyclictest351922-21ssh22:05:250
27982299376,14cyclictest430991-21ssh00:10:262
27982199371,6cyclictest171rcu_preempt19:55:291
279821993710,13cyclictest345596-21ssh21:55:281
279820993730,4cyclictest0-21swapper/021:45:230
279820993729,5cyclictest347667-21systemctl22:00:010
27982099372,30cyclictest316808-21ssh21:10:230
279822993628,5cyclictest0-21swapper/223:50:212
279822993627,6cyclictest311310-21systemd-run20:55:012
279822993627,4cyclictest0-21swapper/221:20:252
279822993615,16cyclictest351ktimers/222:30:262
279822993615,16cyclictest351ktimers/222:30:262
27982199364,26cyclictest351813-21ssh22:05:211
27982199364,25cyclictest335962-21sh21:40:231
279821993629,4cyclictest396742-21tail23:15:341
279821993624,8cyclictest332893-21sh21:35:271
27982099362,29cyclictest418329-21ssh23:50:230
27982099362,28cyclictest665-21snmpd20:35:220
279820993618,11cyclictest12650irq/32-eth022:35:250
27982099361,27cyclictest389889-21rm23:05:230
27982299357,23cyclictest349328-21ssh22:00:372
279822993527,5cyclictest1-21systemd00:25:012
279822993527,5cyclictest1-21systemd00:25:012
279822993526,6cyclictest284343-21systemd-run19:25:022
279822993525,6cyclictest300846-21systemd-run20:20:012
279822993525,6cyclictest300846-21systemd-run20:20:012
279821993528,3cyclictest0-21swapper/123:40:241
279821993527,5cyclictest339724-21unixbench_multi21:45:351
279821993525,7cyclictest448964-21sendmail00:40:001
279821993525,7cyclictest448964-21sendmail00:40:001
279821993525,6cyclictest420-21avahi-daemon19:25:021
279821993525,6cyclictest363542-21systemd-run22:25:011
279821993525,6cyclictest363542-21systemd-run22:25:011
279821993524,8cyclictest290332-21systemd-run19:45:011
27982199352,23cyclictest320104-21ssh21:15:281
279821993510,19cyclictest271ktimers/123:50:281
27982299344,23cyclictest335953-21ssh21:40:232
279822993427,4cyclictest411039-21systemd-run23:40:012
27982299342,24cyclictest329663-21ssh21:30:282
279821993427,4cyclictest0-21swapper/119:40:011
279820993426,5cyclictest0-21swapper/021:40:010
279820993415,10cyclictest151ktimers/020:45:230
27982299334,22cyclictest443-21gpm21:25:272
279822993325,4cyclictest1003-21mta-sts-daemon21:10:152
279822993324,6cyclictest390160-21cat23:05:312
27982299332,24cyclictest665-21snmpd21:05:282
27982199333,23cyclictest354983-21ssh22:10:231
27982199332,13cyclictest28-21ksoftirqd/120:05:211
279820993329,2cyclictest0-21swapper/000:25:000
279820993329,2cyclictest0-21swapper/000:25:000
279820993326,4cyclictest411039-21systemd-run23:40:010
27982099331,5cyclictest364547-21ssh22:25:240
27982099331,5cyclictest364547-21ssh22:25:240
27982099331,5cyclictest171rcu_preempt20:10:300
27982099331,5cyclictest171rcu_preempt20:10:300
279822993227,2cyclictest0-21swapper/223:00:012
279822993224,5cyclictest442784-21ssh00:30:022
279822993223,6cyclictest289329-21grep19:40:172
279822993222,8cyclictest312823-21sh21:00:012
279822993221,8cyclictest1001-21mta-sts-daemon00:17:002
279822993221,8cyclictest1001-21mta-sts-daemon00:17:002
279822993221,6cyclictest447280-21timerwakeupswit00:35:352
279822993221,6cyclictest447280-21timerwakeupswit00:35:352
27982199327,7cyclictest28-21ksoftirqd/100:30:251
27982199326,9cyclictest28-21ksoftirqd/123:30:281
279821993225,4cyclictest425304-21unixbench_singl00:00:351
279821993224,4cyclictest0-21swapper/119:50:011
279821993223,5cyclictest271ktimers/123:02:071
279821993223,3cyclictest171rcu_preempt23:20:301
27982199322,24cyclictest348780-21ssh22:00:281
27982199322,21cyclictest402502-21ssh23:25:241
279821993215,10cyclictest307132-21aten_r2power_fr20:40:141
279821993212,3cyclictest171rcu_preempt20:25:271
279821993210,9cyclictest171rcu_preempt22:25:071
279821993210,9cyclictest171rcu_preempt22:25:071
27982099323,18cyclictest12650irq/32-eth022:45:240
279820993226,4cyclictest405282-21fschecks_time23:30:160
279820993226,4cyclictest287303-21idleruntime-cro19:35:010
27982099322,3cyclictest171rcu_preempt21:50:290
279820993223,7cyclictest303122-21needreboot20:25:200
27982099322,24cyclictest434299-21ssh00:15:340
27982099322,24cyclictest434299-21ssh00:15:340
27982099322,23cyclictest399450-21ssh23:20:270
279820993217,4cyclictest0-21swapper/023:25:150
27982099321,27cyclictest427851-21ssh00:05:230
27982099321,27cyclictest427851-21ssh00:05:230
27982299318,11cyclictest402480-21ssh23:25:232
279822993124,4cyclictest346054-21unixbench_multi21:55:342
279822993123,6cyclictest427486-21fschecks_time00:05:172
279822993123,6cyclictest427486-21fschecks_time00:05:172
279822993123,5cyclictest286033-21/usr/sbin/munin19:30:222
279822993122,6cyclictest0-21swapper/221:35:262
279822993119,7cyclictest360342-21munin-plugin-st22:20:002
279822993119,7cyclictest360342-21munin-plugin-st22:20:002
27982199312,5cyclictest28-21ksoftirqd/119:30:241
279821993123,4cyclictest271ktimers/123:11:421
279821993122,3cyclictest171rcu_preempt20:55:011
279821993122,2cyclictest171rcu_preempt20:00:301
279821993121,7cyclictest312787-21munin-run21:00:011
279821993110,3cyclictest171rcu_preempt19:15:301
27982099315,7cyclictest171rcu_preempt20:30:230
279820993118,10cyclictest446721-21needreboot00:35:210
279820993118,10cyclictest446721-21needreboot00:35:210
27982299304,21cyclictest380444-21ssh22:50:282
27982299303,17cyclictest1003-21mta-sts-daemon19:50:242
279822993023,3cyclictest0-21swapper/221:20:012
27982299302,22cyclictest342448-21ssh21:50:292
279822993021,6cyclictest0-21swapper/222:50:022
279822993019,8cyclictest304611-21needreboot20:30:192
279822993019,8cyclictest294733-21unixbench_singl19:55:352
27982199307,8cyclictest28-21ksoftirqd/100:05:261
27982199307,8cyclictest28-21ksoftirqd/100:05:261
279821993023,4cyclictest292316-21fschecks_count19:50:161
279821993021,6cyclictest271ktimers/122:47:411
279821993021,5cyclictest299326-21systemd-run20:15:011
279821993021,5cyclictest299326-21systemd-run20:15:011
27982199302,11cyclictest28-21ksoftirqd/123:45:221
279821993020,7cyclictest437183-21missed_timers00:20:201
279821993020,7cyclictest437183-21missed_timers00:20:201
279821993013,3cyclictest171rcu_preempt21:50:281
279821993012,4cyclictest261rcuc/119:15:011
27982099308,2cyclictest0-21swapper/022:30:270
27982099308,2cyclictest0-21swapper/022:30:270
27982099303,4cyclictest171rcu_preempt21:05:250
27982099303,20cyclictest665-21snmpd20:55:260
279820993024,4cyclictest288740-21head19:35:340
279820993023,4cyclictest151ktimers/019:20:240
279820993020,6cyclictest439757-21/usr/sbin/munin00:25:320
279820993016,7cyclictest12650irq/32-eth022:50:280
279822992922,4cyclictest1002-21mta-sts-daemon21:45:372
27982299292,18cyclictest374073-21ssh22:40:252
279822992920,7cyclictest309869-21latency_hist20:50:022
27982299291,22cyclictest399435-21ssh23:20:262
27982199297,18cyclictest271ktimers/121:11:051
27982199294,4cyclictest171rcu_preempt20:45:231
27982199293,8cyclictest28-21ksoftirqd/121:05:271
279821992923,3cyclictest0-21swapper/120:20:011
279821992923,3cyclictest0-21swapper/120:20:011
279821992921,4cyclictest271ktimers/121:00:231
279821992919,5cyclictest271ktimers/120:20:241
279821992912,3cyclictest171rcu_preempt20:35:281
27982099293,6cyclictest415134-21ntpq23:45:220
279820992923,4cyclictest310321-21fschecks_time20:50:160
279820992923,3cyclictest1002-21mta-sts-daemon22:24:080
279820992923,3cyclictest1002-21mta-sts-daemon22:24:080
279820992922,4cyclictest281176-21unixbench_multi19:10:350
279820992921,6cyclictest331788-21cron21:35:010
279820992916,9cyclictest12650irq/32-eth020:40:380
27982099291,23cyclictest424708-21sh00:00:230
279822992821,4cyclictest302319-21munin-run20:25:002
279820992821,3cyclictest1002-21mta-sts-daemon19:50:240
279820992820,6cyclictest328602-21ssh21:30:010
27982199272,17cyclictest665-21snmpd20:30:211
27982099275,14cyclictest12650irq/32-eth019:15:270
279820992720,4cyclictest0-21swapper/021:20:340
279822992619,4cyclictest287849-21gltestperf19:35:162
279822992619,4cyclictest1000-21mta-sts-daemon19:10:282
279822992618,6cyclictest303368-21sendmail_mailqu20:25:302
279822992618,6cyclictest295346-21gltestperf20:00:172
279822992616,7cyclictest314103-21timerwakeupswit21:00:362
27982299252,19cyclictest364476-21ssh22:25:212
27982299252,19cyclictest364476-21ssh22:25:212
27982299252,17cyclictest284927-21awk19:25:192
27982199255,6cyclictest28-21ksoftirqd/119:10:011
27982199255,6cyclictest28-21ksoftirqd/119:10:011
27982099259,9cyclictest151ktimers/020:15:240
27982099259,9cyclictest151ktimers/020:15:240
27982099253,11cyclictest428-21mta-sts-daemon20:20:240
27982099249,3cyclictest171rcu_preempt19:40:300
279822992318,2cyclictest0-21swapper/220:40:172
279822992314,6cyclictest291109-21grep19:45:212
279820992215,5cyclictest279929-21latency_hist19:10:020
279820992215,5cyclictest279929-21latency_hist19:10:020
279822992111,6cyclictest279958-21grep19:10:022
279822992111,6cyclictest279958-21grep19:10:022
27982399202,9cyclictest0-21swapper/323:45:293
27982399181,10cyclictest0-21swapper/322:10:273
27982399172,6cyclictest0-21swapper/319:40:233
27982399171,8cyclictest0-21swapper/322:40:243
27982399171,8cyclictest0-21swapper/321:30:283
27982399171,5cyclictest0-21swapper/322:00:253
27982399163,8cyclictest0-21swapper/322:35:233
27982399163,6cyclictest0-21swapper/322:25:223
27982399163,6cyclictest0-21swapper/322:25:223
27982399162,8cyclictest0-21swapper/322:55:263
27982399162,7cyclictest0-21swapper/322:50:243
27982399161,9cyclictest0-21swapper/323:10:263
27982399153,6cyclictest0-21swapper/300:10:313
27982399153,5cyclictest0-21swapper/321:05:243
27982399152,9cyclictest0-21swapper/322:45:323
27982399152,8cyclictest0-21swapper/323:05:233
27982399152,7cyclictest0-21swapper/319:25:023
27982399152,6cyclictest0-21swapper/321:20:133
27982399152,6cyclictest0-21swapper/320:35:223
27982399152,5cyclictest0-21swapper/320:30:203
27982399151,9cyclictest0-21swapper/322:20:173
27982399151,9cyclictest0-21swapper/322:20:173
27982399151,7cyclictest0-21swapper/320:45:263
27982399144,6cyclictest0-21swapper/320:40:273
27982399143,6cyclictest0-21swapper/319:10:233
27982399142,8cyclictest0-21swapper/300:25:003
27982399142,8cyclictest0-21swapper/300:25:003
27982399142,7cyclictest0-21swapper/323:40:263
27982399142,7cyclictest0-21swapper/323:30:183
27982399142,7cyclictest0-21swapper/321:55:223
27982399142,7cyclictest0-21swapper/321:35:223
27982399142,7cyclictest0-21swapper/300:20:013
27982399142,7cyclictest0-21swapper/300:20:013
27982399142,7cyclictest0-21swapper/300:05:253
27982399142,7cyclictest0-21swapper/300:05:253
27982399142,7cyclictest0-21swapper/300:00:033
27982399141,8cyclictest0-21swapper/322:35:013
27982399141,8cyclictest0-21swapper/322:35:013
27982399141,8cyclictest0-21swapper/300:30:283
27982399141,8cyclictest0-21swapper/300:25:253
27982399141,7cyclictest0-21swapper/323:20:243
27982399141,7cyclictest0-21swapper/321:10:263
27982399141,7cyclictest0-21swapper/319:45:293
27982399141,7cyclictest0-21swapper/300:00:273
27982399141,6cyclictest0-21swapper/321:50:243
27982399141,6cyclictest0-21swapper/319:50:233
27982399141,6cyclictest0-21swapper/319:35:213
27982399141,6cyclictest0-21swapper/319:15:273
27982399134,6cyclictest0-21swapper/320:20:213
27982399133,4cyclictest0-21swapper/320:50:243
27982399132,8cyclictest0-21swapper/323:00:273
27982399132,6cyclictest0-21swapper/320:10:013
27982399132,5cyclictest0-21swapper/323:35:223
27982399132,5cyclictest0-21swapper/320:55:263
27982399131,7cyclictest0-21swapper/323:15:223
27982399131,7cyclictest0-21swapper/322:15:273
27982399131,7cyclictest0-21swapper/322:15:273
27982399131,7cyclictest0-21swapper/322:05:153
27982399131,7cyclictest0-21swapper/321:45:273
27982399131,7cyclictest0-21swapper/321:20:013
27982399131,7cyclictest0-21swapper/320:25:223
27982399131,6cyclictest0-21swapper/323:50:133
27982399131,6cyclictest0-21swapper/320:10:223
27982399131,6cyclictest0-21swapper/320:10:223
27982399131,6cyclictest0-21swapper/300:35:133
27982399131,6cyclictest0-21swapper/300:35:133
27982399124,5cyclictest0-21swapper/321:25:233
27982399123,5cyclictest0-21swapper/323:25:233
27982399122,6cyclictest0-21swapper/321:40:233
27982399121,7cyclictest0-21swapper/319:25:273
27982399121,6cyclictest0-21swapper/319:55:263
27982399121,6cyclictest0-21swapper/319:30:283
27982399111,7cyclictest0-21swapper/320:00:243
27982399111,6cyclictest0-21swapper/321:05:013
27982399111,6cyclictest0-21swapper/320:15:243
27982399111,6cyclictest0-21swapper/320:15:243
2798239972,4cyclictest0-21swapper/319:10:013
2798239972,4cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional