You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-10 - 15:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Wed Jun 10, 2026 12:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3537676995851,5cyclictest3706496-21munin-run12:40:010
3537678995648,5cyclictest421-21dbus-daemon07:40:022
3537678995342,7cyclictest0-21swapper/212:20:012
3537678994940,5cyclictest3659145-21awk11:25:012
353767799494,38cyclictest3641324-21ssh10:55:281
353767699492,16cyclictest3609622-21sh10:05:240
3537678994822,5cyclictest171rcu_preempt10:55:272
353767899473,32cyclictest665-21snmpd09:45:212
3537676994739,6cyclictest3685426-21rm12:05:260
353767699472,38cyclictest0-21swapper/011:00:240
3537676994630,10cyclictest12650irq/32-eth011:05:220
3537678994537,3cyclictest0-21swapper/211:50:242
3537678994536,6cyclictest3671768-21cron11:45:012
3537678994516,17cyclictest665-21snmpd08:20:282
353767899442,14cyclictest36-21ksoftirqd/212:35:322
353767699443,34cyclictest3675897-21ssh11:50:230
3537677994334,5cyclictest0-21swapper/107:45:011
353767799433,31cyclictest3685461-21ssh12:05:271
3537677994331,8cyclictest3586431-21ssh09:30:001
3537677994331,8cyclictest3586431-21ssh09:30:001
3537677994233,6cyclictest1-21systemd08:30:021
3537677994232,6cyclictest0-21swapper/112:20:261
353767699424,34cyclictest3587463-21ssh09:30:240
353767699424,34cyclictest3587463-21ssh09:30:240
3537676994227,12cyclictest0-21swapper/010:20:010
353767899414,32cyclictest3606519-21ssh10:00:252
3537678994131,7cyclictest3697033-21cron12:25:012
3537676994135,3cyclictest0-21swapper/011:40:270
3537676994124,14cyclictest3606092-21fschecks_time10:00:160
353767699412,35cyclictest0-21swapper/008:20:260
3537676994123,15cyclictest3641769-21unixbench_multi10:55:340
3537678994033,4cyclictest3576859-21cron09:15:012
3537678994029,4cyclictest0-21swapper/211:00:242
353767799402,32cyclictest3682261-21ssh12:00:241
353767699405,32cyclictest3666478-21ntpq11:35:240
3537676994034,3cyclictest0-21swapper/008:35:010
353767699401,33cyclictest3593869-21ntpq09:40:240
353767699401,23cyclictest3690154-21ssh12:13:250
3537678993932,3cyclictest0-21swapper/209:30:222
3537678993932,3cyclictest0-21swapper/209:30:222
3537678993929,3cyclictest0-21swapper/212:10:252
3537677993931,5cyclictest3612692-21needreboot10:10:181
3537677993930,6cyclictest3564712-21latency_hist08:40:021
353767699398,27cyclictest3561060-21ntpq08:25:260
353767699393,32cyclictest3547561-21ntpq07:40:230
353767699391,35cyclictest3568586-21ntpq08:50:240
353767699391,27cyclictest3612792-21ntpq10:10:220
3537677993830,5cyclictest3537681-21systemd-run07:10:011
3537676993831,4cyclictest3652858-21systemd11:15:020
3537676993831,3cyclictest3693858-21munin-plugin-st12:20:010
3537676993829,7cyclictest421-21dbus-daemon08:20:010
3537676993818,17cyclictest0-21swapper/011:55:270
353767699381,34cyclictest3660263-21ssh11:25:290
353767699381,34cyclictest0-21swapper/009:35:260
3537676993813,21cyclictest0-21swapper/007:20:220
353767699381,13cyclictest3673810-21ssh11:46:430
353767699381,12cyclictest3600264-21ntp_states09:50:280
353767699381,12cyclictest3600264-21ntp_states09:50:280
3537678993729,5cyclictest3570706-21latency_hist09:00:012
3537678993728,4cyclictest0-21swapper/209:25:232
3537678993728,4cyclictest0-21swapper/209:25:232
353767899372,27cyclictest3612816-21ssh10:10:232
353767799373,26cyclictest3666502-21ssh11:35:251
3537677993731,3cyclictest0-21swapper/111:25:011
3537677993726,8cyclictest3555009-21ntp_kernel_err08:05:221
3537677993726,8cyclictest3555009-21ntp_kernel_err08:05:221
3537677993725,9cyclictest3675406-21aten2.4-expect11:50:141
353767799372,23cyclictest3653854-21ssh11:15:271
353767699377,27cyclictest3556538-21ntpq08:10:230
353767699374,28cyclictest3552003-21ntpq07:55:210
3537676993728,7cyclictest3631089-21aten2.4-expect10:40:130
353767699372,31cyclictest3544609-21ntpq07:30:250
353767699372,30cyclictest3573031-21ntpq09:05:210
3537676993721,13cyclictest0-21swapper/009:20:000
3537676993721,13cyclictest0-21swapper/009:20:000
353767699371,6cyclictest3571529-21ntpq09:00:210
353767699371,30cyclictest3543097-21ntpq07:25:250
353767699371,26cyclictest3697982-21ntpq12:25:200
353767699371,11cyclictest3549138-21ntpq07:45:290
353767699370,13cyclictest3636966-21ssh10:49:490
353767699370,12cyclictest3585093-21ssh09:26:090
353767699370,12cyclictest3585093-21ssh09:26:090
353767899363,29cyclictest3600263-21ssh09:50:272
353767899363,29cyclictest3600263-21ssh09:50:272
3537678993628,5cyclictest3552650-21grep08:00:012
353767799363,27cyclictest3631710-21ssh10:40:251
3537677993629,4cyclictest3558653-21cron08:20:001
3537677993628,5cyclictest3580674-21fschecks_count09:20:161
3537677993628,5cyclictest3580674-21fschecks_count09:20:161
3537677993628,5cyclictest3575286-21head09:10:341
353767799362,26cyclictest3672905-21ssh11:45:291
353767799362,25cyclictest3663413-21ssh11:30:291
353767799362,23cyclictest3701256-21ssh12:30:261
353767699369,24cyclictest3683967-21sh12:03:440
353767699366,27cyclictest3622055-21grep10:25:170
353767699366,27cyclictest3570016-21ntpq08:55:210
353767699362,30cyclictest3565170-21df_inode08:40:180
353767699362,30cyclictest3564102-21ntpq08:35:280
353767699361,32cyclictest3663275-21ssh11:30:230
353767699361,32cyclictest3653760-21awk11:15:230
353767699361,32cyclictest3625411-21ntpq10:30:230
353767699361,32cyclictest3574605-21awk09:10:200
353767699361,30cyclictest3638013-21ntpq10:50:220
353767699361,12cyclictest3604031-21ssh09:55:480
353767699361,11cyclictest3553165-21/usr/sbin/munin08:00:160
353767699361,10cyclictest3694904-21ntpq12:20:230
353767699360,10cyclictest3628543-21ntpq10:35:220
353767699360,10cyclictest3546068-21ntpq07:35:240
3537678993527,5cyclictest3682808-21grep12:00:352
3537678993527,5cyclictest3621390-21latency_hist10:25:022
3537678993525,7cyclictest3567664-21munin-plugin-st08:50:022
3537678993525,7cyclictest3567664-21munin-plugin-st08:50:022
3537677993528,3cyclictest0-21swapper/112:25:211
3537677993528,3cyclictest0-21swapper/107:50:011
3537677993526,6cyclictest3569156-21cron08:55:001
353767799352,26cyclictest3577982-21ssh09:15:281
353767799352,26cyclictest3577982-21ssh09:15:281
3537676993530,3cyclictest0-21swapper/009:45:290
353767699351,9cyclictest3550198-21/usr/sbin/munin07:50:160
353767699351,5cyclictest3581042-21ntpq09:20:210
353767699351,5cyclictest3581042-21ntpq09:20:210
353767699351,30cyclictest3538539-21ntpq07:10:220
353767699351,30cyclictest3538539-21ntpq07:10:220
353767699351,30cyclictest0-21swapper/008:45:220
353767699351,30cyclictest0-21swapper/008:45:220
353767699351,12cyclictest3658975-21ssh11:24:330
3537678993427,3cyclictest0-21swapper/209:35:242
3537678993427,3cyclictest0-21swapper/209:15:332
3537678993427,3cyclictest0-21swapper/209:15:332
3537678993426,5cyclictest3681383-21latency_hist12:00:022
3537678993423,8cyclictest3627621-21latency_hist10:35:022
3537678993421,10cyclictest3568469-21needreboot08:50:192
353767799348,20cyclictest3679191-21ssh11:55:291
353767799344,22cyclictest3619201-21ssh10:20:261
3537677993428,3cyclictest0-21swapper/112:40:001
3537677993426,5cyclictest3647527-21ntp_kernel_pll_11:05:231
353767799342,25cyclictest3587450-21ssh09:30:231
353767799342,25cyclictest3587450-21ssh09:30:231
353767699342,29cyclictest3555048-21ntpq08:05:240
353767699342,29cyclictest3555048-21ntpq08:05:240
353767699341,30cyclictest3540014-21ntpq07:15:210
353767699341,30cyclictest3540014-21ntpq07:15:210
353767699340,4cyclictest0-21swapper/010:20:270
353767699340,10cyclictest3701160-21ntpq12:30:230
3537678993327,3cyclictest0-21swapper/208:15:012
3537678993326,4cyclictest0-21swapper/207:20:002
3537678993326,4cyclictest0-21swapper/207:20:002
3537678993325,5cyclictest3675134-21latency_hist11:50:022
3537678993323,7cyclictest3581352-21mailstats09:20:322
3537678993323,7cyclictest3581352-21mailstats09:20:322
353767899332,26cyclictest3628643-21ssh10:35:262
3537678993313,5cyclictest3593869-21ntpq09:40:242
353767799335,22cyclictest3600218-21sh09:50:251
353767799335,22cyclictest3600218-21sh09:50:251
353767799333,23cyclictest3669680-21ssh11:40:261
3537677993325,4cyclictest3660696-21unixbench_multi11:25:351
3537677993320,9cyclictest3570658-21munin-plugin-st09:00:001
353767899327,8cyclictest36-21ksoftirqd/208:15:252
3537678993226,3cyclictest0-21swapper/208:40:012
3537678993225,3cyclictest0-21swapper/211:20:012
353767899322,20cyclictest3634933-21ssh10:45:272
3537678993220,9cyclictest3561331-21switchtime08:25:362
3537678993216,3cyclictest171rcu_preempt08:40:342
3537677993225,4cyclictest3561686-21kworker/u8:6-xprtiod10:05:011
3537677993224,5cyclictest3693846-21cron12:20:011
353767799322,21cyclictest3590801-21ssh09:35:291
3537677993221,3cyclictest0-21swapper/110:45:241
353767899317,9cyclictest36-21ksoftirqd/208:30:242
353767899316,9cyclictest36-21ksoftirqd/207:25:212
353767899314,17cyclictest428-21mta-sts-daemon11:35:242
353767899313,23cyclictest3660135-21ssh11:25:242
3537678993124,4cyclictest0-21swapper/211:30:262
353767899312,22cyclictest3650679-21ssh11:10:232
3537678993120,6cyclictest0-21swapper/209:55:262
3537678993116,4cyclictest171rcu_preempt11:05:262
353767799314,20cyclictest3609668-21ssh10:05:251
353767799314,19cyclictest3638006-21ssh10:50:221
3537677993125,3cyclictest0-21swapper/112:15:011
3537677993125,2cyclictest0-21swapper/110:25:331
353767799312,21cyclictest3628565-21rm10:35:231
3537677993121,8cyclictest3650648-21ntp_kernel_pll_11:10:231
353767699311,2cyclictest0-21swapper/007:10:010
353767899307,8cyclictest36-21ksoftirqd/212:30:222
353767899305,8cyclictest36-21ksoftirqd/208:00:262
3537678993023,4cyclictest3548083-21sed07:40:342
3537678993022,5cyclictest3573660-21cron09:10:012
3537678993018,9cyclictest3632179-21timerandwakeup10:40:402
3537678993016,2cyclictest171rcu_preempt07:50:212
3537677993024,3cyclictest0-21swapper/111:00:261
3537677993024,3cyclictest0-21swapper/110:00:011
3537677993022,5cyclictest3552640-21cron08:00:011
3537677993022,5cyclictest3544193-21uniq07:30:161
3537677993021,7cyclictest3596908-21needreboot09:45:201
3537677993020,5cyclictest0-21swapper/110:35:021
353767899298,5cyclictest351ktimers/210:05:242
353767899295,19cyclictest3685426-21rm12:05:262
353767899293,20cyclictest3622317-21ssh10:25:242
3537678992921,5cyclictest3697633-21ssh12:25:152
3537678992914,3cyclictest171rcu_preempt10:15:192
3537677992922,4cyclictest0-21swapper/107:40:011
3537677992922,3cyclictest0-21swapper/109:10:011
3537677992921,5cyclictest750-21ntpd08:30:231
3537677992921,5cyclictest3541201-21fschecks_time07:20:161
353767899284,8cyclictest36-21ksoftirqd/207:10:242
353767899284,8cyclictest36-21ksoftirqd/207:10:242
353767899283,13cyclictest36-21ksoftirqd/207:30:222
3537678992820,5cyclictest421-21dbus-daemon07:50:022
3537678992816,3cyclictest171rcu_preempt10:50:182
3537678992814,3cyclictest171rcu_preempt09:05:012
3537677992821,4cyclictest3572140-21kworker/u8:16+rpciod09:05:011
3537677992820,6cyclictest1000-21mta-sts-daemon07:50:261
3537677992820,5cyclictest3618092-21sendmail10:20:011
3537677992819,7cyclictest3539891-21missed_timers07:15:181
3537677992819,7cyclictest3539891-21missed_timers07:15:181
3537677992819,4cyclictest0-21swapper/109:40:291
3537678992716,6cyclictest171rcu_preempt08:05:282
3537678992716,6cyclictest171rcu_preempt08:05:282
3537677992721,4cyclictest0-21swapper/108:10:231
3537677992721,3cyclictest0-21swapper/108:05:011
3537677992717,8cyclictest3567510-21unixbench-2d08:45:351
3537677992717,8cyclictest3567510-21unixbench-2d08:45:351
3537678992618,5cyclictest1-21systemd07:10:012
3537678992615,3cyclictest171rcu_preempt07:20:242
353767799264,9cyclictest0-21swapper/108:40:251
3537677992618,5cyclictest0-21swapper/108:20:211
3537677992519,3cyclictest0-21swapper/107:30:001
3537677992418,3cyclictest1001-21mta-sts-daemon07:10:131
3537677992418,3cyclictest1001-21mta-sts-daemon07:10:131
353767999222,12cyclictest0-21swapper/311:50:243
353767999221,3cyclictest0-21swapper/311:35:233
353767999221,3cyclictest0-21swapper/310:45:253
353767999211,3cyclictest0-21swapper/312:25:273
353767999211,3cyclictest0-21swapper/309:35:233
353767999210,3cyclictest0-21swapper/312:20:233
353767999210,3cyclictest0-21swapper/312:05:303
353767999210,3cyclictest0-21swapper/311:10:213
353767999210,3cyclictest0-21swapper/309:25:213
353767999210,3cyclictest0-21swapper/309:25:213
353767999202,13cyclictest0-21swapper/310:20:253
353767999201,3cyclictest0-21swapper/312:30:233
353767999201,3cyclictest0-21swapper/311:30:223
353767999201,3cyclictest0-21swapper/311:05:223
353767999201,3cyclictest0-21swapper/310:30:233
353767999201,3cyclictest0-21swapper/309:15:283
353767999201,3cyclictest0-21swapper/309:15:283
353767999201,3cyclictest0-21swapper/309:10:193
353767999201,3cyclictest0-21swapper/308:55:283
353767999201,3cyclictest0-21swapper/308:25:183
353767999201,3cyclictest0-21swapper/307:55:183
353767999201,3cyclictest0-21swapper/307:50:233
353767999201,3cyclictest0-21swapper/307:20:273
353767999201,3cyclictest0-21swapper/307:10:283
353767999201,3cyclictest0-21swapper/307:10:283
353767999201,14cyclictest0-21swapper/310:55:213
353767999200,3cyclictest0-21swapper/310:40:293
353767999200,3cyclictest0-21swapper/309:50:213
353767999200,3cyclictest0-21swapper/309:50:213
353767999200,3cyclictest0-21swapper/309:45:263
353767999200,3cyclictest0-21swapper/307:40:323
353767999200,14cyclictest0-21swapper/310:35:223
353767999192,13cyclictest0-21swapper/311:20:223
353767999191,3cyclictest0-21swapper/312:35:303
353767999191,3cyclictest0-21swapper/312:10:313
353767999191,3cyclictest0-21swapper/311:00:233
353767999191,3cyclictest0-21swapper/310:10:133
353767999191,3cyclictest0-21swapper/310:05:243
353767999191,3cyclictest0-21swapper/310:00:153
353767999191,3cyclictest0-21swapper/309:20:193
353767999191,3cyclictest0-21swapper/309:20:193
353767999191,3cyclictest0-21swapper/308:45:193
353767999191,3cyclictest0-21swapper/308:45:193
353767999191,3cyclictest0-21swapper/308:10:203
353767999191,3cyclictest0-21swapper/308:00:223
353767999191,3cyclictest0-21swapper/307:25:143
353767999191,14cyclictest0-21swapper/307:30:223
353767999191,13cyclictest0-21swapper/308:35:253
353767999190,3cyclictest0-21swapper/311:55:173
353767999190,3cyclictest0-21swapper/311:40:173
353767999190,3cyclictest0-21swapper/311:25:173
353767999190,3cyclictest0-21swapper/311:15:203
353767999190,3cyclictest0-21swapper/310:50:273
353767999190,3cyclictest0-21swapper/310:25:233
353767999190,3cyclictest0-21swapper/310:15:263
353767999190,3cyclictest0-21swapper/309:55:213
353767999190,3cyclictest0-21swapper/309:30:233
353767999190,3cyclictest0-21swapper/309:30:233
353767999190,3cyclictest0-21swapper/309:05:213
353767999190,3cyclictest0-21swapper/309:00:203
353767999190,3cyclictest0-21swapper/308:50:223
353767999190,3cyclictest0-21swapper/308:30:123
353767999190,3cyclictest0-21swapper/308:20:343
353767999190,3cyclictest0-21swapper/308:15:173
353767999190,3cyclictest0-21swapper/307:45:143
353767999190,3cyclictest0-21swapper/307:35:323
353767999190,3cyclictest0-21swapper/307:15:163
353767999190,3cyclictest0-21swapper/307:15:163
353767999182,12cyclictest0-21swapper/312:15:223
353767999181,3cyclictest0-21swapper/311:45:253
353767999180,3cyclictest0-21swapper/312:00:123
353767999180,3cyclictest0-21swapper/309:40:243
353767999180,3cyclictest0-21swapper/308:40:133
353767999180,3cyclictest0-21swapper/308:05:223
353767999180,3cyclictest0-21swapper/308:05:223
353767999150,2cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional