You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 16:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Jun 28, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
325706821250,2sleep00-21swapper/012:35:120
32159032960,3sleep10-21swapper/110:35:191
139099850,68rtkit-daemon0-21swapper/307:06:003
31412202810,8sleep20-21swapper/207:05:232
31407512800,9sleep126-21ksoftirqd/107:05:151
314073927252,14sleep00-21swapper/007:05:150
31924142560,3sleep20-21swapper/209:30:012
31797882560,7sleep33179787-21sed08:55:013
32522332520,2sleep00-21swapper/012:20:120
31682262500,3sleep13168228-21grep08:20:181
32096342350,14sleep126-21ksoftirqd/110:19:591
32096342350,14sleep126-21ksoftirqd/110:19:591
314197999278,5cyclictest3252144-21expr12:20:101
3141979992711,14cyclictest3211600-21/usr/sbin/munin10:25:241
314197999267,5cyclictest3224419-21grep11:00:171
3141979992613,12cyclictest3234966-21if_err_enp2s011:30:181
3141979992611,13cyclictest3164872-21cut08:10:201
3141979992611,13cyclictest3142278-21cpuspeed_turbos07:10:141
314197999260,1cyclictest0-21swapper/111:24:251
31673302250,3chrt3056-21PK-Backend08:20:013
314197999257,4cyclictest3154582-21wc07:40:211
314197999253,8cyclictest3203163-21cstates10:00:131
314197999252,18cyclictest1788620-21WebKitWebProces12:10:121
3141979992519,4cyclictest3218363-21dump-pmu-power10:45:001
314197999251,11cyclictest3175567-21python308:40:211
139099250,4rtkit-daemon1389-21rtkit-daemon10:00:362
139099250,4rtkit-daemon1389-21rtkit-daemon08:03:212
314198799240,22cyclictest1389-21rtkit-daemon12:08:313
314197999249,4cyclictest3147127-21timerwakeupswit07:20:211
314197999248,6cyclictest3173725-21ntpq08:35:251
314197999247,4cyclictest3167158-21seq08:19:201
314197999241,9cyclictest3221229-21python310:50:201
314197999241,7cyclictest3214922-21unin-run10:35:011
314197999241,5cyclictest3056-21PK-Backend09:55:121
3141979992411,11cyclictest3218610-21/usr/sbin/munin10:45:181
3141977992415,7cyclictest82332-21update-manager10:35:420
3141987992312,9cyclictest3209654-21unin-run10:20:003
3141987992312,9cyclictest3209654-21unin-run10:20:003
3141983992319,3cyclictest3147601-21idleruntime-cro07:25:002
314198399231,2cyclictest978-21snmpd09:54:322
314197999237,3cyclictest3253560-21/usr/sbin/munin12:25:131
314197999235,5cyclictest3180544-21/usr/sbin/munin08:55:161
314197999232,8cyclictest2156-21gnome-shell11:25:421
3141979992319,3cyclictest3056-21PK-Backend08:00:251
314197999231,6cyclictest3157961-21taskset07:50:191
314197799234,4cyclictest13-21ksoftirqd/007:30:210
3141977992315,6cyclictest3220728-21df10:50:110
3141987992218,3cyclictest42-21ksoftirqd/310:25:403
3141987992218,3cyclictest3152828-21taskset07:36:393
314198799221,17cyclictest3229892-21python311:15:213
314198799220,20cyclictest3223808-21/usr/sbin/munin11:00:123
3141983992220,1cyclictest0-21swapper/211:21:322
3141983992220,1cyclictest0-21swapper/207:41:152
3141983992215,6cyclictest1-21systemd08:00:012
314197999228,3cyclictest3176612-21cpuspeed_turbos08:45:121
314197999227,4cyclictest3222397-21expr10:55:111
314197999226,3cyclictest3155423-21sort07:45:111
314197999224,5cyclictest3204741-21cpuspeed10:05:121
314197999222,6cyclictest3148065-21cstates07:25:141
3141979992220,1cyclictest3056-21PK-Backend11:53:041
3141979992220,1cyclictest0-21swapper/108:25:401
314197999221,8cyclictest3144965-21/usr/sbin/munin07:15:171
314197999220,6cyclictest436-20systemd-journal12:10:001
314197999220,19cyclictest0-21swapper/108:05:251
314197799224,4cyclictest13-21ksoftirqd/007:36:040
3141977992220,1cyclictest0-21swapper/012:32:220
3141977992220,1cyclictest0-21swapper/011:11:520
3141977992220,1cyclictest0-21swapper/010:21:080
3141977992220,1cyclictest0-21swapper/007:46:100
314197799220,21cyclictest0-21swapper/008:17:260
139099220,17rtkit-daemon3141968-21cyclictest08:16:442
3141987992118,2cyclictest3056-21PK-Backend12:25:163
3141987992113,6cyclictest3191215-21awk09:25:153
3141983992119,1cyclictest3056-21PK-Backend08:45:172
3141983992119,1cyclictest0-21swapper/212:31:012
3141983992119,1cyclictest0-21swapper/211:53:562
3141983992119,1cyclictest0-21swapper/211:41:322
3141983992119,1cyclictest0-21swapper/211:18:002
3141983992119,1cyclictest0-21swapper/210:50:432
3141983992119,1cyclictest0-21swapper/210:41:102
3141983992119,1cyclictest0-21swapper/210:31:352
3141983992119,1cyclictest0-21swapper/210:26:492
3141983992119,1cyclictest0-21swapper/210:21:092
3141983992119,1cyclictest0-21swapper/210:18:452
3141983992119,1cyclictest0-21swapper/210:18:442
3141983992119,1cyclictest0-21swapper/208:54:022
314198399210,20cyclictest0-21swapper/212:35:052
314198399210,20cyclictest0-21swapper/210:39:402
314198399210,20cyclictest0-21swapper/208:08:252
314198399210,20cyclictest0-21swapper/207:45:282
314197999219,3cyclictest3056-21PK-Backend12:02:301
3141979992119,1cyclictest3056-21PK-Backend11:56:361
3141979992119,1cyclictest3056-21PK-Backend10:21:461
3141979992119,1cyclictest0-21swapper/108:30:541
314197999211,8cyclictest3250489-21/usr/sbin/munin12:15:121
3141979992118,2cyclictest3056-21PK-Backend09:05:281
314197999210,19cyclictest3056-21PK-Backend11:44:121
314197999210,19cyclictest3056-21PK-Backend10:14:151
314197999210,19cyclictest3056-21PK-Backend09:13:311
3141977992119,1cyclictest0-21swapper/011:32:130
3141977992119,1cyclictest0-21swapper/011:17:460
3141977992119,1cyclictest0-21swapper/010:56:440
3141977992119,1cyclictest0-21swapper/010:18:460
3141977992119,1cyclictest0-21swapper/010:18:450
3141977992119,1cyclictest0-21swapper/010:07:580
3141977992119,1cyclictest0-21swapper/009:59:520
3141977992119,1cyclictest0-21swapper/009:44:530
3141977992119,1cyclictest0-21swapper/009:36:070
3141977992119,1cyclictest0-21swapper/008:42:470
3141977992119,1cyclictest0-21swapper/007:41:230
3141977992113,7cyclictest3142152-21cat07:10:120
314197799210,20cyclictest0-21swapper/010:47:450
314197799210,20cyclictest0-21swapper/009:00:520
139099210,3rtkit-daemon1389-21rtkit-daemon12:21:482
3141987992018,1cyclictest3056-21PK-Backend10:50:173
3141987992018,1cyclictest0-21swapper/311:06:073
3141987992017,2cyclictest3056-21PK-Backend10:33:153
3141987992017,2cyclictest3056-21PK-Backend07:48:283
3141987992017,2cyclictest0-21swapper/310:20:573
3141987992012,6cyclictest3231620-21sensors_temp11:20:193
3141987992012,6cyclictest3216624-21cron10:44:563
3141987992012,6cyclictest3215788-21cat10:35:183
3141987992012,6cyclictest3182049-21grep09:00:133
314198799201,17cyclictest3201967-21/usr/sbin/munin09:55:203
314198799200,8cyclictest3223582-21awk11:00:003
314198799200,19cyclictest3056-21PK-Backend12:17:043
314198799200,19cyclictest3056-21PK-Backend08:39:053
314198799200,19cyclictest3056-21PK-Backend08:00:443
314198799200,19cyclictest0-21swapper/311:45:413
314198799200,19cyclictest0-21swapper/311:37:443
314198799200,19cyclictest0-21swapper/309:44:413
314198799200,19cyclictest0-21swapper/309:17:243
314198799200,19cyclictest0-21swapper/307:21:523
314198799200,18cyclictest3168957-21sh08:25:003
314198799200,18cyclictest3056-21PK-Backend11:29:453
314198399203,16cyclictest1095-21runrttasks10:12:042
3141983992018,1cyclictest0-21swapper/212:26:552
3141983992018,1cyclictest0-21swapper/212:08:492
3141983992018,1cyclictest0-21swapper/211:00:152
3141983992018,1cyclictest0-21swapper/210:55:282
3141983992018,1cyclictest0-21swapper/210:45:562
3141983992018,1cyclictest0-21swapper/210:05:332
3141983992018,1cyclictest0-21swapper/209:14:362
3141983992018,1cyclictest0-21swapper/209:09:162
3141983992018,1cyclictest0-21swapper/207:34:472
3141983992012,6cyclictest3245641-21ntpq12:00:182
314198399200,1cyclictest0-21swapper/208:31:312
314198399200,19cyclictest0-21swapper/212:15:322
314198399200,19cyclictest0-21swapper/211:25:032
314198399200,19cyclictest0-21swapper/211:10:372
314198399200,19cyclictest0-21swapper/209:45:122
314198399200,19cyclictest0-21swapper/209:20:592
314198399200,19cyclictest0-21swapper/208:26:182
314198399200,19cyclictest0-21swapper/207:12:492
314197999201,8cyclictest3257927-21python312:35:221
3141979992018,1cyclictest3056-21PK-Backend09:17:441
3141979992018,1cyclictest3056-21PK-Backend09:03:051
314197999201,7cyclictest3198298-21/usr/sbin/munin09:45:161
314197999201,4cyclictest3237101-21python311:35:201
314197999200,8cyclictest3255994-21strings12:30:161
314197999200,8cyclictest3225929-21df11:05:131
314197999200,19cyclictest3056-21PK-Backend11:48:381
314197999200,19cyclictest3056-21PK-Backend07:35:101
314197799209,2cyclictest141rcu_preempt09:25:220
314197799206,5cyclictest13-21ksoftirqd/009:08:590
3141977992018,1cyclictest3056-21PK-Backend08:05:570
3141977992018,1cyclictest0-21swapper/012:16:350
3141977992018,1cyclictest0-21swapper/012:08:230
3141977992018,1cyclictest0-21swapper/012:04:170
3141977992018,1cyclictest0-21swapper/011:05:580
3141977992018,1cyclictest0-21swapper/010:31:370
3141977992018,1cyclictest0-21swapper/008:57:330
3141977992012,6cyclictest3203692-21ntp_states10:00:190
3141977992012,6cyclictest3159688-21sensors_temp07:55:200
3141977992012,2cyclictest13-21ksoftirqd/010:26:180
314197799200,1cyclictest0-21swapper/011:59:370
314197799200,1cyclictest0-21swapper/011:41:400
314197799200,19cyclictest3056-21PK-Backend07:21:140
314197799200,19cyclictest0-21swapper/010:11:160
314197799200,19cyclictest0-21swapper/009:11:540
314197799200,19cyclictest0-21swapper/008:28:130
314197799200,19cyclictest0-21swapper/008:01:310
139099200,3rtkit-daemon1389-21rtkit-daemon09:07:473
139099200,17rtkit-daemon978-21snmpd08:44:593
139099200,16rtkit-daemon3141968-21cyclictest09:59:052
139099200,16rtkit-daemon3141968-21cyclictest08:40:172
3141987991917,1cyclictest3056-21PK-Backend12:22:393
3141987991917,1cyclictest3056-21PK-Backend07:43:423
3141987991917,1cyclictest3056-21PK-Backend07:12:393
3141987991917,1cyclictest0-21swapper/310:06:513
314198799191,5cyclictest3189849-21runrttasks09:20:203
3141987991914,3cyclictest3165102-21users08:10:233
3141987991913,5cyclictest3170794-21cut08:30:003
3141987991912,5cyclictest3200158-21strings09:50:173
314198799191,16cyclictest3186231-21python309:10:223
3141987991910,7cyclictest883-21dbus-daemon11:45:013
314198799190,1cyclictest3056-21PK-Backend12:30:493
314198799190,1cyclictest0-21swapper/312:36:233
314198799190,1cyclictest0-21swapper/311:10:333
314198799190,1cyclictest0-21swapper/309:35:013
314198799190,18cyclictest0-21swapper/310:14:223
314198799190,18cyclictest0-21swapper/310:01:103
314198799190,18cyclictest0-21swapper/308:06:203
314198799190,18cyclictest0-21swapper/307:52:413
314198399199,8cyclictest3181664-21unin-run09:00:012
3141983991917,1cyclictest3056-21PK-Backend08:23:412
3141983991917,1cyclictest0-21swapper/211:06:102
3141983991915,3cyclictest978-21snmpd09:01:022
3141983991915,3cyclictest978-21snmpd08:11:072
3141983991912,6cyclictest1-21systemd12:15:002
3141983991912,5cyclictest3196191-21grep09:40:122
314198399190,1cyclictest0-21swapper/211:38:002
314198399190,1cyclictest0-21swapper/208:38:272
314198399190,18cyclictest0-21swapper/207:52:112
3141979991917,1cyclictest3056-21PK-Backend11:13:091
3141979991917,1cyclictest3056-21PK-Backend09:44:491
3141979991913,5cyclictest3056-21PK-Backend09:22:371
314197999190,8cyclictest3151317-21perf07:35:001
314197999190,1cyclictest3056-21PK-Backend09:34:141
314197999190,18cyclictest3056-21PK-Backend11:18:301
3141977991917,1cyclictest0-21swapper/011:24:150
3141977991917,1cyclictest0-21swapper/010:44:270
3141977991917,1cyclictest0-21swapper/009:52:230
3141977991917,1cyclictest0-21swapper/009:23:220
3141977991917,1cyclictest0-21swapper/007:16:220
3141977991913,5cyclictest3149311-21cron07:29:590
3141977991911,6cyclictest3232490-21awk11:25:100
3141977991911,2cyclictest13-21ksoftirqd/012:25:200
3141977991911,2cyclictest13-21ksoftirqd/009:15:310
3141977991911,2cyclictest13-21ksoftirqd/007:50:100
314197799190,1cyclictest0-21swapper/012:10:260
314197799190,1cyclictest0-21swapper/011:49:030
314197799190,18cyclictest3056-21PK-Backend08:45:310
314198799182,12cyclictest3180132-21apt-get08:55:033
3141987991816,1cyclictest3056-21PK-Backend12:11:303
3141987991816,1cyclictest0-21swapper/312:01:493
3141987991816,1cyclictest0-21swapper/309:47:053
3141987991816,1cyclictest0-21swapper/308:33:163
3141987991816,1cyclictest0-21swapper/307:58:143
314198799181,5cyclictest3149354-21munin-run07:29:593
314198799181,4cyclictest3243761-21memory11:55:163
3141987991811,5cyclictest3193423-21python309:30:183
314198799181,15cyclictest3149793-21cnf-update-db07:30:003
3141987991810,6cyclictest42-21ksoftirqd/311:50:183
3141983991814,3cyclictest3194376-21cpuspeed09:35:112
3141983991812,4cyclictest3240021-21expr11:45:142
3141983991811,6cyclictest3145157-21awk07:15:202
3141983991811,5cyclictest3192878-21expr09:30:102
314198399180,1cyclictest0-21swapper/209:19:102
314197999186,3cyclictest3056-21PK-Backend09:54:451
3141979991811,5cyclictest3194849-21cat09:35:171
314197799185,4cyclictest13-21ksoftirqd/009:33:160
3141977991816,1cyclictest3056-21PK-Backend11:01:310
3141977991816,1cyclictest3056-21PK-Backend08:31:240
3141977991816,1cyclictest0-21swapper/008:20:020
3141977991816,1cyclictest0-21swapper/008:13:540
3141977991811,6cyclictest3179634-21apt-get08:55:000
3141977991811,6cyclictest3174050-21awk08:35:270
3141987991715,1cyclictest0-21swapper/311:30:193
314198799171,12cyclictest436-20systemd-journal07:20:003
314198799170,15cyclictest3176812-21diskstats08:45:153
3141983991715,1cyclictest0-21swapper/211:33:342
3141979991715,1cyclictest3056-21PK-Backend09:27:311
314197999170,7cyclictest3160297-21cron08:00:011
3141977991715,1cyclictest0-21swapper/011:35:450
3141977991715,1cyclictest0-21swapper/009:49:050
314198799169,5cyclictest3219001-21df_inode10:45:123
3141983991610,5cyclictest3243429-21cat11:55:122
3141979991613,2cyclictest3056-21PK-Backend08:54:391
3141983991512,2cyclictest3152441-21sensors_temp07:35:192
3141983991512,2cyclictest3147832-21/usr/sbin/munin07:25:152
3141977991513,1cyclictest0-21swapper/011:51:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional