You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-20 - 23:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Aug 20, 2026 12:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
376322621470,4sleep10-21swapper/109:40:171
376693921370,4sleep20-21swapper/209:50:192
371029021100,1sleep20-21swapper/207:10:162
378451021060,3sleep03784502-21ntp_kernel_pll_10:40:170
372115521010,1sleep30-21swapper/307:40:163
370897427638,13sleep30-21swapper/307:05:363
139099750,63rtkit-daemon0-21swapper/207:05:252
370854826940,22sleep00-21swapper/007:05:200
370904926545,14sleep10-21swapper/107:06:331
37829542490,3chrt0-21swapper/310:35:233
37961222480,3sleep20-21swapper/211:15:122
38228302410,4sleep00-21swapper/012:30:130
370945899391,27cyclictest3813967-21/usr/sbin/munin12:05:173
370945899391,27cyclictest3813967-21/usr/sbin/munin12:05:173
370945899377,21cyclictest42-21ksoftirqd/311:44:043
370945899360,21cyclictest3756908-21taskset09:20:403
370945899350,21cyclictest0-21swapper/308:13:273
370945899350,20cyclictest3795883-21chrt11:15:053
370945899341,19cyclictest3816717-21latency_hist12:15:003
370945899321,19cyclictest0-21swapper/310:17:023
370945899321,16cyclictest0-21swapper/310:20:233
370945899321,16cyclictest0-21swapper/308:45:193
370945899320,19cyclictest3768468-21cstates09:55:143
370945899320,18cyclictest3800774-21taskset11:29:163
370945899320,18cyclictest3710845-21runrttasks07:11:403
3709450993214,4cyclictest13-21ksoftirqd/009:43:120
370945899310,17cyclictest3713163-21runrttasks07:17:533
3709450993115,4cyclictest13-21ksoftirqd/007:26:550
370945899301,27cyclictest3795843-21cat11:15:003
3709458993010,18cyclictest3770879-21sed10:00:213
370945899300,18cyclictest0-21swapper/309:03:123
370945899299,18cyclictest3764875-21ls09:45:143
370945899299,18cyclictest3740484-21ls08:35:173
370945899298,20cyclictest3725028-21sed07:50:183
3709458992910,17cyclictest3774271-21ls10:10:203
3709452992915,12cyclictest3722096-21unin-run07:45:011
370945899288,18cyclictest3757755-21expr09:25:133
370945899288,18cyclictest3750787-21cut09:05:133
370945899287,19cyclictest3754473-21basename09:15:143
370945899282,22cyclictest3810056-21apt11:55:103
370945899280,26cyclictest3806940-21df11:45:123
370945899280,26cyclictest3788955-21/usr/sbin/munin10:55:133
370945899280,26cyclictest3747504-21df_inode08:55:133
370945899277,19cyclictest3729768-21expr08:05:143
370945899277,18cyclictest3817570-21grep12:15:153
370945899277,18cyclictest3803299-21cat11:35:153
3709458992713,12cyclictest3771653-21apt10:05:123
3709458992710,15cyclictest3753795-21latency_hist09:15:003
370945899270,25cyclictest3812356-21sed12:00:163
370945899270,25cyclictest3759473-21cstates09:30:123
370945899270,15cyclictest3798047-21hddtemp_smartct11:20:153
3709450992712,4cyclictest13-21ksoftirqd/010:34:010
370945899269,15cyclictest3761157-21cpuspeed_turbos09:35:133
370945899268,16cyclictest436-20systemd-journal07:35:013
370945899267,17cyclictest436-20systemd-journal10:50:003
370945899267,17cyclictest3738570-21df_inode08:30:153
370945899265,19cyclictest3794035-21sed11:10:003
370945899261,23cyclictest3713586-21/usr/sbin/munin07:20:213
3709458992611,13cyclictest3820691-21/usr/sbin/munin12:25:133
3709458992610,14cyclictest978-21snmpd08:27:503
3709458992610,14cyclictest3781031-21ntp_kernel_pll_10:30:193
370945899260,24cyclictest3792257-21wc11:05:003
370945899260,24cyclictest3745408-21apt08:50:133
370945899260,14cyclictest3822836-21diskstats12:30:133
370945899256,17cyclictest436-20systemd-journal07:45:013
370945899255,18cyclictest3735022-21ls08:20:113
370945899252,20cyclictest3742795-21python308:40:223
370945899252,19cyclictest3801118-21apt11:30:113
370945899251,21cyclictest0-21swapper/310:50:113
370945899250,23cyclictest978-21snmpd12:38:363
370945899250,23cyclictest3718785-21/usr/sbin/munin07:35:153
370945899250,23cyclictest3715346-21apt07:25:123
139099250,3rtkit-daemon1389-21rtkit-daemon07:57:002
370945899247,15cyclictest0-21swapper/312:22:263
370945899245,17cyclictest3763107-21entropy09:40:143
370945899241,21cyclictest3808326-21apt11:50:123
3709458992410,13cyclictest978-21snmpd10:28:123
370945099249,4cyclictest13-21ksoftirqd/009:11:320
370945099248,4cyclictest13-21ksoftirqd/008:32:200
370945099243,11cyclictest13-21ksoftirqd/012:10:000
370945099243,11cyclictest13-21ksoftirqd/012:10:000
370945899236,15cyclictest3726240-21cat07:55:163
370945899235,16cyclictest3784766-21sort10:40:213
370945899231,6cyclictest3728138-21sed08:00:193
370945899230,21cyclictest3766718-21cat09:50:183
370945899230,21cyclictest3733521-21ls08:15:163
3709456992315,6cyclictest436-20systemd-journal12:32:342
370945699230,4cyclictest3713012-21chrt07:15:582
370945099234,5cyclictest13-21ksoftirqd/011:45:000
3709450992312,4cyclictest13-21ksoftirqd/011:06:290
3709450992312,4cyclictest13-21ksoftirqd/010:53:110
3709450992312,4cyclictest13-21ksoftirqd/010:29:480
3709450992312,4cyclictest13-21ksoftirqd/009:58:160
3709450992312,4cyclictest13-21ksoftirqd/008:52:560
3709450992312,4cyclictest13-21ksoftirqd/008:03:150
3709450992312,4cyclictest13-21ksoftirqd/007:33:010
3709450992311,3cyclictest13-21ksoftirqd/009:55:020
3709456992220,1cyclictest0-21swapper/211:41:212
3709456992220,1cyclictest0-21swapper/210:55:392
3709456992220,1cyclictest0-21swapper/208:55:062
3709456992220,1cyclictest0-21swapper/208:26:202
3709456992220,1cyclictest0-21swapper/208:01:262
370945699220,20cyclictest3752269-21latency_hist09:10:012
3709452992220,1cyclictest3056-21PK-Backend08:32:161
3709452992219,2cyclictest3056-21PK-Backend11:38:131
3709452992219,2cyclictest3056-21PK-Backend09:17:391
370945299220,20cyclictest3056-21PK-Backend12:02:591
370945299220,20cyclictest3056-21PK-Backend08:05:051
370945099228,4cyclictest13-21ksoftirqd/008:49:430
3709450992220,1cyclictest0-21swapper/012:23:550
3709450992212,4cyclictest13-21ksoftirqd/012:29:340
3709450992212,3cyclictest13-21ksoftirqd/009:48:380
3709450992212,2cyclictest141rcu_preempt10:35:340
3709450992211,4cyclictest13-21ksoftirqd/012:39:560
3709450992211,4cyclictest13-21ksoftirqd/011:56:130
3709450992211,4cyclictest13-21ksoftirqd/011:23:380
3709450992211,4cyclictest13-21ksoftirqd/011:13:310
3709450992211,4cyclictest13-21ksoftirqd/010:21:470
3709450992211,4cyclictest13-21ksoftirqd/010:14:160
3709450992211,4cyclictest13-21ksoftirqd/009:02:470
3709450992211,4cyclictest13-21ksoftirqd/008:38:080
3709450992211,4cyclictest13-21ksoftirqd/008:12:580
3709450992211,3cyclictest13-21ksoftirqd/008:44:260
3709456992119,1cyclictest3056-21PK-Backend12:15:352
3709456992119,1cyclictest3056-21PK-Backend10:40:182
3709456992119,1cyclictest0-21swapper/212:06:362
3709456992119,1cyclictest0-21swapper/212:06:362
3709456992119,1cyclictest0-21swapper/211:54:532
3709456992119,1cyclictest0-21swapper/211:33:042
3709456992119,1cyclictest0-21swapper/211:20:532
3709456992119,1cyclictest0-21swapper/210:22:202
3709456992119,1cyclictest0-21swapper/210:01:462
3709456992119,1cyclictest0-21swapper/209:13:332
3709456992119,1cyclictest0-21swapper/209:02:092
3709456992119,1cyclictest0-21swapper/208:17:072
3709456992119,1cyclictest0-21swapper/207:32:062
3709456992119,1cyclictest0-21swapper/207:20:362
3709456992118,2cyclictest3787250-21apt10:50:112
3709456992115,4cyclictest3744086-21sort08:45:142
3709456992113,6cyclictest883-21dbus-daemon11:15:002
370945699210,2cyclictest3741932-21apt08:40:132
370945699210,20cyclictest0-21swapper/210:30:322
370945699210,20cyclictest0-21swapper/210:11:312
3709452992119,1cyclictest3056-21PK-Backend12:29:081
3709452992119,1cyclictest3056-21PK-Backend11:04:521
3709452992119,1cyclictest3056-21PK-Backend10:49:451
3709452992119,1cyclictest3056-21PK-Backend09:09:481
3709452992119,1cyclictest3056-21PK-Backend08:19:121
3709452992119,1cyclictest3056-21PK-Backend08:13:301
3709452992119,1cyclictest3056-21PK-Backend07:34:371
3709452992119,1cyclictest3056-21PK-Backend07:25:511
3709452992119,1cyclictest0-21swapper/110:55:121
3709452992119,1cyclictest0-21swapper/107:56:061
3709452992118,2cyclictest3056-21PK-Backend10:16:461
370945299210,20cyclictest3056-21PK-Backend11:57:451
370945299210,19cyclictest3056-21PK-Backend12:31:321
370945299210,19cyclictest3056-21PK-Backend11:53:191
370945299210,19cyclictest3056-21PK-Backend11:16:081
370945299210,19cyclictest3056-21PK-Backend09:29:481
370945299210,19cyclictest3056-21PK-Backend09:23:201
370945299210,19cyclictest3056-21PK-Backend08:50:151
370945299210,19cyclictest3056-21PK-Backend08:37:441
370945299210,19cyclictest3056-21PK-Backend07:21:131
370945099219,4cyclictest13-21ksoftirqd/010:16:170
370945099219,4cyclictest13-21ksoftirqd/007:50:130
370945099218,5cyclictest13-21ksoftirqd/011:16:450
3709450992119,1cyclictest0-21swapper/009:34:490
3709450992119,1cyclictest0-21swapper/007:16:120
3709450992113,6cyclictest3733700-21awk08:15:170
3709450992112,7cyclictest3726652-21sensors_temp07:55:200
3709450992111,3cyclictest13-21ksoftirqd/012:11:180
3709450992111,3cyclictest13-21ksoftirqd/009:37:030
3709450992111,3cyclictest13-21ksoftirqd/007:11:470
3709450992110,4cyclictest13-21ksoftirqd/011:53:250
3709450992110,4cyclictest13-21ksoftirqd/011:29:310
3709450992110,4cyclictest13-21ksoftirqd/011:03:280
3709450992110,4cyclictest13-21ksoftirqd/010:03:370
3709450992110,4cyclictest13-21ksoftirqd/008:21:400
3709450992110,4cyclictest13-21ksoftirqd/007:38:210
37704992200,3sleep126-21ksoftirqd/110:00:171
3709456992018,1cyclictest0-21swapper/212:38:062
3709456992018,1cyclictest0-21swapper/212:25:092
3709456992018,1cyclictest0-21swapper/211:58:332
3709456992018,1cyclictest0-21swapper/211:46:492
3709456992018,1cyclictest0-21swapper/211:28:242
3709456992018,1cyclictest0-21swapper/211:05:332
3709456992018,1cyclictest0-21swapper/209:42:532
3709456992018,1cyclictest0-21swapper/209:31:512
3709456992018,1cyclictest0-21swapper/209:20:482
3709456992018,1cyclictest0-21swapper/208:52:572
3709456992017,2cyclictest3056-21PK-Backend12:20:222
3709456992014,4cyclictest3739411-21sshd08:31:552
3709456992011,7cyclictest3759006-21awk09:30:002
370945699200,1cyclictest0-21swapper/210:16:322
370945699200,19cyclictest0-21swapper/208:10:402
370945699200,19cyclictest0-21swapper/208:09:522
370945699200,19cyclictest0-21swapper/207:44:032
3709452992018,1cyclictest3056-21PK-Backend12:36:191
3709452992018,1cyclictest3056-21PK-Backend11:34:331
3709452992018,1cyclictest3056-21PK-Backend09:30:361
3709452992018,1cyclictest0-21swapper/110:41:421
3709452992017,2cyclictest3056-21PK-Backend11:29:061
3709452992016,3cyclictest3820072-21vmstat12:20:231
3709452992014,5cyclictest26-21ksoftirqd/109:55:041
370945299200,1cyclictest3056-21PK-Backend07:17:551
370945299200,19cyclictest3056-21PK-Backend11:48:531
370945299200,19cyclictest0-21swapper/107:45:461
370945299200,18cyclictest3056-21PK-Backend11:24:261
370945299200,18cyclictest3056-21PK-Backend08:24:061
370945099209,4cyclictest13-21ksoftirqd/012:01:230
370945099209,4cyclictest13-21ksoftirqd/009:28:460
370945099209,4cyclictest13-21ksoftirqd/009:09:200
370945099209,4cyclictest13-21ksoftirqd/007:42:420
370945099208,4cyclictest13-21ksoftirqd/011:46:200
370945099208,4cyclictest13-21ksoftirqd/011:34:520
370945099208,4cyclictest13-21ksoftirqd/007:20:410
370945099204,4cyclictest13-21ksoftirqd/008:26:460
3709450992018,1cyclictest0-21swapper/012:18:350
3709450992018,1cyclictest0-21swapper/010:08:560
3709450992017,2cyclictest3056-21PK-Backend11:35:010
3709450992010,4cyclictest13-21ksoftirqd/008:08:010
3709450992010,3cyclictest13-21ksoftirqd/009:16:220
3709456991916,2cyclictest3056-21PK-Backend12:10:012
3709456991914,3cyclictest3804329-21grep11:40:002
3709456991912,4cyclictest1095-21runrttasks07:50:172
370945699190,1cyclictest3056-21PK-Backend07:48:552
370945699190,1cyclictest0-21swapper/212:02:592
370945699190,1cyclictest0-21swapper/211:01:552
370945699190,18cyclictest0-21swapper/210:49:102
3709452991917,1cyclictest3056-21PK-Backend10:53:231
3709452991917,1cyclictest3056-21PK-Backend09:47:541
3709452991917,1cyclictest0-21swapper/109:10:351
3709452991916,2cyclictest3056-21PK-Backend09:37:051
3709452991915,2cyclictest3818452-21sh12:20:001
370945299190,1cyclictest3056-21PK-Backend11:42:391
370945299190,18cyclictest3056-21PK-Backend11:10:071
370945099198,3cyclictest13-21ksoftirqd/008:56:500
370945099197,4cyclictest13-21ksoftirqd/010:47:400
370945099196,4cyclictest13-21ksoftirqd/009:24:570
3709450991917,1cyclictest0-21swapper/010:57:160
3709456991816,1cyclictest0-21swapper/209:37:462
3709456991816,1cyclictest0-21swapper/209:16:292
3709456991816,1cyclictest0-21swapper/208:20:052
3709456991811,6cyclictest3779501-21sed10:25:202
3709456991810,6cyclictest3765205-21ntp_states09:45:182
3709452991816,1cyclictest3056-21PK-Backend07:50:401
3709452991816,1cyclictest0-21swapper/110:09:231
3709452991812,5cyclictest3815101-21wc12:10:011
3709452991812,5cyclictest3815101-21wc12:10:001
370945699179,6cyclictest3716380-21sed07:25:212
3709456991710,5cyclictest3740107-21grep08:35:142
3709456991710,5cyclictest3719287-21awk07:35:162
370945299179,6cyclictest3778326-21sh10:25:001
3709452991715,1cyclictest3056-21PK-Backend08:28:101
3709452991712,3cyclictest3816717-21latency_hist12:15:001
3709450991710,5cyclictest3722485-21grep07:45:110
3709456991610,5cyclictest3772541-21ls10:05:242
3709452991614,1cyclictest0-21swapper/108:41:151
3709452991613,2cyclictest3056-21PK-Backend11:06:421
3709452991611,4cyclictest3747714-21load08:55:171
3709452991611,3cyclictest1095-21runrttasks07:10:031
3709452991610,4cyclictest3767652-21apt-get09:55:011
139099160,13rtkit-daemon0-21swapper/210:36:402
3709452991513,1cyclictest3056-21PK-Backend10:35:181
3709452991513,1cyclictest3056-21PK-Backend09:03:321
370945699149,3cyclictest978-21snmpd09:57:522
370945299148,5cyclictest3779622-21ls10:25:201
370945299147,5cyclictest3780810-21grep10:30:171
3709452991410,3cyclictest3056-21PK-Backend08:01:191
370945299139,3cyclictest3056-21PK-Backend10:12:081
370945299119,1cyclictest3056-21PK-Backend08:47:311
370945299117,3cyclictest3056-21PK-Backend07:37:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional