You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 11:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Sep 17, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3071229670,8sleep00-21swapper/019:06:500
3068329668,9sleep30-21swapper/319:06:293
3064229379,10sleep20-21swapper/219:05:562
3089728467,13sleep10-21swapper/119:09:161
3104899280,1cyclictest0-21swapper/021:46:020
3105099260,25cyclictest0-21swapper/222:33:562
3105199240,1cyclictest0-21swapper/321:51:253
3105099240,1cyclictest0-21swapper/200:08:522
31051992319,1cyclictest39-21ksoftirqd/322:05:473
3105099211,11cyclictest7471-21ssh23:35:142
3105099210,20cyclictest0-21swapper/219:10:122
31049992119,1cyclictest0-21swapper/123:25:281
31051992019,1cyclictest0-21swapper/322:11:163
31050992019,1cyclictest0-21swapper/222:16:102
3104999200,20cyclictest0-21swapper/122:51:151
3104999200,19cyclictest0-21swapper/123:40:201
3104999200,0cyclictest0-21swapper/121:32:301
3104999200,0cyclictest0-21swapper/100:29:271
3104899205,7cyclictest14050-21munin-run20:50:000
3104899200,1cyclictest0-21swapper/023:36:380
3105199192,16cyclictest39-21ksoftirqd/323:15:493
3104999190,18cyclictest0-21swapper/119:55:131
3105199188,4cyclictest8826-21apt-get21:40:023
31051991811,3cyclictest39-21ksoftirqd/300:12:563
3105199180,15cyclictest0-21swapper/323:00:043
3104899180,4cyclictest0-21swapper/022:00:240
3105199176,2cyclictest225-21systemd-journal22:20:143
31050991714,2cyclictest23310-21load23:05:192
3105099170,17cyclictest0-21swapper/200:39:512
3104999170,16cyclictest26999-21kworker/1:023:15:151
3104999170,13cyclictest0-21swapper/100:23:481
31048991717,0cyclictest0-21swapper/000:25:460
31048991716,1cyclictest13-21ksoftirqd/000:19:100
3105199166,3cyclictest39-21ksoftirqd/322:53:433
3105199162,1cyclictest141rcu_preempt23:14:283
31051991615,1cyclictest0-21swapper/322:01:263
3105199160,5cyclictest13073-21ntp_kernel_pll_20:45:213
31050991611,2cyclictest32-21ksoftirqd/221:15:152
3105099160,1cyclictest0-21swapper/223:15:582
3105099160,16cyclictest0-21swapper/200:34:332
31049991614,0cyclictest0-21swapper/123:08:581
3104999160,2cyclictest0-21swapper/120:05:121
3104899163,6cyclictest16440-21awk20:55:000
3104899161,7cyclictest9293-21munin-run20:40:000
31048991614,2cyclictest13-21ksoftirqd/021:55:190
31051991511,3cyclictest13203-21cut22:40:143
3105099155,9cyclictest25369-21apache222:37:352
3105099150,14cyclictest0-21swapper/200:03:132
3104999155,9cyclictest0-21swapper/121:59:561
31049991510,3cyclictest29807-21ls20:15:001
3104899151,8cyclictest3348-21munin-run19:20:010
3104899150,15cyclictest0-21swapper/000:07:030
31051991413,1cyclictest0-21swapper/300:17:453
31051991411,2cyclictest39-21ksoftirqd/323:55:163
3105199141,12cyclictest11981-21cron22:40:003
3105199140,1cyclictest141rcu_preempt21:55:413
3105199140,0cyclictest141rcu_preempt22:18:123
3105099148,5cyclictest976-21grep23:30:252
3105099147,6cyclictest23567-21diskmemload23:57:332
3105099141,7cyclictest141rcu_preempt21:00:002
31050991410,3cyclictest4102-21sed20:25:262
31049991414,0cyclictest0-21swapper/121:22:461
3104999140,2cyclictest0-21swapper/123:39:441
3104999140,14cyclictest0-21swapper/100:35:401
3104899140,4cyclictest0-21swapper/023:00:210
3104899140,10cyclictest0-21swapper/021:00:120
3105199139,3cyclictest2689-21cat21:35:253
3105199139,3cyclictest24406-21cut21:10:153
3105199139,3cyclictest18127-21cat20:55:263
3105199136,6cyclictest12847-21munin-node00:24:223
3105199134,6cyclictest39-21ksoftirqd/300:37:413
31051991312,1cyclictest381rcuc/323:36:513
3105199131,1cyclictest381rcuc/322:28:253
3105199131,1cyclictest381rcuc/322:28:253
31051991310,2cyclictest7897-21dump-pmu-power23:55:003
31051991310,2cyclictest27757-21grep20:10:143
31051991310,2cyclictest25599-21tr20:05:133
31051991310,2cyclictest25002-21munin-plugin-st20:05:003
31051991310,2cyclictest21232-21cut19:55:163
31051991310,2cyclictest10661-21ls19:35:003
3105099139,3cyclictest2544-21grep00:10:202
3105099137,6cyclictest25369-21apache222:49:002
31050991310,2cyclictest908-21grep23:50:162
31050991310,2cyclictest311rcuc/221:30:152
31050991310,2cyclictest19054-21cat00:20:192
31050991310,2cyclictest1826-21ls21:55:002
31050991310,2cyclictest15093-21sed20:50:172
31050991310,2cyclictest12073-21grep20:45:152
3104999139,3cyclictest11592-21grep19:35:161
3104999132,2cyclictest26704-21ssh00:05:251
31049991313,0cyclictest0-21swapper/122:48:531
31049991310,2cyclictest28567-21ssh22:10:201
31049991310,2cyclictest25223-21wc23:45:191
31049991310,2cyclictest12493-21sh22:00:251
3104999130,12cyclictest4573-21grep22:15:241
3104899139,3cyclictest9800-21wc23:55:210
3104899139,3cyclictest19621-21wc19:50:260
3104899139,2cyclictest21697-21ls19:55:210
3104899137,4cyclictest0-21swapper/022:35:220
3104899133,5cyclictest0-21swapper/023:05:190
31048991310,2cyclictest4199-21tail22:15:210
31048991310,2cyclictest27994-21wc21:50:240
31048991310,2cyclictest23024-21grep22:45:260
31048991310,2cyclictest17954-21grep00:00:200
31048991310,2cyclictest17595-21grep21:25:170
31048991310,2cyclictest11526-21grep21:40:290
3105199129,2cyclictest6922-21sh23:34:593
3105199129,2cyclictest3688-21sed00:30:233
3105199129,2cyclictest26606-21grep00:25:143
3105199129,2cyclictest18723-21head21:45:153
3105199129,2cyclictest17279-21wc19:45:263
3105199129,2cyclictest17061-21tr00:00:153
3105199129,2cyclictest11746-21tail19:35:173
3105199128,3cyclictest24795-21cut23:45:173
3105199128,3cyclictest1992-21cut19:15:193
3105199128,3cyclictest19215-21cut21:00:123
3105199125,3cyclictest225-21systemd-journal23:30:003
3105199123,3cyclictest39-21ksoftirqd/321:15:203
3105199121,0cyclictest141rcu_preempt22:45:243
31051991210,2cyclictest39-21ksoftirqd/323:05:163
31051991210,1cyclictest25532-21tr00:05:173
31051991210,1cyclictest1657-21sed20:20:273
3105199120,11cyclictest21743-21grep21:05:163
3105099129,2cyclictest24322-21grep23:25:202
3105099129,2cyclictest21744-21tr21:05:162
3105099129,2cyclictest21179-21cron21:05:002
3105099129,2cyclictest17021-21grep21:25:142
3105099129,2cyclictest16140-21grep23:20:202
3105099128,3cyclictest9440-21cat20:40:002
3105099126,6cyclictest25356-21apache221:22:422
31050991211,1cyclictest10549-21sh22:57:572
31050991210,2cyclictest0-21swapper/223:13:152
31050991210,1cyclictest24773-21grep23:45:172
3105099120,9cyclictest0-21swapper/220:40:122
3105099120,9cyclictest0-21swapper/220:40:122
3105099120,1cyclictest0-21swapper/222:26:122
3105099120,1cyclictest0-21swapper/222:26:122
3104999129,2cyclictest685-21cat23:30:241
3104999129,2cyclictest4360-21grep19:20:171
3104999129,2cyclictest32037-21tail19:10:161
3104999129,2cyclictest28166-21grep21:50:251
3104999129,2cyclictest20489-21cat21:00:241
3104999129,2cyclictest19640-21tail22:05:161
3104999129,2cyclictest17548-21grep00:00:171
3104999129,2cyclictest15583-21grep23:20:141
3104999129,2cyclictest14118-21head19:40:151
3104999128,4cyclictest38150irq/131-snd_hda22:37:331
3104999128,3cyclictest19214-21rm22:44:081
3104999128,3cyclictest17207-21rm00:19:491
3104999120,2cyclictest0-21swapper/100:32:241
3104999120,12cyclictest0-21swapper/121:42:001
3104999120,10cyclictest5664-21tr22:55:161
3104899129,2cyclictest9158-21grep19:30:170
3104899129,2cyclictest7939-21fgrep23:15:230
3104899129,2cyclictest3652-21grep20:25:210
3104899129,2cyclictest31981-21cron23:49:590
3104899129,2cyclictest28689-21ls22:10:210
3104899129,2cyclictest28572-21tr22:30:140
3104899129,2cyclictest15111-21expr23:20:130
3104899128,3cyclictest447-21cron21:35:000
3104899125,6cyclictest946-21/usr/sbin/munin21:35:160
3104899123,8cyclictest0-21swapper/021:10:140
308012120,0chrt0-21swapper/023:29:520
176132120,0chrt0-21swapper/321:25:173
118502120,1sleep00-21swapper/000:35:210
3105199119,1cyclictest9687-21tr21:20:193
3105199118,2cyclictest9294-21systemctl20:40:003
3105199118,2cyclictest6918-21irqstats19:25:163
3105199118,2cyclictest5286-21ssh22:55:103
3105199118,2cyclictest31904-21taskset20:16:383
3105199118,2cyclictest30136-21grep22:30:253
3105199118,2cyclictest2963-21sed20:25:153
3105199118,2cyclictest26386-21ntpq21:30:193
3105199118,2cyclictest15792-21taskset23:40:143
3105199118,2cyclictest15009-21grep19:40:243
3105199115,5cyclictest5319-21grep19:20:263
3105199114,6cyclictest14721-21apache_volume23:20:113
3105099119,1cyclictest21269-21head19:55:162
3105099119,1cyclictest13015-21df_abs22:40:142
3105099118,2cyclictest9610-21grep19:30:222
3105099118,2cyclictest9381-21ssh00:15:042
3105099118,2cyclictest5922-21cat19:24:592
3105099118,2cyclictest5571-21head20:30:162
3105099118,2cyclictest3876-21ssh21:36:072
3105099118,2cyclictest28534-21cat22:10:202
3105099118,2cyclictest28134-21grep00:25:252
3105099118,2cyclictest27772-21ssh21:11:432
3105099118,2cyclictest21670-21ssh22:06:042
3105099118,2cyclictest20408-21sort19:55:002
3105099118,2cyclictest18050-21tr21:45:122
3105099118,2cyclictest14681-21ssh23:00:172
3105099118,2cyclictest14668-21gzip19:40:202
3105099118,2cyclictest11614-21ssh22:20:132
3105099118,2cyclictest11614-21sed21:40:292
3105099118,2cyclictest1086-21ssh22:52:042
3105099117,3cyclictest32177-21sh20:20:002
3105099113,4cyclictest225-21systemd-journal20:05:002
3105099112,5cyclictest225-21systemd-journal20:15:002
3105099111,3cyclictest225-21systemd-journal19:15:282
3105099110,10cyclictest2044-21ssh21:55:052
3104999119,1cyclictest12355-21expr20:45:151
3104999118,2cyclictest8635-21tr19:30:151
3104999118,2cyclictest7469-21cat20:35:141
3104999118,2cyclictest6145-21grep19:25:121
3104999118,2cyclictest4602-21rm23:52:261
3104999118,2cyclictest3628-21ssh00:10:421
3104999118,2cyclictest3531-21ssh22:34:531
3104999118,2cyclictest3410-21cat20:25:191
3104999118,2cyclictest29180-21rm21:12:501
3104999118,2cyclictest24802-21taskset20:02:461
3104999118,2cyclictest20994-21sh21:46:171
3104999118,2cyclictest1970-21rm23:11:541
3104999118,2cyclictest17576-21cut21:25:171
3104999118,2cyclictest16047-21cat19:45:141
3104999118,2cyclictest10009-21cat20:40:141
3104999118,2cyclictest10009-21cat20:40:141
3104999117,3cyclictest20360-21expr22:25:151
3104999117,3cyclictest20360-21expr22:25:151
3104999117,3cyclictest14054-21ssh22:20:481
3104999110,10cyclictest30043-21acpi20:15:141
3104899118,3cyclictest13-21ksoftirqd/022:06:180
3104899118,2cyclictest9572-21ssh22:57:100
3104899118,2cyclictest9444-21ssh21:20:180
3104899118,2cyclictest3549-21cat00:30:210
3104899118,2cyclictest32653-21grep20:20:150
3104899118,2cyclictest32280-21taskset21:15:050
3104899118,2cyclictest30939-21sendmail_mailst22:50:220
3104899118,2cyclictest2381-21ssh23:12:140
3104899118,2cyclictest18809-21head00:20:170
3104899118,2cyclictest1864-21ls23:50:250
3104899118,2cyclictest18276-21ssh22:43:240
3104899118,2cyclictest14673-21grep19:40:200
3104899118,2cyclictest14580-21ssh22:21:120
3104899118,2cyclictest1276-21ssh00:10:120
3104899118,2cyclictest10264-21grep20:40:160
3104899118,2cyclictest10264-21grep20:40:160
3104899110,10cyclictest5967-21open_files20:30:200
3104899110,10cyclictest32505-21cut23:30:170
3104899110,10cyclictest21948-21ssh22:25:270
3104899110,10cyclictest21948-21ssh22:25:270
3105199108,1cyclictest5901-21cut20:30:193
3105199108,1cyclictest32590-21grep19:10:213
3105199108,1cyclictest113752sleep320:41:213
3105199108,1cyclictest113752sleep320:41:213
3105199107,2cyclictest20199-21taskset19:54:543
3105099108,2cyclictest17133-21python19:45:252
3105099108,1cyclictest1872-21chrt20:21:332
3105099108,1cyclictest15554-21ssh23:40:042
3105099108,1cyclictest10853-21apt-config19:35:002
3105099108,1cyclictest10309-21ssh22:00:032
3105099107,2cyclictest25259-21apt20:05:122
3104999108,1cyclictest911-21if_err_enp0s31f20:20:191
3104999108,1cyclictest5440-21sort20:30:161
3104999108,1cyclictest1251-21cut19:15:151
3104999107,2cyclictest22175-21grep21:05:181
3104999107,2cyclictest1927-21basename21:35:181
3104999107,2cyclictest18487-21expr19:50:161
3104999107,2cyclictest16657-21basename20:55:121
3104999107,2cyclictest13943-21tr23:00:141
3104999107,2cyclictest10423-21ssh23:55:281
3104999100,9cyclictest32482-21ssh21:15:141
3104999100,9cyclictest16208-21taskset20:52:281
3104899108,1cyclictest7207-21cut19:25:190
3104899108,1cyclictest29186-21cat20:10:250
3104899108,1cyclictest16011-21expr19:45:150
3104899108,1cyclictest15597-21ssh23:40:060
3104899107,2cyclictest3910-21expr19:20:150
3104899107,2cyclictest31889-21tune2fs19:10:150
3104899107,2cyclictest30611-21df20:15:180
3104899107,2cyclictest27276-21chrt20:08:160
3104899107,2cyclictest25189-21cut20:05:000
3104899107,2cyclictest16946-21expr20:55:150
3104899106,3cyclictest21525-21grep21:05:150
3104899106,3cyclictest13101-21basename19:40:000
310519997,1cyclictest14606-21wc20:50:153
310509997,1cyclictest6921-21irqstats19:25:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional