You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-07 - 17:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu May 07, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
233112239162,25sleep00-21swapper/007:07:460
234362215163,39sleep10-21swapper/107:09:241
232352204167,24sleep30-21swapper/307:06:493
233032203166,25sleep20-21swapper/207:07:402
2832421680,2sleep32358399cyclictest08:30:213
2357799108104,3cyclictest0-21swapper/209:13:412
2127121070,4sleep22357799cyclictest10:40:262
23577998986,2cyclictest0-21swapper/209:35:272
309752680,5sleep330979-21unixbench_multi08:35:263
2357399375,14cyclictest0-21swapper/108:35:241
2356599378,13cyclictest10696-21munin-run10:20:010
2356599347,13cyclictest22654-21munin-run08:20:010
2356599346,13cyclictest18254-21munin-run08:10:010
2357399331,25cyclictest0-21swapper/109:10:211
2357399331,10cyclictest0-21swapper/108:53:151
2358399321,22cyclictest0-21swapper/310:32:133
2358399318,2cyclictest0-21swapper/307:15:173
23583993110,3cyclictest0-21swapper/308:42:223
2357399311,23cyclictest0-21swapper/112:35:241
2356599316,12cyclictest9127-21awk11:30:020
2358399301,13cyclictest0-21swapper/309:17:173
23573993015,3cyclictest10312-21users10:15:271
2357399301,22cyclictest0-21swapper/111:40:201
2357399301,20cyclictest0-21swapper/108:58:541
2358399291,11cyclictest0-21swapper/312:26:453
2358399290,20cyclictest0-21swapper/312:14:533
2357399291,11cyclictest0-21swapper/108:45:531
2358399281,12cyclictest0-21swapper/309:01:303
23583992810,3cyclictest0-21swapper/311:36:213
23583992810,3cyclictest0-21swapper/311:13:533
2357799284,3cyclictest87650irq/129-enp1s0-12:05:192
2357399281,19cyclictest24077-21perl09:35:141
2357399280,16cyclictest8891-21kworker/u8:012:00:231
2358399271,12cyclictest0-21swapper/310:00:173
2357399271,16cyclictest0-21swapper/108:00:361
2357399271,15cyclictest0-21swapper/111:24:061
2357399270,11cyclictest0-21swapper/110:45:051
2356599272,13cyclictest28636-21munin-run07:20:020
2356599272,10cyclictest30405-21kworker/0:211:21:040
23565992712,3cyclictest20811-21kworker/0:110:00:190
2356599271,13cyclictest4732-21basename11:20:010
23565992711,14cyclictest31464-21perf08:40:010
2356599270,12cyclictest17918-21awk11:50:010
129152270,5sleep011-21rcuc/009:10:140
2358399261,12cyclictest0-21swapper/310:06:533
2358399261,12cyclictest0-21swapper/309:49:453
2358399261,12cyclictest0-21swapper/308:26:103
2358399261,11cyclictest0-21swapper/310:13:413
2358399260,11cyclictest0-21swapper/311:00:303
2357399267,12cyclictest0-21swapper/112:05:091
23573992613,2cyclictest20528-21sed08:15:011
2357399261,20cyclictest0-21swapper/111:19:401
2357399261,18cyclictest0-21swapper/110:31:181
2357399261,13cyclictest10599-21sensors11:30:221
2357399261,13cyclictest10599-21sensors11:30:221
2356599268,15cyclictest32061-21munin-run09:55:020
2356599262,10cyclictest7703-21kworker/0:309:02:470
2356599262,10cyclictest23200-21kworker/0:009:37:320
2356599261,5cyclictest0-21swapper/009:55:250
2356599261,13cyclictest6302-21munin-run10:10:000
2356599261,10cyclictest31421-21kworker/0:308:42:230
2356599261,10cyclictest30405-21kworker/0:211:11:070
2358399253,13cyclictest0-21swapper/309:57:403
2358399251,18cyclictest0-21swapper/309:43:053
2358399251,14cyclictest0-21swapper/311:46:053
2358399251,13cyclictest0-21swapper/310:26:013
2358399251,11cyclictest0-21swapper/311:16:173
2358399250,12cyclictest0-21swapper/309:36:013
2357799259,3cyclictest9091-21needreboot09:00:182
2357799251,7cyclictest0-21swapper/210:29:212
2357399253,14cyclictest18415-21kworker/u8:109:15:121
2357399253,12cyclictest0-21swapper/109:40:261
23573992514,3cyclictest0-21swapper/107:20:001
2357399251,18cyclictest0-21swapper/109:48:581
2357399251,17cyclictest19141-21kworker/u8:211:03:111
2357399251,16cyclictest29499-21kworker/u8:009:56:411
2357399251,16cyclictest0-21swapper/111:25:021
2357399251,15cyclictest0-21swapper/107:36:121
2357399251,14cyclictest12274-21diskmemload09:26:311
2357399251,12cyclictest30049-21memory12:15:181
2357399251,12cyclictest24669-21perl10:50:141
2357399251,11cyclictest28297-21basename11:00:011
2357399251,10cyclictest569-21turbostat.cron07:30:011
2357399251,10cyclictest0-21swapper/112:30:171
2357399250,5cyclictest29499-21kworker/u8:010:10:191
2357399250,16cyclictest8586-21kworker/u8:110:39:481
2356599252,10cyclictest30487-21kworker/0:007:23:350
2358399248,3cyclictest0-21swapper/308:57:423
2358399245,9cyclictest0-21swapper/309:20:313
2358399242,16cyclictest0-21swapper/311:28:323
2358399241,10cyclictest0-21swapper/307:22:253
2358399240,16cyclictest24311-21kworker/u8:312:36:363
2358399240,16cyclictest0-21swapper/312:32:213
2357799242,20cyclictest0-21swapper/212:21:112
2357399249,3cyclictest32402-21ntp_states12:20:211
2357399248,8cyclictest32587-21kworker/u8:009:10:011
2357399242,3cyclictest0-21swapper/109:50:161
2357399241,5cyclictest0-21swapper/109:32:141
2357399241,16cyclictest29499-21kworker/u8:010:02:321
2357399241,15cyclictest0-21swapper/108:23:201
2357399241,12cyclictest0-21swapper/109:22:261
2357399241,11cyclictest28906-21latency_hist12:15:031
2357399241,11cyclictest20668-21df11:55:141
2357399241,11cyclictest18248-21latency_hist08:10:011
2357399241,10cyclictest10792-21pmu-power07:50:211
23573992410,3cyclictest20242-21fschecks_time10:40:151
23573992410,3cyclictest19116-21sed11:50:191
2357399240,15cyclictest0-21swapper/108:44:301
2357399240,11cyclictest11067-21tr10:20:111
2357399240,11cyclictest11067-21tr10:20:111
2357399240,10cyclictest0-21swapper/111:07:131
2356599249,3cyclictest11640-21kworker/0:208:00:110
2356599245,16cyclictest28283-21munin-run11:00:010
2356599242,10cyclictest5538-21kworker/0:008:58:150
2356599242,10cyclictest30487-21kworker/0:007:30:070
2356599241,8cyclictest15044-21kworker/0:210:47:510
2356599241,5cyclictest0-21swapper/012:10:130
2356599241,10cyclictest4926-21kworker/0:007:42:470
23565992410,3cyclictest20811-21kworker/0:109:30:120
2358399238,2cyclictest0-21swapper/311:40:233
2358399237,6cyclictest0-21swapper/309:31:293
2358399236,3cyclictest0-21swapper/310:58:113
23583992319,2cyclictest304342sleep309:50:153
23583992310,3cyclictest0-21swapper/309:14:543
2357799236,4cyclictest8481-21needreboot07:45:182
2357799231,6cyclictest0-21swapper/212:36:522
2357799231,6cyclictest0-21swapper/211:53:222
2357799231,20cyclictest0-21swapper/210:36:012
2357799231,20cyclictest0-21swapper/209:31:542
2357799231,12cyclictest0-21swapper/210:34:382
2357399237,9cyclictest0-21swapper/112:27:541
2357399237,4cyclictest1-21systemd10:30:011
2357399237,3cyclictest12801-21smart_sda11:35:251
2357399232,10cyclictest0-21swapper/109:00:111
23573992312,3cyclictest0-21swapper/108:17:421
2357399231,14cyclictest0-21swapper/108:28:431
2357399231,14cyclictest0-21swapper/107:23:031
2357399231,11cyclictest1709-21irqstats07:30:171
2357399231,11cyclictest1618-21ntp_states11:10:201
2357399231,10cyclictest8431-21missed_timers07:45:171
2356599239,2cyclictest6748-21kworker/0:011:35:070
2356599239,2cyclictest11640-21kworker/0:207:56:270
2356599239,2cyclictest11640-21kworker/0:207:56:270
2356599238,3cyclictest4926-21kworker/0:007:47:270
2356599238,3cyclictest27621-21kworker/0:108:32:310
2356599232,9cyclictest302-21kworker/0:107:36:470
2356599232,11cyclictest31038-21kworker/0:012:25:390
2356599232,10cyclictest8995-21kworker/0:111:30:480
2356599232,10cyclictest8995-21kworker/0:111:30:480
2356599232,10cyclictest28150-21kworker/0:111:06:430
2356599232,10cyclictest25396-21kworker/0:309:40:120
2356599232,10cyclictest10485-21kworker/0:210:21:350
2356599232,10cyclictest10485-21kworker/0:210:21:350
2356599231,10cyclictest15044-21kworker/0:210:42:030
2356599231,10cyclictest12452-21kworker/0:009:24:120
2356599231,10cyclictest12452-21kworker/0:009:15:030
2356599230,10cyclictest25851-21kworker/0:011:00:590
57772220,3sleep02356599cyclictest12:35:130
2358399227,3cyclictest0-21swapper/307:59:333
2358399227,3cyclictest0-21swapper/307:59:333
2358399225,8cyclictest0-21swapper/312:00:383
2358399223,13cyclictest0-21swapper/311:33:353
2358399223,13cyclictest0-21swapper/311:33:353
2358399222,14cyclictest0-21swapper/311:05:033
2358399221,8cyclictest0-21swapper/308:05:293
2358399221,13cyclictest0-21swapper/310:46:343
2358399220,12cyclictest0-21swapper/307:47:293
2357799222,7cyclictest17921-21idleruntime-cro11:50:012
23577992212,2cyclictest87550irq/128-enp1s0-09:45:192
2357799221,20cyclictest0-21swapper/210:57:272
2357799221,19cyclictest0-21swapper/208:08:252
2357399228,3cyclictest4541-21ls10:05:141
23573992212,2cyclictest0-21swapper/107:56:061
23573992212,2cyclictest0-21swapper/107:56:061
2357399221,10cyclictest17000-21ntp_states11:45:211
2357399221,10cyclictest0-21swapper/107:40:171
2356599229,3cyclictest19123-21kworker/0:110:39:150
2356599228,3cyclictest2684-21kworker/0:108:46:190
2356599228,2cyclictest15044-21kworker/0:210:51:070
2356599222,9cyclictest11241-21kworker/0:211:41:110
2356599222,10cyclictest15044-21kworker/0:210:31:110
2356599221,4cyclictest0-21swapper/010:25:020
2356599221,10cyclictest20811-21kworker/0:109:46:240
2356599221,10cyclictest19563-21kworker/0:112:24:350
23565992210,2cyclictest23240-21kworker/0:207:25:190
296352210,5sleep341-21ksoftirqd/312:15:133
2358399217,7cyclictest32146-21irqstats12:20:163
2358399217,7cyclictest19068-21grep10:35:243
2358399217,3cyclictest12282-21awk09:10:023
2358399216,7cyclictest4948-21/usr/sbin/munin11:20:263
2358399215,8cyclictest853-21/usr/sbin/munin07:30:163
2358399214,4cyclictest20425-21cat10:40:163
2358399211,10cyclictest0-21swapper/312:07:093
23583992110,3cyclictest24744-21grep10:50:163
2358399210,16cyclictest0-21swapper/310:23:413
2358399210,16cyclictest0-21swapper/310:23:413
2358399210,12cyclictest0-21swapper/308:45:463
2357799214,4cyclictest22845-21cat08:20:012
2357799211,9cyclictest0-21swapper/210:03:152
2357799211,6cyclictest0-21swapper/209:26:222
23577992112,6cyclictest31050-21head08:35:262
2357799211,19cyclictest0-21swapper/210:08:302
2357799211,19cyclictest0-21swapper/208:30:082
2357799210,19cyclictest0-21swapper/207:59:492
2357799210,19cyclictest0-21swapper/207:59:492
23573992110,3cyclictest0-21swapper/107:10:091
2356599218,3cyclictest1858-21kworker/0:010:10:550
2356599214,3cyclictest5680-21perf08:55:000
2356599212,12cyclictest22507-21kworker/0:008:26:070
2356599212,11cyclictest19563-21kworker/0:112:03:190
2356599212,10cyclictest1140-21kworker/0:212:30:470
2356599211,9cyclictest20811-21kworker/0:109:29:360
2356599210,11cyclictest19563-21kworker/0:111:51:590
2358399205,7cyclictest22872-21/usr/sbin/munin08:20:163
2358399201,9cyclictest0-21swapper/307:42:173
2358399201,10cyclictest5678-21munin-run08:55:013
2358399200,14cyclictest0-21swapper/311:50:053
2358399200,11cyclictest0-21swapper/307:10:293
2357799205,3cyclictest5150-21sed07:40:022
2357799204,9cyclictest827-21runrttasks11:36:062
2357799204,3cyclictest87650irq/129-enp1s0-11:00:162
2357799203,3cyclictest87550irq/128-enp1s0-09:40:252
2357799201,7cyclictest3369-21/usr/sbin/munin12:30:202
2357799201,6cyclictest0-21swapper/207:40:232
2357799201,5cyclictest1487-21/usr/sbin/munin08:45:102
23577992015,3cyclictest6543-21grep08:55:152
2357799201,4cyclictest6090-21pmu-power11:20:232
2357799201,3cyclictest0-21swapper/212:10:342
2357799201,3cyclictest0-21swapper/211:12:242
2357799201,10cyclictest0-21swapper/211:19:212
2356599207,3cyclictest19563-21kworker/0:112:15:150
2356599202,4cyclictest17930-21kworker/0:108:10:250
2356599201,11cyclictest9685-21/usr/sbin/munin07:50:130
2358399195,3cyclictest20673-21sed09:25:243
2358399194,9cyclictest0-21swapper/311:59:453
23583991915,3cyclictest0-21swapper/308:10:233
2358399191,11cyclictest0-21swapper/310:18:293
2357799195,3cyclictest87350irq/126-enp1s0-10:10:182
2357799194,3cyclictest87650irq/129-enp1s0-08:20:172
2357799193,4cyclictest27030-21sshd08:30:012
2357799191,7cyclictest28622-21sh07:20:012
2357799191,6cyclictest1906-21munin-run10:00:002
2357799191,6cyclictest17892-21needreboot09:20:172
2357799191,6cyclictest0-21swapper/212:25:222
2357799191,6cyclictest0-21swapper/210:22:252
2357799191,6cyclictest0-21swapper/210:22:252
2357799191,6cyclictest0-21swapper/207:10:142
2357799191,5cyclictest87350irq/126-enp1s0-12:15:262
2357799191,4cyclictest944-21sort07:30:122
2357799191,4cyclictest24518-21munin-run12:05:012
2357799191,4cyclictest13741-21/usr/sbin/munin11:40:252
2357799191,14cyclictest20517-21sort08:15:012
23573991915,2cyclictest827-21runrttasks08:31:211
2356599196,2cyclictest87350irq/126-enp1s0-09:10:010
2356599194,3cyclictest24769-21kworker/0:208:24:350
2356599192,4cyclictest0-21swapper/012:05:130
2358399181,9cyclictest9685-21/usr/sbin/munin07:50:113
2358399181,3cyclictest20676-21/usr/sbin/munin08:15:213
23583991812,3cyclictest827-21runrttasks07:29:453
2357799184,3cyclictest0-21swapper/209:15:202
2357799184,2cyclictest87550irq/128-enp1s0-11:55:172
2357799182,6cyclictest15148-21ntp_states08:00:212
2357799182,4cyclictest9399-21hddtemp_smartct10:15:152
2357799181,8cyclictest7919-1kworker/u9:011:30:112
2357799181,8cyclictest7919-1kworker/u9:011:30:112
2357799181,6cyclictest0-21swapper/209:52:212
2357799181,5cyclictest21907-21/usr/sbin/munin10:45:172
2357799181,5cyclictest0-21swapper/207:53:442
23577991811,5cyclictest32-21ksoftirqd/207:25:122
2357799181,15cyclictest0-21swapper/208:52:542
2358399176,6cyclictest41-21ksoftirqd/307:39:423
2357799173,3cyclictest87650irq/129-enp1s0-11:25:182
2357799172,3cyclictest31978-21smart_sda11:05:242
2357799171,5cyclictest29049-21expr07:20:132
2357799171,5cyclictest0-21swapper/210:53:052
2357799171,3cyclictest10312-21/usr/sbin/munin09:05:262
2356599171,10cyclictest0-21swapper/011:55:250
2358399162,9cyclictest0-21swapper/308:00:103
2357799161,5cyclictest0-21swapper/208:41:012
2356599161,12cyclictest23775-21/usr/sbin/munin07:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional