You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-07 - 16:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Sep 07, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
284442242166,24sleep30-21swapper/307:09:373
281442236160,25sleep00-21swapper/007:05:430
281512234163,57sleep20-21swapper/207:05:502
282872233163,57sleep10-21swapper/107:07:341
19232620,5sleep081ktimersoftd/009:45:240
72642590,2sleep00-21swapper/008:45:160
201372530,2sleep20-21swapper/210:30:012
69232490,3sleep011-21rcuc/007:30:200
135242460,1sleep30-21swapper/307:45:213
2854999329,11cyclictest5670-21awk07:30:010
2854999313,13cyclictest12873-21idleruntime-cro09:00:000
2854999302,13cyclictest16311-21idleruntime-cro11:35:010
2856999291,21cyclictest0-21swapper/310:43:133
2856499293,16cyclictest0-21swapper/209:45:232
2856499285,10cyclictest12050irq/123-ahci[0008:05:172
2856499283,14cyclictest3722-21crond12:20:012
28564992810,8cyclictest24493-21timerandwakeup11:50:242
2854999281,13cyclictest11-21rcuc/009:30:010
2856499275,13cyclictest7597-21sed12:25:252
2856499274,12cyclictest8810-21cut12:30:142
2856499273,10cyclictest5544-21/usr/sbin/munin11:10:252
2856499271,15cyclictest0-21swapper/211:55:232
2856499267,11cyclictest21480-21tune2fs11:45:172
2856499265,13cyclictest3693-21diskstats11:05:162
2856499263,14cyclictest24088-21/usr/sbin/munin09:25:182
28564992611,6cyclictest17735-21expr09:10:132
2855999261,16cyclictest19403-21sh11:40:171
2855999261,16cyclictest19403-21sh11:40:171
2854999263,11cyclictest22910-21perf11:50:010
2856499258,7cyclictest22293-21basename10:35:012
2856499258,7cyclictest22293-21basename10:35:012
2856499256,5cyclictest865-21latency_hist11:00:022
2856499256,5cyclictest15909-21/usr/sbin/munin10:20:162
2856499255,13cyclictest1120-21/usr/sbin/munin11:00:242
2856499254,18cyclictest25436-21sh08:15:012
2856499254,18cyclictest25436-21sh08:15:012
2856499253,8cyclictest26285-21/usr/sbin/munin09:30:242
2856499252,9cyclictest0-21swapper/211:30:192
2856499252,20cyclictest16218-21cat09:05:192
2856499252,15cyclictest10545-21/usr/sbin/munin12:35:212
2856499251,13cyclictest15964-21sort07:50:242
2856499250,11cyclictest18775-21cpu11:40:132
2856499250,11cyclictest18775-21cpu11:40:132
2855999254,9cyclictest0-21swapper/109:25:261
2855999251,20cyclictest79562sleep112:28:041
2855999251,16cyclictest0-21swapper/112:00:261
2855999251,11cyclictest26383-21ntp_states11:55:211
2854999251,12cyclictest3127-21munin-run11:05:010
2856499245,8cyclictest0-21swapper/211:29:422
28564992414,6cyclictest3772-40dnf08:36:182
2855999241,19cyclictest17266-21munin-run09:10:011
2855999241,11cyclictest5157-21sed07:25:251
2854999240,11cyclictest15698-21perf10:20:010
2856999231,5cyclictest2334-21grep12:15:153
2856499237,8cyclictest32002-21/usr/sbin/munin07:15:142
2856499236,5cyclictest23766-21smart_sda10:35:242
2856499235,9cyclictest11442-21perl08:55:152
2856499235,15cyclictest9176-21wc11:15:262
2856499234,6cyclictest87350irq/126-enp1s0-09:40:132
2856499232,8cyclictest27311-21tune2fs10:45:152
2856499232,11cyclictest31256-21awk12:05:262
2856499231,13cyclictest11518-21/usr/sbin/munin10:10:182
2856499230,9cyclictest17262-21diskmemload10:17:072
2856499230,13cyclictest7318-21kworker/u8:211:20:122
2855999234,11cyclictest87550irq/128-enp1s0-10:00:011
2855999232,18cyclictest27420-21fschecks_time10:45:161
2855999232,11cyclictest9343-21smart_sda07:35:231
2855999231,3cyclictest30798-21ntp_states12:05:221
2855999230,15cyclictest0-21swapper/109:13:291
2856999222,11cyclictest19692-21/usr/sbin/munin09:15:233
2856999221,16cyclictest0-21swapper/310:19:253
2856999221,13cyclictest0-21swapper/312:30:153
2856499225,8cyclictest2739-21bc09:50:132
2856499224,9cyclictest5023-21ntp_states12:20:212
2856499222,8cyclictest25668-21/usr/sbin/munin08:15:152
2856499222,7cyclictest15227-21cat09:05:022
2856499222,11cyclictest0-21swapper/209:20:272
2856499222,11cyclictest0-21swapper/209:20:262
2856499221,9cyclictest27514-21/usr/sbin/munin12:00:152
2856499221,6cyclictest87650irq/129-enp1s0-07:30:152
28564992214,5cyclictest25968-21smart_sda10:40:242
28564992213,6cyclictest29461-21diskstats10:50:202
2856499220,7cyclictest0-21swapper/209:15:192
2855999227,6cyclictest0-21swapper/110:55:261
2855999222,9cyclictest0-21swapper/111:35:011
2855999221,18cyclictest6631-21if_enp2s007:30:171
2855999221,18cyclictest14934-21/usr/sbin/munin10:15:221
2855999221,16cyclictest87550irq/128-enp1s0-07:40:171
2855999221,12cyclictest3064-21df_inode09:50:141
2855999221,10cyclictest0-21swapper/112:40:021
2855999220,5cyclictest0-21swapper/112:21:541
2855999220,11cyclictest217202fschecks_count08:05:141
2854999221,4cyclictest28185-21chrt09:34:160
2856999216,2cyclictest87350irq/126-enp1s0-10:14:293
2856999213,10cyclictest23470-21/usr/sbin/munin08:10:203
2856999213,10cyclictest23470-21/usr/sbin/munin08:10:203
2856999211,7cyclictest22061-21grep11:45:243
2856499215,13cyclictest6448-21grep09:55:242
28564992113,5cyclictest4648-21diskstats08:40:142
28564992113,5cyclictest29680-21memory07:10:192
28564992112,6cyclictest17683-21irqstats07:55:172
28564992112,5cyclictest29136-21smart_sda08:20:242
2855999218,6cyclictest28153-21kworker/u8:007:55:171
2855999211,3cyclictest21996-21ntp_states11:45:231
2855999211,3cyclictest21567-21sensors_temp10:30:241
2855999211,3cyclictest21567-21sensors_temp10:30:241
2855999211,3cyclictest10322-21munin-run12:35:011
2855999211,3cyclictest0-21swapper/111:05:351
28559992113,3cyclictest0-21swapper/111:29:421
28559992113,3cyclictest0-21swapper/111:16:041
2855999211,14cyclictest0-21swapper/109:34:501
2855999211,11cyclictest23768-21smart_sda10:35:251
2855999211,10cyclictest579-21systemd-journal08:15:011
2855999211,10cyclictest579-21systemd-journal08:15:011
2855999211,10cyclictest1759-21gltestperf11:00:161
2855999211,10cyclictest0-21swapper/111:52:531
28559992110,3cyclictest2774-21hddtemp_smartct08:35:161
2854999213,10cyclictest22504-21/usr/sbin/munin10:35:150
2854999212,11cyclictest8079-21/usr/sbin/munin07:35:220
2854999211,6cyclictest27638-21munin-run08:20:020
2854999211,12cyclictest0-21swapper/012:15:190
2856999209,3cyclictest30070-21/usr/sbin/munin08:25:193
2856999208,3cyclictest14081-21head09:00:193
2856999206,4cyclictest0-21swapper/311:50:323
2856999206,3cyclictest0-21swapper/311:05:153
2856999205,2cyclictest87350irq/126-enp1s0-09:53:563
2856999201,9cyclictest0-21swapper/308:40:133
28569992017,2cyclictest0-21swapper/311:40:233
28569992017,2cyclictest0-21swapper/311:40:223
2856999201,11cyclictest7117-21/usr/sbin/munin10:00:243
2856999201,11cyclictest28482-21/usr/sbin/munin09:35:203
2856999200,7cyclictest0-21swapper/312:38:043
2856499208,8cyclictest2127-21gltestperf07:20:172
2856499203,6cyclictest13080-21kworker/2:208:25:172
28564992013,4cyclictest28835-21diskstats09:35:142
28564992011,6cyclictest17331-21/usr/sbin/munin11:35:162
28564992011,5cyclictest4949-21smart_sda07:25:232
28564992011,5cyclictest10279-21/usr/sbin/munin07:40:272
2856499201,13cyclictest0-21swapper/212:14:022
2856499201,11cyclictest0-21swapper/208:51:552
2855999209,3cyclictest29462-21diskstats10:50:191
2855999206,3cyclictest0-21swapper/109:35:051
2855999201,15cyclictest0-21swapper/112:11:021
2855999201,14cyclictest0-21swapper/112:17:241
2855999201,14cyclictest0-21swapper/111:31:161
2855999201,11cyclictest8694-21/usr/sbin/munin08:50:191
2855999201,11cyclictest31533-21ntp_states07:15:051
2855999201,11cyclictest0-21swapper/110:13:071
2855999200,13cyclictest0-21swapper/110:03:551
2854999205,5cyclictest626-21sshd08:30:170
2854999202,10cyclictest12473-21/usr/sbin/munin07:45:210
2854999201,13cyclictest0-21swapper/012:35:140
2854999201,11cyclictest9322-21/usr/sbin/munin10:05:230
2854999201,11cyclictest7739-21/usr/sbin/munin11:15:170
2854999201,11cyclictest3347-21/usr/sbin/munin11:05:180
2854999201,10cyclictest31753-21ntp_states09:40:200
2856999199,3cyclictest31655-21/usr/sbin/munin07:15:223
2856999198,3cyclictest0-21swapper/311:01:283
2856999198,2cyclictest2490-21idleruntime-cro09:50:013
2856999198,2cyclictest0-21swapper/312:20:483
2856999196,5cyclictest0-21swapper/310:09:093
2856999192,7cyclictest6154-21/usr/sbin/munin12:25:173
2856999192,7cyclictest0-21swapper/310:55:183
2856999191,9cyclictest22286-21munin-run10:35:013
2856999191,9cyclictest22286-21munin-run10:35:013
2856999191,8cyclictest0-21swapper/312:13:213
2856999191,5cyclictest0-21swapper/309:10:443
28569991914,3cyclictest28161-21smartctl10:45:243
2856999191,11cyclictest0-21swapper/310:52:503
2856999191,11cyclictest0-21swapper/309:58:103
2856999191,10cyclictest0-21swapper/311:15:253
2856999190,6cyclictest0-21swapper/312:02:243
2856499199,7cyclictest32266-21/usr/sbin/munin08:30:222
28564991912,4cyclictest8746-21sed07:35:152
28564991912,4cyclictest7481-21forks10:00:152
28564991912,4cyclictest19981-21memory08:00:172
28564991910,6cyclictest9322-21/usr/sbin/munin10:05:162
28564991910,6cyclictest7521-21/usr/sbin/munin08:45:192
28564991910,6cyclictest12809-21/usr/sbin/munin07:45:122
2855999196,12cyclictest28153-21kworker/u8:009:42:501
2855999194,6cyclictest21173-21unixbench-2d09:15:251
28559991915,3cyclictest0-21swapper/110:42:121
28559991915,3cyclictest0-21swapper/110:22:421
2855999191,4cyclictest22962-21ntp_states09:20:221
2855999191,4cyclictest22962-21ntp_states09:20:221
2855999191,2cyclictest0-21swapper/107:47:481
2855999191,17cyclictest0-21swapper/108:47:321
2855999191,17cyclictest0-21swapper/108:25:011
2855999191,16cyclictest0-21swapper/108:17:541
2855999191,10cyclictest5332-21awk08:40:221
2855999191,10cyclictest3455-21latency_hist07:25:021
2855999191,10cyclictest29835-21ntp_states07:10:221
2855999191,10cyclictest18436-21cat10:25:141
2855999190,18cyclictest0-21swapper/108:03:071
2854999199,2cyclictest31542-21sshd08:25:250
2854999197,4cyclictest7603-21cat12:25:250
2854999195,3cyclictest11-21rcuc/008:35:150
2854999192,4cyclictest11-21rcuc/010:57:460
2854999191,8cyclictest0-21swapper/011:40:120
2854999191,8cyclictest0-21swapper/011:40:120
2854999191,5cyclictest0-21swapper/011:24:400
2854999191,4cyclictest27514-21/usr/sbin/munin12:00:120
281832190,3sleep081ktimersoftd/010:45:240
2856999187,3cyclictest19103-21strings10:25:193
2856999186,4cyclictest0-21swapper/311:20:053
2856999186,3cyclictest0-21swapper/311:35:343
2856999185,4cyclictest0-21swapper/309:21:473
2856999185,4cyclictest0-21swapper/309:21:463
2856999183,5cyclictest579-21systemd-journal09:10:003
2856999182,6cyclictest1052-21/usr/sbin/munin08:30:223
2856999181,9cyclictest30076-21diskstats12:05:153
2856999181,9cyclictest0-21swapper/311:25:103
2856999181,7cyclictest6879-21cut07:30:203
2856999181,7cyclictest26660-21grep08:15:193
2856999181,4cyclictest16293-21sh11:35:023
28569991813,3cyclictest20630-21grep08:00:233
2856999181,2cyclictest87350irq/126-enp1s0-10:20:513
28569991811,4cyclictest0-21swapper/309:41:373
2856999181,11cyclictest0-21swapper/309:32:043
2856999181,10cyclictest8694-21/usr/sbin/munin08:50:233
2856999181,10cyclictest0-21swapper/311:10:473
2855999181,6cyclictest0-21swapper/110:05:561
28559991814,3cyclictest0-21swapper/109:46:021
28559991814,3cyclictest0-21swapper/108:31:511
2855999181,3cyclictest15014-21df_inode07:50:141
2855999181,16cyclictest0-21swapper/111:20:321
2855999181,16cyclictest0-21swapper/109:00:561
2855999181,16cyclictest0-21swapper/108:55:461
2855999180,9cyclictest31554-21unixbench-2d08:25:261
2854999181,6cyclictest6904-21awk10:00:020
2854999181,6cyclictest0-21swapper/010:53:100
2854999181,5cyclictest0-21swapper/012:21:070
2854999181,5cyclictest0-21swapper/012:05:440
2854999181,5cyclictest0-21swapper/010:41:050
2854999181,4cyclictest25319-21/usr/sbin/munin11:55:170
28549991812,3cyclictest11-21rcuc/010:25:230
2854999181,15cyclictest23259-21perf08:10:020
2854999181,15cyclictest0-21swapper/009:05:140
2854999181,14cyclictest29107-21find08:20:240
28549991811,4cyclictest17438-21unixbench_multi10:20:260
2854999181,12cyclictest0-21swapper/012:30:220
2854999180,10cyclictest0-21swapper/009:10:170
222502181,2sleep00-21swapper/010:34:490
222502181,2sleep00-21swapper/010:34:490
12652181,2sleep00-21swapper/012:11:580
303622170,2sleep00-21swapper/009:38:550
294352170,1sleep30-21swapper/307:10:173
2856999176,2cyclictest8005-21grep08:45:243
2856999173,3cyclictest8079-21/usr/sbin/munin07:35:223
2856999172,9cyclictest0-21swapper/309:26:043
2856999171,7cyclictest25461-21expr11:55:133
28569991712,3cyclictest28700-21kernelversion08:20:183
2856999171,10cyclictest0-21swapper/310:35:133
2855999171,8cyclictest6998-21sed11:10:261
2854999172,7cyclictest20245-21perl09:15:140
2854999171,7cyclictest0-21swapper/011:53:300
2854999171,7cyclictest0-21swapper/009:53:130
2854999171,7cyclictest0-21swapper/008:40:260
2854999171,7cyclictest0-21swapper/007:55:220
2854999171,6cyclictest0-21swapper/010:03:390
2854999171,6cyclictest0-21swapper/007:41:220
2854999171,3cyclictest0-21swapper/011:25:130
28549991712,3cyclictest2558-21ntp_states07:20:220
2854999171,15cyclictest0-21swapper/011:14:470
2854999171,15cyclictest0-21swapper/008:50:440
305102160,2sleep00-21swapper/007:12:200
2856999161,6cyclictest21610-21df08:05:123
28569991611,3cyclictest14673-21/usr/sbin/munin07:50:163
2856999161,12cyclictest1469-21/usr/sbin/munin07:20:213
2856999161,12cyclictest10279-21/usr/sbin/munin07:40:153
2856999160,9cyclictest0-21swapper/307:55:123
2854999162,5cyclictest15075-21munin-run09:05:010
2854999161,6cyclictest16522-21/usr/sbin/munin11:35:240
2854999161,3cyclictest19979-21memory08:00:160
28549991612,3cyclictest0-21swapper/008:10:410
28549991612,3cyclictest0-21swapper/008:10:410
28549991611,3cyclictest31655-21/usr/sbin/munin07:15:210
28549991611,3cyclictest14673-21/usr/sbin/munin07:50:260
2854999161,13cyclictest12193-21hddtemp_smartct10:10:180
2854999161,12cyclictest21890-21/usr/sbin/munin09:20:230
2854999161,12cyclictest21890-21/usr/sbin/munin09:20:230
2856999151,7cyclictest0-21swapper/308:35:173
2856999151,4cyclictest0-21swapper/307:25:163
28569991510,2cyclictest827-21runrttasks08:55:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional