You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-10 - 02:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 10, 2026 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30532241165,24sleep30-21swapper/319:08:513
30472234162,24sleep20-21swapper/219:08:462
30622232162,57sleep10-21swapper/119:08:581
31082221190,21sleep00-21swapper/019:09:330
1792321780,5sleep3324399cyclictest00:35:143
626021270,5sleep2323899cyclictest21:40:192
3238999996,2cyclictest0-21swapper/223:39:422
3238997471,2cyclictest0-21swapper/222:10:542
3238996853,12cyclictest17080-21perf19:40:012
144892630,2sleep2323899cyclictest22:00:172
3238995753,3cyclictest0-21swapper/221:09:382
249522530,1sleep10-21swapper/122:25:011
3238994643,2cyclictest0-21swapper/200:25:542
25442450,2sleep20-21swapper/200:00:132
319862440,3sleep230-21rcuc/220:10:262
57002410,1sleep30-21swapper/320:25:193
57002410,1sleep30-21swapper/320:25:193
324399387,3cyclictest87350irq/126-enp1s0-22:05:203
3238993813,22cyclictest12907-21users20:40:252
324399373,3cyclictest12050irq/123-ahci[0019:45:133
3243993712,3cyclictest26090-21/usr/sbin/munin20:00:253
324399361,12cyclictest25361-21sshd23:38:163
3238993611,22cyclictest30798-21hddtemp_smartct23:50:142
322599369,13cyclictest9522-21awk21:50:020
322599356,14cyclictest10138-21munin-run23:05:010
324399348,4cyclictest14449-21smart_sda00:25:243
324399342,29cyclictest26716-21/usr/sbin/munin21:15:193
324399341,4cyclictest0-21swapper/323:46:383
3243993411,3cyclictest16758-21timerandwakeup00:30:293
323899346,5cyclictest27637-21chrt23:44:162
3238993412,19cyclictest7538-21/usr/sbin/munin21:45:152
3238993412,19cyclictest3135-21/usr/sbin/munin21:35:212
3238993412,19cyclictest3135-21/usr/sbin/munin21:35:212
3238993410,21cyclictest5190-21cat00:05:172
156002340,1sleep10-21swapper/120:50:011
324399333,9cyclictest19801-21grep23:25:153
324399333,9cyclictest0-21swapper/322:03:263
324399332,3cyclictest29934-21sshd20:05:513
3243993311,4cyclictest17397-21cut19:40:113
324399331,13cyclictest4431-21sshd20:24:153
324399331,13cyclictest21799-21tr19:50:123
3243993311,3cyclictest1534-21/usr/sbin/munin22:45:233
3243993311,3cyclictest1534-21/usr/sbin/munin22:45:233
323899331,6cyclictest12657-21/usr/sbin/munin21:55:172
323899331,6cyclictest0-21swapper/223:55:182
323899331,6cyclictest0-21swapper/223:55:172
323899331,28cyclictest21992-21tail23:30:152
322599335,13cyclictest28687-21munin-run21:20:000
322599334,14cyclictest2926-21awk21:35:010
322599334,13cyclictest4505-21munin-run20:25:000
324399329,2cyclictest827-21runrttasks00:10:063
324399327,6cyclictest23467-21pmu-power21:05:223
324399326,8cyclictest12330-21ls23:10:013
324399321,6cyclictest3431-21/usr/sbin/munin19:10:153
324399321,6cyclictest3431-21/usr/sbin/munin19:10:143
3243993211,3cyclictest26653-21cat21:15:023
323899329,4cyclictest29402-21/usr/sbin/munin20:05:222
323899328,4cyclictest0-21swapper/200:10:252
323899321,15cyclictest0-21swapper/220:20:522
3238993210,3cyclictest2198-21sensors_temp21:30:222
3238993210,3cyclictest16945-21/usr/sbin/munin23:20:232
3238993210,19cyclictest29943-21cat23:50:022
322599322,14cyclictest4172-21munin-run00:05:010
324399319,3cyclictest30577-21sshd21:21:173
324399319,2cyclictest19147-21find20:55:253
324399318,3cyclictest827-21runrttasks22:43:123
324399316,5cyclictest0-21swapper/321:30:123
324399316,3cyclictest87350irq/126-enp1s0-00:00:143
324399311,9cyclictest0-21swapper/320:35:163
324399311,4cyclictest31102-21/usr/sbin/munin21:25:133
324399311,4cyclictest0-21swapper/300:15:193
324399311,3cyclictest9244-21open_files23:00:213
3243993112,16cyclictest23036-21tr22:20:113
3243993111,3cyclictest4967-21processes22:50:243
3243993111,3cyclictest32571-21cut20:15:013
3243993111,3cyclictest1965-21wc00:00:023
3243993111,3cyclictest1965-21wc00:00:023
324399311,12cyclictest27335-21/usr/sbin/munin22:30:293
3243993110,3cyclictest5393-21cpu21:40:133
3243993110,3cyclictest3487-21sh21:35:123
3243993110,3cyclictest3487-21sh21:35:123
323899319,4cyclictest17318-21/usr/sbin/munin22:05:192
323899319,4cyclictest14531-21/usr/sbin/munin23:15:012
323899319,2cyclictest15712-21/usr/sbin/munin20:50:172
323899318,3cyclictest18556-21cut20:55:172
323899311,6cyclictest31161-21cpu21:25:112
323899311,3cyclictest27529-21irqstats21:15:202
3238993112,16cyclictest2080-21grep22:45:142
3238993112,16cyclictest2080-21grep22:45:142
3238993111,17cyclictest25807-21sh22:25:142
322599312,13cyclictest10503-21idleruntime-cro19:25:020
324399308,3cyclictest25312-21cstates22:25:113
324399302,5cyclictest22093-21munin-run21:05:023
324399301,9cyclictest13172-21sendmail_mailqu21:55:243
3243993013,14cyclictest11334-21df_inode00:20:143
324399301,14cyclictest0-21swapper/321:51:323
3243993010,4cyclictest22888-21unixbench_singl23:30:243
3243993010,3cyclictest9163-21cut19:20:153
3243993010,3cyclictest14997-21grep20:45:243
3243993010,3cyclictest13324-21date20:45:013
323899309,3cyclictest29558-21grep21:20:162
323899308,3cyclictest21268-21proc_pri21:00:232
323899308,3cyclictest17579-21/usr/sbin/munin00:35:172
323899306,5cyclictest5773-21tune2fs19:15:022
323899306,5cyclictest5773-21tune2fs19:15:022
323899306,4cyclictest14855-21timerandwakeup20:45:242
323899306,4cyclictest0-21swapper/219:55:132
323899304,3cyclictest87550irq/128-enp1s0-22:22:162
323899302,5cyclictest7216-21smart_sda22:55:232
323899301,6cyclictest0-21swapper/200:30:282
323899301,5cyclictest0-21swapper/223:17:392
3238993011,16cyclictest9270-21if_enp2s019:20:162
3238993011,16cyclictest10991-21/usr/sbin/munin00:20:162
324399299,3cyclictest21367-21gltestperf22:15:123
324399299,3cyclictest18306-21timerandwakeup23:20:233
324399299,2cyclictest20648-21cat22:15:013
324399298,3cyclictest827-21runrttasks23:19:563
324399298,3cyclictest15401-21expr19:35:113
324399298,2cyclictest8919-21cat20:35:013
324399294,3cyclictest87350irq/126-enp1s0-23:44:153
3243992910,3cyclictest8320-21cat21:45:173
323899299,3cyclictest20846-21tr22:15:092
323899298,3cyclictest1476-21proc_pri20:15:242
323899297,3cyclictest827-21runrttasks23:07:122
323899296,4cyclictest12219-21grep19:25:242
323899296,3cyclictest4233-21interrupts19:10:182
323899296,3cyclictest4233-21interrupts19:10:182
323899291,5cyclictest21563-21cat19:50:012
323899291,5cyclictest0-21swapper/222:44:102
3238992911,15cyclictest9263-21fschecks_time00:15:132
3238992910,3cyclictest827-21runrttasks23:29:412
3238992910,3cyclictest5846-21cut22:55:012
3238992910,16cyclictest29365-21cut22:35:012
323399291,21cyclictest0-21swapper/123:21:401
324399289,3cyclictest6849-21cat22:55:173
324399289,3cyclictest30910-21cat22:35:263
324399287,4cyclictest6370-21turbostat00:10:023
324399287,3cyclictest14345-21seq23:12:513
324399282,4cyclictest0-21swapper/319:15:033
324399282,4cyclictest0-21swapper/319:15:033
324399281,3cyclictest11385-21hddtemp_smartct19:25:153
323899288,3cyclictest14077-21proc_pri19:30:222
323899288,2cyclictest827-21runrttasks20:28:292
323899288,2cyclictest827-21runrttasks20:28:292
323899283,5cyclictest24864-21diskstats21:10:152
323899283,3cyclictest87650irq/129-enp1s0-23:02:262
3238992810,15cyclictest30986-21grep22:35:262
324399279,2cyclictest827-21runrttasks23:53:163
324399275,4cyclictest25424-21unixbench_multi19:55:243
323899279,3cyclictest9950-21kernelversion20:35:192
323899279,2cyclictest827-21runrttasks20:30:352
323899277,3cyclictest21801-21expr19:50:122
323899276,3cyclictest26354-21expr20:00:132
323899271,6cyclictest0-21swapper/221:51:072
323899271,2cyclictest87550irq/128-enp1s0-19:43:232
322599279,15cyclictest23668-21munin-run19:55:010
323399262,15cyclictest5688-21diskstats21:40:131
323399260,19cyclictest0-21swapper/121:14:151
322599262,12cyclictest0-21swapper/023:10:020
323399251,4cyclictest0-21swapper/100:35:121
322599251,13cyclictest31788-21switchtime20:10:240
322599251,12cyclictest21338-21/usr/sbin/munin23:30:230
322599251,12cyclictest11096-21munin-run20:40:010
99292240,4sleep011-21rcuc/021:50:130
3243992410,2cyclictest401ktimersoftd/320:50:153
3233992415,5cyclictest0-21swapper/100:21:111
323399241,17cyclictest0-21swapper/122:36:071
323399241,13cyclictest0-21swapper/123:30:461
323399241,13cyclictest0-21swapper/121:39:201
323399241,13cyclictest0-21swapper/121:39:201
322599248,14cyclictest25880-21perf20:00:010
322599246,15cyclictest24914-21awk22:25:010
322599245,16cyclictest1307-21munin-run22:45:010
324399233,2cyclictest88950irq/132-enp2s0-19:30:393
323399237,8cyclictest0-21swapper/122:57:251
323399232,15cyclictest87650irq/129-enp1s0-22:27:501
323399231,4cyclictest0-21swapper/123:25:241
3233992312,5cyclictest0-21swapper/123:15:321
323399231,16cyclictest0-21swapper/122:49:561
323399231,16cyclictest0-21swapper/122:49:561
323399231,16cyclictest0-21swapper/121:21:411
322599239,4cyclictest0-21swapper/022:05:440
322599236,5cyclictest27237-21head20:00:220
322599230,8cyclictest24688-21kworker/u8:222:45:140
322599230,8cyclictest24688-21kworker/u8:222:45:130
324399221,1cyclictest0-21swapper/320:15:163
323399228,2cyclictest24282-21sh21:10:011
323399227,8cyclictest0-21swapper/123:37:521
323399227,7cyclictest0-21swapper/100:18:581
323399225,8cyclictest0-21swapper/122:18:431
323399225,8cyclictest0-21swapper/121:58:101
323399224,8cyclictest0-21swapper/123:52:041
323399224,8cyclictest0-21swapper/123:10:261
323399223,6cyclictest0-21swapper/123:44:021
323399223,12cyclictest26737-21egrep21:15:141
323399221,19cyclictest0-21swapper/119:45:561
323399221,14cyclictest0-21swapper/121:50:261
323399220,4cyclictest0-21swapper/121:31:221
322599226,6cyclictest0-21swapper/000:12:410
322599221,7cyclictest0-21swapper/023:35:200
322599221,6cyclictest0-21swapper/023:20:080
322599221,19cyclictest0-21swapper/023:16:320
322599221,12cyclictest13295-21if_enp1s023:10:160
322599221,11cyclictest20729-21/usr/sbin/munin22:15:120
322599221,11cyclictest15869-21kworker/u8:222:10:200
322599221,10cyclictest28724-21sshd23:45:150
323399217,4cyclictest0-21swapper/121:26:411
323399215,8cyclictest0-21swapper/122:51:071
323399212,10cyclictest8193-21users00:10:281
323399211,8cyclictest0-21swapper/123:47:331
323399211,5cyclictest0-21swapper/100:00:381
323399211,3cyclictest0-21swapper/100:30:501
3233992111,3cyclictest0-21swapper/122:00:341
323399211,13cyclictest0-21swapper/100:05:231
323399211,11cyclictest0-21swapper/122:08:241
323399210,6cyclictest0-21swapper/123:57:051
323399210,6cyclictest0-21swapper/123:57:051
322599219,2cyclictest0-21swapper/021:35:230
322599219,2cyclictest0-21swapper/021:35:230
322599216,5cyclictest0-21swapper/023:55:530
322599216,5cyclictest0-21swapper/023:55:530
322599216,5cyclictest0-21swapper/023:40:060
322599211,7cyclictest0-21swapper/000:15:350
3225992116,3cyclictest31486-21grep21:25:140
322599211,3cyclictest0-21swapper/023:53:050
322599211,18cyclictest0-21swapper/000:07:050
322599211,11cyclictest8135-21latency_hist23:00:010
322599211,10cyclictest0-21swapper/019:30:190
322599210,8cyclictest0-21swapper/022:39:160
322599210,6cyclictest0-21swapper/022:00:350
323399208,4cyclictest0-21swapper/122:43:491
323399206,3cyclictest19892-21awk21:00:011
323399204,3cyclictest0-21swapper/122:34:331
3233992016,3cyclictest0-21swapper/120:50:221
323399201,3cyclictest0-21swapper/123:05:511
3233992012,3cyclictest0-21swapper/120:33:531
323399201,13cyclictest0-21swapper/122:13:281
3233992011,3cyclictest0-21swapper/121:46:081
323399201,13cyclictest0-21swapper/100:26:051
322599208,4cyclictest0-21swapper/021:42:290
322599206,6cyclictest0-21swapper/000:27:050
322599205,3cyclictest0-21swapper/021:11:450
322599204,6cyclictest0-21swapper/000:37:110
3225992016,3cyclictest0-21swapper/019:29:110
322599201,11cyclictest0-21swapper/022:50:050
322599201,11cyclictest0-21swapper/022:30:020
322599201,11cyclictest0-21swapper/022:29:080
322599201,11cyclictest0-21swapper/021:20:240
322599201,11cyclictest0-21swapper/000:30:150
322599201,10cyclictest11934-21/usr/sbin/munin21:55:210
323399195,7cyclictest8516-21/usr/sbin/munin19:20:171
323399194,8cyclictest30493-21/usr/sbin/munin20:10:141
323399194,4cyclictest9035-21wc23:00:191
323399194,4cyclictest0-21swapper/119:40:021
323399191,9cyclictest808-21snmpd20:05:241
3233991916,2cyclictest0-21swapper/119:10:421
3233991916,2cyclictest0-21swapper/119:10:411
3233991914,4cyclictest0-21swapper/119:26:381
323399191,2cyclictest0-21swapper/120:29:471
323399191,2cyclictest0-21swapper/120:29:471
322599191,9cyclictest0-21swapper/019:10:260
322599191,9cyclictest0-21swapper/019:10:260
322599191,7cyclictest0-21swapper/020:48:300
3225991912,4cyclictest11324-21df00:20:130
322599191,10cyclictest0-21swapper/023:26:110
323399189,2cyclictest0-21swapper/119:15:231
323399189,2cyclictest0-21swapper/119:15:231
323399185,4cyclictest20833-21idleruntime21:00:171
323399184,4cyclictest21839-21grep19:50:131
323399183,7cyclictest26090-21/usr/sbin/munin20:00:241
323399182,5cyclictest18893-21users19:40:261
323399181,9cyclictest773-21NetworkManager20:43:521
3233991815,2cyclictest0-21swapper/120:39:151
323399181,4cyclictest471-21expr20:15:131
323399181,3cyclictest0-21swapper/119:57:551
323399180,16cyclictest0-21swapper/119:31:351
322599187,3cyclictest6323-21/usr/sbin/munin19:15:190
322599187,3cyclictest6323-21/usr/sbin/munin19:15:180
322599186,2cyclictest12050irq/123-ahci[0021:00:010
322599181,7cyclictest808-21snmpd21:00:330
3225991811,4cyclictest24288-21ls21:10:010
3225991811,4cyclictest16082-21needreboot19:35:180
195002180,1sleep00-21swapper/019:45:120
323399171,2cyclictest0-21swapper/120:23:421
322599178,2cyclictest0-21swapper/020:50:140
322599175,3cyclictest12050irq/123-ahci[0019:40:240
322599171,5cyclictest808-21snmpd20:05:230
322599171,5cyclictest1226-21memory20:15:190
3225991711,3cyclictest12669-21expr20:40:220
3225991611,3cyclictest8893-21latency_hist20:35:010
3225991610,3cyclictest4727-21/usr/sbin/munin20:25:140
3225991610,3cyclictest4727-21/usr/sbin/munin20:25:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional