You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 02:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 06, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
312762239167,58sleep10-21swapper/119:06:291
312772238161,26sleep20-21swapper/219:06:302
312732203167,24sleep30-21swapper/319:06:273
295822203165,25sleep00-21swapper/019:05:030
31651996555,9cyclictest0-21swapper/223:01:332
213252610,3sleep13164699cyclictest22:25:161
276902600,4sleep03163999cyclictest21:25:180
3165899441,26cyclictest2496-21latency_hist00:10:013
3165899421,22cyclictest7158-21sh20:40:013
3165899411,25cyclictest26590-21df23:50:133
3165899401,27cyclictest23104-21awk23:40:243
3165899392,26cyclictest16187-21/usr/sbin/munin21:00:263
3165899381,33cyclictest19950-21/usr/sbin/munin19:55:183
3165899381,27cyclictest29798-21munin-run22:45:013
3165899377,19cyclictest2729-21/usr/sbin/munin00:10:223
3165899372,31cyclictest9930-21smart_sda23:10:243
3165899372,23cyclictest9101-21munin-run00:25:013
3165899371,26cyclictest3643-21cpu21:45:123
3165899371,22cyclictest24357-21/usr/sbin/munin20:05:133
3165899367,19cyclictest86550irq/126-enp1s0-22:00:163
3165899363,22cyclictest0-21swapper/321:22:413
3165899361,27cyclictest16852-21egrep22:15:153
3165899361,26cyclictest15264-21/usr/sbin/munin23:25:173
3165899361,25cyclictest8119-21sort21:55:123
3165899355,20cyclictest22986-21cat21:15:143
3165899355,20cyclictest22986-21cat21:15:143
3165899351,26cyclictest14405-21/usr/sbin/munin22:10:013
3165899351,24cyclictest32229-21/usr/sbin/munin22:50:233
3165899351,21cyclictest12744-21gunzip00:30:263
3165899351,20cyclictest13833-21cpuspeed_turbos00:35:143
3165899351,19cyclictest13124-21latency_hist19:40:023
3163999352,9cyclictest8425-21kworker/0:120:49:380
3165899341,29cyclictest19358-40dnf19:50:323
3165899341,28cyclictest25128-21ntp_states23:45:233
3165899341,22cyclictest28335-1kworker/3:1H20:14:563
3165899341,22cyclictest28335-1kworker/3:1H20:14:563
3165899341,20cyclictest4928-21/usr/sbin/munin00:15:153
3165899341,20cyclictest20904-21expr21:10:133
3165899341,19cyclictest15030-21latency_hist23:25:023
3165899341,19cyclictest15030-21latency_hist23:25:023
3165899341,18cyclictest20370-21diskmemload21:41:003
3163999347,12cyclictest6258-21idleruntime-cro23:05:000
3163999346,13cyclictest21003-21munin-run22:25:010
3163999346,13cyclictest21003-21munin-run22:25:010
3163999346,13cyclictest19436-21munin-run23:35:010
3163999341,12cyclictest20370-21diskmemload23:40:210
3165899332,27cyclictest29109-21diskstats20:15:153
3165899332,27cyclictest20728-21ntp_states23:35:223
3165899331,24cyclictest7639-21ps23:05:233
3165899331,22cyclictest86850irq/129-enp1s0-19:40:253
3165899331,20cyclictest3105-21ntp_kernel_pll_22:55:223
3165899331,19cyclictest2768-21latency_hist20:30:023
3165899331,18cyclictest29505-21expr21:30:133
31658993311,19cyclictest18459-21netstat23:30:203
3165899330,23cyclictest0-21swapper/319:20:173
3165899330,21cyclictest0-21swapper/319:45:213
3163999338,12cyclictest18804-21munin-run22:20:010
3165899322,27cyclictest9311-21diskstats19:30:153
3165899322,18cyclictest20370-21diskmemload22:35:223
31658993214,15cyclictest23165-21netstat20:00:213
3165899321,22cyclictest5543-21switchtime23:00:243
3165899321,22cyclictest27194-21/usr/sbin/munin21:25:173
31658993212,17cyclictest14637-21acpi22:10:133
3165899321,18cyclictest7865-21latency_hist21:55:023
3165899320,23cyclictest12050irq/123-ahci[0023:20:013
3163999321,14cyclictest25363-21kworker/0:222:48:230
3165899312,25cyclictest387-21smart_sda21:35:243
3165899312,25cyclictest10848-21smart_sda20:45:223
3165899311,27cyclictest30960-21/usr/sbin/munin20:20:143
3165899311,27cyclictest28458-21/usr/sbin/munin23:55:193
3165899311,22cyclictest31841-21/usr/sbin/munin19:10:163
3165899311,16cyclictest0-21swapper/319:25:153
3163999312,14cyclictest21924-21munin-run20:00:010
3163999312,12cyclictest12192-21kworker/0:322:11:400
31639993119,3cyclictest1130-21ls22:50:250
3163999311,23cyclictest10622-21kworker/0:119:45:180
3163999310,11cyclictest11462-21kworker/0:321:18:440
3163999310,11cyclictest11462-21kworker/0:321:18:440
3165899302,24cyclictest30163-21tr22:45:123
3165899301,26cyclictest3003-21/usr/sbin/munin20:30:193
3165899301,26cyclictest2370-21/usr/sbin/munin19:15:143
3165899301,26cyclictest21234-21/usr/sbin/munin22:25:163
3165899301,26cyclictest19034-21/usr/sbin/munin22:20:243
3165899301,26cyclictest19034-21/usr/sbin/munin22:20:243
3165899301,26cyclictest18742-21/usr/sbin/munin21:05:163
3165899301,26cyclictest13988-21/usr/sbin/munin20:55:223
3165899301,17cyclictest11790-21/usr/sbin/munin20:50:243
31651993019,2cyclictest29388-21kworker/2:122:03:062
3163999301,13cyclictest11462-21kworker/0:321:13:220
3163999301,12cyclictest9802-21kworker/0:222:03:380
3163999301,12cyclictest30261-21kworker/0:000:18:010
3163999301,11cyclictest25363-21kworker/0:222:37:460
3163999300,12cyclictest20893-21kworker/0:222:26:020
3163999300,10cyclictest29092-21kworker/0:321:39:520
3165899291,25cyclictest9325-21/usr/sbin/munin00:25:193
3165899291,25cyclictest9325-21/usr/sbin/munin00:25:193
3165899291,25cyclictest7395-21/usr/sbin/munin20:40:133
3165899291,25cyclictest30666-21/usr/sbin/munin00:00:143
3165899291,25cyclictest23429-21/usr/sbin/munin22:30:193
3163999291,13cyclictest8747-21awk19:30:010
3163999291,13cyclictest22787-21kworker/0:121:22:040
3163999282,21cyclictest29092-21kworker/0:321:34:360
31639992813,3cyclictest11462-21kworker/0:321:05:240
3163999281,21cyclictest24055-21kworker/0:023:47:590
3163999281,13cyclictest29092-21kworker/0:321:46:480
3163999281,11cyclictest6547-21kworker/0:100:34:190
3163999281,11cyclictest12062-21kworker/0:022:06:000
3163999281,10cyclictest11026-21kworker/0:000:29:060
3163999281,10cyclictest11026-21kworker/0:000:29:060
3164699273,2cyclictest86850irq/129-enp1s0-00:26:411
3164699273,2cyclictest86850irq/129-enp1s0-00:26:411
3163999271,9cyclictest13261-21kworker/0:200:36:190
3163999271,22cyclictest21642-21kworker/0:223:57:150
3163999271,21cyclictest17077-21kworker/0:023:29:550
3163999271,12cyclictest7558-21kworker/0:121:52:520
3163999270,12cyclictest6179-21kworker/0:319:30:180
3163999270,11cyclictest28021-21kworker/0:323:54:110
3163999270,11cyclictest23917-21kworker/0:020:08:040
3164699262,12cyclictest2102-21hddtemp_smartct21:40:131
31646992614,3cyclictest0-21swapper/119:45:161
3164699261,17cyclictest0-21swapper/120:43:471
3164699260,17cyclictest0-21swapper/121:23:151
3164699260,17cyclictest0-21swapper/119:17:451
3163999266,3cyclictest30261-21kworker/0:000:08:090
3163999261,12cyclictest7558-21kworker/0:121:58:120
31639992610,2cyclictest30395-21kworker/0:220:25:260
3163999260,11cyclictest8782-21kworker/0:200:24:170
3164699251,17cyclictest0-21swapper/120:00:271
3163999259,3cyclictest11462-21kworker/0:321:00:500
3163999258,2cyclictest21669-21kworker/0:220:00:540
3163999251,11cyclictest0-21swapper/022:41:020
3163999250,16cyclictest2643-21kworker/0:020:35:160
3163999250,10cyclictest5923-21kworker/0:223:05:450
3165199243,5cyclictest0-21swapper/221:14:232
3165199241,7cyclictest0-21swapper/222:39:532
3165199241,3cyclictest29388-21kworker/2:123:11:042
3164699241,4cyclictest9013-21df_inode23:10:141
3164699241,3cyclictest20785-21cstates21:10:121
3164699241,3cyclictest1495-21cat19:15:011
31646992410,12cyclictest31955-21kworker/u8:123:31:441
3163999249,3cyclictest2741-21kworker/0:122:55:230
3163999245,3cyclictest5923-21kworker/0:223:22:350
3163999245,3cyclictest5923-21kworker/0:223:22:350
3163999242,10cyclictest29092-21kworker/0:321:41:340
3163999241,8cyclictest23917-21kworker/0:020:15:400
3163999241,11cyclictest21642-21kworker/0:200:01:130
3165199236,4cyclictest26624-21uniq23:50:142
3165199231,12cyclictest0-21swapper/200:12:402
31646992317,3cyclictest7461-21df00:20:121
31646992315,2cyclictest0-21swapper/120:55:121
3164699231,3cyclictest8439-21latency_hist23:10:011
3164699231,15cyclictest0-21swapper/122:23:421
3164699231,15cyclictest0-21swapper/122:23:421
3163999238,2cyclictest30261-21kworker/0:000:14:530
3163999236,2cyclictest23126-21kworker/0:022:31:400
3163999234,4cyclictest836-21runrttasks19:21:260
3163999234,4cyclictest5923-21kworker/0:223:10:390
3163999233,11cyclictest637-21kworker/0:119:12:150
3163999232,7cyclictest17077-21kworker/0:023:36:140
31639992311,3cyclictest2643-21kworker/0:020:34:160
31639992311,3cyclictest12896-21kworker/0:019:40:580
3163999231,11cyclictest637-21kworker/0:119:15:400
31651992217,3cyclictest10716-21ls20:45:202
3165199221,4cyclictest0-21swapper/223:36:582
3164699229,6cyclictest5742-21kworker/u8:019:25:061
31646992213,3cyclictest0-21swapper/119:23:071
3164699221,19cyclictest0-21swapper/123:44:021
3164699221,19cyclictest0-21swapper/122:16:111
3164699221,19cyclictest0-21swapper/121:15:381
3164699221,19cyclictest0-21swapper/121:15:381
3164699221,15cyclictest0-21swapper/123:16:041
3164699221,15cyclictest0-21swapper/123:00:261
3164699221,15cyclictest0-21swapper/122:46:491
3164699221,15cyclictest0-21swapper/121:27:441
3164699221,15cyclictest0-21swapper/120:16:001
3164699221,15cyclictest0-21swapper/120:13:441
3164699221,15cyclictest0-21swapper/120:13:441
3164699221,15cyclictest0-21swapper/119:51:061
3164699221,10cyclictest11913-21tr20:50:131
3164699220,14cyclictest0-21swapper/123:25:261
3163999227,3cyclictest28062-21unixbench_multi20:10:300
3163999227,3cyclictest28062-21unixbench_multi20:10:300
3163999221,13cyclictest4947-21kworker/0:220:40:160
3163999221,12cyclictest10622-21kworker/0:119:36:420
3163999221,10cyclictest10379-21kworker/0:023:15:110
3163999220,10cyclictest11462-21kworker/0:320:57:200
117922221,2sleep014634-21snmpd20:50:140
3165199211,11cyclictest22926-21ntp_states23:40:222
3164699219,3cyclictest31887-21ps00:00:251
3164699211,9cyclictest0-21swapper/121:50:011
3164699211,3cyclictest0-21swapper/121:51:381
31646992113,3cyclictest0-21swapper/120:38:331
31646992113,2cyclictest0-21swapper/121:07:581
31646992113,2cyclictest0-21swapper/119:36:511
3164699211,2cyclictest11216-21strings22:00:191
31646992112,3cyclictest0-21swapper/122:52:401
3164699211,18cyclictest0-21swapper/123:45:271
3164699211,18cyclictest0-21swapper/123:35:331
3164699211,18cyclictest0-21swapper/121:00:291
3164699211,17cyclictest20370-21diskmemload00:05:251
3164699211,17cyclictest16565-21sshd22:14:391
3164699211,15cyclictest0-21swapper/123:21:281
3164699211,15cyclictest0-21swapper/123:21:281
3164699211,15cyclictest0-21swapper/119:40:191
3164699211,15cyclictest0-21swapper/119:31:061
3164699211,11cyclictest13697-21smart_sda22:05:241
3164699211,11cyclictest1338-21perl20:25:171
3164699210,19cyclictest0-21swapper/121:30:281
3164699210,15cyclictest0-21swapper/122:42:451
3164699210,15cyclictest0-21swapper/100:38:121
3163999211,11cyclictest10622-21kworker/0:119:52:140
3165199203,8cyclictest31955-21kworker/u8:123:32:182
3165199202,5cyclictest24133-21munin-run20:05:012
3165199201,9cyclictest0-21swapper/222:21:312
3165199201,9cyclictest0-21swapper/222:21:312
3165199201,9cyclictest0-21swapper/200:35:322
3165199201,10cyclictest6624-21cat19:25:002
3165199201,10cyclictest11497-21gltestperf23:15:172
31646992013,2cyclictest0-21swapper/120:49:101
31646992012,3cyclictest0-21swapper/122:34:361
3164699201,18cyclictest0-21swapper/120:07:171
3164699201,18cyclictest0-21swapper/100:12:331
3164699201,17cyclictest0-21swapper/121:35:071
3164699201,15cyclictest0-21swapper/120:24:531
3164699201,10cyclictest27030-21if_err_enp1s023:50:161
3164699200,18cyclictest0-21swapper/123:56:081
3164699200,18cyclictest0-21swapper/100:16:141
3164699200,15cyclictest0-21swapper/122:36:461
3163999201,11cyclictest30960-21/usr/sbin/munin20:20:130
28362201,2sleep10-21swapper/122:55:201
3165199196,3cyclictest22552-1kworker/u9:100:20:432
3165199195,4cyclictest16419-21kworker/u8:122:30:232
3165199194,6cyclictest8950-21grep20:40:282
3165199194,5cyclictest1097-21grep00:05:172
3165199194,5cyclictest0-21swapper/222:15:152
3165199193,9cyclictest26047-21kworker/u8:120:17:132
3165199192,5cyclictest22552-1kworker/u9:121:37:122
3165199191,9cyclictest0-21swapper/222:27:002
3165199191,5cyclictest32149-21sed00:00:272
3165199191,5cyclictest1434-21users22:50:282
3165199191,4cyclictest3394-21crond21:45:012
3165199191,10cyclictest3930-21chrt22:58:492
3165199191,10cyclictest0-21swapper/223:49:412
3165199191,10cyclictest0-21swapper/222:12:122
3165199191,10cyclictest0-21swapper/221:47:562
3164699191,16cyclictest0-21swapper/120:32:261
3164699191,10cyclictest0-21swapper/100:30:161
3164699190,15cyclictest0-21swapper/119:57:231
98092180,2sleep20-21swapper/221:57:442
3165199185,5cyclictest28458-21/usr/sbin/munin23:55:202
3165199185,3cyclictest12050irq/123-ahci[0023:30:012
3165199184,3cyclictest12050irq/123-ahci[0020:15:012
3165199184,3cyclictest12050irq/123-ahci[0020:15:012
3165199182,9cyclictest0-21swapper/222:45:462
3165199182,4cyclictest581-21systemd-journal21:20:012
3165199182,4cyclictest581-21systemd-journal21:20:012
3165199181,9cyclictest86850irq/129-enp1s0-00:25:252
3165199181,9cyclictest86850irq/129-enp1s0-00:25:252
3165199181,9cyclictest5148-21taskset00:15:132
3165199181,8cyclictest6469-21/usr/sbin/munin23:05:232
3165199181,6cyclictest0-21swapper/223:22:152
3165199181,6cyclictest0-21swapper/223:22:152
3165199181,5cyclictest6007-21irqstats20:35:182
3165199181,4cyclictest30-21rcuc/219:25:142
31651991813,3cyclictest27958-21if_enp2s021:25:202
31651991812,3cyclictest5153-21sort20:35:022
3165199181,15cyclictest0-21swapper/221:50:182
3165199181,11cyclictest0-21swapper/222:43:212
3165199181,11cyclictest0-21swapper/219:30:132
3165199181,11cyclictest0-21swapper/200:33:332
3165199181,10cyclictest32-21ksoftirqd/220:20:182
31646991812,3cyclictest8839-21irqstats21:55:171
200752181,3sleep20-21swapper/221:06:302
3165199173,3cyclictest86550irq/126-enp1s0-19:55:222
3165199173,3cyclictest2435-21cpuspeed_turbos19:15:122
3165199172,9cyclictest0-21swapper/222:06:302
3165199171,9cyclictest0-21swapper/221:00:022
3165199171,8cyclictest31841-21/usr/sbin/munin19:10:172
3165199171,5cyclictest26346-21awk20:10:012
3165199171,5cyclictest2499-21sshd20:26:412
31651991712,3cyclictest14189-21cut19:40:192
3165199171,14cyclictest0-21swapper/219:36:092
31651991711,3cyclictest29524-21cpuspeed_turbos21:30:132
3165199171,10cyclictest27383-21kworker/u8:219:45:532
3165199170,9cyclictest25895-21sed21:20:192
196632171,2sleep20-21swapper/219:54:242
3165199162,3cyclictest12050irq/123-ahci[0020:50:152
3165199161,9cyclictest0-21swapper/220:56:572
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional