You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 08:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Jul 14, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
254852241165,24sleep00-21swapper/019:07:480
256182232162,57sleep30-21swapper/319:09:313
256022232164,55sleep20-21swapper/219:09:192
254762227158,56sleep10-21swapper/119:07:411
3228621540,4sleep22575399cyclictest00:15:182
1288321540,5sleep02574399cyclictest19:50:220
25753999491,2cyclictest0-21swapper/219:38:362
237072560,1sleep10-21swapper/121:30:121
194922560,4sleep232-21ksoftirqd/223:45:242
49632530,1sleep20-21swapper/220:45:252
77232480,2sleep10-21swapper/122:05:231
160372430,2sleep00-21swapper/020:00:010
2575799381,8cyclictest26583-21memory21:35:193
2575799381,8cyclictest26583-21memory21:35:193
2575799381,19cyclictest0-21swapper/323:36:463
2574399389,13cyclictest5381-21munin-run00:30:020
2575799371,10cyclictest21189-21taskset22:35:243
2575799361,11cyclictest29444-21/usr/sbin/munin20:30:183
2575799351,7cyclictest11558-21/usr/sbin/munin23:30:153
2575799351,6cyclictest0-21swapper/322:15:013
25757993511,21cyclictest6566-21hwlatdetect20:50:183
25757993511,20cyclictest26605-21gltestperf19:10:153
25757993511,20cyclictest26605-21gltestperf19:10:153
2575799351,10cyclictest3409-21/usr/sbin/munin00:25:223
2574399357,13cyclictest20129-21munin-run23:50:010
2575799345,3cyclictest87650irq/129-enp1s0-23:05:163
2575799342,6cyclictest3552-21timerwakeupswit21:55:263
2575799341,30cyclictest808-21snmpd23:00:553
2575799341,30cyclictest22470-21sshd20:13:143
25757993410,16cyclictest5387-21cut23:15:153
2574799341,25cyclictest0-21swapper/120:34:151
2575799336,6cyclictest0-21swapper/321:19:013
2575799336,4cyclictest15739-21ntp_states21:10:213
2575799336,3cyclictest87350irq/126-enp1s0-19:25:193
2575799333,6cyclictest808-21snmpd23:49:173
2575799332,5cyclictest29634-21/usr/sbin/munin19:15:183
2575799331,9cyclictest0-21swapper/322:08:033
2575799331,3cyclictest0-21swapper/321:47:053
2575799331,29cyclictest10940-21/usr/sbin/munin22:15:203
2575799331,11cyclictest26870-21sshd20:23:153
25757993310,20cyclictest9703-21df_inode23:25:153
2574799330,25cyclictest0-21swapper/120:00:161
2575799328,3cyclictest12686-21expr21:05:133
2575799322,6cyclictest27037-21cpuspeed_turbos00:05:153
2575799322,26cyclictest7929-21cpuspeed_turbos00:35:123
2575799321,3cyclictest0-21swapper/322:45:303
2575799321,28cyclictest808-21snmpd22:34:143
25757993212,17cyclictest18225-21ls20:05:003
25757993210,19cyclictest827-21runrttasks21:22:473
25757993210,19cyclictest21268-21acpi21:25:123
2575799320,6cyclictest0-21swapper/321:31:513
2575799319,3cyclictest14446-21diskmemload21:43:333
2575799319,2cyclictest26462-21seq00:01:363
2575799315,17cyclictest0-21swapper/322:41:323
2575799314,6cyclictest0-21swapper/321:50:183
2575799312,4cyclictest28508-21smart_sda20:25:253
2575799312,4cyclictest28508-21smart_sda20:25:253
2575799311,9cyclictest0-21swapper/322:50:413
2575799311,20cyclictest24939-21cut20:20:023
2575799311,10cyclictest0-21swapper/323:11:213
25757993110,18cyclictest827-21runrttasks23:21:383
2575799310,10cyclictest0-21swapper/322:24:383
2574799311,21cyclictest0-21swapper/119:49:341
2574799311,10cyclictest21220-21sort21:25:021
2575799309,3cyclictest9655-21/usr/sbin/munin19:45:173
2575799309,3cyclictest31636-21/usr/sbin/munin20:35:163
2575799309,3cyclictest31368-21/usr/sbin/munin00:15:213
2575799309,18cyclictest5697-21sort00:30:113
2575799305,3cyclictest87350irq/126-enp1s0-22:55:153
2575799303,7cyclictest17657-21kworker/u8:123:51:293
2575799301,9cyclictest1818-21grep00:20:133
2575799301,4cyclictest5499-21sshd20:48:143
2575799301,11cyclictest0-21swapper/300:12:403
25757993010,17cyclictest827-21runrttasks20:42:063
25757993010,17cyclictest4678-21sh22:00:123
2575399301,14cyclictest14390-21df19:55:162
2575399301,13cyclictest0-21swapper/223:59:532
2574799301,22cyclictest0-21swapper/120:57:091
2574799301,13cyclictest0-21swapper/119:53:321
2574799300,23cyclictest0-21swapper/122:51:501
2574799300,14cyclictest0-21swapper/100:25:221
2574399303,13cyclictest7-21ksoftirqd/021:10:020
286472290,8sleep230-21rcuc/219:15:032
2575799297,2cyclictest0-21swapper/323:40:343
2575799295,3cyclictest87650irq/129-enp1s0-20:09:423
2575799294,5cyclictest9355-21sshd19:44:133
2575799292,4cyclictest32294-21smart_sda19:20:243
2575799291,5cyclictest0-21swapper/320:57:383
25757992910,16cyclictest15944-21grep22:25:153
2575399291,14cyclictest0-21swapper/221:02:552
2575399291,14cyclictest0-21swapper/220:32:352
2575399291,13cyclictest0-21swapper/220:35:552
2575399290,10cyclictest0-21swapper/223:11:542
2574799295,16cyclictest0-21swapper/122:15:081
2574799291,9cyclictest0-21swapper/123:38:051
2574799291,7cyclictest0-21swapper/119:59:541
2574799290,13cyclictest0-21swapper/120:27:511
2574799290,13cyclictest0-21swapper/120:27:511
25743992910,16cyclictest4117-21munin-run22:00:020
260352280,4sleep2311ktimersoftd/222:49:192
2575799287,3cyclictest14049-21/usr/sbin/munin19:55:173
2575799285,3cyclictest6512-21sendmail_mailtr19:35:243
25757992810,15cyclictest3445-21cut19:30:153
25757992810,15cyclictest11274-21netstat21:00:213
2575399281,13cyclictest0-21swapper/223:33:122
2575399281,11cyclictest0-21swapper/221:27:592
2574799281,17cyclictest0-21swapper/123:09:571
2574799281,12cyclictest0-21swapper/123:19:131
2574799281,11cyclictest31335-21cat19:20:141
2574799281,11cyclictest0-21swapper/120:17:121
2574799281,10cyclictest0-21swapper/120:20:491
2574799280,12cyclictest0-21swapper/119:27:121
2575799271,12cyclictest0-21swapper/323:55:063
2575399271,12cyclictest0-21swapper/223:16:402
2575399271,12cyclictest0-21swapper/219:42:152
2575399271,11cyclictest0-21swapper/221:11:112
2575399271,10cyclictest0-21swapper/223:26:442
2575399271,10cyclictest0-21swapper/200:01:032
2575399270,14cyclictest0-21swapper/221:36:592
2575399270,14cyclictest0-21swapper/221:36:592
2575399270,11cyclictest0-21swapper/222:25:362
2575399270,11cyclictest0-21swapper/220:00:402
25747992716,3cyclictest21847-21perl21:25:171
2574799271,11cyclictest0-21swapper/122:40:421
2574799271,10cyclictest0-21swapper/123:49:391
2574799271,10cyclictest0-21swapper/119:40:061
2574799270,12cyclictest0-21swapper/123:30:251
2574799270,12cyclictest0-21swapper/119:32:181
2574799270,11cyclictest0-21swapper/120:53:011
2574799270,10cyclictest0-21swapper/120:11:581
2574399279,15cyclictest26118-21idleruntime-cro22:50:000
2575799269,3cyclictest13102-21mailstats19:50:253
2575399261,11cyclictest0-21swapper/222:44:082
2575399261,10cyclictest14446-21diskmemload23:10:002
2575399260,12cyclictest0-21swapper/222:53:522
2575399260,12cyclictest0-21swapper/200:20:032
2575399260,11cyclictest0-21swapper/219:10:162
2575399260,11cyclictest0-21swapper/219:10:162
2575399260,10cyclictest0-21swapper/222:14:322
2574799260,16cyclictest0-21swapper/121:56:361
2574799260,12cyclictest0-21swapper/119:18:331
2575399251,9cyclictest0-21swapper/221:06:552
2575399251,11cyclictest0-21swapper/220:12:242
2575399251,10cyclictest0-21swapper/219:51:442
2575399250,12cyclictest0-21swapper/222:04:562
2575399250,12cyclictest0-21swapper/221:21:312
2574799258,2cyclictest0-21swapper/123:55:241
2574799257,7cyclictest0-21swapper/121:05:061
2574799251,7cyclictest0-21swapper/121:02:051
2574799251,16cyclictest31137-21idleruntime-cro00:15:011
2574799251,12cyclictest0-21swapper/120:41:011
2574799251,11cyclictest0-21swapper/123:00:501
2574799251,11cyclictest0-21swapper/119:39:281
2574799251,10cyclictest909-21sssd_nss20:50:011
2575399241,9cyclictest0-21swapper/222:23:162
2575399241,12cyclictest0-21swapper/219:30:272
2575399241,11cyclictest0-21swapper/220:25:122
2575399241,11cyclictest0-21swapper/220:25:122
2575399241,10cyclictest0-21swapper/222:18:122
2575399240,11cyclictest0-21swapper/221:51:122
2575399240,11cyclictest0-21swapper/200:06:472
2574799241,9cyclictest0-21swapper/122:35:401
2574799241,15cyclictest0-21swapper/122:25:241
2574799241,13cyclictest0-21swapper/120:36:591
2574799241,11cyclictest0-21swapper/122:31:401
2574799241,11cyclictest0-21swapper/119:13:381
2574799241,11cyclictest0-21swapper/119:13:371
25747992410,3cyclictest0-21swapper/121:35:101
25747992410,3cyclictest0-21swapper/121:35:101
2575399231,9cyclictest0-21swapper/200:27:212
2575399231,11cyclictest0-21swapper/221:30:152
2575399230,10cyclictest0-21swapper/223:52:352
2574799231,9cyclictest0-21swapper/123:52:211
2574799231,7cyclictest0-21swapper/100:19:211
2574799231,15cyclictest0-21swapper/123:20:271
2574799231,11cyclictest0-21swapper/100:07:561
2574799231,10cyclictest25961-40dnf22:48:241
2574799231,10cyclictest14446-21diskmemload22:59:041
2574799230,10cyclictest0-21swapper/121:19:461
2574399231,11cyclictest7-21ksoftirqd/022:30:010
2574399231,11cyclictest2557-21perf23:10:010
158552230,1sleep00-21swapper/023:40:010
2575399220,9cyclictest0-21swapper/200:11:192
2575399220,11cyclictest0-21swapper/223:22:322
2574799227,13cyclictest24539-21kworker/u8:000:20:311
25747992212,3cyclictest32614-21diskstats21:50:151
2574799221,16cyclictest0-21swapper/100:02:351
2574799221,15cyclictest0-21swapper/122:00:121
2574799221,15cyclictest0-21swapper/100:30:271
2574799221,10cyclictest0-21swapper/120:08:281
2575399210,10cyclictest0-21swapper/223:40:542
2575399210,10cyclictest0-21swapper/222:58:542
2574799211,9cyclictest0-21swapper/122:20:201
25747992113,3cyclictest0-21swapper/121:46:151
2574799211,15cyclictest0-21swapper/123:25:101
2574799211,11cyclictest0-21swapper/100:35:251
2574799210,10cyclictest0-21swapper/121:13:461
2574399218,3cyclictest26483-21tr22:50:120
2574399214,8cyclictest0-21swapper/000:30:140
2574399211,11cyclictest21921-21/usr/sbin/munin22:40:180
2574399210,5cyclictest0-21swapper/022:09:570
2575399209,3cyclictest20430-21awk20:10:012
2575399201,10cyclictest0-21swapper/221:46:102
25747992015,3cyclictest0-21swapper/121:40:141
2574799201,11cyclictest17142-21processes23:40:221
2574799201,10cyclictest3238-21tune2fs23:10:141
2574799200,11cyclictest0-21swapper/122:11:191
2574399209,3cyclictest8742-21/usr/sbin/munin22:10:190
2574399204,13cyclictest3735-21cut19:30:170
2574399201,6cyclictest0-21swapper/021:31:130
2574399201,4cyclictest12252-21perf21:05:020
2574399201,11cyclictest20644-21/usr/sbin/munin20:10:200
2574399201,10cyclictest32475-21sshd23:02:140
114012200,4sleep011403-21latency_hist23:30:010
2575399191,11cyclictest0-21swapper/223:38:122
2575399191,11cyclictest0-21swapper/221:16:072
2575399190,9cyclictest0-21swapper/221:43:222
2574399199,3cyclictest9655-21/usr/sbin/munin19:45:210
2574399199,3cyclictest31029-21/usr/sbin/munin19:20:210
2574399198,3cyclictest7155-21sshd19:39:140
2574399191,5cyclictest0-21swapper/000:19:500
2574399191,4cyclictest0-21swapper/021:42:490
2574399191,10cyclictest7801-21/usr/sbin/munin00:35:130
25743991910,3cyclictest27917-21munin-run19:15:000
25743991910,3cyclictest27917-21munin-run19:15:000
133212191,2sleep00-21swapper/023:32:320
2575399183,5cyclictest0-21swapper/221:55:202
2575399181,8cyclictest8072-21/usr/sbin/munin20:55:242
2575399181,15cyclictest0-21swapper/219:27:042
2575399181,11cyclictest0-21swapper/223:00:232
2575399181,11cyclictest0-21swapper/219:49:042
2574399188,3cyclictest31427-21awk20:35:010
2574399182,7cyclictest0-21swapper/022:00:250
2574399181,9cyclictest21267-21/usr/sbin/munin21:25:270
2574399181,9cyclictest0-21swapper/023:15:130
2574399181,7cyclictest31636-21/usr/sbin/munin20:35:110
2574399181,3cyclictest810-21sshd21:47:150
25743991813,3cyclictest7849-21munin-run20:55:020
2574399181,15cyclictest0-21swapper/021:22:500
2575399173,6cyclictest17531-21/usr/sbin/munin22:30:152
2575399172,14cyclictest25730-21cyclictest00:33:582
25753991712,3cyclictest28060-21munin-node20:40:122
25753991711,3cyclictest6701-21tr22:05:132
2575399171,10cyclictest0-21swapper/222:37:282
2575399171,10cyclictest0-21swapper/220:23:352
2574399176,3cyclictest7456-21/usr/sbin/munin19:40:230
2574399172,3cyclictest0-21swapper/022:16:320
2574399171,7cyclictest823-21/usr/sbin/munin19:25:220
2574399171,7cyclictest5665-21awk20:50:020
2574399171,6cyclictest0-21swapper/023:51:090
2574399171,6cyclictest0-21swapper/021:51:430
2574399171,6cyclictest0-21swapper/021:16:530
2574399171,5cyclictest0-21swapper/023:40:260
2574399171,4cyclictest14446-21diskmemload22:36:460
2574399171,3cyclictest87550irq/128-enp1s0-00:20:250
2574399171,3cyclictest7696-21fschecks_time23:20:150
25743991713,3cyclictest0-21swapper/020:25:280
25743991713,3cyclictest0-21swapper/020:25:270
25743991712,3cyclictest2949-21cstates23:10:120
2574399171,15cyclictest0-21swapper/021:37:020
2574399171,15cyclictest0-21swapper/021:37:020
25743991711,4cyclictest29805-21grep00:10:160
25743991711,3cyclictest28623-21grep22:55:130
25743991711,3cyclictest26158-21cat00:00:260
2574399171,10cyclictest0-21swapper/021:12:380
179612171,2sleep00-21swapper/020:01:470
2575399161,7cyclictest0-21swapper/200:38:162
2575399161,14cyclictest0-21swapper/220:50:202
25753991610,3cyclictest31148-21cpuspeed_turbos19:20:122
2574399165,3cyclictest13134-21/usr/sbin/munin22:20:180
2574399162,12cyclictest8072-21/usr/sbin/munin20:55:120
2574399161,9cyclictest0-21swapper/020:40:170
25743991612,3cyclictest0-21swapper/020:22:290
25743991612,3cyclictest0-21swapper/020:19:320
25743991612,3cyclictest0-21swapper/019:15:480
25743991612,2cyclictest27106-21tr00:05:150
25743991611,3cyclictest18773-21sendmail_mailst22:30:240
2574399161,10cyclictest0-21swapper/023:58:540
245492160,2sleep20-21swapper/220:16:462
25743991510,3cyclictest827-21runrttasks20:07:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional