You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 12:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Apr 07, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5202232165,55sleep20-21swapper/219:06:072
5662203166,25sleep00-21swapper/019:06:420
5772201165,24sleep10-21swapper/119:06:511
313232201164,24sleep30-21swapper/319:05:113
1320221730,4sleep299599cyclictest00:30:102
2142821620,5sleep098299cyclictest22:20:170
2142821620,5sleep098299cyclictest22:20:170
1899521450,5sleep299599cyclictest22:15:142
995991080,106cyclictest0-21swapper/221:50:422
995999490,3cyclictest0-21swapper/223:27:392
995997168,2cyclictest0-21swapper/219:21:192
995997067,2cyclictest0-21swapper/220:07:522
995997067,2cyclictest0-21swapper/220:07:522
982994111,14cyclictest30869-21munin-run21:30:000
99599321,11cyclictest0-21swapper/223:20:332
99599320,13cyclictest0-21swapper/221:15:092
99599320,11cyclictest0-21swapper/220:21:212
98299329,11cyclictest0-21swapper/000:10:020
99599310,13cyclictest0-21swapper/221:24:012
99599301,21cyclictest0-21swapper/200:09:532
99599301,11cyclictest0-21swapper/222:51:042
99599301,11cyclictest0-21swapper/200:01:572
99599300,22cyclictest0-21swapper/222:12:252
99599291,9cyclictest0-21swapper/221:36:012
99599291,10cyclictest0-21swapper/223:39:012
99599291,10cyclictest0-21swapper/223:39:002
99599291,10cyclictest0-21swapper/221:07:132
99599290,13cyclictest0-21swapper/223:34:562
99599290,11cyclictest0-21swapper/221:55:292
99599290,10cyclictest0-21swapper/222:41:162
98299294,12cyclictest32104-21munin-run00:00:010
99599281,13cyclictest0-21swapper/222:06:092
99599281,12cyclictest0-21swapper/223:02:122
99599281,12cyclictest0-21swapper/221:13:452
99599281,12cyclictest0-21swapper/220:25:532
99599281,10cyclictest0-21swapper/219:47:442
99599280,9cyclictest0-21swapper/221:29:452
99599280,22cyclictest0-21swapper/223:47:412
99599280,17cyclictest0-21swapper/220:35:012
99599280,12cyclictest0-21swapper/200:11:372
99599280,11cyclictest0-21swapper/221:43:092
99599280,10cyclictest0-21swapper/221:47:052
99599280,10cyclictest0-21swapper/200:20:332
98299281,6cyclictest23299-21strings21:10:190
99599271,11cyclictest0-21swapper/200:19:012
99599271,10cyclictest30937-21awk23:55:172
99599271,10cyclictest0-21swapper/222:46:322
99599270,9cyclictest0-21swapper/223:15:002
99599270,9cyclictest0-21swapper/219:54:132
99599270,12cyclictest0-21swapper/223:53:372
99599270,12cyclictest0-21swapper/222:21:332
99599270,12cyclictest0-21swapper/222:21:332
99599270,11cyclictest0-21swapper/223:14:482
99599270,11cyclictest0-21swapper/222:25:532
99599270,10cyclictest0-21swapper/222:38:042
99599270,10cyclictest0-21swapper/220:12:172
98899271,23cyclictest10312-21sed00:20:251
99599261,10cyclictest0-21swapper/220:52:292
99599261,10cyclictest0-21swapper/200:38:292
99599260,18cyclictest0-21swapper/223:06:572
99599260,11cyclictest0-21swapper/222:04:092
99599260,11cyclictest0-21swapper/221:32:132
99599260,11cyclictest0-21swapper/220:35:252
99599260,10cyclictest0-21swapper/222:59:482
99599260,10cyclictest0-21swapper/220:17:372
99999254,2cyclictest87350irq/126-enp1s0-22:07:483
99999252,8cyclictest12050irq/123-ahci[0020:05:013
99599251,9cyclictest0-21swapper/220:41:212
99599251,10cyclictest0-21swapper/200:28:012
99599250,9cyclictest0-21swapper/222:31:212
99599250,11cyclictest0-21swapper/223:40:572
99599250,11cyclictest0-21swapper/221:01:492
99599250,10cyclictest0-21swapper/220:55:052
99599250,10cyclictest0-21swapper/220:46:532
98899253,17cyclictest31958-40dnf23:58:251
988992518,4cyclictest4205-21ntp_states21:40:221
988992511,4cyclictest4668-21memory22:55:181
99999243,6cyclictest0-21swapper/321:31:533
99599241,9cyclictest0-21swapper/219:40:402
98899249,13cyclictest748-21kworker/u8:020:40:231
988992419,3cyclictest7309-21munin-run21:50:011
98899240,22cyclictest0-21swapper/119:37:131
98899240,18cyclictest0-21swapper/119:49:131
98899240,16cyclictest0-21swapper/119:27:131
98299243,9cyclictest0-21swapper/023:20:550
99999237,8cyclictest12050irq/123-ahci[0000:35:013
99999235,7cyclictest755-21rngd00:26:103
99999233,15cyclictest39-21rcuc/321:25:223
99999231,9cyclictest0-21swapper/319:15:353
99999231,5cyclictest0-21swapper/321:16:453
999992311,4cyclictest401ktimersoftd/300:00:003
99999231,11cyclictest0-21swapper/300:35:143
99599238,3cyclictest0-21swapper/219:39:482
988992319,2cyclictest23411-40dnf19:57:131
98899231,3cyclictest25858-21munin-run20:05:011
98899230,4cyclictest0-21swapper/100:35:081
98899230,21cyclictest0-21swapper/123:20:131
98899230,13cyclictest0-21swapper/123:10:161
98299231,7cyclictest5733-21munin-run23:00:020
98299231,6cyclictest0-21swapper/022:36:020
98299231,6cyclictest0-21swapper/000:20:120
99999228,6cyclictest87350irq/126-enp1s0-22:20:013
99999228,3cyclictest19142-21/usr/sbin/munin23:30:223
99999223,11cyclictest3755-21/usr/sbin/munin22:55:233
99999223,11cyclictest0-21swapper/323:10:043
99999221,6cyclictest0-21swapper/321:54:353
99999221,12cyclictest87350irq/126-enp1s0-22:28:273
99599226,3cyclictest0-21swapper/219:58:532
99599224,3cyclictest0-21swapper/219:13:352
98899222,13cyclictest87550irq/128-enp1s0-21:26:071
98899221,9cyclictest0-21swapper/123:44:341
98899221,7cyclictest9754-21vmstat23:05:271
98899221,7cyclictest20008-21unixbench-2d22:15:231
988992216,2cyclictest0-21swapper/121:15:071
98899221,3cyclictest8598-21ntp_states21:50:211
98899221,3cyclictest28727-21fschecks_count20:10:141
98899221,19cyclictest5475-21ntp_states00:10:221
98899221,18cyclictest18325-21ls23:25:251
98899221,18cyclictest109802sleep100:25:031
98899221,10cyclictest27700-21latency_hist23:50:021
98899220,6cyclictest0-21swapper/122:24:231
98899220,6cyclictest0-21swapper/122:24:231
98899220,6cyclictest0-21swapper/122:12:041
98899220,6cyclictest0-21swapper/121:34:001
98899220,5cyclictest0-21swapper/122:51:291
98299225,3cyclictest0-21swapper/023:19:240
98299221,6cyclictest0-21swapper/000:29:350
98299221,5cyclictest0-21swapper/022:15:380
98299221,5cyclictest0-21swapper/000:19:280
98299221,10cyclictest11343-21head23:10:180
98299220,11cyclictest0-21swapper/021:53:200
98299220,10cyclictest0-21swapper/022:05:150
99999219,4cyclictest0-21swapper/322:46:183
99999217,3cyclictest0-21swapper/323:46:303
99999216,6cyclictest0-21swapper/323:16:053
99999216,4cyclictest0-21swapper/321:57:103
99999215,3cyclictest32358-21grep00:00:113
99999214,9cyclictest39-21rcuc/300:05:123
99999213,9cyclictest3120750kworker/u8:023:08:293
99999211,9cyclictest8888-21perf20:40:013
99999211,11cyclictest12375-21ntp_states20:45:223
999992110,3cyclictest12050irq/123-ahci[0019:35:023
99599217,3cyclictest0-21swapper/220:04:272
99599217,3cyclictest0-21swapper/219:27:092
98899217,3cyclictest0-21swapper/122:02:541
98899215,4cyclictest4448-21sshd19:15:131
98899211,9cyclictest0-21swapper/100:02:411
988992117,2cyclictest22151-21irqstats23:35:181
988992117,2cyclictest22151-21irqstats23:35:181
98899211,18cyclictest0-21swapper/123:53:241
98899210,9cyclictest0-21swapper/121:14:421
98899210,6cyclictest0-21swapper/122:39:311
98899210,5cyclictest0-21swapper/123:02:261
98899210,5cyclictest0-21swapper/122:41:061
98899210,4cyclictest0-21swapper/122:25:281
98899210,2cyclictest0-21swapper/122:46:311
98899210,19cyclictest0-21swapper/121:55:121
98299217,1cyclictest0-21swapper/022:00:130
98299215,3cyclictest87550irq/128-enp1s0-22:25:220
98299214,3cyclictest0-21swapper/022:30:130
98299212,3cyclictest31168-21runrttasks22:41:070
98299211,6cyclictest0-21swapper/022:47:400
982992116,3cyclictest1932-21needreboot21:35:160
982992111,3cyclictest8579-21sshd20:36:190
98299211,11cyclictest16946-21/usr/sbin/munin23:25:130
99999208,4cyclictest0-21swapper/323:37:113
99999208,4cyclictest0-21swapper/323:37:113
99999206,5cyclictest0-21swapper/300:20:123
99999206,3cyclictest87350irq/126-enp1s0-19:20:253
99999206,3cyclictest0-21swapper/323:22:483
99999203,11cyclictest0-21swapper/322:40:293
99999202,10cyclictest26080-21/usr/sbin/munin20:05:173
99999202,10cyclictest26080-21/usr/sbin/munin20:05:173
99999201,11cyclictest808-21snmpd22:10:193
99999201,11cyclictest6601-21/usr/sbin/munin00:15:223
99999201,11cyclictest31319-21latency20:15:173
99999201,11cyclictest0-21swapper/323:40:083
99999201,11cyclictest0-21swapper/323:00:053
99999201,11cyclictest0-21swapper/322:51:233
99999201,11cyclictest0-21swapper/322:21:043
99999201,11cyclictest0-21swapper/322:21:043
99999201,11cyclictest0-21swapper/321:40:223
99999201,11cyclictest0-21swapper/300:11:373
99999200,5cyclictest87350irq/126-enp1s0-23:50:183
99999200,3cyclictest0-21swapper/322:35:363
98899207,2cyclictest6179-21kworker/u8:200:30:241
98899204,4cyclictest87550irq/128-enp1s0-20:50:251
98899203,7cyclictest87550irq/128-enp1s0-23:30:061
98899202,6cyclictest0-21swapper/100:08:101
98899201,18cyclictest0-21swapper/120:45:181
98899201,17cyclictest0-21swapper/120:59:071
98899201,17cyclictest0-21swapper/119:42:171
98899201,10cyclictest0-21swapper/119:10:021
98899200,6cyclictest0-21swapper/122:08:321
98899200,5cyclictest0-21swapper/121:25:001
98899200,3cyclictest0-21swapper/123:15:391
98899200,2cyclictest87550irq/128-enp1s0-00:15:181
98299209,9cyclictest87350irq/126-enp1s0-19:57:130
98299208,3cyclictest20549-21ntp_states19:50:220
98299201,6cyclictest31746-21gltestperf21:30:140
98299201,6cyclictest0-21swapper/022:50:050
98299201,11cyclictest30178-21sshd20:14:180
98299201,10cyclictest0-21swapper/022:14:000
99999195,3cyclictest0-21swapper/323:25:093
99999192,12cyclictest41-21ksoftirqd/321:35:163
99999191,9cyclictest9444-21df_inode20:40:123
99999191,9cyclictest2292-21munin-run20:25:003
99999191,9cyclictest22443-21kworker/u8:022:30:233
99999191,11cyclictest839-21lldpd21:47:073
99999191,10cyclictest4715-21/usr/sbin/munin20:30:233
99999191,10cyclictest28273-21/usr/sbin/munin20:10:203
99999191,10cyclictest0-21swapper/321:20:073
99999191,10cyclictest0-21swapper/321:10:403
99599194,3cyclictest0-21swapper/219:33:352
98899196,3cyclictest0-21swapper/120:07:201
98899196,3cyclictest0-21swapper/120:07:201
98899191,8cyclictest0-21swapper/122:32:311
98899191,5cyclictest20109-21/usr/sbin/munin21:05:211
988991915,3cyclictest0-21swapper/120:28:401
98899191,3cyclictest0-21swapper/120:20:231
98899191,17cyclictest0-21swapper/120:34:531
98299191,7cyclictest13501-21/usr/sbin/munin20:50:110
98299191,6cyclictest0-21swapper/023:38:010
98299191,6cyclictest0-21swapper/023:38:010
98299191,6cyclictest0-21swapper/000:14:370
98299191,6cyclictest0-21swapper/000:03:400
98299191,5cyclictest87550irq/128-enp1s0-21:21:400
98299191,5cyclictest0-21swapper/023:02:570
98299191,5cyclictest0-21swapper/021:59:170
98299191,4cyclictest0-21swapper/000:31:320
98299191,3cyclictest0-21swapper/000:37:060
982991911,2cyclictest32668-21/usr/sbin/munin20:20:210
99999187,3cyclictest17904-21/usr/sbin/munin21:00:253
99999186,8cyclictest17450-40dnf20:57:213
99999183,7cyclictest401ktimersoftd/320:50:243
99999183,10cyclictest87350irq/126-enp1s0-21:10:013
99999182,7cyclictest2517-21/usr/sbin/munin20:25:233
99999181,4cyclictest1184-21/usr/sbin/munin19:10:183
99999180,2cyclictest0-21swapper/322:03:113
99599184,3cyclictest0-21swapper/219:16:072
988991815,2cyclictest0-21swapper/121:01:271
98899181,2cyclictest0-21swapper/119:20:281
988991812,3cyclictest1249-21df21:35:101
98899181,16cyclictest0-21swapper/120:35:151
98899181,16cyclictest0-21swapper/119:53:021
98899181,16cyclictest0-21swapper/119:34:441
98899181,14cyclictest32449-21sleep20:20:011
98299182,13cyclictest19142-21/usr/sbin/munin23:30:170
98299181,5cyclictest0-21swapper/023:51:040
98299181,3cyclictest0-21swapper/021:47:360
982991813,3cyclictest4728-21users21:40:270
98299181,2cyclictest0-21swapper/019:44:130
98299181,15cyclictest0-21swapper/023:05:120
982991811,4cyclictest6689-21grep20:35:010
982991811,4cyclictest24124-21gltestperf23:40:150
99999171,9cyclictest808-21snmpd19:59:533
999991711,3cyclictest13050-21tr19:35:123
99999170,6cyclictest41-21ksoftirqd/319:25:163
98299174,3cyclictest12868-21unixbench_singl20:45:250
98299173,3cyclictest22072-21dump-pmu-power21:10:000
98299171,5cyclictest0-21swapper/021:15:120
982991711,3cyclictest26314-21fw_packets23:45:150
98299171,13cyclictest23872-21/usr/sbin/munin20:00:120
191792170,3sleep30-21swapper/319:49:163
99999162,5cyclictest15082-21/usr/sbin/munin19:40:173
98299163,2cyclictest87650irq/129-enp1s0-19:25:240
98299162,12cyclictest2517-21/usr/sbin/munin20:25:160
98299161,3cyclictest87550irq/128-enp1s0-20:15:220
98299161,3cyclictest0-21swapper/019:38:590
98299161,14cyclictest0-21swapper/020:57:390
98299161,14cyclictest0-21swapper/019:12:490
982991611,3cyclictest17279-21/usr/sbin/munin19:45:110
98299161,13cyclictest10101-21head20:40:170
98299161,12cyclictest10692-21/usr/sbin/munin19:30:110
98299160,2cyclictest0-21swapper/020:05:170
98299160,2cyclictest0-21swapper/020:05:170
98299160,14cyclictest0-21swapper/021:00:540
40962161,2sleep00-21swapper/019:15:080
999991510,3cyclictest19475-21/usr/sbin/munin19:50:143
98299151,3cyclictest87550irq/128-enp1s0-19:20:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional