You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 07:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 14, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
49262242167,24sleep20-21swapper/219:08:432
30542238162,25sleep00-21swapper/019:05:040
48802236163,24sleep30-21swapper/319:08:123
49652223192,20sleep10-21swapper/119:09:141
3205621520,3sleep1511499cyclictest21:20:251
5121997269,2cyclictest0-21swapper/219:45:272
181642520,5sleep1511499cyclictest19:35:221
146312520,5sleep0510999cyclictest21:55:190
137282520,5sleep313729-21grep23:05:263
511499350,17cyclictest12053-21latency_hist23:05:011
510999347,13cyclictest7017-21perf21:40:020
510999347,12cyclictest9832-21perf23:00:010
511499305,17cyclictest0-21swapper/121:50:011
511499305,17cyclictest0-21swapper/121:50:011
511499301,23cyclictest0-21swapper/119:41:051
510999302,13cyclictest7287-21munin-run19:15:010
511499292,21cyclictest0-21swapper/121:59:441
511499291,22cyclictest0-21swapper/122:38:481
5114992911,12cyclictest0-21swapper/100:22:591
510999295,12cyclictest21773-21munin-run21:00:010
5114992824,3cyclictest0-21swapper/120:58:021
5114992824,3cyclictest0-21swapper/120:09:241
511499281,24cyclictest8486-21grep22:55:151
511499281,24cyclictest2807-21grep23:55:171
511499281,21cyclictest0-21swapper/122:09:011
511499281,21cyclictest0-21swapper/119:52:541
511499280,22cyclictest0-21swapper/122:13:421
511499280,22cyclictest0-21swapper/100:07:191
511499280,22cyclictest0-21swapper/100:07:191
510999283,12cyclictest18000-21munin-run22:05:010
5114992713,8cyclictest0-21swapper/119:22:071
5114992713,8cyclictest0-21swapper/119:22:071
511499271,21cyclictest0-21swapper/123:42:091
511499271,20cyclictest0-21swapper/122:42:171
510999274,11cyclictest1621-21idleruntime-cro23:55:020
510999271,13cyclictest7620-21munin-run22:55:010
510999271,13cyclictest7620-21munin-run22:55:010
510999271,13cyclictest11396-21munin-run21:50:020
510999271,13cyclictest11396-21munin-run21:50:020
46002270,4sleep339-21rcuc/300:00:153
46002270,4sleep339-21rcuc/300:00:153
84352260,5sleep339-21rcuc/319:15:153
512199267,5cyclictest12050irq/123-ahci[0021:00:012
512199263,3cyclictest0-21swapper/221:54:162
512199261,3cyclictest0-21swapper/222:15:362
512199260,2cyclictest12050irq/123-ahci[0019:15:022
5114992623,2cyclictest0-21swapper/120:18:381
511499262,19cyclictest0-21swapper/123:23:031
5114992614,7cyclictest0-21swapper/119:27:191
511499261,22cyclictest11148-21fschecks_count00:15:171
511499261,20cyclictest0-21swapper/122:47:291
511499260,20cyclictest0-21swapper/123:46:301
512599250,11cyclictest0-21swapper/321:12:143
511499253,6cyclictest14517-21tr23:10:121
511499251,21cyclictest25765-21missed_timers22:20:191
511499251,19cyclictest0-21swapper/123:36:231
511499251,15cyclictest0-21swapper/119:16:241
5114992510,3cyclictest26840-21ntp_states19:55:221
510999251,12cyclictest10173-21munin-run19:20:010
512199248,9cyclictest0-21swapper/223:43:512
512199242,14cyclictest311ktimersoftd/223:45:202
512199241,10cyclictest0-21swapper/219:52:092
511499243,18cyclictest221ktimersoftd/100:25:181
511499241,21cyclictest0-21swapper/120:33:171
511499241,21cyclictest0-21swapper/120:20:201
511499241,12cyclictest30099-21latency22:30:181
511499241,11cyclictest8252-21/usr/sbin/munin21:40:201
511499241,11cyclictest32449-21grep23:50:141
511499241,10cyclictest9827-21/usr/sbin/munin00:10:251
510999241,12cyclictest5096-21cyclictest22:40:000
511499234,9cyclictest87650irq/129-enp1s0-22:15:211
511499232,19cyclictest0-21swapper/121:10:531
511499232,15cyclictest87650irq/129-enp1s0-23:08:521
511499231,20cyclictest21-21rcuc/119:10:161
511499231,20cyclictest0-21swapper/121:25:091
511499231,20cyclictest0-21swapper/121:08:071
511499231,20cyclictest0-21swapper/120:02:591
511499231,19cyclictest11921-21/usr/sbin/munin20:35:191
511499231,14cyclictest0-21swapper/123:25:491
511499231,14cyclictest0-21swapper/100:37:591
511499231,11cyclictest6437-21timerwakeupswit21:35:261
511499231,11cyclictest23437-21expr23:30:121
511499231,11cyclictest17614-21cpu20:50:111
510999232,3cyclictest8117-21kworker/0:100:39:190
510999232,13cyclictest26561-21kworker/0:322:28:370
307942231,18sleep031005-21expr21:20:130
65292221,2sleep20-21swapper/221:35:282
512599225,3cyclictest22972grep23:55:133
512599221,13cyclictest0-21swapper/300:25:253
512199227,3cyclictest0-21swapper/220:13:312
512199226,4cyclictest26173-21diskmemload00:07:262
512199226,4cyclictest26173-21diskmemload00:07:262
512199226,4cyclictest0-21swapper/221:20:252
512199225,6cyclictest0-21swapper/219:15:012
512199225,5cyclictest1-21systemd23:00:012
512199223,8cyclictest12050irq/123-ahci[0020:55:012
512199221,8cyclictest87350irq/126-enp1s0-19:40:012
512199221,11cyclictest4407-21/usr/sbin/munin20:20:162
511499228,3cyclictest16816-21if_err_enp2s022:00:181
511499221,9cyclictest17276-21munin-node00:30:121
511499221,8cyclictest3746-21chrt21:30:201
5114992213,2cyclictest0-21swapper/122:51:341
5114992213,2cyclictest0-21swapper/122:51:341
5114992212,2cyclictest21-21rcuc/100:00:181
5114992212,2cyclictest21-21rcuc/100:00:181
511499221,19cyclictest0-21swapper/120:25:231
511499221,19cyclictest0-21swapper/120:12:161
511499221,10cyclictest0-21swapper/120:40:111
511499220,20cyclictest0-21swapper/121:00:061
511499220,20cyclictest0-21swapper/120:45:021
511499220,19cyclictest221ktimersoftd/121:50:151
511499220,14cyclictest0-21swapper/119:46:301
510999220,5cyclictest0-21swapper/021:30:190
512599214,7cyclictest26173-21diskmemload22:28:023
512599211,6cyclictest0-21swapper/323:29:443
512599211,6cyclictest0-21swapper/323:11:093
512599211,6cyclictest0-21swapper/322:01:283
5125992111,8cyclictest0-21swapper/322:30:363
512599211,17cyclictest8488-21/usr/sbin/munin00:10:133
512199218,3cyclictest0-21swapper/222:52:572
512199218,3cyclictest0-21swapper/222:52:572
512199216,9cyclictest0-21swapper/200:00:012
512199216,4cyclictest15576-21chrt21:57:262
512199216,3cyclictest0-21swapper/220:45:552
512199214,9cyclictest12050irq/123-ahci[0022:40:022
512199213,5cyclictest32-21ksoftirqd/222:25:132
512199211,7cyclictest0-21swapper/200:16:172
512199211,6cyclictest29514-21memory21:15:192
512199210,11cyclictest0-21swapper/223:31:532
511499215,14cyclictest0-21swapper/122:25:491
511499212,7cyclictest0-21swapper/121:15:121
510999213,3cyclictest0-21swapper/023:31:540
510999211,12cyclictest0-21swapper/022:13:340
510999211,11cyclictest19290-21ntp_states22:05:220
510999211,11cyclictest12593-21diskstats23:05:130
510999211,11cyclictest11007-21/usr/sbin/munin20:35:190
510999210,4cyclictest0-21swapper/022:34:130
5125992013,4cyclictest16990-21/usr/sbin/munin23:15:153
512599201,16cyclictest6376-21ntp_states19:10:213
512599201,11cyclictest6285-21/usr/sbin/munin00:05:293
512599201,11cyclictest6285-21/usr/sbin/munin00:05:293
512599201,11cyclictest17598-21/usr/sbin/munin20:50:263
512599201,11cyclictest1136-21sshd23:50:263
512599201,11cyclictest10396-21/usr/sbin/munin19:20:143
512599201,11cyclictest10396-21/usr/sbin/munin19:20:143
512599201,10cyclictest0-21swapper/323:30:213
512199208,3cyclictest0-21swapper/222:01:572
512199206,4cyclictest827-21runrttasks23:11:122
512199206,3cyclictest0-21swapper/220:01:032
512199204,2cyclictest89250irq/135-enp2s0-19:55:212
512199203,10cyclictest32-21ksoftirqd/200:20:242
512199201,9cyclictest0-21swapper/222:45:212
512199201,7cyclictest20115-21missed_timers19:40:182
512199201,6cyclictest4118-21cpu00:00:122
512199201,6cyclictest4118-21cpu00:00:122
512199201,6cyclictest0-21swapper/222:30:202
512199201,5cyclictest0-21swapper/222:23:022
512199201,12cyclictest0-21swapper/223:35:452
512199201,12cyclictest0-21swapper/200:37:592
512199201,11cyclictest0-21swapper/200:30:012
512199201,10cyclictest13200-21/usr/sbin/munin20:40:132
5121992010,2cyclictest87650irq/129-enp1s0-22:10:012
512199200,12cyclictest0-21swapper/220:29:482
5114992010,2cyclictest0-21swapper/123:16:171
5114992010,2cyclictest0-21swapper/119:31:461
510999209,3cyclictest26173-21diskmemload22:45:170
510999204,13cyclictest0-21swapper/023:40:220
510999203,15cyclictest18972-21perf19:40:020
510999202,6cyclictest0-21swapper/019:55:250
510999201,16cyclictest81ktimersoftd/021:15:230
510999201,11cyclictest12294-21sshd19:24:150
510999201,11cyclictest12294-21sshd19:24:140
512599192,5cyclictest8191-21diskstats22:55:143
5125991912,4cyclictest13178-21date20:40:023
512599191,12cyclictest0-21swapper/319:50:173
512599191,10cyclictest8815-21/usr/sbin/munin20:30:183
512599191,10cyclictest13820-21/usr/sbin/munin21:55:263
512599191,10cyclictest12592-21/usr/sbin/munin19:25:203
512199197,3cyclictest87650irq/129-enp1s0-00:30:172
512199196,6cyclictest87350irq/126-enp1s0-23:50:122
512199196,5cyclictest12050irq/123-ahci[0021:15:022
512199193,5cyclictest30-21rcuc/200:13:312
512199192,8cyclictest808-21snmpd20:30:062
512199191,7cyclictest0-21swapper/223:15:432
512199191,6cyclictest19976-21tr23:20:222
512199191,6cyclictest14790-21/usr/sbin/munin19:30:252
512199191,6cyclictest1469-21expr22:40:132
512199191,5cyclictest10955-21memory23:00:182
512199191,12cyclictest0-21swapper/222:11:392
512199191,10cyclictest2832-21/usr/sbin/munin21:30:262
512199191,10cyclictest26173-21diskmemload23:27:232
510999195,5cyclictest0-21swapper/021:10:360
510999192,3cyclictest0-21swapper/000:21:540
510999192,14cyclictest21990-21grep19:45:170
510999192,11cyclictest0-21swapper/021:26:560
510999191,9cyclictest3445-21users23:55:250
510999191,6cyclictest0-21swapper/000:16:430
510999191,3cyclictest0-21swapper/023:01:250
510999191,10cyclictest25427-21/usr/sbin/munin23:35:260
510999191,10cyclictest18036-21ethtool00:30:170
510999191,10cyclictest0-21swapper/023:46:300
510999190,10cyclictest19792-21gltestperf19:40:160
512599182,5cyclictest23354-21munin-run19:50:013
512599181,8cyclictest22537-21latency_hist22:15:023
512599181,7cyclictest17320-21df19:35:133
512599181,6cyclictest2801-21cat21:30:013
512599181,6cyclictest0-21swapper/300:18:483
512599181,4cyclictest29486-21sed20:00:263
5125991813,3cyclictest30704-21grep20:05:123
5125991812,3cyclictest3444-21/usr/sbin/munin22:45:223
5125991812,3cyclictest26416-21gltestperf19:55:153
512599181,10cyclictest0-21swapper/321:50:233
512599181,10cyclictest0-21swapper/321:34:323
512599180,10cyclictest10373-21missed_timers21:45:213
512599180,10cyclictest10373-21missed_timers21:45:213
512199185,3cyclictest12050irq/123-ahci[0021:30:012
512199185,3cyclictest12050irq/123-ahci[0020:40:012
512199182,3cyclictest11464-21ntp_states19:20:212
512199182,3cyclictest11464-21ntp_states19:20:212
512199181,7cyclictest0-21swapper/221:40:372
512199181,5cyclictest12247-21/usr/sbin/munin23:05:172
512199181,4cyclictest3241-21awk20:15:222
510999188,3cyclictest0-21swapper/020:25:150
510999187,2cyclictest16394-21strings20:45:210
510999184,3cyclictest0-21swapper/023:22:210
510999183,6cyclictest21035-21/usr/sbin/munin23:25:260
510999183,6cyclictest0-21swapper/021:40:190
510999183,2cyclictest87350irq/126-enp1s0-23:10:060
510999182,14cyclictest15189-21perf20:45:010
510999181,6cyclictest13820-21gzip19:25:240
510999181,3cyclictest25439-21find21:05:240
510999181,14cyclictest31704-21grep20:05:220
510999181,14cyclictest27973-21/usr/sbin/munin20:00:230
510999181,10cyclictest0-21swapper/022:16:040
39682181,3sleep00-21swapper/020:17:240
89302171,2sleep30-21swapper/321:41:433
512599179,6cyclictest87350irq/126-enp1s0-23:02:393
512599178,2cyclictest32376-21/usr/sbin/munin20:10:173
512599177,2cyclictest5027-21/usr/sbin/munin21:35:193
512599176,3cyclictest87350irq/126-enp1s0-22:50:293
512599176,3cyclictest87350irq/126-enp1s0-22:50:293
512599175,3cyclictest24583-21crond22:20:013
512599171,9cyclictest0-21swapper/323:35:353
512599171,7cyclictest521-21sed22:35:263
512599171,7cyclictest31046-21expr21:20:143
512599171,7cyclictest20865-21ntp_states20:55:213
512599171,6cyclictest0-21swapper/323:48:543
512599171,6cyclictest0-21swapper/323:24:483
512599171,4cyclictest0-21swapper/322:23:093
5125991713,3cyclictest0-21swapper/320:46:393
512599171,15cyclictest0-21swapper/321:09:253
512599171,15cyclictest0-21swapper/300:36:093
512599170,15cyclictest0-21swapper/319:30:373
512199174,3cyclictest87350irq/126-enp1s0-21:45:192
512199174,3cyclictest87350irq/126-enp1s0-21:45:192
512199172,9cyclictest0-21swapper/221:05:132
512199171,5cyclictest30175-21/usr/sbin/munin20:05:202
512199171,5cyclictest12592-21/usr/sbin/munin19:25:152
510999178,6cyclictest5606-21unixbench_multi00:00:270
510999178,6cyclictest5606-21unixbench_multi00:00:270
510999175,3cyclictest12921-21cat21:50:280
510999174,5cyclictest0-21swapper/022:20:270
510999171,8cyclictest1585-21diskstats22:40:140
510999171,7cyclictest15081-21/usr/sbin/munin00:25:180
510999171,6cyclictest17690-21ntp_kernel_pll_23:15:200
510999171,4cyclictest5010-21grep20:20:140
510999171,3cyclictest23551-21unixbench_singl21:00:230
510999171,11cyclictest0-21swapper/000:05:170
510999171,11cyclictest0-21swapper/000:05:170
510999170,5cyclictest0-21swapper/000:13:280
252832170,2sleep00-21swapper/019:51:490
107292171,2sleep00-21swapper/020:34:300
512599161,6cyclictest0-21swapper/323:44:133
512599161,6cyclictest0-21swapper/320:40:163
512599161,5cyclictest0-21swapper/322:05:063
5125991612,3cyclictest0-21swapper/319:42:353
5125991612,3cyclictest0-21swapper/300:22:333
5125991611,3cyclictest5924-21grep20:20:243
5125991611,3cyclictest2440-21ls20:15:133
5125991611,3cyclictest21995-21/usr/sbin/munin21:00:153
512599161,12cyclictest808-21snmpd20:25:043
512599160,9cyclictest0-21swapper/321:15:133
5121991611,3cyclictest22899-21load21:00:162
5109991611,3cyclictest17598-21/usr/sbin/munin20:50:210
510999161,12cyclictest15397-21tail19:30:160
5109991610,4cyclictest0-21swapper/020:10:150
512599151,7cyclictest0-21swapper/300:30:123
512599151,6cyclictest0-21swapper/322:40:043
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional