You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-31 - 11:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun May 31, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
66242239162,25sleep00-21swapper/019:06:570
66242239162,25sleep00-21swapper/019:06:570
65792225153,22sleep20-21swapper/219:06:232
65792225153,22sleep20-21swapper/219:06:232
66002213181,21sleep10-21swapper/119:06:381
66002213181,21sleep10-21swapper/119:06:381
66872200165,23sleep30-21swapper/319:07:453
66872200165,23sleep30-21swapper/319:07:443
2468521790,3sleep2695699cyclictest22:15:132
123222770,10sleep339-21rcuc/300:15:023
71192500,4sleep121-21rcuc/100:00:231
6962994513,18cyclictest18286-21ls23:15:023
696299409,17cyclictest4827-21pmu-power23:55:233
696299409,17cyclictest4827-21pmu-power23:55:233
6962994016,13cyclictest32406-21munin-run21:20:013
6962994016,13cyclictest10029-21diskstats22:55:143
6962994016,10cyclictest808-21snmpd00:00:403
6962994015,13cyclictest19214-21perf20:50:023
6962994012,15cyclictest18486-21/usr/sbin/munin23:15:133
696299398,14cyclictest6280-21grep22:45:203
696299398,14cyclictest6280-21grep22:45:203
696299398,11cyclictest0-21swapper/300:25:253
6962993914,11cyclictest21298-21fgrep22:05:203
6962993811,10cyclictest31420-21cut22:30:153
696299374,23cyclictest0-21swapper/321:55:573
6962993714,8cyclictest6026-21perf20:20:003
696299365,22cyclictest87350irq/126-enp1s0-23:21:303
6962993615,10cyclictest3282-21irqstats21:25:193
6962993613,14cyclictest2227-21/usr/sbin/munin22:35:253
6962993613,11cyclictest31785-21latency_hist20:05:003
696299355,17cyclictest11444-21cstates21:45:133
6962993515,6cyclictest808-21snmpd21:50:173
6962993514,10cyclictest12538-21/usr/sbin/munin00:15:283
6962993513,4cyclictest808-21snmpd22:20:223
696299349,11cyclictest6133-21grep21:30:243
6962993415,11cyclictest32010-21df23:45:133
6962993413,12cyclictest14730-21/usr/sbin/munin00:20:113
6962993413,10cyclictest25064-21cat23:30:013
695699341,5cyclictest30-21rcuc/223:50:232
6962993322,8cyclictest22279-21smart_sda19:40:233
6962993312,8cyclictest12004-21latency_hist19:20:023
6962993312,7cyclictest8662-21fschecks_time00:05:153
6962993312,12cyclictest32016-21/usr/sbin/munin20:05:213
694599334,13cyclictest8839-21munin-run21:40:010
694599334,13cyclictest8839-21munin-run21:40:010
696299325,18cyclictest8731-21kworker/u8:022:00:023
696299324,17cyclictest25960-21latency23:30:173
6962993215,9cyclictest10127-21ntp_states21:40:203
6962993215,5cyclictest12050irq/123-ahci[0019:55:013
6962993214,5cyclictest9296-21cat19:15:023
6962993213,7cyclictest17006-21latency_hist20:45:023
6962993211,17cyclictest28397-21pmu-power23:35:223
6962993211,12cyclictest21322-21/usr/sbin/munin00:35:163
6962993211,11cyclictest16800-21cstates19:30:133
6962993210,13cyclictest18822-21/usr/sbin/munin19:35:213
694599324,13cyclictest31797-21munin-run20:05:010
696299318,9cyclictest12624-21awk20:35:013
6962993115,7cyclictest2926-21ntp_states20:10:203
6962993114,14cyclictest20195-21/usr/sbin/munin00:30:213
6962993112,10cyclictest23605-21munin-run21:00:023
6962993111,17cyclictest11675-21kworker/u8:121:10:153
6962993110,18cyclictest23215-21/usr/sbin/munin19:45:213
6962993022,5cyclictest7542-21hddtemp_smartct21:35:153
6962993022,5cyclictest7542-21hddtemp_smartct21:35:153
696299302,21cyclictest0-21swapper/323:40:163
6962993016,11cyclictest14763-21hddtemp_smartct23:05:173
695199303,3cyclictest12050irq/123-ahci[0021:50:021
694599305,12cyclictest12023-21idleruntime-cro19:20:010
6945993011,16cyclictest14819-21munin-run20:40:000
696299299,7cyclictest12797-21memory23:00:193
696299298,8cyclictest0-21swapper/322:25:123
696299298,17cyclictest22260-21/usr/sbin/munin22:10:263
6962992911,6cyclictest26037-21/usr/sbin/munin21:05:253
6962992911,4cyclictest12050irq/123-ahci[0020:00:003
6962992910,11cyclictest87350irq/126-enp1s0-22:40:143
6962992910,11cyclictest808-21snmpd21:03:333
6945992912,13cyclictest5681-40updatedb00:00:010
6945992912,13cyclictest5681-40updatedb00:00:010
696299287,8cyclictest0-21swapper/319:25:023
696299287,10cyclictest2587-21sh23:50:223
696299284,13cyclictest0-21swapper/322:50:163
6962992819,5cyclictest20495-21ntp_states20:50:213
695199288,3cyclictest6024-21munin-run20:20:011
695199281,6cyclictest0-21swapper/122:12:261
6962992718,6cyclictest25996-21/usr/sbin/munin22:15:263
6962992710,6cyclictest808-21snmpd21:24:353
695699273,6cyclictest30845-21cut21:15:182
695199271,17cyclictest0-21swapper/121:13:461
6962992619,4cyclictest10617-21sed20:30:023
6962992618,5cyclictest15605-21/usr/sbin/munin19:25:223
6962992617,6cyclictest12846-21/usr/sbin/munin20:35:143
6962992613,10cyclictest7517-21/usr/sbin/munin20:20:243
6962992613,10cyclictest7517-21/usr/sbin/munin20:20:243
695199261,4cyclictest0-21swapper/122:02:201
695199261,15cyclictest0-21swapper/100:35:261
695199261,11cyclictest2863-21perf22:44:551
695199260,4cyclictest0-21swapper/121:29:381
695199260,18cyclictest0-21swapper/100:05:341
694599261,4cyclictest0-21swapper/022:43:420
695199251,13cyclictest26045-21users22:15:261
695199251,12cyclictest16969-21df_inode19:30:131
695699249,4cyclictest12050irq/123-ahci[0022:40:012
695699241,6cyclictest0-21swapper/223:19:242
695699241,5cyclictest0-21swapper/223:58:452
695699241,5cyclictest0-21swapper/223:58:452
695699241,4cyclictest0-21swapper/222:20:142
695699240,9cyclictest0-21swapper/219:34:142
695199244,9cyclictest15487-21if_enp2s000:20:161
695199242,14cyclictest18616-21sshd00:26:341
695199242,11cyclictest2239-21timerandwakeup22:35:251
695199241,20cyclictest31965-21memory22:30:191
695199241,20cyclictest20047-21gltestperf20:50:141
695199241,11cyclictest917-21hddtemp_smartct21:20:161
695199241,11cyclictest12872-21df00:15:141
695199241,10cyclictest0-21swapper/122:51:271
6951992410,3cyclictest2194-21df_inode20:10:131
695199240,16cyclictest0-21swapper/123:56:351
695199240,16cyclictest0-21swapper/123:56:351
695199240,11cyclictest0-21swapper/121:50:331
6945992414,3cyclictest808-21snmpd21:40:150
76042230,1sleep00-21swapper/019:10:130
695699238,5cyclictest32-21ksoftirqd/221:34:202
695699238,4cyclictest0-21swapper/200:28:032
695199239,3cyclictest0-21swapper/123:25:121
695199236,3cyclictest16565-21cstates23:10:121
6951992317,3cyclictest6969-21cpuspeed_turbos21:35:121
6951992317,3cyclictest6969-21cpuspeed_turbos21:35:121
6951992312,2cyclictest0-21swapper/123:40:141
695199231,19cyclictest15498-21ntp_states19:25:211
695199231,14cyclictest0-21swapper/122:05:381
695199230,16cyclictest0-21swapper/123:54:431
695199230,13cyclictest0-21swapper/123:36:391
694599235,9cyclictest28096-21ls22:20:250
694599234,3cyclictest12050irq/123-ahci[0022:00:020
695699229,5cyclictest12050irq/123-ahci[0000:35:022
695699227,4cyclictest32-21ksoftirqd/223:10:012
695699223,9cyclictest32-21ksoftirqd/200:15:022
695199229,2cyclictest5272-21taskset22:45:041
695199229,2cyclictest5272-21taskset22:45:041
695199221,6cyclictest0-21swapper/120:34:071
695199221,5cyclictest0-21swapper/122:21:561
695199221,3cyclictest18859-21taskset20:45:431
695199221,19cyclictest0-21swapper/123:00:191
695199221,18cyclictest15682-21gltestperf20:40:151
695199221,10cyclictest0-21swapper/121:33:411
695199220,4cyclictest0-21swapper/122:57:001
695199220,16cyclictest0-21swapper/122:27:001
694599221,14cyclictest0-21swapper/021:21:410
285682220,5sleep011-21rcuc/021:10:140
695699216,3cyclictest32-21ksoftirqd/223:35:122
695699214,8cyclictest311ktimersoftd/220:20:012
695699212,13cyclictest87450irq/127-enp1s0-00:20:022
695199214,3cyclictest30662-21cstates21:15:151
695199211,18cyclictest0-21swapper/121:04:221
695199211,18cyclictest0-21swapper/120:35:181
695199211,18cyclictest0-21swapper/120:00:181
695199211,18cyclictest0-21swapper/119:10:351
695199211,10cyclictest31771-21tr23:45:131
694599217,9cyclictest0-21swapper/023:37:290
694599217,8cyclictest0-21swapper/020:11:330
694599211,6cyclictest0-21swapper/000:17:340
694599211,5cyclictest0-21swapper/021:25:140
6945992114,4cyclictest19192-21uniq19:35:140
694599211,2cyclictest87550irq/128-enp1s0-23:28:210
6945992112,4cyclictest0-21swapper/020:30:240
6945992112,2cyclictest10387-21idleruntime22:55:170
694599211,12cyclictest0-21swapper/021:50:180
193402210,1sleep10-21swapper/100:30:131
695699209,2cyclictest87350irq/126-enp1s0-19:55:012
695699208,4cyclictest87350irq/126-enp1s0-21:25:192
695699208,2cyclictest87350irq/126-enp1s0-21:55:182
695699207,5cyclictest0-21swapper/221:47:332
695699204,3cyclictest30-21rcuc/223:10:242
695699203,4cyclictest32-21ksoftirqd/223:25:132
695699202,6cyclictest87550irq/128-enp1s0-21:40:232
695699202,3cyclictest87550irq/128-enp1s0-21:15:012
695699202,2cyclictest87350irq/126-enp1s0-23:33:442
695699202,10cyclictest0-21swapper/223:20:132
695699201,8cyclictest0-21swapper/223:00:252
695699201,6cyclictest0-21swapper/222:55:192
695699201,4cyclictest0-21swapper/223:44:002
695699201,11cyclictest0-21swapper/222:13:182
6956992010,2cyclictest87350irq/126-enp1s0-21:52:482
695199209,3cyclictest15351-21smart_sda23:05:231
695199204,7cyclictest0-21swapper/123:22:081
6951992016,3cyclictest0-21swapper/119:20:191
6951992015,3cyclictest25372-21sort19:50:021
695199201,4cyclictest27400-21awk19:55:021
695199201,18cyclictest0-21swapper/119:40:191
695199201,12cyclictest0-21swapper/123:17:101
695199201,12cyclictest0-21swapper/100:10:341
694599209,2cyclictest6031-21awk20:20:000
694599207,3cyclictest8124-21kworker/0:221:00:150
694599201,5cyclictest6524-21cstates20:20:120
694599201,5cyclictest6524-21cstates20:20:120
694599201,5cyclictest0-21swapper/000:21:270
694599201,4cyclictest9572-21sed00:05:250
6945992012,5cyclictest19512-21fschecks_count00:30:140
694599201,13cyclictest0-21swapper/023:45:380
694599201,12cyclictest0-21swapper/023:22:290
694599201,11cyclictest0-21swapper/023:32:370
694599201,11cyclictest0-21swapper/022:07:220
694599200,10cyclictest19476-21strings23:15:210
278972200,8sleep07-21ksoftirqd/019:55:130
695699196,5cyclictest87650irq/129-enp1s0-21:20:152
695699195,4cyclictest29532-21sshd19:59:342
695699193,9cyclictest0-21swapper/221:35:022
695699193,9cyclictest0-21swapper/221:35:022
695699193,5cyclictest0-21swapper/200:08:502
695699191,8cyclictest0-21swapper/222:32:502
695699191,7cyclictest19582-21grep20:50:122
695699191,7cyclictest0-21swapper/222:50:112
6956991912,2cyclictest87350irq/126-enp1s0-22:08:552
695699190,8cyclictest87350irq/126-enp1s0-00:20:262
695699190,4cyclictest28565-21kworker/u8:122:27:142
695199195,2cyclictest0-21swapper/120:55:241
695199191,7cyclictest10036-21/usr/sbin/munin19:15:241
695199191,3cyclictest0-21swapper/121:56:391
695199191,17cyclictest0-21swapper/121:08:281
695199191,16cyclictest0-21swapper/120:26:461
695199190,18cyclictest0-21swapper/123:32:131
694599199,4cyclictest0-21swapper/000:26:330
694599195,3cyclictest87350irq/126-enp1s0-19:25:210
694599194,12cyclictest11-21rcuc/019:50:200
694599193,8cyclictest827-21runrttasks22:50:510
694599191,9cyclictest6091-21irqstats22:45:180
694599191,9cyclictest6091-21irqstats22:45:170
694599191,7cyclictest0-21swapper/021:45:150
694599191,6cyclictest31312-21expr22:30:120
694599191,6cyclictest30780-21df_inode21:15:170
694599191,6cyclictest28932-21cpuspeed_turbos22:25:110
694599191,6cyclictest0-21swapper/022:12:450
694599191,6cyclictest0-21swapper/000:01:520
694599191,4cyclictest21669-21diskstats00:35:140
694599191,4cyclictest18196-21df_inode22:00:150
694599191,3cyclictest0-21swapper/023:06:320
6945991912,2cyclictest0-21swapper/019:30:280
6945991911,5cyclictest16926-21tail23:10:150
6945991911,2cyclictest0-21swapper/021:05:250
694599191,11cyclictest1899-21fschecks_count23:50:160
694599191,11cyclictest0-21swapper/021:30:190
694599191,10cyclictest845-21/usr/sbin/munin22:35:190
694599191,10cyclictest24458-21/usr/sbin/munin22:15:140
694599191,10cyclictest0-21swapper/000:10:130
212132190,2sleep00-21swapper/020:52:400
695699185,5cyclictest87650irq/129-enp1s0-21:05:142
695699182,6cyclictest0-21swapper/222:02:472
695699182,14cyclictest31789-21ls20:05:012
695699181,8cyclictest0-21swapper/220:20:222
695699181,8cyclictest0-21swapper/220:20:222
695699181,7cyclictest0-21swapper/220:30:202
695699181,6cyclictest1853-21/usr/sbin/munin20:10:112
695699181,6cyclictest0-21swapper/222:48:222
695699181,6cyclictest0-21swapper/222:48:222
695699181,6cyclictest0-21swapper/219:40:202
695699181,4cyclictest0-21swapper/219:15:012
6956991813,3cyclictest7659-21taskset00:02:022
6956991811,4cyclictest18226-21needreboot20:45:202
695699181,10cyclictest0-21swapper/219:35:212
695699180,8cyclictest87350irq/126-enp1s0-19:45:242
695699180,5cyclictest30-21rcuc/223:45:142
695199185,12cyclictest1912-21kworker/u8:220:20:351
695199185,12cyclictest1912-21kworker/u8:220:20:351
695199182,13cyclictest19912-21/usr/sbin/munin19:35:211
695199181,9cyclictest0-21swapper/121:40:501
695199181,16cyclictest0-21swapper/120:05:051
695199181,16cyclictest0-21swapper/119:56:441
694599187,2cyclictest0-21swapper/019:20:170
694599184,9cyclictest0-21swapper/019:40:250
694599183,3cyclictest87550irq/128-enp1s0-23:40:150
694599182,4cyclictest0-21swapper/020:40:220
694599181,6cyclictest23238-21cpu19:45:120
694599181,4cyclictest8450-21/usr/sbin/munin20:25:230
6945991812,3cyclictest327-21if_err_enp1s020:05:180
694599181,11cyclictest0-21swapper/023:02:530
146882181,2sleep20-21swapper/220:38:342
695699176,2cyclictest30-21rcuc/200:35:102
695699171,5cyclictest0-21swapper/219:25:262
6956991714,2cyclictest32-21ksoftirqd/221:00:022
6956991712,3cyclictest12235-21/usr/sbin/munin19:20:152
695699171,10cyclictest0-21swapper/222:41:182
695699170,15cyclictest0-21swapper/220:28:022
694599175,3cyclictest87550irq/128-enp1s0-20:45:250
694599172,4cyclictest827-21runrttasks20:55:170
117392171,2sleep20-21swapper/219:16:422
695699164,2cyclictest12050irq/123-ahci[0020:05:172
695699163,3cyclictest87350irq/126-enp1s0-20:40:192
695699151,2cyclictest0-21swapper/221:00:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional