You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-18 - 08:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Mar 18, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
211792239163,24sleep10-21swapper/119:06:371
213022236163,24sleep20-21swapper/219:08:162
212552208137,59sleep00-21swapper/019:07:360
214452204166,25sleep30-21swapper/319:09:503
2394521280,2sleep12155599cyclictest22:53:351
21565991091,106cyclictest0-21swapper/300:22:403
21565998986,2cyclictest0-21swapper/322:38:373
21565998481,2cyclictest0-21swapper/321:54:083
21565997976,2cyclictest0-21swapper/322:27:243
21565997976,2cyclictest0-21swapper/320:39:393
21565996562,2cyclictest0-21swapper/323:01:393
21565996361,1cyclictest0-21swapper/322:06:303
268472530,4sleep32156599cyclictest20:30:263
37662490,4sleep123-21ksoftirqd/100:35:121
2154999421,29cyclictest16800-21switchtime22:35:230
2154999411,25cyclictest17482-21fschecks_count21:25:120
2154999401,28cyclictest21471-21/usr/sbin/munin21:35:130
2154999393,25cyclictest28476-21munin-run23:05:010
2154999392,22cyclictest812-21snmpd00:35:250
2154999381,33cyclictest28665-21grep21:50:150
2154999381,25cyclictest25607-21kworker/u8:021:45:000
2154999374,26cyclictest26245-21sshd19:15:300
2154999371,21cyclictest30264-21/usr/sbin/munin21:55:230
2155599361,12cyclictest13924-21gltestperf22:30:161
2154999367,20cyclictest21748-21/usr/sbin/munin19:10:240
2154999361,31cyclictest29336-21/usr/sbin/munin00:20:170
2154999361,26cyclictest14157-21expr23:45:130
2154999361,25cyclictest18311-21/usr/sbin/munin23:55:220
2154999361,24cyclictest0-21swapper/020:45:130
2154999361,20cyclictest23565-21unixbench-2d22:50:240
2154999356,11cyclictest20887-21sshd21:30:330
2154999351,22cyclictest20854-21/usr/sbin/munin20:20:130
2154999351,21cyclictest25765-21grep22:55:250
2154999351,17cyclictest0-21swapper/022:00:210
2156199341,16cyclictest0-21swapper/223:13:162
2154999348,23cyclictest8590-21smart_sda23:30:230
2154999341,29cyclictest14879-21/usr/sbin/munin21:20:240
2154999341,21cyclictest26764-21chrt00:13:160
2154999341,21cyclictest2314-21/usr/sbin/munin22:05:270
2154999341,19cyclictest0-21swapper/022:12:130
2154999340,25cyclictest0-21swapper/023:05:160
2154999340,24cyclictest0-21swapper/020:00:250
2154999340,20cyclictest0-21swapper/021:45:170
2154999335,19cyclictest29163-21expr19:25:120
2154999333,17cyclictest771-21rngd23:22:210
2154999333,17cyclictest771-21rngd23:22:210
2154999329,20cyclictest1-21systemd21:05:340
2154999323,14cyclictest32413-21latency00:25:180
2154999322,27cyclictest7103-21munin-run23:30:020
2154999321,28cyclictest1687-21/usr/sbin/munin20:50:140
2154999321,28cyclictest15953-21sshd20:06:330
2154999321,19cyclictest23780-21ntpq00:05:200
2154999320,19cyclictest28015-21latency00:15:180
2154999320,17cyclictest0-21swapper/023:50:250
2156199318,8cyclictest2745-21ntp_offset20:50:202
2156199311,13cyclictest0-21swapper/221:30:172
21555993114,3cyclictest6428-21df_inode21:00:141
2155599311,12cyclictest0-21swapper/121:10:401
2154999316,22cyclictest11105-21/usr/sbin/munin19:55:230
2154999312,21cyclictest19892-21/usr/sbin/munin22:45:140
2154999311,22cyclictest0-21swapper/021:16:330
2154999310,23cyclictest0-21swapper/023:20:020
2154999310,21cyclictest0-21swapper/023:35:190
2156199301,15cyclictest0-21swapper/219:40:352
2156199301,13cyclictest0-21swapper/221:26:042
2156199301,13cyclictest0-21swapper/220:18:322
2156199301,12cyclictest12050-21missed_timers22:25:182
2154999307,20cyclictest3336-21crond00:35:010
2154999307,20cyclictest3336-21crond00:35:010
2154999306,17cyclictest0-21swapper/019:45:140
2154999304,18cyclictest771-21rngd22:15:190
2154999303,15cyclictest86750irq/128-enp1s0-23:40:150
2154999301,26cyclictest6084-21/usr/sbin/munin21:00:180
2154999301,26cyclictest27455-21/usr/sbin/munin20:35:150
2154999301,26cyclictest26835-21/usr/sbin/munin19:20:140
2154999301,26cyclictest15271-21idleruntime-cro22:35:010
2154999301,16cyclictest86550irq/126-enp1s0-22:40:260
21549993010,17cyclictest16463-21/usr/sbin/munin20:10:150
2154999300,20cyclictest86850irq/129-enp1s0-20:30:160
126862300,4sleep32156599cyclictest21:15:093
2155599295,12cyclictest0-21swapper/123:42:051
2154999291,25cyclictest7664-21/usr/sbin/munin19:50:250
2154999291,25cyclictest3268-21/usr/sbin/munin19:40:130
2154999291,25cyclictest31233-21/usr/sbin/munin19:30:120
2154999291,25cyclictest29648-21/usr/sbin/munin20:40:140
2154999291,25cyclictest23060-21/usr/sbin/munin20:25:240
2154999291,25cyclictest20516-21/usr/sbin/munin00:00:150
2154999291,25cyclictest19177-21tune2fs20:15:150
2154999291,25cyclictest10488-21/usr/sbin/munin21:10:120
2154999291,25cyclictest1048-21/usr/sbin/munin19:35:140
2154999291,24cyclictest8903-21/usr/sbin/munin22:20:120
2154999290,26cyclictest30894-21/usr/sbin/munin23:10:200
2154999281,24cyclictest3883-21/usr/sbin/munin20:55:150
2154999281,24cyclictest3883-21/usr/sbin/munin20:55:150
2154999281,24cyclictest11107-21/usr/sbin/munin22:25:140
2156199271,13cyclictest0-21swapper/219:35:412
2155599271,11cyclictest12276-21taskset20:00:121
2156599266,7cyclictest0-21swapper/300:35:143
2156199261,11cyclictest0-21swapper/223:32:122
2156199261,10cyclictest0-21swapper/219:55:152
2155599261,9cyclictest0-21swapper/121:32:271
2155599261,15cyclictest17460-21grep23:50:241
2156599252,2cyclictest88450irq/134-enp2s0-23:19:583
2156199256,3cyclictest10615-40dnf22:21:512
2156199251,9cyclictest10259-21diskmemload00:26:172
2156199251,8cyclictest0-21swapper/222:05:332
2156199251,12cyclictest0-21swapper/223:28:362
2156199251,11cyclictest0-21swapper/221:56:492
2156199251,10cyclictest10259-21diskmemload23:00:402
2155599257,2cyclictest0-21swapper/121:55:121
2156599242,3cyclictest41-21ksoftirqd/321:45:003
21565992412,3cyclictest12471-21awk21:15:013
2156199249,4cyclictest0-21swapper/222:45:222
2156199248,3cyclictest0-21swapper/219:54:212
2156199241,6cyclictest0-21swapper/223:42:072
21561992412,3cyclictest4032-21grep20:55:122
21561992412,3cyclictest4032-21grep20:55:122
2156199241,11cyclictest0-21swapper/221:09:022
2156199241,11cyclictest0-21swapper/220:06:362
2156199241,11cyclictest0-21swapper/219:32:412
2156199241,10cyclictest0-21swapper/219:17:242
21561992410,3cyclictest0-21swapper/221:01:402
2155599247,7cyclictest0-21swapper/122:45:201
2155599246,3cyclictest0-21swapper/120:25:181
2155599241,5cyclictest3485-21grep23:20:171
2155599241,5cyclictest3485-21grep23:20:171
2155599241,4cyclictest5800-21df19:45:111
2155599241,20cyclictest22095-21diskstats19:10:131
2155599241,20cyclictest15490-21unixbench_singl23:45:271
2155599241,13cyclictest27962-21sed23:00:261
2155599241,10cyclictest0-21swapper/119:17:001
2156599238,7cyclictest0-21swapper/322:00:223
2156599231,16cyclictest0-21swapper/320:50:133
21565992311,3cyclictest21572-21grep00:00:213
2156199239,3cyclictest0-21swapper/222:16:052
2156199238,4cyclictest5551-21awk00:40:012
2156199235,3cyclictest29968-21sed19:25:192
2156199231,13cyclictest0-21swapper/200:02:232
2156199231,11cyclictest0-21swapper/222:00:052
2156199231,11cyclictest0-21swapper/221:17:322
2156199231,11cyclictest0-21swapper/200:10:132
2156199231,10cyclictest1585-21sort23:15:182
2156199231,10cyclictest0-21swapper/221:45:092
2156199231,10cyclictest0-21swapper/221:10:032
2156199230,11cyclictest0-21swapper/200:06:292
2155599238,8cyclictest0-21swapper/122:16:541
2155599237,4cyclictest19674-21munin-run22:45:011
2155599236,3cyclictest0-21swapper/122:24:181
2155599231,4cyclictest26579-21taskset00:10:541
2155599231,3cyclictest26222-21diskstats21:45:151
2155599231,19cyclictest31011-21munin-run19:30:011
2155599231,10cyclictest0-21swapper/122:56:371
21555992310,3cyclictest14909-21tail20:05:151
2155599230,10cyclictest0-21swapper/123:55:201
2156599224,10cyclictest39-21rcuc/300:30:153
2156599224,10cyclictest39-21rcuc/300:30:153
21565992212,5cyclictest25607-21kworker/u8:022:21:503
2156199228,3cyclictest0-21swapper/223:58:292
2156199228,3cyclictest0-21swapper/220:03:162
2156199223,11cyclictest0-21swapper/219:20:242
2156199221,12cyclictest0-21swapper/222:55:202
2156199221,11cyclictest0-21swapper/219:10:562
2156199221,10cyclictest0-21swapper/223:54:412
2156199221,10cyclictest0-21swapper/221:35:292
2156199220,9cyclictest10259-21diskmemload22:11:412
2156199220,11cyclictest0-21swapper/222:50:282
2155599229,3cyclictest0-21swapper/120:40:521
2155599226,3cyclictest0-21swapper/123:05:141
2155599224,3cyclictest0-21swapper/119:36:521
2155599222,2cyclictest0-21swapper/121:35:231
21555992215,4cyclictest31950-21cut20:45:111
2155599221,3cyclictest5849-21threads22:10:251
21555992213,3cyclictest0-21swapper/121:16:151
2155599221,13cyclictest0-21swapper/120:34:121
2155599221,11cyclictest0-21swapper/123:10:341
2155599221,10cyclictest782-21in:imjournal19:55:011
2155599220,16cyclictest0-21swapper/119:20:351
2156599219,10cyclictest86550irq/126-enp1s0-22:18:323
2156599216,6cyclictest0-21swapper/321:39:153
2156599216,4cyclictest0-21swapper/322:43:323
2156599211,12cyclictest0-21swapper/321:21:323
2156599210,7cyclictest0-21swapper/322:34:343
2156599210,6cyclictest0-21swapper/323:24:493
2156599210,6cyclictest0-21swapper/323:24:493
2156199218,2cyclictest86650irq/127-enp1s0-20:40:102
2156199216,3cyclictest0-21swapper/222:40:042
2156199212,6cyclictest0-21swapper/222:35:192
2156199211,14cyclictest0-21swapper/220:35:362
2156199211,14cyclictest0-21swapper/220:20:522
2156199211,11cyclictest25320-21cpu20:30:122
2156199211,11cyclictest0-21swapper/221:24:492
2156199211,10cyclictest0-21swapper/220:46:002
2155599218,3cyclictest25632-21cat21:44:571
2155599218,1cyclictest0-21swapper/121:50:161
2155599217,3cyclictest0-21swapper/123:39:281
2155599215,8cyclictest0-21swapper/120:24:161
2155599214,3cyclictest0-21swapper/119:41:401
21555992117,3cyclictest0-21swapper/120:11:201
2155599211,19cyclictest0-21swapper/100:25:291
2155599211,17cyclictest0-21swapper/100:08:301
2155599211,15cyclictest0-21swapper/121:22:131
2155599211,11cyclictest32526-21grep22:00:131
2155599211,11cyclictest2680-21seq23:19:551
2155599211,11cyclictest19046-21latency_hist21:30:021
2155599211,11cyclictest18925-21expr20:15:121
118192211,3sleep312033-21latency_hist20:00:023
2156599208,4cyclictest0-21swapper/321:48:273
2156599207,3cyclictest8284-21kworker/3:119:35:063
2156599206,4cyclictest0-21swapper/323:37:513
2156599205,6cyclictest86550irq/126-enp1s0-20:45:193
2156599205,5cyclictest0-21swapper/321:25:233
2156599203,9cyclictest39-21rcuc/323:50:103
2156599202,6cyclictest9-21rcu_preempt00:19:243
2156599202,10cyclictest22096-21/usr/sbin/munin22:50:143
2156599201,9cyclictest32490-21missed_timers00:25:193
2156599201,11cyclictest25266-21sed22:55:183
2156599201,11cyclictest0-21swapper/321:56:283
2156599201,10cyclictest0-21swapper/323:45:213
21565992010,2cyclictest12050irq/123-ahci[0023:10:003
2156199204,9cyclictest771-21rngd20:26:182
2156199204,4cyclictest27454-21expr00:15:132
2156199200,11cyclictest0-21swapper/200:33:092
2156199200,11cyclictest0-21swapper/200:33:082
2155599209,3cyclictest31302-21latency_hist00:25:011
2155599204,3cyclictest86650irq/127-enp1s0-19:55:181
21555992016,3cyclictest0-21swapper/122:26:051
21555992013,4cyclictest29040-21users20:35:271
2155599201,18cyclictest0-21swapper/120:51:351
2155599201,17cyclictest0-21swapper/121:08:381
2155599201,13cyclictest0-21swapper/123:34:091
2155599201,10cyclictest0-21swapper/119:31:281
2155599201,10cyclictest0-21swapper/100:02:511
2155599200,18cyclictest0-21swapper/122:05:151
2155599200,10cyclictest0-21swapper/100:34:501
2155599200,10cyclictest0-21swapper/100:34:501
2156599199,2cyclictest401ktimersoftd/319:45:003
2156599198,3cyclictest30894-21/usr/sbin/munin23:10:143
2156599194,2cyclictest88550irq/135-enp2s0-20:45:003
2156599191,9cyclictest0-21swapper/323:55:123
21565991911,6cyclictest41-21ksoftirqd/319:15:023
2156599191,15cyclictest15725-21grep20:05:253
2156599191,11cyclictest31233-21/usr/sbin/munin19:30:153
2156599191,10cyclictest7329-21/usr/sbin/munin23:30:143
2156599191,10cyclictest26835-21/usr/sbin/munin19:20:133
2156599191,10cyclictest10259-21diskmemload23:25:183
2156599191,10cyclictest0-21swapper/322:13:493
2156199197,3cyclictest0-21swapper/221:44:292
2156199196,2cyclictest0-21swapper/222:30:262
2156199195,3cyclictest28407-21df21:50:132
2156199192,4cyclictest29932-21grep00:20:172
2156199191,4cyclictest11496-21crond23:40:012
21561991913,3cyclictest15027-21ls23:45:242
2155599199,2cyclictest29103-21latency_hist00:20:011
2155599195,4cyclictest16151-21gltestperf22:35:141
2155599191,12cyclictest0-21swapper/120:57:141
2155599191,12cyclictest0-21swapper/120:57:141
2155599191,10cyclictest6033-21load23:25:181
2156599187,4cyclictest0-21swapper/300:10:123
2156599181,8cyclictest12969-21load20:00:193
2156599181,7cyclictest23407-21diskstats20:25:153
2156599181,10cyclictest0-21swapper/322:46:143
2156599181,10cyclictest0-21swapper/321:32:563
2156599181,10cyclictest0-21swapper/300:08:343
2156599180,9cyclictest13243-21unixbench_multi23:40:243
2156199182,6cyclictest4698-40dnf23:22:332
2156199182,6cyclictest4698-40dnf23:22:332
2156199181,13cyclictest0-21swapper/223:05:082
2156199181,13cyclictest0-21swapper/220:11:402
2156199180,11cyclictest0-21swapper/219:45:152
2156599174,2cyclictest41-21ksoftirqd/320:20:193
2156599171,9cyclictest0-21swapper/319:25:173
2156599171,3cyclictest0-21swapper/320:59:233
2156599171,3cyclictest0-21swapper/320:59:233
21565991712,3cyclictest8817-21grep21:05:153
2156599171,14cyclictest0-21swapper/319:45:083
2156599161,7cyclictest0-21swapper/320:15:133
2156599161,5cyclictest0-21swapper/319:15:143
21565991611,3cyclictest8095-21cut19:50:143
2156599161,10cyclictest0-21swapper/321:00:153
2156599161,10cyclictest0-21swapper/320:11:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional