You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 04:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jan 10, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
157952230153,24sleep00-21swapper/019:06:080
159762221191,20sleep10-21swapper/119:08:311
157382203166,25sleep30-21swapper/319:05:283
139582201163,24sleep20-21swapper/219:05:022
1447221690,5sleep21619199cyclictest22:45:022
1324521550,2sleep00-21swapper/022:40:180
1857921460,4sleep31619599cyclictest20:25:173
247442960,4sleep3401ktimersoftd/319:25:233
70452530,5sleep21619199cyclictest23:40:152
182132500,1sleep30-21swapper/321:40:023
16191994412,10cyclictest26351-21kernelversion23:10:182
16191994316,17cyclictest509-21unixbench-2d22:10:232
16191994315,11cyclictest4904-21sleep21:10:012
16195994012,13cyclictest32453-21/usr/sbin/munin19:45:143
16191994015,13cyclictest21543-21grep23:00:162
16191993812,14cyclictest16009-21latency_hist21:35:012
16191993811,3cyclictest29902-21kworker/2:119:20:542
16191993715,11cyclictest32224-21latency_hist19:45:012
16191993710,10cyclictest13921-21needreboot23:55:182
16191993618,6cyclictest21182-21chrt21:45:142
16191993615,11cyclictest11856-21sshd22:37:002
1619199358,9cyclictest8054-21if_enp1s021:15:162
1619199358,9cyclictest8054-21if_enp1s021:15:162
16191993515,12cyclictest16109-21/usr/sbin/munin21:35:142
16191993512,12cyclictest9519-21acpi21:20:112
1619599341,20cyclictest0-21swapper/300:10:143
29092330,3sleep216167-21cyclictest21:05:022
16191993311,8cyclictest26166-21/usr/sbin/munin00:25:152
16191993311,12cyclictest28692-21expr00:30:132
1619599328,10cyclictest32070-21latency_hist23:25:023
1619599326,15cyclictest26870-21latency_hist22:00:013
16195993213,15cyclictest13575-21gltestperf23:55:143
16191993212,12cyclictest10490-21taskset23:49:462
16191993211,9cyclictest7520-21runrttasks22:27:432
16191993211,9cyclictest7520-21runrttasks22:27:432
16191993210,9cyclictest31120-21fschecks_time00:35:152
1618599321,19cyclictest25567-21cut21:55:141
16195993123,5cyclictest14305-21munin-run22:45:013
1619199318,14cyclictest0-21swapper/223:50:222
16191993113,15cyclictest17020-21head19:10:142
1618199312,13cyclictest28454-21munin-run20:50:000
1619599308,9cyclictest15142-21/usr/sbin/munin00:00:243
16195993021,6cyclictest16843-21cut22:50:123
16195993020,6cyclictest19013-40dnf00:06:003
16195993019,7cyclictest2675-21chrt21:04:543
1619199307,3cyclictest29902-21kworker/2:100:01:052
16191993013,14cyclictest24199-21unixbench_multi21:50:242
1619599297,14cyclictest26982-21/usr/sbin/munin00:25:163
16195992914,6cyclictest11848-21/usr/sbin/munin23:50:203
1619199298,12cyclictest0-21swapper/220:40:122
16191992921,5cyclictest24967-40dnf23:05:532
16191992921,5cyclictest20013-21grep20:30:132
16191992916,10cyclictest325-21/usr/sbin/munin23:25:152
16191992911,6cyclictest20895-21latency_hist23:00:022
1618599291,17cyclictest19152-21unixbench-2d20:25:251
1619599289,7cyclictest9512-21/usr/sbin/munin21:20:223
1619599289,16cyclictest22018-21tune2fs19:20:133
1619599289,15cyclictest27096-21/usr/sbin/munin22:00:123
1619599289,11cyclictest814-21snmpd22:30:023
1619599289,10cyclictest9616-21ntp_states23:45:203
16195992812,13cyclictest28185-21tr19:35:123
1619199289,11cyclictest20008-21chrt21:41:372
1619199287,10cyclictest0-21swapper/200:17:162
16191992821,4cyclictest9242-21diskstats20:05:132
16191992820,5cyclictest28868-21chrt19:35:172
16191992820,5cyclictest18035-21diskstats20:25:132
16191992819,6cyclictest987-21/usr/sbin/munin19:45:172
16191992819,6cyclictest31782-21/usr/sbin/munin20:55:182
16191992819,6cyclictest30107-21/usr/sbin/munin22:05:162
16191992819,6cyclictest15483-21/usr/sbin/munin20:20:102
16191992816,9cyclictest3604-21/usr/sbin/munin22:20:112
16191992815,10cyclictest12293-21/usr/sbin/munin21:25:142
16191992813,5cyclictest25926-21sshd00:25:002
1618599281,18cyclictest0-21swapper/120:05:121
1619599277,12cyclictest4172-21sshd19:53:593
16195992718,6cyclictest29278-21/usr/sbin/munin00:30:183
16195992714,10cyclictest21949-21/usr/sbin/munin23:00:183
1619199279,6cyclictest7936-21/usr/sbin/munin22:30:222
1619199278,13cyclictest0-21swapper/221:14:162
16191992720,4cyclictest5284-21/usr/sbin/munin19:55:172
16191992720,4cyclictest26546-21/usr/sbin/munin19:30:162
16191992719,5cyclictest30272-21diskstats23:20:142
16191992719,5cyclictest27899-21/usr/sbin/munin23:15:142
16191992719,5cyclictest27510-21/usr/sbin/munin20:45:212
16191992719,5cyclictest17705-21/usr/sbin/munin00:05:122
16191992719,5cyclictest14614-21/usr/sbin/munin22:45:132
16191992718,6cyclictest7738-21/usr/sbin/munin20:00:212
16191992718,6cyclictest7738-21/usr/sbin/munin20:00:212
16191992718,6cyclictest29013-21/usr/sbin/munin20:50:132
16191992718,6cyclictest24900-21/usr/sbin/munin21:55:152
16191992718,6cyclictest22083-21/usr/sbin/munin20:35:172
16191992718,6cyclictest20520-21/usr/sbin/munin00:10:172
16191992718,6cyclictest1961-21/usr/sbin/munin23:30:202
16191992718,6cyclictest1668-21/usr/sbin/munin22:15:122
16191992718,6cyclictest11090-21/usr/sbin/munin20:10:212
16191992716,7cyclictest5629-21grep23:35:232
16191992716,7cyclictest5629-21grep23:35:232
1618199272,12cyclictest31898-21munin-run23:25:000
1618199271,13cyclictest15885-21munin-run21:35:010
16195992619,4cyclictest11872-21cat20:10:163
16195992617,6cyclictest6349-21sendmail_mailqu21:10:233
16195992616,7cyclictest23516-21/usr/sbin/munin21:50:173
16195992613,10cyclictest6777-21fschecks_count23:40:143
16191992619,4cyclictest24249-21/usr/sbin/munin19:25:172
16191992618,5cyclictest14195-21load20:15:172
16191992617,6cyclictest27446-21dnf22:00:122
16191992617,6cyclictest17418-21/usr/sbin/munin22:50:172
16191992616,7cyclictest18859-21unixbench-2d19:15:022
16191992615,8cyclictest2949-21head19:50:152
1618599264,14cyclictest0-21swapper/123:40:231
1619599259,13cyclictest13908-21/usr/sbin/munin21:30:153
1619599258,14cyclictest30875-21/usr/sbin/munin20:55:113
1619599256,16cyclictest18929-21/usr/sbin/munin22:55:123
16195992517,5cyclictest8381-21ntp_states21:15:213
16195992517,5cyclictest8381-21ntp_states21:15:213
16195992517,5cyclictest25877-21diskstats23:10:133
16195992516,6cyclictest24104-21cpuspeed_turbos00:20:123
16195992516,6cyclictest20506-21/usr/sbin/munin21:45:223
16195992516,6cyclictest18309-21/usr/sbin/munin21:40:173
16195992516,6cyclictest16282-21/usr/sbin/munin20:20:153
16195992516,5cyclictest28800-21ntp_states23:15:233
16195992515,7cyclictest24120-21wc20:40:013
16195992514,9cyclictest6475-21awk20:00:013
16195992512,10cyclictest5921-21taskset23:37:283
16195992512,10cyclictest5921-21taskset23:37:283
1618599258,4cyclictest12941-21fgrep21:25:211
1618599253,7cyclictest4891-21diskmemload23:15:111
1618599251,5cyclictest0-21swapper/123:26:021
16185992514,2cyclictest0-21swapper/119:20:141
1618599251,16cyclictest0-21swapper/123:10:201
1618199255,6cyclictest0-21swapper/021:11:180
1618199253,7cyclictest87150irq/126-enp1s0-00:25:000
1619599243,12cyclictest0-21swapper/322:35:233
16195992417,4cyclictest3877-21df22:20:123
16195992416,6cyclictest15702-21/usr/sbin/munin22:45:223
16195992416,5cyclictest6086-21diskstats22:25:133
16195992416,5cyclictest6086-21diskstats22:25:133
16195992416,5cyclictest2307-21diskstats23:30:153
16195992415,6cyclictest7382-21/usr/sbin/munin20:00:183
16195992415,6cyclictest7382-21/usr/sbin/munin20:00:183
16195992415,6cyclictest2923-21/usr/sbin/munin21:05:153
16195992415,6cyclictest28677-21/usr/sbin/munin20:50:223
16195992415,6cyclictest26200-21/usr/sbin/munin19:30:133
16195992415,6cyclictest21776-21/usr/sbin/munin00:15:163
16195992414,7cyclictest21344-21cat19:20:013
16195992414,7cyclictest13898-21head20:15:153
16195992413,8cyclictest24400-21tr20:40:103
16195992412,7cyclictest860-21grep23:25:233
16185992415,3cyclictest0-21swapper/121:37:181
1618599241,16cyclictest0-21swapper/121:07:121
1618599241,13cyclictest5195-21kworker/u8:222:36:421
1618599241,10cyclictest22627-21cat00:15:181
1618599240,2cyclictest0-21swapper/123:50:191
16195992317,4cyclictest20230-21df_inode20:30:143
16195992316,4cyclictest31843-21df_inode22:10:123
16195992315,5cyclictest26873-21tune2fs20:45:153
16195992315,5cyclictest17634-21/usr/sbin/munin19:10:223
16195992314,6cyclictest9526-21sed20:05:163
16195992314,6cyclictest30627-21fschecks_count19:40:133
16195992314,6cyclictest29650-21df_inode22:05:123
16195992314,6cyclictest11710-21/usr/sbin/munin21:25:233
16195992314,5cyclictest24682-21sort23:05:233
16195992312,8cyclictest31180-21head00:35:143
1618599233,11cyclictest15323-21latency_hist20:20:011
1618599231,9cyclictest29639-21kworker/u8:223:20:431
1618599231,9cyclictest14963-21ntpq21:30:201
16185992315,3cyclictest0-21swapper/100:26:441
1618599231,4cyclictest23606-21kworker/u8:200:23:391
1618599231,3cyclictest4350-21irqstats22:20:171
1618599231,3cyclictest0-21swapper/121:10:131
1618599231,16cyclictest0-21swapper/122:13:111
1618599231,16cyclictest0-21swapper/120:58:521
1618599231,15cyclictest0-21swapper/123:07:351
1618599231,12cyclictest4809-21cut23:35:141
1618599231,12cyclictest4809-21cut23:35:141
1618599230,4cyclictest21256-21kworker/u8:100:14:381
1618599230,17cyclictest12409-21kworker/u8:000:30:411
1618599230,17cyclictest0-21swapper/121:42:311
1618199235,9cyclictest768-21rngd23:15:110
1618199233,13cyclictest0-21swapper/021:59:590
1618199231,6cyclictest5522-21awk22:25:010
1618199231,6cyclictest0-21swapper/023:36:550
1618199231,6cyclictest0-21swapper/023:36:550
1618199231,5cyclictest18730-21timerandwakeup00:05:250
1618199230,11cyclictest30665-21perf20:55:000
16195992211,8cyclictest1432-21cpuspeed_turbos22:15:113
1618599229,3cyclictest21854-21latency_hist20:35:021
1618599222,2cyclictest13545-21kworker/u8:021:47:501
1618599221,8cyclictest0-21swapper/121:52:401
1618599221,5cyclictest0-21swapper/122:58:031
1618599221,5cyclictest0-21swapper/122:50:391
16185992214,2cyclictest0-21swapper/122:03:031
16185992213,3cyclictest0-21swapper/121:03:041
16185992213,3cyclictest0-21swapper/120:54:541
1618599221,11cyclictest10527-21munin-run23:50:011
1618599221,10cyclictest24006-21chrt20:39:331
1618599221,10cyclictest23233-21cat23:05:021
1618599221,10cyclictest20333-21ntp_states19:15:201
1618199224,7cyclictest0-21swapper/022:09:090
1618199222,13cyclictest0-21swapper/000:02:520
1618199222,12cyclictest768-21rngd23:32:110
16181992212,2cyclictest12409-21kworker/u8:023:59:040
1618599211,7cyclictest10803-21switchtime21:20:221
16185992114,1cyclictest0-21swapper/119:25:271
1618599211,3cyclictest5195-21kworker/u8:222:32:421
1618599211,18cyclictest0-21swapper/120:11:531
1618599211,18cyclictest0-21swapper/119:40:091
1618599211,15cyclictest0-21swapper/122:09:431
1618599211,15cyclictest0-21swapper/121:16:461
1618599211,15cyclictest0-21swapper/121:16:461
1618599211,15cyclictest0-21swapper/119:51:231
1618599211,15cyclictest0-21swapper/119:46:151
1618599211,15cyclictest0-21swapper/100:38:131
16185992111,3cyclictest0-21swapper/122:42:071
16185992111,3cyclictest0-21swapper/120:44:211
1618599211,11cyclictest6653-21memory22:25:181
1618599211,11cyclictest6653-21memory22:25:181
1618599211,11cyclictest15753-21cat20:20:121
1618599211,11cyclictest13104-21cut23:55:121
1618599211,10cyclictest18430-21ntp_states00:05:211
1618599211,10cyclictest0-21swapper/100:04:181
1618599210,3cyclictest2974-21kworker/u8:022:45:581
1618599210,15cyclictest0-21swapper/122:19:011
1618599210,15cyclictest0-21swapper/120:03:241
1618599210,15cyclictest0-21swapper/120:03:241
1618599210,15cyclictest0-21swapper/119:38:441
1618599210,15cyclictest0-21swapper/119:31:201
1618199217,3cyclictest0-21swapper/022:16:380
1618199216,3cyclictest0-21swapper/022:30:220
1618199215,9cyclictest26853-21kworker/u8:122:01:170
1618199212,9cyclictest8986-21kworker/u8:221:25:420
1618199211,8cyclictest768-21rngd21:49:580
1618199211,6cyclictest0-21swapper/021:40:260
1618199211,5cyclictest27630-21grep00:25:230
1618199211,4cyclictest0-21swapper/022:13:330
1618199211,3cyclictest0-21swapper/022:56:470
1618199211,11cyclictest11732-21kworker/u8:119:35:270
1618199211,10cyclictest0-21swapper/000:13:530
1618199210,3cyclictest7594-21sendmail_mailqu23:40:230
1618599201,17cyclictest0-21swapper/120:45:171
1618599200,19cyclictest0-21swapper/123:30:261
1618199203,8cyclictest0-21swapper/021:21:230
1618199202,10cyclictest4891-21diskmemload22:50:490
1618199201,9cyclictest768-21rngd23:09:230
1618199201,7cyclictest22023-21expr00:15:120
1618199201,7cyclictest0-21swapper/023:01:000
1618199201,3cyclictest0-21swapper/023:13:300
1618199201,2cyclictest0-21swapper/021:50:140
1618199201,2cyclictest0-21swapper/000:39:440
1618199201,11cyclictest5739-21/usr/sbin/munin22:25:200
1618199201,11cyclictest5739-21/usr/sbin/munin22:25:190
1618199201,11cyclictest24280-21/usr/sbin/munin20:40:140
1618199201,11cyclictest0-21swapper/021:35:200
1618199201,10cyclictest8386-21ntp_states21:15:200
1618199201,10cyclictest8386-21ntp_states21:15:200
16181992010,2cyclictest12409-21kworker/u8:000:32:080
1618199200,3cyclictest0-21swapper/023:45:320
1618599191,8cyclictest17047-21gltestperf19:10:141
1618199194,4cyclictest17044-21gltestperf19:10:150
1618199192,5cyclictest20949-21ntp_states20:30:210
1618199192,2cyclictest87150irq/126-enp1s0-22:46:130
1618199191,9cyclictest31224-21diskstats20:55:130
1618199191,5cyclictest1064-21strings19:45:180
1618199191,12cyclictest87150irq/126-enp1s0-21:10:000
1618199191,12cyclictest0-21swapper/023:52:030
1618199191,10cyclictest814-21snmpd19:54:330
1618199191,10cyclictest4891-21diskmemload22:37:200
1618199191,10cyclictest32122-21/usr/sbin/munin23:25:160
1618199191,10cyclictest19271-21/usr/sbin/munin19:15:240
233672181,2sleep00-21swapper/019:24:150
16185991815,2cyclictest0-21swapper/119:57:141
1618199185,3cyclictest16722-21find20:20:220
1618199181,4cyclictest0-21swapper/019:55:130
16181991812,3cyclictest6686-21/usr/sbin/munin20:00:170
16181991812,3cyclictest6686-21/usr/sbin/munin20:00:170
1618199181,15cyclictest0-21swapper/020:36:480
1618199181,14cyclictest13877-21grep20:15:160
1618199178,2cyclictest87150irq/126-enp1s0-19:30:230
1618199171,7cyclictest23667-21/usr/sbin/munin19:25:150
1618199171,3cyclictest0-21swapper/020:05:140
1618199171,2cyclictest0-21swapper/020:30:020
1618199171,10cyclictest0-21swapper/021:00:240
130512170,2sleep00-21swapper/020:15:010
1618199161,7cyclictest0-21swapper/019:40:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional