You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 12:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Apr 28, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
219832236163,25sleep00-21swapper/019:07:570
219022214163,39sleep30-21swapper/319:06:563
218532202164,25sleep20-21swapper/219:06:182
219202201166,23sleep10-21swapper/119:07:101
222399910097,2cyclictest0-21swapper/221:24:462
30182660,1sleep10-21swapper/100:30:201
22239995956,2cyclictest0-21swapper/222:02:592
286812540,4sleep230-21rcuc/220:35:152
102972370,1sleep10-21swapper/122:20:171
281082360,1sleep30-21swapper/321:45:273
2224599351,22cyclictest0-21swapper/321:12:413
2224599341,26cyclictest0-21swapper/323:50:243
2224599341,23cyclictest0-21swapper/319:47:123
2224599330,26cyclictest0-21swapper/321:41:483
2224599330,15cyclictest0-21swapper/323:34:363
2222899331,26cyclictest0-21swapper/021:13:400
2222899331,24cyclictest0-21swapper/022:29:190
2224599323,12cyclictest0-21swapper/319:35:163
22245993214,3cyclictest0-21swapper/300:30:213
2224599321,24cyclictest0-21swapper/322:07:333
2224599321,24cyclictest0-21swapper/300:15:453
2224599321,22cyclictest0-21swapper/321:01:493
2224599321,21cyclictest0-21swapper/320:08:213
2224599321,21cyclictest0-21swapper/300:07:053
2224599321,20cyclictest0-21swapper/323:07:333
2224599321,19cyclictest0-21swapper/323:44:123
2224599321,12cyclictest0-21swapper/300:01:453
2224599321,12cyclictest0-21swapper/300:01:443
2222899321,24cyclictest87350irq/126-enp1s0-21:16:030
37182310,3sleep12223399cyclictest23:20:131
2224599311,21cyclictest0-21swapper/319:14:533
2224599311,21cyclictest0-21swapper/319:14:533
2224599311,18cyclictest0-21swapper/322:37:333
2224599311,13cyclictest0-21swapper/322:26:453
2224599310,22cyclictest0-21swapper/319:29:573
2224599310,19cyclictest0-21swapper/323:57:443
2222899311,22cyclictest26138-21kworker/u8:221:52:510
2222899311,22cyclictest0-21swapper/000:22:370
22245993018,3cyclictest0-21swapper/320:32:373
2224599301,21cyclictest0-21swapper/322:40:493
2224599301,20cyclictest0-21swapper/322:49:173
2224599301,20cyclictest0-21swapper/320:37:213
22245993010,9cyclictest0-21swapper/321:20:133
2224599300,21cyclictest0-21swapper/319:41:003
2224599300,15cyclictest0-21swapper/300:13:203
22228993011,16cyclictest4335-21munin-run20:55:010
2224599291,9cyclictest0-21swapper/319:15:233
2224599291,26cyclictest0-21swapper/321:15:093
2224599291,18cyclictest0-21swapper/320:03:293
2224599290,19cyclictest0-21swapper/323:16:453
22245992817,3cyclictest0-21swapper/323:49:363
22245992815,8cyclictest0-21swapper/300:35:493
2224599281,12cyclictest10375-21cat23:35:123
2224599281,12cyclictest0-21swapper/322:15:173
2224599281,12cyclictest0-21swapper/321:33:213
2224599281,11cyclictest0-21swapper/322:24:253
2224599281,11cyclictest0-21swapper/322:00:573
176202280,1sleep00-21swapper/023:50:180
2224599271,19cyclictest0-21swapper/321:58:213
2224599271,18cyclictest0-21swapper/323:20:053
2224599271,16cyclictest0-21swapper/320:55:253
2224599271,13cyclictest30670-21diskstats20:40:153
2224599271,13cyclictest0-21swapper/323:14:573
2224599271,12cyclictest0-21swapper/322:56:573
2224599271,11cyclictest0-21swapper/323:28:443
2224599271,11cyclictest0-21swapper/321:36:123
2224599271,11cyclictest0-21swapper/321:36:123
2224599271,10cyclictest0-21swapper/320:54:373
2224599270,17cyclictest0-21swapper/300:29:293
2224599270,14cyclictest23319-21seq20:22:453
2224599270,14cyclictest0-21swapper/321:51:213
2224599270,12cyclictest0-21swapper/322:30:413
2224599270,11cyclictest0-21swapper/319:57:493
2222899272,12cyclictest27911-21awk20:35:000
2224599261,14cyclictest0-21swapper/323:01:053
2224599261,14cyclictest0-21swapper/320:46:053
2224599261,12cyclictest0-21swapper/320:15:093
2224599261,11cyclictest0-21swapper/322:54:493
2224599260,24cyclictest0-21swapper/300:20:493
22239992615,6cyclictest0-21swapper/221:47:472
2222899267,16cyclictest30100-21munin-run20:40:010
22228992610,14cyclictest0-21swapper/021:50:010
2224599258,9cyclictest0-21swapper/321:29:253
2224599251,11cyclictest27859-21df_inode19:20:133
2224599250,12cyclictest0-21swapper/322:10:553
2223999251,15cyclictest14036-21smartctl21:15:172
2222899253,15cyclictest0-21swapper/023:01:210
2222899252,11cyclictest31568-21/usr/sbin/munin23:10:150
2222899251,16cyclictest0-21swapper/022:56:440
256452240,3sleep121-21rcuc/100:10:131
2224599241,15cyclictest0-21swapper/320:10:093
2224599241,13cyclictest0-21swapper/319:31:003
2224599240,12cyclictest0-21swapper/319:50:023
2223999244,7cyclictest17601-21irqstats23:50:172
2223399241,4cyclictest1821-21munin-run00:30:021
2223399241,15cyclictest87650irq/129-enp1s0-23:45:411
2222899249,3cyclictest808-21snmpd20:01:050
2222899243,11cyclictest10931-21diskmemload23:21:530
2222899241,21cyclictest0-21swapper/022:50:270
2222899241,20cyclictest0-21swapper/020:41:440
2222899241,17cyclictest0-21swapper/000:25:230
2222899241,14cyclictest29416-21kworker/u8:200:36:410
2222899241,11cyclictest684-21kworker/u8:022:30:220
2222899241,11cyclictest32520-21/usr/sbin/munin20:45:240
2222899241,11cyclictest21526-21/usr/sbin/munin20:20:140
2222899240,7cyclictest0-21swapper/023:26:060
2222899240,12cyclictest22556-21perf22:50:010
89302230,4sleep1221ktimersoftd/119:50:161
2224599231,12cyclictest0-21swapper/321:08:453
2223399234,7cyclictest0-21swapper/123:25:231
2223399233,16cyclictest0-21swapper/122:37:271
22233992316,4cyclictest17838-21grep23:50:211
22233992312,3cyclictest29386-21smart_sda20:35:241
2223399231,19cyclictest24401-21munin-run19:15:011
2223399231,19cyclictest24401-21munin-run19:15:001
2223399231,10cyclictest4051-21grep19:40:131
2223399230,18cyclictest0-21swapper/122:40:411
2223399230,18cyclictest0-21swapper/121:14:451
2222899238,3cyclictest27515-21/usr/sbin/munin19:20:210
2222899236,9cyclictest27302-21basename19:20:010
2222899235,8cyclictest10931-21diskmemload00:01:260
2222899235,8cyclictest10931-21diskmemload00:01:260
2222899235,7cyclictest4010-21crond00:35:010
2222899234,10cyclictest0-21swapper/021:55:090
2222899231,6cyclictest0-21swapper/021:35:220
2222899231,6cyclictest0-21swapper/021:35:220
2222899231,16cyclictest0-21swapper/000:10:370
2222899231,15cyclictest0-21swapper/022:22:190
2222899231,15cyclictest0-21swapper/022:12:210
2222899231,11cyclictest30443-21ntp_states23:05:220
2222899230,11cyclictest25717-21perf20:30:000
2222899230,11cyclictest25717-21perf20:30:000
33332221,2sleep23510-21grep00:30:262
2224599221,14cyclictest0-21swapper/320:26:373
2224599221,14cyclictest0-21swapper/320:26:373
2223999225,8cyclictest87350irq/126-enp1s0-21:40:152
2223999225,10cyclictest10931-21diskmemload22:46:542
2223999221,9cyclictest0-21swapper/223:15:012
2223999221,7cyclictest10937-21munin-run21:10:022
2223999221,4cyclictest0-21swapper/221:56:242
2223999221,16cyclictest0-21swapper/219:45:462
2223999221,15cyclictest0-21swapper/219:34:472
2223399225,8cyclictest0-21swapper/122:57:201
2223399222,3cyclictest22183-40updatedb00:00:211
2223399222,3cyclictest22183-40updatedb00:00:211
22233992216,3cyclictest827-21runrttasks21:37:101
22233992216,3cyclictest827-21runrttasks21:37:101
2223399221,18cyclictest23714-21latency_hist20:25:021
22233992210,3cyclictest30667-21diskstats20:40:141
2223399220,15cyclictest0-21swapper/122:51:531
2222899226,3cyclictest16009-21ntp_states20:05:210
2222899225,4cyclictest7518-21grep22:15:120
2222899223,10cyclictest19360-21kworker/u8:221:30:520
2222899221,8cyclictest2988-21/usr/sbin/munin22:05:210
2222899221,15cyclictest0-21swapper/023:48:180
2222899221,14cyclictest0-21swapper/023:31:330
2222899221,10cyclictest10931-21diskmemload22:00:270
2222899221,10cyclictest0-21swapper/023:17:270
2222899220,13cyclictest0-21swapper/023:40:190
2223999215,8cyclictest87650irq/129-enp1s0-22:52:422
2223999212,3cyclictest12138-21needreboot21:10:192
2223999211,8cyclictest13974-21/usr/sbin/munin22:30:192
2223999211,7cyclictest22271-21ntp_states00:00:242
2223999211,7cyclictest22271-21ntp_states00:00:242
2223999211,6cyclictest25399-21awk00:10:012
2223999211,6cyclictest0-21swapper/223:44:032
2223999211,11cyclictest6746-21/usr/sbin/munin21:00:142
2223999211,11cyclictest31597-21users00:20:272
2223999211,11cyclictest28233-21ntp_states23:00:202
2223999211,11cyclictest20995-21latency_hist00:00:012
2223999211,11cyclictest18726-21needreboot21:25:172
2223399218,4cyclictest30031-21expr19:25:131
2223399212,9cyclictest29724-21diskstats23:05:151
2223399212,15cyclictest9357-21munin-run22:20:021
22233992115,4cyclictest28814-21netstat00:15:221
2223399211,3cyclictest808-21snmpd21:06:171
2223399211,3cyclictest29155-21perf23:05:011
2223399211,2cyclictest27609-21ntpq21:45:231
2223399211,2cyclictest25139-21if_err_enp2s021:40:181
2223399211,18cyclictest0-21swapper/123:35:171
2223399211,18cyclictest0-21swapper/121:04:171
2223399211,11cyclictest463-21awk22:00:011
2223399211,11cyclictest29481-21latency_hist19:25:011
2223399211,11cyclictest19102-21latency_hist20:15:011
2222899218,2cyclictest8338-21/usr/sbin/munin19:50:150
2222899217,4cyclictest0-21swapper/021:41:040
2222899212,7cyclictest11787-21users23:35:270
2222899212,13cyclictest87350irq/126-enp1s0-23:57:000
2222899211,11cyclictest0-21swapper/000:06:470
2222899210,19cyclictest0-21swapper/021:00:230
2223999206,3cyclictest12050irq/123-ahci[0019:20:002
2223999205,4cyclictest16795-21sshd20:08:442
2223999205,4cyclictest10931-21diskmemload22:07:302
2223999203,9cyclictest24703-21missed_timers20:25:202
2223999203,9cyclictest24703-21missed_timers20:25:202
2223999203,9cyclictest0-21swapper/223:25:332
2223999203,9cyclictest0-21swapper/222:42:422
2223999202,10cyclictest0-21swapper/222:21:172
2223999201,6cyclictest0-21swapper/221:36:232
2223999201,6cyclictest0-21swapper/221:36:232
2223999201,11cyclictest2386-21needreboot23:15:192
2223999201,11cyclictest0-21swapper/220:45:212
2223399207,11cyclictest26501-21kworker/u8:119:59:151
2223399204,3cyclictest0-21swapper/119:16:371
2223399204,13cyclictest8105-21crond19:50:021
2223399202,14cyclictest29669-21wc21:50:201
2223399201,8cyclictest164832cat21:20:181
22233992016,3cyclictest0-21swapper/123:44:121
22233992016,3cyclictest0-21swapper/122:29:311
2223399201,3cyclictest0-21swapper/121:26:531
2223399201,2cyclictest808-21snmpd20:03:191
2223399201,17cyclictest0-21swapper/122:45:401
2223399201,17cyclictest0-21swapper/120:55:061
2223399201,17cyclictest0-21swapper/120:45:351
2223399201,13cyclictest0-21swapper/100:22:341
2223399201,11cyclictest3388-21ntp_kernel_err20:50:181
2223399201,11cyclictest3374-21chrt19:35:471
2223399201,11cyclictest20427-21open_inodes20:15:241
2223399201,11cyclictest14462-21chrt21:15:241
2223399200,1cyclictest0-21swapper/120:25:161
2223399200,1cyclictest0-21swapper/120:25:161
2223399200,18cyclictest0-21swapper/123:30:101
2223399200,18cyclictest0-21swapper/122:31:321
2223399200,18cyclictest0-21swapper/122:05:131
2222899204,8cyclictest19397-21cpu20:15:130
2222899202,7cyclictest10931-21diskmemload00:18:140
2222899201,10cyclictest0-21swapper/022:35:080
2222899200,1cyclictest0-21swapper/021:20:120
2223999194,8cyclictest87350irq/126-enp1s0-20:20:222
2223999193,8cyclictest0-21swapper/200:27:132
2223999192,6cyclictest10931-21diskmemload22:58:252
2223999191,7cyclictest0-21swapper/223:08:242
2223999191,7cyclictest0-21swapper/219:55:192
2223999191,6cyclictest18134-21latency_hist22:40:012
2223999191,6cyclictest0-21swapper/221:53:592
2223999191,5cyclictest8969-21users22:15:272
2223999191,5cyclictest4239-21/usr/sbin/munin00:35:182
2223999191,5cyclictest10545-21diskstats23:35:152
22239991915,3cyclictest0-21swapper/222:10:032
2223999191,2cyclictest87350irq/126-enp1s0-00:18:412
2223999191,10cyclictest26005-21grep20:30:122
2223399192,3cyclictest0-21swapper/122:11:391
2223399192,16cyclictest0-21swapper/120:05:511
22233991915,3cyclictest0-21swapper/123:58:191
2223399191,16cyclictest0-21swapper/100:35:441
2223399191,10cyclictest31923-21egrep19:30:111
2223399191,10cyclictest1451-21cpuspeed_turbos23:15:131
2223399191,10cyclictest1177-21fschecks_count22:00:161
2223399190,18cyclictest0-21swapper/123:13:101
2222899196,4cyclictest0-21swapper/021:25:040
2222899196,3cyclictest87550irq/128-enp1s0-19:10:260
2222899196,3cyclictest87550irq/128-enp1s0-19:10:250
2222899191,8cyclictest87350irq/126-enp1s0-19:45:190
2222899191,15cyclictest0-21swapper/022:41:430
83882180,4sleep232-21ksoftirqd/223:30:132
2223999185,3cyclictest87650irq/129-enp1s0-19:35:192
2223999184,3cyclictest87650irq/129-enp1s0-21:30:172
2223999184,3cyclictest87650irq/129-enp1s0-19:29:432
2223999183,9cyclictest0-21swapper/223:45:442
2223999183,9cyclictest0-21swapper/223:24:332
2223999183,3cyclictest12050irq/123-ahci[0019:15:012
2223999183,3cyclictest12050irq/123-ahci[0019:15:012
2223999182,4cyclictest4328-21latency_hist20:55:022
2223999181,6cyclictest3943-21/usr/sbin/munin19:40:122
2223999181,6cyclictest0-21swapper/222:25:232
2223999181,6cyclictest0-21swapper/200:10:232
2223999181,5cyclictest4556-21/usr/sbin/munin20:55:182
2223999181,4cyclictest18009-21/usr/sbin/munin20:10:192
22239991812,3cyclictest20408-21ntpq20:15:242
2223399181,3cyclictest0-21swapper/100:07:541
2223399181,16cyclictest0-21swapper/121:31:231
2223399180,2cyclictest0-21swapper/120:31:201
2222899183,4cyclictest4052-21tr19:40:130
2222899181,5cyclictest808-21snmpd19:36:010
2222899181,5cyclictest721-21/usr/sbin/munin19:30:230
2222899181,16cyclictest87350irq/126-enp1s0-20:55:140
2222899180,9cyclictest0-21swapper/019:55:260
309932170,14sleep20-21swapper/220:40:182
2223999174,3cyclictest87350irq/126-enp1s0-19:50:212
2223999171,6cyclictest0-21swapper/219:20:262
2222899174,3cyclictest29889-21cat19:25:120
2222899171,4cyclictest8961-21/usr/sbin/munin21:05:190
2222899170,15cyclictest19121-21perf20:15:000
2223999161,3cyclictest12732-21/usr/sbin/munin20:00:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional