You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 06:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 03, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
42782241166,24sleep20-21swapper/219:08:082
41382238163,24sleep10-21swapper/119:06:181
24592235163,24sleep30-21swapper/319:05:043
24612201166,23sleep00-21swapper/019:05:040
4517991080,106cyclictest0-21swapper/200:00:582
451799103100,2cyclictest0-21swapper/219:56:272
11602500,4sleep121-21rcuc/123:55:011
451799391,6cyclictest0-21swapper/221:25:162
4505993914,8cyclictest25600-21basename21:10:020
4505993818,8cyclictest32745-21df_inode21:25:140
4505993813,13cyclictest14096-21expr23:10:180
4505993811,18cyclictest32180-21munin-run21:25:010
451799371,26cyclictest31142-21expr22:35:132
451799361,4cyclictest14749-21grep21:55:252
451799361,3cyclictest0-21swapper/223:25:172
4505993615,12cyclictest20790-21/usr/sbin/munin19:45:160
4522993524,3cyclictest0-21swapper/323:25:233
451799355,3cyclictest87350irq/126-enp1s0-23:30:252
451799351,4cyclictest87350irq/126-enp1s0-20:35:162
450599359,12cyclictest87550irq/128-enp1s0-21:35:170
4505993514,9cyclictest14389-21head21:55:210
4505993511,13cyclictest3820-21grep21:30:250
451799342,7cyclictest0-21swapper/200:20:232
451799342,5cyclictest18114-40dnf20:50:252
451799341,4cyclictest0-21swapper/223:55:252
451799341,22cyclictest827-21runrttasks19:43:052
451799341,18cyclictest6021-21/usr/sbin/munin20:25:222
4505993411,14cyclictest25585-21diskmemload22:40:160
4505993411,11cyclictest808-21snmpd20:49:180
4522993325,3cyclictest0-21swapper/320:27:193
452299330,14cyclictest14505-21/usr/sbin/munin00:25:163
451799331,5cyclictest28891-21kworker/u8:023:51:252
451799331,4cyclictest22756-21latency_hist19:50:022
451799331,3cyclictest28949-21awk20:00:272
451799331,3cyclictest25639-21cut21:10:012
451799331,3cyclictest0-21swapper/219:30:162
450599338,16cyclictest25585-21diskmemload23:09:230
450599338,13cyclictest0-21swapper/021:40:140
4505993312,11cyclictest87550irq/128-enp1s0-22:30:230
4505993311,9cyclictest7239-21latency_hist22:55:020
452299321,25cyclictest0-21swapper/321:49:343
452299321,18cyclictest0-21swapper/321:40:383
452299321,15cyclictest0-21swapper/322:51:593
451799328,21cyclictest5713-21/usr/sbin/munin00:05:212
451799324,3cyclictest87650irq/129-enp1s0-00:10:242
451799324,3cyclictest87650irq/129-enp1s0-00:10:242
451799321,5cyclictest28969-21df22:30:132
451799321,4cyclictest0-21swapper/222:50:172
451799321,4cyclictest0-21swapper/221:48:022
451799321,3cyclictest27077-21smart_sda21:10:232
4517993211,18cyclictest8854-21users22:55:272
450599328,10cyclictest0-21swapper/023:25:260
450599327,16cyclictest0-21swapper/022:56:350
450599327,15cyclictest0-21swapper/000:15:240
4505993217,7cyclictest24233-21/usr/sbin/munin22:20:100
4505993213,7cyclictest9178-21/usr/sbin/munin21:45:130
4505993213,5cyclictest23909-21sensors_temp23:30:240
450599321,16cyclictest24021-21awk22:20:010
4505993210,9cyclictest18897-21/usr/sbin/munin00:35:140
4505993210,14cyclictest808-21snmpd00:05:470
4505993210,14cyclictest19826-21unixbench_singl23:20:280
452299311,27cyclictest0-21swapper/320:07:053
452299311,22cyclictest0-21swapper/323:03:503
452299311,22cyclictest0-21swapper/323:03:503
452299311,22cyclictest0-21swapper/300:00:123
452299311,21cyclictest0-21swapper/319:13:513
452299311,19cyclictest0-21swapper/319:23:223
451799319,19cyclictest1564-21cat22:40:202
451799317,21cyclictest15176-21hddtemp_smartct00:25:172
451799312,4cyclictest17050-21diskstats00:30:152
451799311,4cyclictest0-21swapper/219:29:022
451799311,4cyclictest0-21swapper/219:29:022
451799311,27cyclictest8131-21fschecks_time19:15:142
4517993112,16cyclictest1-21systemd20:55:232
4517993111,17cyclictest16872-21irqstats23:15:152
4517993110,18cyclictest20598-21if_enp2s022:10:222
4517993110,18cyclictest10781-21cut00:15:172
450599319,14cyclictest17642-21/usr/sbin/munin22:05:230
450599318,11cyclictest87550irq/128-enp1s0-23:15:190
450599316,12cyclictest12392-21munin-run20:40:010
4505993112,7cyclictest22986-21/usr/sbin/munin19:50:260
4522993017,8cyclictest0-21swapper/300:05:203
452299301,14cyclictest0-21swapper/300:22:363
452299301,14cyclictest0-21swapper/300:18:003
452299301,13cyclictest0-21swapper/323:16:043
452299301,12cyclictest0-21swapper/323:43:023
452299300,21cyclictest0-21swapper/320:38:273
451799309,18cyclictest11993-21missed_timers21:50:242
451799309,18cyclictest11993-21missed_timers21:50:242
451799309,11cyclictest29594-21diskstats23:45:132
451799308,19cyclictest16323-21unixbench_multi20:45:242
451799308,19cyclictest11853-21cstates23:05:112
451799307,20cyclictest5976-21smart_sda19:10:242
451799307,19cyclictest22311-21latency21:00:182
451799305,22cyclictest0-21swapper/220:20:252
451799303,18cyclictest17765-21cut22:05:122
451799301,5cyclictest10139-21smartctl23:00:152
451799301,5cyclictest10139-21smartctl23:00:142
451799301,4cyclictest5627-21processes21:35:232
451799301,4cyclictest31965-21/usr/sbin/munin20:10:112
451799301,3cyclictest19537-21/usr/sbin/munin23:20:252
4517993012,15cyclictest20867-21dump-pmu-power00:40:022
4517993011,16cyclictest13873-21/usr/sbin/munin23:10:272
451799300,27cyclictest26549-21tr22:25:122
450599307,13cyclictest87650irq/129-enp1s0-19:15:230
4505993021,6cyclictest678-40dnf23:51:280
4505993019,8cyclictest26446-21smart_sda19:55:240
4505993016,5cyclictest25198-21diskstats23:35:150
4505993013,6cyclictest12825-21tune2fs00:20:130
4505993013,5cyclictest29187-21sshd21:15:250
4505993012,9cyclictest26996-21grep22:25:140
4522992916,8cyclictest0-21swapper/319:26:353
4522992916,8cyclictest0-21swapper/319:26:343
452299291,19cyclictest0-21swapper/320:12:273
452299291,19cyclictest0-21swapper/319:34:233
452299291,12cyclictest0-21swapper/323:30:303
452299291,12cyclictest0-21swapper/322:42:313
452299291,12cyclictest0-21swapper/321:15:583
452299290,14cyclictest0-21swapper/300:33:523
451799296,20cyclictest28479-21cut21:15:172
451799295,4cyclictest87350irq/126-enp1s0-19:20:252
451799291,4cyclictest6641-21/usr/sbin/munin21:40:292
451799291,4cyclictest25942-21/usr/sbin/munin23:35:222
451799291,3cyclictest24589-21/usr/sbin/munin22:20:142
451799291,3cyclictest22949-21memory22:15:172
4517992912,14cyclictest29687-21sort20:05:172
451799290,4cyclictest3344-21/usr/sbin/munin21:30:212
450599299,6cyclictest14521-21cpu00:25:120
450599299,12cyclictest18589-21/usr/sbin/munin19:40:190
450599296,12cyclictest808-21snmpd19:13:560
4505992913,8cyclictest20761-21memory22:10:220
4505992912,9cyclictest1326-21/usr/sbin/munin23:55:110
4505992910,11cyclictest16703-21/usr/sbin/munin00:30:170
4522992815,7cyclictest0-21swapper/321:05:273
452299281,14cyclictest19331-21ntp_states23:20:233
452299281,13cyclictest0-21swapper/322:28:473
452299281,12cyclictest0-21swapper/323:46:543
452299281,12cyclictest0-21swapper/321:36:593
452299281,11cyclictest0-21swapper/321:34:353
452299281,10cyclictest0-21swapper/320:52:573
452299281,10cyclictest0-21swapper/320:44:313
452299281,10cyclictest0-21swapper/319:51:193
451799288,17cyclictest32343-21date21:25:012
451799282,3cyclictest3211-21diskstats22:45:152
451799282,3cyclictest16142-21/usr/sbin/munin22:00:172
451799281,4cyclictest27629-21/usr/sbin/munin23:40:152
451799281,4cyclictest13664-21/usr/sbin/munin20:40:202
451799281,3cyclictest9250-21/usr/sbin/munin20:30:202
451799281,3cyclictest2798-21/usr/sbin/munin20:15:222
451799281,3cyclictest23666-21/usr/sbin/munin19:50:172
450599287,14cyclictest10246-21if_err_enp2s023:00:160
450599287,14cyclictest10246-21if_err_enp2s023:00:160
4505992820,5cyclictest22747-21/usr/sbin/munin21:00:250
450599281,3cyclictest0-21swapper/021:51:400
450599281,3cyclictest0-21swapper/021:51:390
4505992813,12cyclictest4425-21unixbench_singl22:45:260
4505992810,5cyclictest0-21swapper/000:10:250
4505992810,5cyclictest0-21swapper/000:10:250
4505992810,11cyclictest25585-21diskmemload21:10:580
4522992712,3cyclictest0-21swapper/323:52:313
4522992711,3cyclictest24278-21cpu22:20:123
452299271,12cyclictest15157-21diskstats20:45:133
452299271,12cyclictest0-21swapper/321:10:203
452299271,11cyclictest0-21swapper/321:28:063
4517992710,14cyclictest16576-21cat19:35:122
450599277,11cyclictest0-21swapper/020:55:130
450599275,16cyclictest0-21swapper/022:00:200
4505992719,5cyclictest2687-21ntp_states20:15:200
4505992719,5cyclictest14534-21df19:30:120
4505992718,6cyclictest29247-21/usr/sbin/munin23:45:220
4505992718,6cyclictest12003-21/usr/sbin/munin19:25:150
4505992718,6cyclictest12003-21/usr/sbin/munin19:25:140
4505992717,7cyclictest9799-21/usr/sbin/munin19:20:210
4505992717,7cyclictest3490-21/usr/sbin/munin00:00:180
4505992710,6cyclictest31096-21expr22:35:120
452299261,13cyclictest0-21swapper/320:22:133
452299261,13cyclictest0-21swapper/319:59:233
452299261,12cyclictest0-21swapper/321:52:433
452299261,12cyclictest0-21swapper/321:52:433
452299261,11cyclictest0-21swapper/321:58:463
4522992610,3cyclictest21856-21ntp_states19:45:233
452299260,6cyclictest0-21swapper/322:00:383
452299260,12cyclictest0-21swapper/300:10:563
452299260,12cyclictest0-21swapper/300:10:563
451399262,20cyclictest32117-21df20:10:111
450599266,10cyclictest0-21swapper/020:09:100
4505992619,4cyclictest339-21/usr/sbin/munin20:10:180
4505992619,4cyclictest18276-21smart_sda20:50:250
4505992619,4cyclictest17306-21memory19:35:180
4505992618,5cyclictest9809-21users20:30:250
4505992618,5cyclictest6021-21/usr/sbin/munin20:25:120
4505992618,5cyclictest3817-21/usr/sbin/munin20:20:240
4505992618,5cyclictest28075-21/usr/sbin/munin23:40:200
4505992618,5cyclictest27382-21/usr/sbin/munin20:00:130
4505992617,6cyclictest12614-21/usr/sbin/munin20:40:130
452299258,10cyclictest0-21swapper/322:05:193
4522992514,3cyclictest0-21swapper/319:15:073
452299251,17cyclictest29883-21kworker/u8:121:20:403
452299251,15cyclictest0-21swapper/319:36:053
452299251,13cyclictest0-21swapper/322:19:483
452299251,11cyclictest0-21swapper/320:15:233
452299251,10cyclictest0-21swapper/300:38:163
452299250,10cyclictest0-21swapper/323:59:033
451399251,17cyclictest0-21swapper/123:31:491
452299241,9cyclictest0-21swapper/323:38:273
452299241,9cyclictest0-21swapper/322:58:203
452299241,9cyclictest0-21swapper/320:34:053
4522992411,3cyclictest0-21swapper/322:12:123
452299241,13cyclictest0-21swapper/320:57:493
4522992410,3cyclictest0-21swapper/322:38:393
451399247,14cyclictest19033-21kworker/u8:020:32:541
451399241,11cyclictest0-21swapper/123:05:151
452299236,3cyclictest0-21swapper/323:06:233
452299231,14cyclictest27728-21diskstats20:00:133
452299231,10cyclictest0-21swapper/322:49:593
451399232,18cyclictest7997-21munin-run20:30:001
4513992319,3cyclictest0-21swapper/100:35:071
451399231,4cyclictest7301-21users00:05:311
451399231,19cyclictest20804-21diskstats23:25:151
451399231,13cyclictest0-21swapper/121:17:081
451399230,15cyclictest0-21swapper/121:12:241
452299227,3cyclictest0-21swapper/322:32:233
4513992214,2cyclictest25585-21diskmemload22:47:401
451399221,3cyclictest17425-21munin-run22:05:011
451399221,16cyclictest87550irq/128-enp1s0-21:36:541
451399221,13cyclictest0-21swapper/121:27:141
451399221,11cyclictest6803-21grep21:40:131
451399221,11cyclictest25745-21cat21:10:021
451399221,10cyclictest16767-21cpuspeed_turbos00:30:121
451399220,15cyclictest0-21swapper/120:50:261
452299211,12cyclictest0-21swapper/323:10:043
452299211,10cyclictest22472-21ntp_states21:00:213
451399212,14cyclictest0-21swapper/121:23:081
451399211,3cyclictest24811-21perl22:20:151
451399211,18cyclictest0-21swapper/120:15:081
451399211,18cyclictest0-21swapper/119:48:161
451399211,15cyclictest0-21swapper/122:57:021
451399211,15cyclictest0-21swapper/120:40:081
451399211,14cyclictest0-21swapper/120:35:311
451399211,11cyclictest27155-21awk20:00:011
451399211,11cyclictest23621-21users22:15:241
451399210,6cyclictest0-21swapper/123:37:461
451399210,16cyclictest0-21swapper/100:17:541
451399210,12cyclictest0-21swapper/122:34:051
451399205,2cyclictest10360-21sed23:00:161
451399205,2cyclictest10360-21sed23:00:161
451399204,10cyclictest19562-21kworker/u8:022:50:191
451399201,3cyclictest0-21swapper/100:00:191
4513992013,2cyclictest0-21swapper/122:26:101
451399201,14cyclictest0-21swapper/122:42:321
451399201,12cyclictest0-21swapper/123:55:301
451399201,11cyclictest12323-21grep00:20:091
451399201,10cyclictest18191-21date23:20:021
451399201,10cyclictest0-21swapper/123:49:061
451399200,18cyclictest0-21swapper/122:14:531
451399200,14cyclictest0-21swapper/120:45:111
452299191,11cyclictest0-21swapper/319:41:473
451399198,3cyclictest14050-21irqstats21:55:181
451399197,3cyclictest21183-21munin-run21:00:011
4513991916,2cyclictest0-21swapper/119:20:211
4513991915,3cyclictest0-21swapper/120:20:011
4513991914,3cyclictest15908-21taskset19:31:501
4513991913,1cyclictest0-21swapper/100:25:191
451399191,17cyclictest0-21swapper/119:17:091
451399191,16cyclictest0-21swapper/121:45:231
451399191,14cyclictest0-21swapper/123:12:161
451399191,11cyclictest23878-21latency19:50:181
451399191,11cyclictest0-21swapper/122:38:551
451399191,11cyclictest0-21swapper/121:50:031
451399191,11cyclictest0-21swapper/121:50:031
451399191,10cyclictest8977-21ntpq00:10:221
451399191,10cyclictest8977-21ntpq00:10:221
451399191,10cyclictest4735-21cpu19:10:111
451399191,10cyclictest18779-21fschecks_time23:20:171
451399191,10cyclictest0-21swapper/120:05:281
451399190,4cyclictest17217-21kworker/u8:122:05:091
451399190,12cyclictest29883-21kworker/u8:121:33:191
451399181,16cyclictest0-21swapper/121:01:471
451399181,16cyclictest0-21swapper/120:01:281
451399181,16cyclictest0-21swapper/119:40:271
451399181,10cyclictest12848-21latency19:25:181
451399181,10cyclictest12848-21latency19:25:181
451399180,2cyclictest0-21swapper/119:36:111
287342180,2sleep10-21swapper/123:41:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional