You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 19:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 31, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36602237161,24sleep00-21swapper/007:10:000
34062236163,24sleep20-21swapper/207:06:442
34642235164,59sleep30-21swapper/307:07:293
33432205167,26sleep10-21swapper/107:05:551
3762991070,105cyclictest0-21swapper/311:14:443
37629910399,3cyclictest0-21swapper/309:34:093
3762999390,2cyclictest0-21swapper/310:14:593
3762999188,2cyclictest0-21swapper/307:16:303
3762998784,2cyclictest0-21swapper/307:31:213
193062560,4sleep121-21rcuc/107:40:261
40982550,3sleep3376299cyclictest08:20:213
3762994542,2cyclictest0-21swapper/309:52:343
375699401,27cyclictest8513-21fschecks_count09:45:152
375699401,19cyclictest24826-21idleruntime-cro09:10:012
375699391,26cyclictest10898-21/usr/sbin/munin11:05:252
375699391,24cyclictest27009-21latency_hist09:15:012
375699391,21cyclictest19918-21cat07:45:022
375699383,30cyclictest1709-40dnf10:43:122
375699382,25cyclictest15903-21/usr/sbin/munin12:30:102
375699381,27cyclictest24231-21cat07:55:022
375699372,22cyclictest5926-21munin-run07:15:012
375699372,22cyclictest5926-21munin-run07:15:012
375699371,26cyclictest10607-21taskset11:04:222
375699371,26cyclictest0-21swapper/211:55:162
375699371,26cyclictest0-21swapper/211:55:162
375699371,26cyclictest0-21swapper/209:15:162
375699371,25cyclictest9991-21cat07:20:172
375699371,24cyclictest22932-21awk11:30:202
375699371,21cyclictest31145-21fschecks_time11:50:152
375699364,18cyclictest18218-21perf08:55:012
375699362,25cyclictest812-21snmpd09:52:462
375699361,27cyclictest6086-21chrt10:52:492
375699361,27cyclictest0-21swapper/210:05:252
375699361,26cyclictest13963-21cstates12:25:122
375699361,25cyclictest3403-21diskstats08:20:152
375699361,25cyclictest15365-21cat08:45:252
375699361,23cyclictest19012-21memory12:35:162
375699361,22cyclictest27878-21chrt11:40:262
375699361,21cyclictest26048-21latency_hist11:40:012
375699361,20cyclictest25426-21latency_hist10:25:012
375699361,20cyclictest13475-21latency_hist12:25:012
375699361,15cyclictest0-21swapper/210:39:592
374099367,13cyclictest7854-21munin-run09:45:000
3740993610,12cyclictest14438-21munin-run10:00:010
375699352,22cyclictest14163-21ntp_states11:10:212
375699351,22cyclictest10384-21ntp_states12:15:212
375699351,20cyclictest6841-21/usr/sbin/munin07:15:172
375699351,20cyclictest3773-21grep12:00:202
375699351,20cyclictest1315-21cut09:30:012
375699343,28cyclictest14662-21find07:30:242
375699342,23cyclictest6889-21latency_hist12:10:022
375699341,24cyclictest0-21swapper/209:55:262
375699341,22cyclictest807-21/usr/sbin/munin08:15:272
375699341,21cyclictest30238-21cat09:20:172
375699341,21cyclictest24413-21/usr/sbin/munin07:55:172
375699341,20cyclictest3693-21cpu09:35:132
375699341,20cyclictest3693-21cpu09:35:132
375699341,17cyclictest25658-21/usr/sbin/munin10:25:252
374099349,12cyclictest17874-21munin-run12:35:020
374099341,24cyclictest0-21swapper/011:50:220
375699331,23cyclictest20020-21/usr/sbin/munin07:45:242
375699331,20cyclictest22332-21ntp_states10:15:232
375699331,19cyclictest28819-21/usr/sbin/munin08:05:172
375699331,19cyclictest0-21swapper/210:45:192
375699331,19cyclictest0-21swapper/210:45:192
375699330,22cyclictest0-21swapper/208:38:302
375699323,17cyclictest86850irq/129-enp1s0-10:30:202
375699322,19cyclictest86550irq/126-enp1s0-10:10:152
375699321,23cyclictest17487-21/usr/sbin/munin11:20:202
375699321,21cyclictest16733-21cat10:05:022
375699321,21cyclictest0-21swapper/208:35:022
375699321,18cyclictest24814-21diskmemload12:14:232
375699321,17cyclictest12050irq/123-ahci[0011:20:022
374099323,14cyclictest6899-21munin-run12:10:010
375699319,19cyclictest31491-21fschecks_time08:10:162
375699313,15cyclictest3083-21runrttasks09:30:292
375699311,22cyclictest24814-21diskmemload10:55:182
375699311,22cyclictest19683-21/usr/sbin/munin11:25:272
375699311,19cyclictest812-21snmpd11:48:102
375699311,16cyclictest86550irq/126-enp1s0-09:40:132
375699311,16cyclictest86550irq/126-enp1s0-08:00:142
375699301,26cyclictest18439-21/usr/sbin/munin08:55:162
375699301,26cyclictest13112-21/usr/sbin/munin08:40:242
375699301,15cyclictest12050irq/123-ahci[0007:30:022
375699291,25cyclictest5259-21/usr/sbin/munin08:25:152
375699291,25cyclictest20634-21/usr/sbin/munin09:00:152
375699291,25cyclictest15627-21/usr/sbin/munin07:35:152
374099294,12cyclictest17603-21munin-run07:40:020
3740992911,15cyclictest13484-21munin-run12:25:020
374999281,17cyclictest0-21swapper/111:36:061
3740992811,8cyclictest15035-21needreboot08:45:210
374099281,10cyclictest0-21swapper/007:43:430
374099280,13cyclictest0-21swapper/011:45:060
374999271,18cyclictest0-21swapper/110:52:231
374999271,11cyclictest8710-21users12:10:271
374099271,12cyclictest0-21swapper/009:54:300
374999261,15cyclictest86750irq/128-enp1s0-07:50:101
374999261,10cyclictest20457-21cat09:00:021
374999260,6cyclictest0-21swapper/111:46:141
374099261,19cyclictest0-21swapper/007:46:420
374099261,12cyclictest0-21swapper/010:01:020
374099260,12cyclictest0-21swapper/011:08:510
59462250,2sleep30-21swapper/308:25:193
376299252,10cyclictest18580-21kworker/3:110:58:123
374999250,6cyclictest0-21swapper/109:38:261
374999250,6cyclictest0-21swapper/109:38:261
374099251,12cyclictest11638-21idleruntime-cro08:40:010
374099251,11cyclictest0-21swapper/012:04:430
376299242,9cyclictest18580-21kworker/3:111:54:073
376299242,9cyclictest18580-21kworker/3:111:39:583
376299241,9cyclictest18580-21kworker/3:110:39:033
376299241,7cyclictest0-21swapper/311:34:113
376299241,7cyclictest0-21swapper/310:40:083
3762992412,5cyclictest24172-21idleruntime10:20:163
376299240,10cyclictest18580-21kworker/3:111:06:183
374999249,3cyclictest6382-21sed07:15:021
374999241,2cyclictest0-21swapper/109:52:211
374999241,20cyclictest32228-21gltestperf09:25:161
374999241,11cyclictest10099-21ntp_states07:20:191
374999240,7cyclictest0-21swapper/109:48:271
374999240,22cyclictest0-21swapper/111:01:541
374099249,14cyclictest0-21swapper/009:10:010
3740992412,3cyclictest25187-21cpuspeed_turbos09:10:110
374099241,13cyclictest0-21swapper/012:10:150
374099241,10cyclictest0-21swapper/010:25:310
374099240,12cyclictest0-21swapper/010:32:070
376299237,2cyclictest18580-21kworker/3:108:02:533
376299235,9cyclictest0-21swapper/311:45:133
376299232,9cyclictest18580-21kworker/3:110:15:373
376299232,9cyclictest18580-21kworker/3:107:57:433
376299232,7cyclictest18580-21kworker/3:107:25:363
376299231,7cyclictest0-21swapper/312:08:573
376299231,5cyclictest0-21swapper/309:10:343
374999235,3cyclictest1690250kworker/u8:212:33:241
374999231,8cyclictest0-21swapper/111:15:451
374999231,19cyclictest16070-21grep07:35:171
374999231,10cyclictest6940-21cat12:10:011
374999231,10cyclictest1814-21sed08:15:191
374999231,10cyclictest15706-21awk10:00:201
374999230,15cyclictest0-21swapper/109:40:451
374099234,8cyclictest0-21swapper/007:24:210
374099231,9cyclictest0-21swapper/011:57:190
374099231,9cyclictest0-21swapper/011:57:190
374099230,15cyclictest0-21swapper/011:20:550
374099230,11cyclictest0-21swapper/011:11:590
376299226,2cyclictest18580-21kworker/3:112:14:453
376299222,8cyclictest18580-21kworker/3:112:35:053
376299222,8cyclictest18580-21kworker/3:110:51:083
376299222,6cyclictest18580-21kworker/3:108:58:203
376299221,9cyclictest18580-21kworker/3:110:05:113
376299221,9cyclictest18580-21kworker/3:108:33:513
376299221,8cyclictest18580-21kworker/3:109:58:573
376299221,8cyclictest0-21swapper/312:25:333
376299221,4cyclictest0-21swapper/312:35:003
3762992212,2cyclictest18580-21kworker/3:110:45:193
3762992212,2cyclictest18580-21kworker/3:110:45:193
376299221,10cyclictest18580-21kworker/3:109:23:063
374999221,4cyclictest2957-21unixbench-2d09:30:241
374999221,3cyclictest26969-21wc10:25:231
374999221,2cyclictest47792chrt08:21:591
374999221,19cyclictest0-21swapper/112:35:161
374999221,19cyclictest0-21swapper/110:59:231
374999221,19cyclictest0-21swapper/108:26:271
374999221,11cyclictest13204-21munin-run07:30:011
374999220,6cyclictest0-21swapper/110:39:111
374099221,9cyclictest0-21swapper/010:37:030
374099221,7cyclictest0-21swapper/012:35:270
3740992214,5cyclictest5595-21systemd-journal09:39:310
3740992214,5cyclictest5595-21systemd-journal09:39:310
376299212,9cyclictest18580-21kworker/3:107:55:013
376299212,8cyclictest18580-21kworker/3:112:23:293
376299212,5cyclictest86850irq/129-enp1s0-10:04:553
376299211,9cyclictest18580-21kworker/3:109:00:443
376299211,9cyclictest18580-21kworker/3:108:13:573
376299211,8cyclictest18580-21kworker/3:109:36:293
376299211,8cyclictest18580-21kworker/3:109:36:293
376299211,8cyclictest18580-21kworker/3:108:17:093
3762992118,2cyclictest0-21swapper/309:17:093
376299211,5cyclictest0-21swapper/311:20:403
376299211,14cyclictest0-21swapper/309:40:213
376299210,9cyclictest18580-21kworker/3:112:18:093
374999219,3cyclictest851-21sssd_nss11:40:241
374999219,3cyclictest14962-21sensors12:25:231
374999217,4cyclictest6023-21cat07:15:011
374999217,4cyclictest6023-21cat07:15:001
374999214,4cyclictest0-21swapper/111:22:071
374999211,19cyclictest0-21swapper/109:23:181
374999211,18cyclictest0-21swapper/110:32:281
374999211,15cyclictest0-21swapper/108:07:431
374999211,11cyclictest24814-21diskmemload11:50:141
374999210,5cyclictest0-21swapper/111:56:361
374999210,5cyclictest0-21swapper/111:56:361
374099218,3cyclictest20751-21ntp_states11:25:220
374099213,11cyclictest10109-21latency_hist09:50:010
374099211,13cyclictest27500-21kworker/u8:210:47:320
374099211,13cyclictest27500-21kworker/u8:210:47:320
3740992111,2cyclictest3874-21kworker/0:210:55:470
376299207,2cyclictest18580-21kworker/3:107:24:343
376299206,2cyclictest18580-21kworker/3:108:08:193
376299205,2cyclictest18580-21kworker/3:107:45:423
376299201,8cyclictest18580-21kworker/3:111:03:443
376299201,8cyclictest18580-21kworker/3:108:41:173
376299201,7cyclictest0-21swapper/309:27:433
376299201,4cyclictest0-21swapper/311:29:273
376299201,4cyclictest0-21swapper/309:49:153
376299201,11cyclictest407-21/usr/sbin/munin11:55:163
376299201,11cyclictest407-21/usr/sbin/munin11:55:163
376299200,9cyclictest18580-21kworker/3:110:28:493
374999208,11cyclictest11498-21kworker/u8:008:52:411
374999201,8cyclictest0-21swapper/111:05:491
374999201,8cyclictest0-21swapper/110:18:471
3749992016,3cyclictest0-21swapper/111:11:491
3749992016,2cyclictest0-21swapper/109:14:491
374999201,3cyclictest0-21swapper/111:25:241
374999201,3cyclictest0-21swapper/110:24:081
374999201,3cyclictest0-21swapper/108:40:181
374999201,18cyclictest0-21swapper/109:57:591
374999201,17cyclictest0-21swapper/112:15:111
374999201,17cyclictest0-21swapper/111:33:431
374999201,17cyclictest0-21swapper/109:05:241
374999201,17cyclictest0-21swapper/108:48:111
374999201,10cyclictest361-21sshd08:13:091
374999201,10cyclictest28418-21processes09:15:231
374999200,9cyclictest0-21swapper/112:01:181
374099209,2cyclictest86850irq/129-enp1s0-09:22:520
374099205,6cyclictest0-21swapper/010:13:210
374099205,4cyclictest27863-21users11:40:240
374099204,8cyclictest0-21swapper/010:40:260
374099202,15cyclictest4590-21sed07:10:150
374099202,15cyclictest4590-21sed07:10:140
374099201,9cyclictest0-21swapper/009:16:190
374099201,4cyclictest0-21swapper/008:08:090
374099201,3cyclictest21598-21df10:15:150
374099201,16cyclictest2109-21sed09:30:140
374099201,13cyclictest11-21rcuc/012:16:390
374099201,11cyclictest0-21swapper/007:57:180
374099200,13cyclictest0-21swapper/011:16:350
376299198,3cyclictest2711-21/usr/sbin/munin12:00:233
376299198,3cyclictest15627-21/usr/sbin/munin07:35:253
376299197,2cyclictest86550irq/126-enp1s0-10:34:593
376299191,7cyclictest18580-21kworker/3:109:06:223
376299191,12cyclictest0-21swapper/308:47:273
376299191,10cyclictest16238-21/usr/sbin/munin08:50:183
374999198,2cyclictest0-21swapper/109:03:411
374999196,3cyclictest27208-21grep08:00:151
374999192,5cyclictest11625-21munin-run08:40:001
3749991915,3cyclictest0-21swapper/110:05:231
374999191,4cyclictest0-21swapper/110:41:051
374999191,2cyclictest0-21swapper/107:45:121
3749991912,4cyclictest19771-21idleruntime10:10:161
374999191,17cyclictest0-21swapper/110:46:581
374999191,17cyclictest0-21swapper/110:46:581
374999191,17cyclictest0-21swapper/107:34:201
374999191,16cyclictest0-21swapper/107:57:491
374099195,4cyclictest0-21swapper/011:37:270
374099194,3cyclictest0-21swapper/010:05:270
374099192,9cyclictest16818-21perl08:50:130
374099191,3cyclictest9605-21memory11:00:180
3740991913,3cyclictest24128-21tail10:20:150
374099191,12cyclictest0-21swapper/010:50:230
3740991910,2cyclictest22948-21ntp_states11:30:200
374099190,3cyclictest0-21swapper/012:28:550
376299186,3cyclictest18580-21kworker/3:107:12:593
376299186,3cyclictest18580-21kworker/3:107:12:593
376299186,2cyclictest18580-21kworker/3:111:18:323
376299186,2cyclictest18580-21kworker/3:107:40:443
376299185,2cyclictest18580-21kworker/3:111:41:273
374999181,2cyclictest0-21swapper/112:20:021
374999181,16cyclictest0-21swapper/108:30:021
374099187,4cyclictest0-21swapper/009:26:590
374099182,3cyclictest786-21NetworkManager08:44:220
374099181,5cyclictest8403-21unixbench_singl07:15:270
374099181,5cyclictest18569-21expr08:55:120
3740991812,3cyclictest11903-21hddtemp_smartct07:25:160
374099181,14cyclictest1306-21grep08:15:160
376299171,6cyclictest11238-21users08:35:263
374099175,3cyclictest21805-21ps09:00:210
374099171,4cyclictest3609-21perl08:20:160
374099165,3cyclictest8452-21needreboot08:30:210
374099161,5cyclictest5259-21/usr/sbin/munin08:25:160
374099161,3cyclictest0-21swapper/008:00:200
374099161,12cyclictest22218-21/usr/sbin/munin07:50:180
374099151,5cyclictest13432-21/usr/sbin/munin07:30:150
374099150,2cyclictest0-21swapper/008:10:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional