You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 08:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue May 12, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15658262726,6243sleep14839-21kworker/1:123:59:341
208782194814,1919sleep029441-21kworker/u8:400:14:520
208782194814,1919sleep029441-21kworker/u8:400:14:520
17267267518,640sleep217299-21/usr/sbin/munin22:20:012
17267267518,640sleep217299-21/usr/sbin/munin22:20:012
8215241812,386sleep18249-21awk20:09:351
34512303134,24sleep10-21swapper/121:35:061
1308228194,23sleep10-21swapper/121:29:531
22027227214,243sleep022029-21sshd20:52:050
3294227171,187sleep13263-21latency_hist19:54:361
4910223956,165sleep15062-21latency_hist19:59:371
2311222873,82sleep12358-21ntpq23:14:581
27951222223,41sleep10-21swapper/119:29:451
25971221932,171sleep125995-21fschecks_count21:04:541
1042221851,29sleep11172-21timerwakeupswit23:10:061
5660221715,188sleep35791-21latency_hist21:44:373
5660221715,188sleep35791-21latency_hist21:44:373
14109221513,182sleep114142-21awk23:54:341
14109221513,182sleep114142-21awk23:54:341
24281220850,22sleep10-21swapper/100:24:581
28693220214,18sleep10-21swapper/100:39:361
28693220214,18sleep10-21swapper/100:39:361
26862219283,29sleep126951-21apt-key00:34:351
3043621909,169sleep0363-21snmpd19:35:090
2401218912,162sleep02421-21open_inodes23:15:010
22721218511,127sleep10-21swapper/120:54:451
15994218513,155sleep116009-21modprobe20:34:341
664217815,21sleep10-21swapper/119:45:041
664217815,21sleep10-21swapper/119:45:041
5767217324,131sleep10-21swapper/120:00:061
29080217218,136sleep029145-21fschecks_time21:14:540
16753216818,64sleep116874-21unixbench_singl00:00:071
13469216317,20sleep00-21swapper/023:50:050
13469216317,20sleep00-21swapper/023:50:050
17907215515,127sleep117945-21ntpq00:04:591
2792821508,65sleep128079-21latency_hist22:54:361
2792821508,65sleep128079-21latency_hist22:54:361
27449214819,84sleep10-21swapper/121:09:511
24935214812,17sleep10-21swapper/121:00:061
24935214812,17sleep10-21swapper/121:00:061
4437214641,27sleep10-21swapper/121:39:551
4437214641,27sleep10-21swapper/121:39:551
3061121468,127sleep1363-21snmpd19:37:201
30297214613,75sleep10-21swapper/123:00:051
22371213965,58sleep122479-21sort20:54:371
20718213825,16sleep10-21swapper/120:49:361
20718213825,16sleep10-21swapper/120:49:361
31977213797,22sleep10-21swapper/123:05:061
7206213612,80sleep10-21swapper/121:49:361
12589213696,29sleep112631-21processes22:05:021
8855213562,48sleep10-21swapper/123:35:041
14136213190,21sleep10-21swapper/122:10:031
32130212566,46sleep132120-21apt-get21:24:531
18640212567,46sleep118590-21apt-get22:25:211
14446211918,81sleep214450-21modprobe20:29:342
10055211813,71sleep10-21swapper/120:14:511
31041211716,82sleep131080-21wc21:20:041
23445211781,17sleep10-21swapper/122:39:551
23445211781,17sleep10-21swapper/122:39:551
3924211520,78sleep03965-21ntpq23:19:580
29362211558,41sleep10-21swapper/121:15:001
28771211468,17sleep10-21swapper/122:55:051
2487921127,90sleep10-21swapper/119:19:481
17475211213,28sleep117605-21sed22:20:041
17475211213,28sleep117605-21sed22:20:041
28750211117,76sleep228785-21latency_hist21:14:362
17650211121,26sleep30-21swapper/320:39:373
10645211112,16sleep10-21swapper/121:59:531
20841211014,80sleep320363-21apt-get00:14:423
20841211014,80sleep320363-21apt-get00:14:423
1953211014,80sleep31470-21apt-get19:49:463
1342421109,36sleep10-21swapper/120:24:571
9288210916,78sleep29361-21memory21:54:582
9288210916,78sleep29361-21memory21:54:582
28368210916,75sleep228394-21sed19:29:582
23580210818,73sleep10-21swapper/119:15:001
23580210818,73sleep10-21swapper/119:15:001
2374221078,82sleep00-21swapper/022:40:020
2374221078,82sleep00-21swapper/022:40:020
9469210614,75sleep09528-21/usr/sbin/munin21:55:040
9469210614,75sleep09528-21/usr/sbin/munin21:55:040
5699210615,69sleep10-21swapper/123:25:071
21443210666,16sleep10-21swapper/100:15:091
21443210666,16sleep10-21swapper/100:15:091
15215210613,74sleep315277-21/usr/sbin/munin20:30:043
17944210512,79sleep017955-21fschecks_count20:39:540
12285210413,58sleep30-21swapper/320:20:063
903210123,53sleep01053-21sed19:45:070
903210123,53sleep01053-21sed19:45:070
25038210116,67sleep324554-21apt-get22:44:453
25038210116,67sleep324554-21apt-get22:44:453
24881210111,73sleep324948-21df19:19:503
21454210119,65sleep10-21swapper/119:09:551
324529914,72sleep23284-21processes21:35:032
2008929911,69sleep020249-21awk20:45:070
2008929911,69sleep020249-21awk20:45:070
1964229970,16sleep10-21swapper/100:10:041
1964229970,16sleep10-21swapper/100:10:041
47762988,16sleep1363-21snmpd23:24:231
156832986,11sleep00-21swapper/022:15:020
2925929717,66sleep329276-21uname21:14:573
2516329720,63sleep10-21swapper/122:44:591
2516329720,63sleep10-21swapper/122:44:591
830029617,52sleep00-21swapper/020:09:380
1020629612,30sleep210212-21ntp_states23:40:022
760829531,42sleep26928-21/usr/sbin/munin20:05:112
162832957,14sleep10-21swapper/120:34:501
115512957,13sleep10-21swapper/123:44:571
1033329513,12sleep10-21swapper/123:40:051
157072948,73sleep10-21swapper/122:15:041
116952947,14sleep10-21swapper/120:19:521
2255529314,57sleep222595-21fschecks_count00:19:542
2255529314,57sleep222595-21fschecks_count00:19:542
2063829117,53sleep320121-21/usr/sbin/munin22:30:043
1990429012,65sleep019946-21latency_hist22:29:360
170229012,62sleep31815-21latency_hist23:14:363
84472897,11sleep10-21swapper/120:09:391
227812898,10sleep10-21swapper/122:37:161
1864128915,62sleep218673-21cpu22:25:222
1025428911,63sleep010332-21run-parts23:40:040
981528813,60sleep29847-21cat20:14:362
74452887,64sleep10-21swapper/121:49:481
229132887,13sleep10-21swapper/100:20:011
229132887,13sleep10-21swapper/100:20:011
1730028816,19sleep30-21swapper/322:20:023
1730028816,19sleep30-21swapper/322:20:023
276132877,10sleep10-21swapper/119:28:361
2652028720,46sleep326605-21fschecks_time19:24:553
179562877,62sleep10-21swapper/120:39:551
1394728711,60sleep014037-21load22:09:580
320782868,10sleep10-21swapper/119:41:001
2216528615,57sleep222180-21/usr/sbin/munin22:35:032
162228610,61sleep31833-21latency_hist19:49:383
162228610,61sleep31833-21latency_hist19:49:383
1132128617,46sleep011375-21tr20:19:350
100222855,11sleep10-21swapper/121:56:381
100222855,11sleep10-21swapper/121:56:381
77032846,11sleep10-21swapper/123:32:141
56928418,51sleep3581-21vmstat21:25:113
3110228421,37sleep031096-21apt-get19:39:500
1985628423,21sleep30-21swapper/322:29:353
2695328321,42sleep20-21swapper/200:34:362
1917528314,55sleep319270-21fschecks_time00:09:543
1917528314,55sleep319270-21fschecks_time00:09:543
210982825,11sleep10-21swapper/122:30:381
1729928225,26sleep20-21swapper/200:04:362
2506028119,27sleep00-21swapper/022:44:460
2506028119,27sleep00-21swapper/022:44:460
1500228117,27sleep00-21swapper/023:55:050
1103128117,43sleep311054-21/usr/sbin/munin22:00:043
2588828013,53sleep225951-21/usr/sbin/munin21:04:512
1140128015,48sleep211503-21cut20:19:372
1264427913,16sleep00-21swapper/023:49:360
817527813,29sleep20-21swapper/223:34:492
2852627812,51sleep328589-21switchtime19:30:043
2537727726,29sleep30-21swapper/300:29:363
2343827727,26sleep20-21swapper/220:55:222
2105027719,34sleep30-21swapper/320:49:583
858927614,47sleep08592-21/usr/sbin/munin20:09:540
115732767,15sleep20-21swapper/223:44:582
1998727521,26sleep00-21swapper/000:10:070
1998727521,26sleep00-21swapper/000:10:070
460127421,32sleep24602-21sshd23:22:102
711727317,25sleep20-21swapper/223:30:052
2690627317,32sleep00-21swapper/019:25:040
2609327314,14sleep30-21swapper/321:04:563
2259627313,27sleep30-21swapper/322:35:083
2066627321,31sleep320684-21apt20:49:343
2066627321,31sleep320684-21apt20:49:343
2047927332,24sleep30-21swapper/319:05:213
1586427317,37sleep315997-21wc22:15:063
1395627315,26sleep30-21swapper/320:25:073
943527221,23sleep30-21swapper/323:39:343
2837727216,22sleep00-21swapper/022:54:550
1295127219,22sleep20-21swapper/220:24:382
3189527123,25sleep00-21swapper/023:05:040
2942527115,38sleep229430-21modprobe22:59:332
1780727115,41sleep317865-21sed00:04:573
1579027123,24sleep20-21swapper/222:15:052
74182707,52sleep00-21swapper/021:49:430
644527021,26sleep00-21swapper/021:45:070
2815827019,29sleep327424-21/usr/sbin/munin21:10:093
2787527020,21sleep30-21swapper/300:35:303
2787527020,21sleep30-21swapper/300:35:303
1173527021,21sleep00-21swapper/020:19:550
715326923,31sleep37171-21modprobe21:49:343
639026919,24sleep20-21swapper/223:29:362
2824726914,20sleep00-21swapper/019:29:560
253826920,22sleep00-21swapper/021:34:370
1476626915,29sleep20-21swapper/223:54:592
663826815,36sleep26776-21sed21:45:092
646926821,26sleep30-21swapper/320:04:363
3158626820,28sleep231670-21grep19:40:042
274222687,11sleep00-21swapper/021:09:460
2040626815,24sleep00-21swapper/022:29:560
14532689,37sleep30-21swapper/321:29:553
742426717,31sleep37459-21grep20:05:083
3180226720,25sleep30-21swapper/323:05:033
310602679,46sleep031177-21latency_hist23:04:360
2465426727,18sleep20-21swapper/200:25:072
1241226712,36sleep012490-21memory22:04:570
728426625,23sleep00-21swapper/023:30:070
1339626632,21sleep30-21swapper/323:50:043
1339626632,21sleep30-21swapper/323:50:043
1300926626,23sleep20-21swapper/223:49:552
1300926626,23sleep20-21swapper/223:49:552
781126516,32sleep27837-21netstat21:49:572
3221626516,23sleep20-21swapper/221:24:582
2422726514,27sleep00-21swapper/000:24:570
2264526522,22sleep30-21swapper/319:14:373
2051226515,25sleep20-21swapper/222:29:582
411026416,25sleep30-21swapper/321:39:363
1761226418,26sleep20-21swapper/220:39:362
144052646,12sleep00-21swapper/020:29:040
1342726423,25sleep20-21swapper/222:09:352
487926319,23sleep30-21swapper/323:24:373
2992626321,21sleep30-21swapper/319:35:003
2248426316,30sleep222238-21apt-get20:54:382
1606026320,27sleep316067-21/usr/sbin/munin23:59:533
151726314,29sleep21609-21apt-config19:49:372
151726314,29sleep21609-21apt-config19:49:372
1272626316,25sleep30-21swapper/322:05:043
2704126216,19sleep20-21swapper/219:25:062
2528226215,21sleep00-21swapper/019:19:590
2030426223,22sleep10-21swapper/119:05:191
376326115,21sleep00-21swapper/021:36:100
3167226111,26sleep30-21swapper/319:40:053
3034226117,21sleep20-21swapper/221:19:372
254472617,21sleep30-21swapper/321:02:383
254472617,21sleep30-21swapper/321:02:383
1971026113,31sleep219236-21apt-get19:04:392
1139426113,22sleep30-21swapper/323:44:553
487026017,25sleep00-21swapper/019:59:360
3189026016,22sleep20-21swapper/221:24:372
3066226017,27sleep330664-21diskstats21:19:543
275492605,12sleep20-21swapper/222:50:212
275492605,12sleep20-21swapper/222:50:212
231422606,34sleep30-21swapper/320:55:153
2974625915,20sleep30-21swapper/322:59:513
2971825917,25sleep20-21swapper/222:59:462
2475325915,25sleep024782-21ntp_states21:00:010
2475325915,25sleep024782-21ntp_states21:00:010
2456525912,33sleep024594-21/usr/sbin/munin19:19:350
2456525912,33sleep024594-21/usr/sbin/munin19:19:350
2376625911,34sleep0363-21snmpd20:56:250
225112597,40sleep00-21swapper/019:13:290
1991525924,16sleep00-21swapper/019:05:100
193825914,28sleep31933-21apt-get23:14:483
98425814,31sleep01041-21latency_hist21:29:360
970325817,18sleep30-21swapper/321:55:073
970325817,18sleep30-21swapper/321:55:073
905725817,19sleep30-21swapper/320:10:053
3125425817,17sleep00-21swapper/021:20:070
2466725818,19sleep20-21swapper/219:19:372
2466725818,19sleep20-21swapper/219:19:372
2355025817,22sleep30-21swapper/322:39:573
2355025817,22sleep30-21swapper/322:39:573
2349925813,18sleep20-21swapper/222:39:562
2349925813,18sleep20-21swapper/222:39:562
1183525814,30sleep211884-21unin-run22:04:352
63382579,34sleep00-21swapper/023:29:340
361725718,22sleep30-21swapper/319:54:553
258362579,27sleep00-21swapper/000:29:580
31372566,10sleep20-21swapper/219:53:122
231832566,15sleep30-21swapper/319:14:453
231832566,15sleep30-21swapper/319:14:453
2182025610,18sleep20-21swapper/219:10:042
1811825610,26sleep30-21swapper/320:39:573
16432564,11sleep20-21swapper/223:14:272
1409125614,25sleep314127-21ntp_states22:10:013
989825513,18sleep30-21swapper/323:39:553
509425517,16sleep20-21swapper/221:40:082
509425517,16sleep20-21swapper/221:40:082
278052557,10sleep00-21swapper/022:53:380
278052557,10sleep00-21swapper/022:53:380
2441625514,23sleep224511-21perl20:59:542
2441625514,23sleep224511-21perl20:59:542
201525512,28sleep22022-21diskstats23:14:512
1935925519,18sleep20-21swapper/200:09:572
1935925519,18sleep20-21swapper/200:09:572
1606925515,18sleep00-21swapper/023:59:540
1574525514,23sleep30-21swapper/323:59:363
117502554,10sleep00-21swapper/022:03:280
2849125416,24sleep20-21swapper/200:38:032
2849125416,24sleep20-21swapper/200:38:032
2816925413,25sleep328236-21df22:54:503
2765925417,18sleep00-21swapper/000:35:230
2765925417,18sleep00-21swapper/000:35:230
2084525412,24sleep20-21swapper/200:14:452
2084525412,24sleep20-21swapper/200:14:452
1055525413,19sleep30-21swapper/320:15:063
60832535,37sleep20-21swapper/220:00:472
2579625313,25sleep325834-21irqstats00:29:573
2485425317,18sleep20-21swapper/219:19:452
2378025316,22sleep323881-21sed00:24:363
2378025316,22sleep323881-21sed00:24:363
2246825312,24sleep022463-21apt-get00:19:480
2246825312,24sleep022463-21apt-get00:19:480
406625214,18sleep20-21swapper/219:55:062
3144425214,17sleep20-21swapper/223:04:542
2632925212,23sleep026339-21ntp_states21:05:010
2170425215,20sleep00-21swapper/022:34:490
20312529,16sleep20-21swapper/221:30:072
180652525,13sleep20-21swapper/220:39:562
1680625214,23sleep016801-21apt-get22:19:470
1680625214,23sleep016801-21apt-get22:19:470
817325113,22sleep08168-21apt-get23:34:470
554325112,23sleep05566-21ntp_states20:00:010
2662125110,25sleep326616-21apt-get22:49:503
2662125110,25sleep326616-21apt-get22:49:503
1635725111,23sleep016366-21diskstats20:34:530
159512516,11sleep20-21swapper/220:34:022
662925011,23sleep36622-21apt-get23:29:503
2952525013,20sleep20-21swapper/219:34:512
2582625011,22sleep20-21swapper/222:45:132
2582625011,22sleep20-21swapper/222:45:132
206625011,22sleep02076-21df_inode19:50:030
1817525012,21sleep00-21swapper/000:05:060
84624912,21sleep0848-21ntp_states23:10:010
1885224914,16sleep20-21swapper/200:09:372
515324813,21sleep35160-21df_inode23:24:533
1514824810,15sleep00-21swapper/020:30:010
1415924612,20sleep20-21swapper/222:10:042
1981024113,13sleep20-21swapper/220:45:012
1981024113,13sleep20-21swapper/220:45:012
74802399,14sleep30-21swapper/321:49:503
47702398,17sleep04806-21ntp_states21:40:020
47702398,17sleep04806-21ntp_states21:40:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional