You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 23:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 30, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6964263589,6327sleep24670-21kworker/u8:309:08:292
139982163512,1609sleep06228-1kworker/0:1H11:13:230
14382283118,795sleep314391-21/usr/sbin/munin11:13:473
1372270918,670sleep31491-21/usr/sbin/munin12:13:413
1372270918,670sleep31491-21/usr/sbin/munin12:13:413
31264260112,575sleep131308-21ntpq08:43:361
10726249940,406sleep210816-21fschecks_time11:03:322
391248123,445sleep2435-21ntpq08:48:382
391248123,445sleep2435-21ntpq08:48:382
22486233914,310sleep322490-21ntpq11:38:393
880223047,261sleep28777-21apt-get09:13:342
782922950,58sleep231036-21cyclictest10:53:332
25043229441,149sleep225175-21unixbench_multi10:03:422
21937229135,148sleep222086-21unixbench_multi09:53:452
8358226529,198sleep28512-21date07:33:132
32590226352,63sleep232645-21munin-node07:08:132
13446225920,215sleep313486-21apt-get11:13:103
4181225327,187sleep222530-21kworker/u8:110:43:122
5660225154,143sleep25707-21sshd07:23:322
23542224250,106sleep223685-21unixbench_singl09:58:462
26124222936,119sleep225887-21/usr/sbin/munin10:08:312
17882222811,163sleep217900-21modprobe08:03:102
20166222745,49sleep20-21swapper/211:33:122
4929221149,99sleep20-21swapper/209:03:122
23235219545,86sleep20-21swapper/208:18:372
29721219414,164sleep029729-21needreboot08:38:500
2369421946,151sleep223296-21apt-get11:43:142
2369421946,151sleep223296-21apt-get11:43:142
9456219143,136sleep29497-21ntpq10:58:352
20311219029,145sleep220430-21timerwakeupswit09:48:442
20311219029,145sleep220430-21timerwakeupswit09:48:442
12956218921,47sleep112972-21apt-get07:48:101
12956218921,47sleep112972-21apt-get07:48:101
30315217946,85sleep20-21swapper/212:03:342
2674217641,48sleep20-21swapper/212:18:362
2674217641,48sleep20-21swapper/212:18:362
6463217026,134sleep25666-21snmpd12:30:322
2147217030,127sleep22183-21cat08:54:152
27631216932,125sleep227699-21fschecks_time08:33:312
27631216932,125sleep227699-21fschecks_time08:33:312
27074216924,88sleep20-21swapper/211:53:322
130321657,146sleep21369-21cat12:13:402
130321657,146sleep21369-21cat12:13:402
24281216422,119sleep30-21swapper/311:44:283
24281216422,119sleep30-21swapper/311:44:283
6212216013,133sleep16354-21awk07:23:431
129021579,39sleep20-21swapper/210:33:272
19554215649,48sleep20-21swapper/211:28:442
3161421528,106sleep231616-21cpu12:08:272
5713215013,101sleep25783-21swap09:03:402
14824215047,58sleep214898-21latency_hist07:53:122
14824215047,58sleep214898-21latency_hist07:53:122
26704214916,83sleep20-21swapper/211:53:122
8599214854,54sleep28573-21/usr/sbin/munin07:33:292
18520214831,75sleep218556-21ntpq09:43:352
12583214833,100sleep212630-21ntp_offset11:08:372
1229021488,103sleep212328-21ntpq09:23:362
1164821476,105sleep211685-21latency_hist09:23:112
3939214633,72sleep24024-21fschecks_time12:23:292
1061214316,109sleep01150-21load07:08:350
29596214136,44sleep20-21swapper/208:38:482
21315214136,54sleep20-21swapper/208:13:282
1731921419,86sleep20-21swapper/209:38:422
16285213633,45sleep20-21swapper/209:38:102
9244213518,21sleep00-21swapper/009:14:210
720221339,25sleep27246-21awk10:53:102
1799213017,93sleep31271-21apt-get08:53:233
31800212517,89sleep0369-21sshd08:43:470
30310212418,87sleep330326-21latency_hist10:23:103
29075212317,42sleep20-21swapper/211:58:412
15534212224,58sleep20-21swapper/207:53:412
4014211928,42sleep20-21swapper/207:18:352
2135211914,89sleep12195-21users10:33:461
19814211916,87sleep319854-21sort08:08:323
2562211813,89sleep12620-21if_err_eth007:13:371
17475211525,44sleep20-21swapper/211:23:352
32141211418,63sleep20-21swapper/208:48:012
29663211130,49sleep129682-21apt-get06:58:231
9470210912,23sleep39596-21cat09:14:283
27033210516,74sleep127070-21fschecks_count11:53:311
16338210518,70sleep016421-21wc09:38:120
785210417,71sleep1806-21cpu07:08:281
6204210416,71sleep16216-21/usr/sbin/munin10:48:321
28230210411,78sleep028259-21systemd-run11:58:110
13716210430,41sleep213747-21grep09:28:332
10115210327,41sleep210134-21apt-get07:38:222
10115210327,41sleep210134-21apt-get07:38:222
8998210220,64sleep38573-21/usr/sbin/munin07:33:383
447821026,82sleep14492-21grep10:43:321
2723521028,54sleep2284-21dbus-daemon08:33:112
23362210221,51sleep00-21swapper/011:43:120
28180210110,79sleep228300-21cat10:13:442
22526210113,35sleep10-21swapper/111:38:411
30375210012,71sleep030542-21unixbench_singl06:58:420
2065121005,30sleep20-21swapper/208:08:512
255522998,12sleep30-21swapper/311:48:323
698229811,18sleep00-21swapper/009:08:390
1298429720,35sleep20-21swapper/207:48:112
1298429720,35sleep20-21swapper/207:48:112
3267229614,66sleep332749-21readlink10:28:433
155582968,41sleep20-21swapper/211:18:262
2802029520,58sleep328081-21grep08:33:403
2802029520,58sleep328081-21grep08:33:403
2295229516,21sleep10-21swapper/108:18:321
50029442,13sleep20-21swapper/207:08:172
2914329418,62sleep029173-21fschecks_count10:18:350
3116229310,41sleep20-21swapper/210:23:462
244912939,70sleep124499-21df08:23:281
1660829312,21sleep016619-21apt-get07:58:230
366929222,33sleep10-21swapper/112:23:121
1556329144,27sleep30-21swapper/307:53:423
336029021,18sleep10-21swapper/108:58:141
2777129016,61sleep127793-21kernelversion10:13:351
359928911,58sleep33740-21/usr/sbin/munin10:38:453
3058928814,58sleep330608-21idleruntime-cro08:43:093
229142880,9sleep10-21swapper/109:58:311
1409228812,60sleep314215-21timerwakeupswit09:28:423
3196128713,51sleep031255-21/usr/sbin/munin07:03:420
3036128718,55sleep130440-21date10:23:121
1555728710,17sleep10-21swapper/111:18:251
533328522,32sleep10-21swapper/109:03:311
486828521,41sleep04870-21cron09:03:100
2722128520,46sleep127234-21modprobe08:33:101
1541828521,43sleep10-21swapper/109:33:361
2920128422,41sleep128821-21apt-get08:38:151
2920128422,41sleep128821-21apt-get08:38:151
2184728416,50sleep321919-21sensors_temp08:13:413
1397328428,39sleep20-21swapper/211:13:212
2560228311,57sleep05666-21snmpd08:27:150
1114928320,42sleep011172-21grep09:18:450
1474528226,32sleep014778-21modprobe07:53:100
1474528226,32sleep014778-21modprobe07:53:100
209412815,65sleep00-21swapper/008:12:290
45928019,44sleep1496-21ps08:48:411
45928019,44sleep1496-21ps08:48:411
290842807,12sleep00-21swapper/008:38:140
290842807,12sleep00-21swapper/008:38:140
2076228017,44sleep120446-21/usr/sbin/munin11:33:481
1662628012,49sleep30-21swapper/307:58:263
126528022,37sleep3284-21dbus-daemon08:53:113
126528022,37sleep3284-21dbus-daemon08:53:113
518327921,25sleep00-21swapper/012:28:120
3010627818,51sleep20-21swapper/210:20:312
2821027819,40sleep328225-21modprobe11:58:103
563427718,35sleep00-21swapper/009:03:380
32082779,12sleep00-21swapper/012:18:470
32082779,12sleep00-21swapper/012:18:470
1005027717,41sleep110083-21munin-plugin-st09:18:091
1821027640,18sleep00-21swapper/009:43:290
1511227612,49sleep315172-21fschecks_time09:33:303
75582758,50sleep30-21swapper/310:53:273
2128527417,41sleep121304-21apt-get08:13:231
74622730,5sleep00-21swapper/007:28:380
674427316,35sleep00-21swapper/012:33:120
329427315,47sleep00-21swapper/007:13:520
2545427312,46sleep125466-21/usr/sbin/munin11:48:301
2412327319,27sleep00-21swapper/010:03:120
1920927317,35sleep319278-21grep11:28:413
399927213,45sleep04008-21/usr/sbin/munin07:18:330
2987727224,30sleep229941-21if_eth006:58:322
2933727219,36sleep129328-21apt-get08:38:271
2796827210,13sleep30-21swapper/310:13:413
1127227215,41sleep311273-21sshd09:18:563
97127120,36sleep20-21swapper/210:33:112
792327117,33sleep07968-21/usr/sbin/munin10:53:360
758327126,24sleep30-21swapper/309:09:233
50727122,21sleep30-21swapper/312:13:123
2633127113,14sleep00-21swapper/008:28:360
1302627117,31sleep30-21swapper/307:48:123
1302627117,31sleep30-21swapper/307:48:123
878827019,35sleep1225-21systemd-journal10:58:101
577327020,27sleep30-21swapper/312:28:353
339927010,47sleep13468-21switchtime10:38:431
245827021,34sleep20-21swapper/207:13:342
2192127011,42sleep021962-21/usr/sbin/munin08:13:420
515426915,39sleep35178-21sed12:28:113
3275126911,13sleep00-21swapper/010:28:440
3186726920,22sleep30-21swapper/307:03:413
3103926911,15sleep30-21swapper/307:03:133
3005526911,39sleep030123-21/usr/sbin/munin12:03:280
2454826919,21sleep10-21swapper/111:44:351
2454826919,21sleep10-21swapper/111:44:351
1688226922,23sleep30-21swapper/311:23:123
145632696,51sleep10-21swapper/107:50:491
145632696,51sleep10-21swapper/107:50:491
1118526922,23sleep00-21swapper/011:03:420
232832687,45sleep223296-21apt-get11:43:102
204926817,35sleep32073-21munin-run07:13:103
925826723,24sleep10-21swapper/107:33:441
2648826720,31sleep30-21swapper/310:08:403
2567426718,20sleep30-21swapper/310:08:123
1941326715,26sleep10-21swapper/111:28:431
187962678,39sleep20-21swapper/208:03:422
1788426715,26sleep30-21swapper/309:43:123
121226712,20sleep00-21swapper/012:13:380
121226712,20sleep00-21swapper/012:13:380
1197626714,37sleep311981-21/usr/sbin/munin09:23:293
54512667,50sleep25666-21snmpd10:45:352
522626617,24sleep30-21swapper/307:23:123
2600026617,30sleep026058-21fschecks_count10:08:290
24972667,11sleep00-21swapper/010:37:290
1737626611,21sleep00-21swapper/011:23:330
1042626612,37sleep110430-21diskstats09:18:301
268812655,11sleep10-21swapper/110:09:231
2567226512,38sleep20-21swapper/208:28:102
206126516,19sleep10-21swapper/112:18:111
206126516,19sleep10-21swapper/112:18:111
1226826516,23sleep00-21swapper/011:08:310
410126413,37sleep04147-21systemctl10:43:100
192192646,42sleep10-21swapper/108:05:231
176402646,13sleep30-21swapper/311:23:403
1576626415,32sleep015816-21/usr/sbin/munin11:18:320
834426314,36sleep18357-21latency_hist07:33:121
1354726322,23sleep10-21swapper/111:13:121
1033726319,28sleep010339-21cron11:03:100
973426214,27sleep10-21swapper/110:58:421
669626215,30sleep16780-21apt-config09:08:161
277326216,24sleep30-21swapper/312:18:423
277326216,24sleep30-21swapper/312:18:423
1381826217,31sleep013849-21/usr/sbin/munin09:28:350
1039526215,32sleep010424-21irqstats07:38:360
1039526215,32sleep010424-21irqstats07:38:360
742126116,20sleep30-21swapper/307:28:383
3253526118,19sleep00-21swapper/008:48:320
3253526118,19sleep00-21swapper/008:48:320
3128226119,21sleep10-21swapper/107:03:271
2877426117,25sleep328868-21memory11:58:343
2593826116,29sleep026010-21sed11:48:420
239952615,12sleep30-21swapper/310:02:103
237826116,28sleep32368-21apt-get07:13:313
2258126117,22sleep10-21swapper/109:58:121
2097126119,21sleep30-21swapper/311:33:543
1709426115,29sleep117118-21proc_pri07:58:411
161922615,11sleep00-21swapper/007:56:210
95732609,12sleep30-21swapper/310:58:403
389826016,27sleep30-21swapper/308:58:493
317992606,43sleep00-21swapper/010:27:150
2790626017,28sleep027954-21ntp_offset10:13:380
2623826013,19sleep30-21swapper/308:28:353
1869926012,32sleep318768-21switchtime09:43:403
1188726020,19sleep10-21swapper/109:23:221
1157526016,23sleep10-21swapper/107:43:181
1157526016,23sleep10-21swapper/107:43:181
666325915,26sleep16687-21munin-run12:33:091
2252025916,25sleep322548-21/usr/sbin/munin09:58:103
1858625921,19sleep10-21swapper/109:43:381
180425917,25sleep01797-21apt-get08:53:240
3071025816,24sleep130810-21perl10:23:321
2450325816,17sleep30-21swapper/308:23:303
1817325813,27sleep018167-21apt-get08:03:240
1182025815,16sleep00-21swapper/007:43:330
1182025815,16sleep00-21swapper/007:43:330
2014125716,26sleep320164-21pluginstate09:48:413
2014125716,26sleep320164-21pluginstate09:48:413
1707925710,19sleep10-21swapper/111:23:221
129125711,22sleep30-21swapper/310:33:283
1111525717,23sleep311183-21readlink11:03:413
1063725713,31sleep310718-21grep07:38:433
1063725713,31sleep310718-21grep07:38:433
3149825614,21sleep30-21swapper/308:43:423
3072425617,16sleep30-21swapper/312:03:433
2057525614,26sleep020589-21vmstat09:48:460
2057525614,26sleep020589-21vmstat09:48:460
181482569,19sleep30-21swapper/308:03:233
487925517,23sleep30-21swapper/310:43:423
3161325512,25sleep131605-21apt-get12:08:261
214772558,37sleep00-21swapper/011:34:520
2133225515,25sleep021338-21diskstats09:53:310
1997725514,25sleep020004-21apt-config11:33:100
1394025518,20sleep10-21swapper/109:28:401
570925414,24sleep35743-21irqstats07:23:333
53912546,36sleep00-21swapper/010:44:480
498625413,21sleep30-21swapper/309:03:133
2980825415,18sleep00-21swapper/012:03:120
242502547,12sleep10-21swapper/110:03:131
1552725411,20sleep10-21swapper/107:53:401
1063125413,24sleep110623-21apt-get11:03:271
859725312,24sleep08592-21apt-get07:33:270
631125312,25sleep36341-21awk10:48:353
593925313,23sleep05935-21apt-get10:48:230
437025314,23sleep14381-21ntp_states07:18:421
2869325318,18sleep10-21swapper/111:58:321
276025317,19sleep10-21swapper/112:18:391
276025317,19sleep10-21swapper/112:18:391
2286325314,24sleep322857-21apt-get08:18:253
1654925313,24sleep10-21swapper/109:38:211
3086125218,16sleep30-21swapper/310:23:343
265125216,15sleep10-21swapper/108:54:261
2446325114,15sleep00-21swapper/011:44:340
2446325114,15sleep00-21swapper/011:44:340
1345225112,23sleep312972-21apt-get07:48:213
1345225112,23sleep312972-21apt-get07:48:213
1216625112,24sleep012260-21memory09:23:340
608725012,22sleep06211-21sed07:23:420
321482509,16sleep00-21swapper/012:08:410
2436225012,23sleep024355-21apt-get10:03:240
3262124912,22sleep132627-21ntp_states10:28:411
1979924912,22sleep119805-21df_inode08:08:301
304532478,20sleep130495-21awk12:03:371
1425824713,18sleep10-21swapper/111:13:451
2024924617,16sleep10-21swapper/109:48:431
2024924617,16sleep10-21swapper/109:48:431
2415524413,16sleep00-21swapper/008:23:110
1657424412,17sleep00-21swapper/009:38:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional