You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 08:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun Dec 21, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
254652501815,4991sleep12360-21kworker/1:220:18:411
128212239512,2363sleep110744-21kworker/u8:323:04:451
17504285864,774sleep227880-1kworker/2:2H21:34:582
17191257014,510sleep217225-21munin-plugin-st23:19:312
2266250515,471sleep02302-21ntpq19:04:560
2266250515,471sleep02302-21ntpq19:04:560
3207724117,369sleep232102-21awk00:04:582
3045323088,286sleep230473-21apt-get20:34:452
5836226619,230sleep35742-21latency_hist20:59:333
9930226460,22sleep20-21swapper/221:10:062
26189222632,181sleep226206-21latency_hist22:04:332
3214222416,21sleep20-21swapper/220:50:162
3214222416,21sleep20-21swapper/220:50:162
25291222014,191sleep125434-21cat23:44:341
12384220124,97sleep212501-21grep23:04:332
864721950,6sleep30-21swapper/300:34:343
512421955,177sleep20-21swapper/222:39:542
512421955,177sleep20-21swapper/222:39:542
9866219416,162sleep19929-21/usr/sbin/munin21:10:051
1015521907,168sleep210304-21latency_hist00:39:342
23276218987,86sleep223320-21mailstats20:10:062
1809321889,92sleep218121-21kernelversion19:54:562
4789218512,152sleep24793-21munin-node19:14:342
2652521788,155sleep226418-21/usr/sbin/munin22:04:542
2652521788,155sleep226418-21/usr/sbin/munin22:04:542
3609216972,32sleep23641-21modprobe00:19:312
29618216920,133sleep329709-21fschecks_time22:14:523
32731216112,76sleep20-21swapper/222:24:522
5151215849,47sleep20-21swapper/220:55:082
4535215112,73sleep24621-21timerandwakeup00:20:042
25949215111,69sleep20-21swapper/223:45:032
7752215010,78sleep27907-21modprobe22:49:352
22376214913,123sleep322421-21grep23:34:513
28394214646,14sleep20-21swapper/220:26:172
31192214510,119sleep231285-21grep22:19:522
7440214130,11sleep20-21swapper/200:30:262
3571214141,78sleep13647-21fschecks_time19:09:571
2237521417,15sleep20-21swapper/223:34:502
4463214029,13sleep20-21swapper/222:38:332
4463214029,13sleep20-21swapper/222:38:332
1585721409,20sleep20-21swapper/221:29:572
1585721409,20sleep20-21swapper/221:29:572
13422213946,11sleep20-21swapper/221:20:162
13422213946,11sleep20-21swapper/221:20:162
6762213850,70sleep20-21swapper/200:29:332
6762213850,70sleep20-21swapper/200:29:332
28912213825,97sleep24793-21munin-node20:29:442
25013213646,73sleep225163-21grep20:15:072
6149213410,107sleep06159-21pluginstate22:44:330
6149213410,107sleep06159-21pluginstate22:44:330
24973213429,13sleep20-21swapper/223:41:402
20157213349,58sleep20-21swapper/223:25:252
26059213229,88sleep226103-21if_eth020:19:552
817213030,12sleep20-21swapper/220:44:142
817213030,12sleep20-21swapper/220:44:142
29055213052,66sleep229119-21awk23:55:012
29055213052,66sleep229119-21awk23:55:012
1396212720,90sleep31415-21uname20:44:563
10509212726,11sleep20-21swapper/222:56:342
26997212633,72sleep20-21swapper/223:49:402
7143212419,82sleep07292-21grep22:45:050
9136212320,82sleep20-21swapper/221:09:342
29610212119,28sleep20-21swapper/222:14:512
26209212115,90sleep326356-21latency_hist22:04:343
11378212142,43sleep211387-21munin-run19:34:332
11378212142,43sleep211387-21munin-run19:34:332
6362212035,17sleep20-21swapper/220:59:532
8669211920,42sleep20-21swapper/219:24:562
8669211920,42sleep20-21swapper/219:24:562
31462211845,16sleep231469-21munin-run00:04:332
28204211633,17sleep20-21swapper/222:09:542
22458211662,35sleep20-21swapper/221:50:042
22458211662,35sleep20-21swapper/221:50:042
22132211618,75sleep20-21swapper/223:34:332
22132211618,75sleep20-21swapper/223:34:332
32132211513,74sleep032212-21tune2fs20:39:520
32132211513,74sleep032212-21tune2fs20:39:520
5536211422,67sleep35676-21sed22:40:033
5536211422,67sleep35676-21sed22:40:033
14821211312,18sleep20-21swapper/221:25:082
14821211312,18sleep20-21swapper/221:25:082
1347221126,10sleep00-21swapper/021:20:540
1347221126,10sleep00-21swapper/021:20:540
1569421118,22sleep215671-21latency_hist23:14:352
11396211112,27sleep20-21swapper/221:15:032
11396211112,27sleep20-21swapper/221:15:032
2884210816,74sleep12399-21apt-get20:49:421
2884210816,74sleep12399-21apt-get20:49:421
11519210820,69sleep311526-21open_files23:00:023
3222210623,69sleep23318-21latency_hist19:09:342
3222210623,69sleep23318-21latency_hist19:09:342
30024210614,76sleep030103-21awk22:15:010
23774210628,14sleep20-21swapper/221:55:002
31588210515,75sleep30-21swapper/322:20:013
1739210429,20sleep20-21swapper/222:29:492
31708210332,39sleep30-21swapper/300:04:473
2540210310,77sleep32582-21cat00:14:523
22729210330,45sleep022725-21apt-get20:09:480
5698210213,70sleep05792-21/usr/sbin/munin19:14:570
3969210217,71sleep14114-21sed22:35:051
3969210217,71sleep14114-21sed22:35:051
15931210228,47sleep015927-21apt-get23:14:450
7278210119,64sleep07243-21apt-get00:30:070
1804921016,41sleep20-21swapper/223:20:042
1446921017,14sleep10-21swapper/123:09:531
7586210023,47sleep37670-21latency_hist21:04:343
1896721006,12sleep00-21swapper/021:39:560
3089529816,66sleep131049-21latency_hist22:19:341
746529718,61sleep36981-21/usr/sbin/munin19:20:003
746529718,61sleep36981-21/usr/sbin/munin19:20:003
505729719,47sleep20-21swapper/219:14:382
96329514,67sleep11021-21sort00:09:531
741429518,62sleep17438-21fschecks_count00:30:251
205472946,77sleep20-21swapper/220:03:082
1683229422,52sleep016852-21apt21:34:310
1683229422,52sleep016852-21apt21:34:310
2117929332,16sleep20-21swapper/220:04:452
252729214,61sleep12727-21latency_hist20:49:341
159029216,61sleep01596-21ntp_states20:45:010
1163129215,60sleep111776-21/usr/sbin/munin21:15:061
1163129215,60sleep111776-21/usr/sbin/munin21:15:061
1869529125,50sleep20-21swapper/221:39:462
3030928916,23sleep00-21swapper/000:00:150
2481828920,29sleep00-21swapper/020:15:050
1748828913,63sleep117502-21netstat21:34:571
119152888,68sleep211950-21latency19:35:092
119152888,68sleep211950-21latency19:35:092
198328719,52sleep32012-21grep19:04:513
198328719,52sleep32012-21grep19:04:513
1474128629,14sleep20-21swapper/219:44:462
1474128629,14sleep20-21swapper/219:44:462
145422869,16sleep214477-21latency_hist19:44:342
3240728513,52sleep032531-21cat22:24:340
1902628320,38sleep118776-21apt-get23:24:351
2715328224,29sleep10-21swapper/120:24:351
93228123,32sleep30-21swapper/322:25:063
214372816,63sleep20-21swapper/221:48:512
1332328128,12sleep20-21swapper/219:39:542
808028022,10sleep20-21swapper/219:22:132
808028022,10sleep20-21swapper/219:22:132
1780028013,50sleep317814-21acpi19:54:453
1476327918,29sleep00-21swapper/023:10:000
172227822,9sleep20-21swapper/200:10:152
1568727827,20sleep30-21swapper/321:29:543
1568727827,20sleep30-21swapper/321:29:543
141502789,18sleep214286-21latency_hist23:09:342
924727712,23sleep00-21swapper/022:54:340
89227721,26sleep30-21swapper/320:44:353
89227721,26sleep30-21swapper/320:44:353
641927713,46sleep06463-21fschecks_time21:00:110
506927721,34sleep10-21swapper/120:55:071
2846127720,27sleep10-21swapper/122:10:021
1994627720,40sleep319956-21modprobe21:44:323
162842777,14sleep20-21swapper/219:49:472
3147027620,30sleep30-21swapper/300:04:343
2749027611,16sleep10-21swapper/123:49:591
1259427612,30sleep30-21swapper/321:19:523
1259427612,30sleep30-21swapper/321:19:523
1214527619,31sleep10-21swapper/119:35:171
1214527619,31sleep10-21swapper/119:35:171
2662527513,20sleep00-21swapper/022:04:560
2662527513,20sleep00-21swapper/022:04:560
1534227512,46sleep315469-21latency_hist21:29:343
1534227512,46sleep315469-21latency_hist21:29:343
756027423,40sleep20-21swapper/222:48:182
401627420,26sleep30-21swapper/300:19:533
2593827420,27sleep00-21swapper/020:19:530
2094027424,30sleep10-21swapper/120:04:361
1759727412,44sleep117743-21date19:54:351
1287927422,35sleep012908-21munin-run19:39:310
1287927422,35sleep012908-21munin-run19:39:310
770627321,36sleep10-21swapper/122:49:341
417527318,34sleep34223-21systemd-run20:54:333
417527318,34sleep34223-21systemd-run20:54:333
3221527318,22sleep10-21swapper/120:39:531
3221527318,22sleep10-21swapper/120:39:531
2094927322,22sleep10-21swapper/123:29:521
2094927322,22sleep10-21swapper/123:29:521
149927318,34sleep00-21swapper/022:29:350
984527222,21sleep00-21swapper/021:10:040
597327218,25sleep00-21swapper/020:59:340
492727212,30sleep10-21swapper/122:39:491
492727212,30sleep10-21swapper/122:39:491
3057527220,28sleep30-21swapper/320:34:543
2694127221,27sleep10-21swapper/122:05:051
2694127221,27sleep10-21swapper/122:05:051
237927222,40sleep20-21swapper/220:49:272
2310127213,44sleep123144-21tr20:10:011
209027217,40sleep02107-21cat00:14:330
2769127118,34sleep027822-21timerwakeupswit23:50:020
2825327012,41sleep328336-21sed22:09:553
244427022,31sleep10-21swapper/119:05:011
244427022,31sleep10-21swapper/119:05:011
1534827021,20sleep30-21swapper/319:45:043
1534827021,20sleep30-21swapper/319:45:043
1304927012,16sleep00-21swapper/023:05:080
1069927023,26sleep30-21swapper/319:30:063
1069927023,26sleep30-21swapper/319:30:063
957926823,16sleep30-21swapper/322:54:533
809526822,20sleep00-21swapper/021:04:550
753526823,26sleep10-21swapper/121:04:321
2464926819,22sleep30-21swapper/321:59:343
2421826818,28sleep00-21swapper/021:55:070
1915826810,13sleep10-21swapper/121:40:011
2387026717,26sleep10-21swapper/121:55:041
1000926716,36sleep310139-21sed21:10:073
543826611,29sleep10-21swapper/100:24:391
543826611,29sleep10-21swapper/100:24:391
389126619,30sleep00-21swapper/022:35:040
389126619,30sleep00-21swapper/022:35:040
2912326613,40sleep329199-21sed23:55:023
2912326613,40sleep329199-21sed23:55:023
2680326622,22sleep30-21swapper/322:04:593
2680326622,22sleep30-21swapper/322:04:593
2527526617,31sleep125315-21awk21:59:561
2071026619,26sleep320800-21systemd-run20:04:343
1881626620,28sleep318907-21apt-config23:24:333
840426520,23sleep30-21swapper/322:50:013
700826511,35sleep07078-21/usr/sbin/munin19:19:490
700826511,35sleep07078-21/usr/sbin/munin19:19:490
525826519,22sleep00-21swapper/000:24:340
2534926512,38sleep025372-21pluginstate21:59:590
2060826519,21sleep10-21swapper/121:44:581
828626419,21sleep10-21swapper/121:05:011
375426414,35sleep03842-21sed19:10:000
2833826414,23sleep00-21swapper/022:09:560
683926318,24sleep36979-21apt-config00:29:343
683926318,24sleep36979-21apt-config00:29:343
314162634,48sleep10-21swapper/120:35:291
215982636,12sleep00-21swapper/020:05:320
2051026324,17sleep30-21swapper/321:44:563
1890926316,32sleep0287-21dbus-daemon23:24:340
887126210,37sleep18877-21diskstats00:34:491
379426219,18sleep30-21swapper/322:35:033
379426219,18sleep30-21swapper/322:35:033
234226216,20sleep00-21swapper/022:30:030
2178126213,28sleep30-21swapper/321:49:493
2178126213,28sleep30-21swapper/321:49:493
2174426211,32sleep320809-21/usr/sbin/munin23:30:063
2174426211,32sleep320809-21/usr/sbin/munin23:30:063
1803526213,31sleep10-21swapper/119:54:551
1305626219,21sleep10-21swapper/121:20:031
1305626219,21sleep10-21swapper/121:20:031
917826110,35sleep39326-21sed21:09:353
422626119,20sleep00-21swapper/020:54:340
422626119,20sleep00-21swapper/020:54:340
2432226117,27sleep324346-21/usr/sbin/munin20:14:523
1926326114,31sleep119272-21/usr/sbin/munin23:24:431
1514726116,29sleep0228-21systemd-journal21:27:410
1514726116,29sleep0228-21systemd-journal21:27:410
1474526116,28sleep114815-21/usr/sbin/munin19:44:491
1474526116,28sleep114815-21/usr/sbin/munin19:44:491
2905726015,23sleep30-21swapper/320:29:543
2760126016,18sleep30-21swapper/323:50:013
2402826018,18sleep30-21swapper/323:39:513
2043226012,35sleep020433-21sshd20:01:400
369025918,20sleep00-21swapper/000:19:340
3264625915,27sleep132640-21apt-get22:24:471
3226325921,18sleep00-21swapper/000:05:040
2285925913,32sleep022927-21systemctl23:35:040
2115525918,23sleep10-21swapper/120:04:441
194625914,27sleep32003-21awk22:29:543
1401925920,19sleep10-21swapper/121:24:471
1401925920,19sleep10-21swapper/121:24:471
1379925914,30sleep013877-21/usr/sbin/munin19:40:040
1379925914,30sleep013877-21/usr/sbin/munin19:40:040
3031325812,20sleep10-21swapper/100:00:161
249362587,39sleep00-21swapper/023:41:100
169712584,12sleep30-21swapper/321:34:343
1629725819,25sleep316309-21/usr/sbin/munin23:14:563
1507025817,15sleep30-21swapper/323:10:083
1132525824,18sleep30-21swapper/321:15:003
1132525824,18sleep30-21swapper/321:15:003
390325716,18sleep10-21swapper/100:19:431
2597325714,17sleep30-21swapper/323:45:043
2270625711,19sleep30-21swapper/320:09:473
2153825718,18sleep00-21swapper/021:49:330
2080825712,29sleep020809-21munin-node23:29:430
2080825712,29sleep020809-21munin-node23:29:430
1915525715,19sleep00-21swapper/019:59:340
172712577,39sleep00-21swapper/019:51:060
879525619,19sleep30-21swapper/300:34:473
87725616,23sleep3881-21diskstats00:09:513
640225616,24sleep36396-21apt-get22:44:483
1546225618,17sleep00-21swapper/019:45:050
1546225618,17sleep00-21swapper/019:45:050
879225517,19sleep00-21swapper/000:34:440
846625515,23sleep18548-21mailstats22:50:031
3171125513,24sleep131705-21apt-get00:04:491
293322556,12sleep10-21swapper/123:55:041
293322556,12sleep10-21swapper/123:55:041
2893725514,24sleep128932-21apt-get20:29:471
2739625512,27sleep127391-21apt-get20:24:511
2679025515,23sleep00-21swapper/023:49:330
855025412,25sleep18544-21apt-get19:24:511
855025412,25sleep18544-21apt-get19:24:511
700425413,24sleep16999-21apt-get19:19:461
700425413,24sleep16999-21apt-get19:19:461
3110625412,25sleep131100-21apt-get22:19:471
2920525411,17sleep00-21swapper/023:55:030
2920525411,17sleep00-21swapper/023:55:030
1929225411,25sleep019284-21apt-get23:24:460
1152925412,23sleep011593-21/usr/sbin/munin23:00:030
1092325412,25sleep010917-21apt-get21:14:490
1092325412,25sleep010917-21apt-get21:14:490
1080925417,20sleep010948-21grep19:30:070
1080925417,20sleep010948-21grep19:30:070
1073725413,25sleep110769-21munin-run22:59:321
987025319,18sleep00-21swapper/019:29:340
987025319,18sleep00-21swapper/019:29:340
1636125313,23sleep116385-21ntp_states23:14:581
1370625315,15sleep30-21swapper/319:40:033
1370625315,15sleep30-21swapper/319:40:033
1210025315,22sleep312106-21ntp_states19:35:143
1210025315,22sleep312106-21ntp_states19:35:143
998225212,24sleep09463-21/usr/sbin/munin22:55:020
2228425218,19sleep00-21swapper/021:50:020
2228425218,19sleep00-21swapper/021:50:020
2177925211,25sleep121774-21apt-get21:49:471
2177925211,25sleep121774-21apt-get21:49:471
1628525212,24sleep316280-21apt-get19:49:483
1584225216,16sleep10-21swapper/121:29:561
1584225216,16sleep10-21swapper/121:29:561
1285125213,23sleep312842-21apt-get23:04:513
1133825212,19sleep10-21swapper/119:34:321
1133825212,19sleep10-21swapper/119:34:321
504625115,21sleep15056-21ntp_states19:14:371
2900725113,22sleep029014-21df_inode20:29:510
2736025113,24sleep327361-21sshd20:24:413
2118325112,22sleep321175-21apt-get20:04:463
203725112,22sleep12104-21memory22:29:561
1159425112,22sleep10-21swapper/123:00:041
626925013,18sleep00-21swapper/000:25:070
626925013,18sleep00-21swapper/000:25:070
34832507,23sleep30-21swapper/319:09:513
279425013,22sleep12800-21ntp_states00:14:581
3055124914,20sleep030558-21df_inode20:34:510
235322495,33sleep10-21swapper/123:38:201
2869624813,20sleep00-21swapper/020:29:340
2576024812,17sleep10-21swapper/123:44:581
1972624811,19sleep319768-21ntp_states23:25:183
1438524814,20sleep114386-21sshd19:43:401
1438524814,20sleep114386-21sshd19:43:401
984024711,17sleep30-21swapper/319:29:333
984024711,17sleep30-21swapper/319:29:333
2593024712,19sleep325936-21diskstats20:19:523
1712524713,20sleep017119-21apt-get21:34:470
567324313,14sleep30-21swapper/319:14:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional