You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Jul 29, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29677270311,678sleep129680-21sshd19:19:161
29677270311,678sleep129680-21sshd19:19:161
223022508469,24sleep322304-21sshd20:38:363
7382245613,429sleep17446-21cat21:37:541
1603124516,430sleep016075-21ntpq22:03:170
31274235925,309sleep131334-21fschecks_time22:53:121
192052329105,199sleep019265-21sendmail_mailqu22:13:220
2691622906,269sleep00-21swapper/020:53:270
315032283118,50sleep031525-21cron19:27:510
315032283118,50sleep031525-21cron19:27:510
199632273118,16sleep00-21swapper/023:58:410
28416225117,217sleep327480-21/usr/sbin/munin00:23:263
28416225117,217sleep327480-21/usr/sbin/munin00:23:263
3026922175,200sleep00-21swapper/021:05:550
234102211123,21sleep00-21swapper/022:28:080
687022089,174sleep07018-21latency_hist19:52:550
687022089,174sleep07018-21latency_hist19:52:550
4591220613,31sleep00-21swapper/021:28:120
21773219210,169sleep021809-21ntp_kernel_pll_20:38:170
27263219120,146sleep227311-21cat00:22:532
3170218819,153sleep03222-21latency_hist23:07:530
30341218815,155sleep130435-21fschecks_time19:23:121
30341218815,155sleep130435-21fschecks_time19:23:121
3961218511,161sleep04027-21switchtime23:08:220
116421858,162sleep01307-21date21:17:540
22329218345,57sleep022363-21latency_hist00:07:530
23121218114,143sleep00-21swapper/020:43:140
1374521798,158sleep3195-21jbd2/mmcblk2p2-21:57:423
31933217820,137sleep00-21swapper/000:37:530
9949217512,28sleep00-21swapper/020:02:520
2325221757,108sleep023386-21unixbench_multi00:08:210
13338217510,20sleep013405-21df20:13:100
6308216914,124sleep00-21swapper/023:17:540
719221619,36sleep07276-21fschecks_time19:53:110
719221619,36sleep07276-21fschecks_time19:53:110
2539421619,138sleep0225-21systemd-journal00:14:510
2539421619,138sleep0225-21systemd-journal00:14:510
10302216014,125sleep00-21swapper/023:28:240
26223215810,91sleep026305-21latency_hist19:12:530
11440215832,19sleep00-21swapper/021:48:230
1119321588,133sleep00-21swapper/023:33:090
1119321588,133sleep00-21swapper/023:33:090
29252215610,132sleep029327-21switchtime19:18:220
29252215610,132sleep029327-21switchtime19:18:220
2940121559,123sleep029532-21latency_hist22:47:540
2733821550,35sleep327342-21fschecks_time19:14:083
2733821550,35sleep327342-21fschecks_time19:14:083
3210521525,11sleep00-21swapper/022:53:320
27116215120,43sleep00-21swapper/000:21:380
2447721509,80sleep024525-21awk20:48:100
2447721509,80sleep024525-21awk20:48:100
30893214911,77sleep030909-21idleruntime-cro22:52:510
2005214919,66sleep024707-21cyclictest21:18:230
820721477,14sleep00-21swapper/021:38:210
29424214721,110sleep129436-21kernelversion21:03:131
17726214613,113sleep00-21swapper/020:27:510
25575214410,119sleep025698-21timerandwakeup22:33:210
29184214219,107sleep329249-21awk19:18:213
29184214219,107sleep329249-21awk19:18:213
1407821408,58sleep014143-21cpuspeed21:58:080
18489213912,109sleep00-21swapper/022:12:540
3216421387,111sleep00-21swapper/019:28:160
28338213710,68sleep028486-21unixbench_singl20:58:240
1309521369,115sleep013168-21latency_hist20:12:530
1637221359,14sleep00-21swapper/023:48:190
1927521337,68sleep00-21swapper/023:57:560
18636213310,63sleep018648-21needreboot20:28:320
18224213321,92sleep317740-21apt-get20:28:033
438121319,18sleep00-21swapper/019:43:180
2750521298,17sleep00-21swapper/000:23:070
2750521298,17sleep00-21swapper/000:23:070
935221287,88sleep09397-21systemd-run23:27:520
20688212810,95sleep00-21swapper/020:33:250
16311212725,76sleep216461-21unixbench_multi22:03:232
20795212617,95sleep220857-21mailstats22:18:212
2718921258,16sleep00-21swapper/019:14:020
2718921258,16sleep00-21swapper/019:14:020
12762212315,46sleep00-21swapper/023:38:050
23161212223,81sleep323248-21find00:08:203
76321218,70sleep00-21swapper/022:58:200
76321218,70sleep00-21swapper/022:58:200
466321200,10sleep30-21swapper/319:43:243
214721207,98sleep01-21systemd19:37:530
3119821195,58sleep00-21swapper/021:08:210
3119821195,58sleep00-21swapper/021:08:210
2338221198,27sleep30-21swapper/322:28:033
27001211815,86sleep20-21swapper/222:38:192
968021178,50sleep00-21swapper/021:43:180
1138221165,100sleep00-21swapper/020:06:230
1535121157,48sleep00-21swapper/020:18:180
1535121157,48sleep00-21swapper/020:18:180
2654021116,46sleep00-21swapper/022:38:050
25286211113,81sleep325323-21sed19:08:203
22940211119,66sleep10-21swapper/100:08:141
2311921108,46sleep00-21swapper/022:27:520
1914621080,8sleep312-21rcu_sched23:57:543
709321076,40sleep00-21swapper/023:18:230
28066210628,38sleep30-21swapper/322:43:053
23205210616,74sleep323359-21latency_hist22:27:553
24332210417,65sleep30-21swapper/319:04:373
11276210323,56sleep311364-21tune2fs23:33:123
11276210323,56sleep311364-21tune2fs23:33:123
3774210223,64sleep13821-21latency_hist19:42:541
18436210211,76sleep218470-21idleruntime-cro22:12:522
25506210115,23sleep30-21swapper/322:33:203
20785210111,76sleep120877-21latency_hist00:02:531
97829912,59sleep20-21swapper/219:33:102
2915929924,51sleep229153-21apt-get21:03:062
1370529930,58sleep00-21swapper/021:57:110
2963329818,39sleep10-21swapper/122:48:031
2185829718,64sleep221880-21cpu22:23:052
1900029715,69sleep219054-21latency22:13:162
921629512,67sleep19359-21sed21:42:551
65452948,47sleep00-21swapper/021:33:210
371829236,45sleep00-21swapper/019:42:490
1226029118,53sleep212335-21latency_hist21:52:532
396129028,28sleep00-21swapper/021:24:410
226628915,59sleep32309-21/usr/sbin/munin23:03:203
3010928831,45sleep00-21swapper/000:30:180
1542628746,32sleep0364-21snmpd23:46:010
2031128640,31sleep00-21swapper/022:18:070
39128528,29sleep10-21swapper/122:58:131
39128528,29sleep10-21swapper/122:58:131
3234428513,53sleep20-21swapper/221:13:122
2760828514,54sleep127704-21fschecks_time00:23:121
2760828514,54sleep127704-21fschecks_time00:23:121
2687028320,40sleep226979-21cut19:12:592
2687028320,40sleep226979-21cut19:12:592
245832837,64sleep1364-21snmpd19:07:501
2654128119,36sleep126535-21apt-get22:38:061
2859728013,47sleep10-21swapper/122:43:231
1757628013,49sleep317618-21/usr/sbin/munin22:08:163
1512428013,48sleep315174-21/usr/sbin/munin20:18:133
1512428013,48sleep315174-21/usr/sbin/munin20:18:133
3080727924,26sleep30-21swapper/321:08:123
3080727924,26sleep30-21swapper/321:08:123
2107527926,30sleep121185-21grep22:18:241
622727819,42sleep16270-21cat23:17:511
2668927823,31sleep10-21swapper/100:18:241
2383127822,34sleep223845-21apt-get00:12:512
1569727819,34sleep315833-21gpgv23:47:533
1205527824,25sleep20-21swapper/220:08:152
943427720,26sleep20-21swapper/223:27:542
77182770,7sleep30-21swapper/319:53:213
77182770,7sleep30-21swapper/319:53:213
3035827721,34sleep10-21swapper/100:32:531
1172327715,36sleep111166-21/usr/sbin/munin23:33:221
1172327715,36sleep111166-21/usr/sbin/munin23:33:221
764927616,29sleep20-21swapper/219:53:212
764927616,29sleep20-21swapper/219:53:212
50442760,7sleep20-21swapper/221:28:212
156272768,15sleep30-21swapper/322:03:033
1172527613,43sleep311854-21users21:48:263
343027515,18sleep10-21swapper/121:23:201
2990427521,24sleep30-21swapper/322:48:143
270027521,36sleep32717-21latency_hist21:22:543
2268727523,26sleep10-21swapper/120:42:541
137532758,15sleep20-21swapper/220:13:202
939827416,43sleep19432-21latency_hist23:27:531
2931727413,45sleep2364-21snmpd00:28:142
108127423,28sleep10-21swapper/121:17:511
92952735,48sleep10-21swapper/119:58:211
493527315,40sleep24936-21/usr/sbin/munin23:13:042
278832730,15sleep227480-21/usr/sbin/munin00:23:162
278832730,15sleep227480-21/usr/sbin/munin00:23:162
2371927310,46sleep323824-21users20:43:253
1904027323,27sleep20-21swapper/223:57:542
1244527325,32sleep212474-21apt23:37:512
1244527325,32sleep212474-21apt23:37:512
2236427229,24sleep10-21swapper/100:07:541
1897727220,28sleep10-21swapper/123:57:531
790027113,41sleep17984-21memory21:38:141
1726227113,40sleep117287-21grep22:08:101
1658327118,22sleep20-21swapper/223:48:252
425827024,27sleep20-21swapper/221:27:532
322427019,21sleep10-21swapper/123:07:541
2981727019,31sleep329945-21/usr/sbin/munin00:28:233
2496427018,25sleep10-21swapper/122:33:061
1895227017,25sleep10-21swapper/120:28:531
1791327019,27sleep30-21swapper/320:27:543
3254926913,23sleep20-21swapper/222:57:542
2740626914,19sleep30-21swapper/320:57:543
2650026818,30sleep326566-21/usr/sbin/munin00:18:223
1943726816,23sleep30-21swapper/323:58:033
785426728,20sleep20-21swapper/223:22:532
497826719,32sleep15043-21awk21:28:211
2118326711,40sleep221332-21latency_hist20:37:552
1903126721,29sleep219161-21sed20:28:542
1428326717,28sleep10-21swapper/121:58:131
1250026725,20sleep10-21swapper/121:53:081
254926613,35sleep22586-21/usr/sbin/munin19:38:112
2431926614,35sleep223383-21/usr/sbin/munin22:28:262
3200226519,26sleep10-21swapper/119:28:131
3094526517,28sleep20-21swapper/222:52:532
1406426515,34sleep314089-21cut23:42:533
730126418,30sleep37306-21modprobe21:37:523
2760226416,29sleep127597-21apt-get20:58:051
2360326420,22sleep10-21swapper/122:28:131
19542649,29sleep10-21swapper/123:03:141
149026415,32sleep11554-21sed19:33:211
136062646,12sleep30-21swapper/320:13:173
1069926414,35sleep310736-21ps20:03:193
977926319,29sleep29823-21ntpq21:43:202
817926320,19sleep30-21swapper/323:23:103
40926323,23sleep3489-21awk21:13:253
321726322,19sleep30-21swapper/321:23:143
3195526321,17sleep10-21swapper/100:37:541
2573826324,16sleep30-21swapper/300:17:533
2573826324,16sleep30-21swapper/300:17:533
64826225,20sleep20-21swapper/219:32:542
2630026215,30sleep226393-21/usr/sbin/munin20:53:172
2443826220,26sleep124447-21diskstats00:13:111
2443826220,26sleep124447-21diskstats00:13:111
989226114,30sleep39954-21/usr/sbin/munin21:43:253
312602616,13sleep30-21swapper/322:53:103
3064126119,17sleep20-21swapper/219:23:172
3064126119,17sleep20-21swapper/219:23:172
2644526114,20sleep10-21swapper/120:53:201
2630826114,21sleep10-21swapper/119:12:541
2544326119,22sleep20-21swapper/222:33:192
2249526124,20sleep10-21swapper/122:23:201
137626119,20sleep10-21swapper/121:18:031
801626015,31sleep37594-21/usr/sbin/munin21:38:163
290652605,11sleep20-21swapper/219:18:162
290652605,11sleep20-21swapper/219:18:162
2674726012,25sleep30-21swapper/322:38:123
1617726015,20sleep10-21swapper/120:22:521
1617726015,20sleep10-21swapper/120:22:521
1415326016,29sleep214159-21diskstats21:58:102
1286826013,33sleep1364-21snmpd20:10:361
88425915,27sleep3881-21apt-get19:33:033
681625919,20sleep10-21swapper/123:18:161
389425920,22sleep30-21swapper/323:08:203
2489125914,27sleep20-21swapper/219:08:072
2349825917,17sleep10-21swapper/120:43:231
224825915,25sleep22301-21users21:18:252
1694225923,17sleep10-21swapper/122:07:541
1095225921,18sleep10-21swapper/123:32:531
556125818,21sleep20-21swapper/219:48:082
556125818,21sleep20-21swapper/219:48:082
40622588,33sleep14086-21df19:43:111
2768725813,20sleep20-21swapper/220:58:102
1870425825,16sleep30-21swapper/322:13:053
1645825821,18sleep10-21swapper/123:48:201
1224725816,24sleep312262-21open_inodes20:08:203
1139825821,21sleep10-21swapper/120:06:371
681625713,27sleep16842-21munin-run19:52:521
681625713,27sleep16842-21munin-run19:52:521
3186325715,26sleep231868-21modprobe00:37:512
3053825716,25sleep130682-21modprobe21:07:541
3053825716,25sleep130682-21modprobe21:07:541
267225719,21sleep22767-21users23:03:272
2332025715,24sleep223318-21apt-get19:03:032
157325718,26sleep31611-21modprobe23:02:513
157325718,26sleep31611-21modprobe23:02:513
1491525718,24sleep214992-21grep23:43:242
1405625719,20sleep20-21swapper/223:42:522
1032225720,21sleep20-21swapper/220:03:102
1021725720,18sleep30-21swapper/323:28:233
962925616,18sleep30-21swapper/319:58:513
3133225615,19sleep20-21swapper/221:08:232
3133225615,19sleep20-21swapper/221:08:232
3059225615,24sleep330589-21apt-get00:33:033
2824125615,19sleep30-21swapper/320:58:233
2418225613,19sleep20-21swapper/220:47:522
2228425615,24sleep3278-21dbus-daemon00:07:523
2086725624,18sleep30-21swapper/322:18:223
1779725615,24sleep217912-21cmp20:27:532
887725515,21sleep20-21swapper/221:42:512
710825515,24sleep17178-21df19:53:081
710825515,24sleep17178-21df19:53:081
33862559,18sleep20-21swapper/223:08:032
292472556,11sleep20-21swapper/222:46:352
253282558,36sleep20-21swapper/200:14:012
253282558,36sleep20-21swapper/200:14:012
1980125513,25sleep319796-21apt-get20:33:073
1651825516,23sleep316554-21ntp_states23:48:223
1370925518,23sleep113752-21/usr/sbin/munin20:13:191
137082555,10sleep10-21swapper/123:38:511
559025417,22sleep35632-21df19:48:093
559025417,22sleep35632-21df19:48:093
401225414,24sleep24008-21apt-get19:43:082
2927225417,17sleep30-21swapper/321:03:113
2378425413,17sleep00-21swapper/019:03:170
2103325413,14sleep20-21swapper/200:03:112
1431225412,27sleep114307-21apt-get23:43:071
1387225413,24sleep214025-21latency_hist21:57:542
495425314,23sleep34955-21apt-get23:13:053
494825310,17sleep30-21swapper/321:28:193
2727425318,21sleep10-21swapper/119:14:071
2727425318,21sleep10-21swapper/119:14:071
2021525314,23sleep120252-21ntp_states20:33:171
604325212,24sleep26037-21apt-get21:33:062
2503725218,17sleep20-21swapper/220:48:252
2503725218,17sleep20-21swapper/220:48:252
2204925213,28sleep222167-21timerwakeupswit20:38:232
1508525217,21sleep315086-21sshd21:59:363
985125114,24sleep29852-21sshd20:01:372
3179525112,23sleep331789-21apt-get19:28:073
303212516,14sleep30-21swapper/319:23:103
303212516,14sleep30-21swapper/319:23:103
245525112,21sleep32458-21apt-get19:38:043
1956925116,16sleep20-21swapper/223:58:222
2486625015,19sleep10-21swapper/120:48:201
2486625015,19sleep10-21swapper/120:48:201
1776224912,18sleep10-21swapper/120:27:521
691624711,22sleep36523-21/usr/sbin/munin23:18:183
1503724715,16sleep20-21swapper/220:18:122
1503724715,16sleep20-21swapper/220:18:122
1119824711,20sleep111226-21irqstats21:48:161
189662469,18sleep10-21swapper/122:13:151
3276724511,20sleep2377-21df22:58:102
3276724511,20sleep2377-21df22:58:102
2511624512,15sleep30-21swapper/320:48:263
2511624512,15sleep30-21swapper/320:48:263
1004124312,16sleep20-21swapper/223:28:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional