You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 07:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sun Apr 05, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30503253517,501sleep130538-21/usr/sbin/munin00:02:191
1981324626,442sleep13120-1kworker/1:2H20:02:111
1981324626,442sleep13120-1kworker/1:2H20:02:111
25525244738,228sleep225539-21fschecks_count23:47:142
6925239620,357sleep37049-21/usr/sbin/munin21:02:263
2807323908,343sleep228158-21cut20:27:242
2807323908,343sleep228158-21cut20:27:242
15269231614,193sleep20-21swapper/219:47:182
2152722889,267sleep318375-21kworker/3:021:51:393
19791227710,254sleep219874-21sed23:27:232
9118225446,80sleep29272-21unixbench_multi19:27:272
19284224615,20sleep10-21swapper/120:01:551
20097223618,201sleep020162-21grep20:02:310
20097223618,201sleep020162-21grep20:02:310
30906222634,173sleep230918-21sh22:21:542
21332221311,48sleep20-21swapper/220:07:052
31967220712,150sleep231970-21cron20:41:542
31967220712,150sleep231970-21cron20:41:542
31682220733,132sleep231728-21grep00:07:092
6526219325,56sleep20-21swapper/222:46:572
32628217320,137sleep032674-21grep19:02:100
29852217211,147sleep129895-21unin-run00:01:551
2457216454,43sleep20-21swapper/200:17:142
2457216454,43sleep20-21swapper/200:17:142
26420215913,131sleep126462-21sed20:22:221
26420215913,131sleep126462-21sed20:22:221
365521519,19sleep23738-21tr20:52:242
365521519,19sleep23738-21tr20:52:242
1569321518,106sleep20-21swapper/221:32:122
749214615,53sleep20-21swapper/200:12:052
14538214571,35sleep20-21swapper/223:12:092
13543214314,25sleep20-21swapper/223:07:242
936621429,118sleep29376-21find21:12:062
402221428,72sleep24058-21cat19:12:172
1858421415,12sleep00-21swapper/023:22:290
16526214142,80sleep116578-21fschecks_count19:52:141
22370214016,61sleep20-21swapper/221:52:222
26504213912,23sleep00-21swapper/020:22:250
26504213912,23sleep00-21swapper/020:22:250
24096213623,92sleep124246-21unixbench_multi21:57:261
7335213518,94sleep00-21swapper/022:47:230
29410213515,16sleep10-21swapper/122:16:561
1257921338,76sleep213441-21snmpd23:05:182
3144521308,103sleep20-21swapper/200:06:562
7223212916,61sleep27252-21proc_pri22:47:212
1920212819,13sleep20-21swapper/222:32:112
5717212796,15sleep20-21swapper/219:17:182
2057212731,60sleep22101-21ntpq20:48:042
2057212731,60sleep22101-21ntpq20:48:042
17044212620,86sleep017125-21timerandwakeup19:52:250
1968321255,10sleep20-21swapper/221:42:502
1968321255,10sleep20-21swapper/221:42:502
5955212429,57sleep26034-21mailstats00:27:242
9695212314,25sleep09590-21latency_hist22:56:560
13318212331,56sleep213330-21apt-get19:42:052
13318212331,56sleep213330-21apt-get19:42:052
22612212228,56sleep222652-21latency23:37:162
14549212215,89sleep314613-21find21:27:253
10351212257,19sleep20-21swapper/222:57:232
2327321216,92sleep223320-21ntpq20:12:182
2327321216,92sleep223320-21ntpq20:12:182
21391212111,58sleep20-21swapper/223:32:242
1055321219,55sleep20-21swapper/219:32:242
1571212017,87sleep11589-21/usr/sbin/munin20:47:061
1571212017,87sleep11589-21/usr/sbin/munin20:47:061
8689211833,27sleep21524-21cyclictest22:52:182
24969211810,42sleep21524-21cyclictest22:02:072
16738211742,38sleep216864-21cat23:17:262
1253621177,60sleep213441-21snmpd19:38:222
28065211618,54sleep20-21swapper/222:12:092
19996211613,84sleep020118-21/usr/sbin/munin23:27:250
811221115,10sleep20-21swapper/219:26:302
2449621115,50sleep20-21swapper/221:59:332
24287211012,82sleep124196-21latency_hist20:16:571
24287211012,82sleep124196-21latency_hist20:16:571
32085210912,79sleep032133-21/usr/sbin/munin00:07:190
591221085,91sleep20-21swapper/221:00:512
25057210717,72sleep025125-21fschecks_time22:02:130
9659210519,72sleep19686-21kernelversion21:12:171
1862121059,78sleep018698-21/usr/sbin/munin19:57:260
2337210317,65sleep02476-21grep20:48:100
2337210317,65sleep02476-21grep20:48:100
17854210350,37sleep20-21swapper/223:22:142
26290210242,44sleep20-21swapper/220:22:192
26290210242,44sleep20-21swapper/220:22:192
443721015,63sleep20-21swapper/222:38:452
443721015,63sleep20-21swapper/222:38:452
20678210131,14sleep20-21swapper/220:02:412
20678210131,14sleep20-21swapper/220:02:412
15443210118,69sleep315515-21ls19:47:233
603629912,74sleep16067-21cut21:01:551
302132995,64sleep20-21swapper/220:34:392
1643729720,25sleep20-21swapper/219:52:072
9722965,79sleep20-21swapper/220:45:452
1382529629,55sleep213834-21munin-run21:26:552
77929524,56sleep2884-21timerandwakeup19:02:242
701529316,58sleep20-21swapper/200:32:082
372829313,58sleep33729-21/usr/sbin/munin19:12:063
237329344,35sleep22498-21cat19:07:242
2350029311,15sleep00-21swapper/023:40:380
159829341,41sleep21607-21munin-run22:31:552
354229228,35sleep33627-21tune2fs22:37:133
354229228,35sleep33627-21tune2fs22:37:133
89729111,14sleep30-21swapper/300:12:143
312429120,54sleep03142-21/usr/sbin/munin20:52:050
312429120,54sleep03142-21/usr/sbin/munin20:52:050
1845529122,47sleep318468-21cron21:41:543
996529013,61sleep09976-21apt-get19:32:060
637629036,29sleep00-21swapper/021:02:150
1977929019,51sleep319298-21apt-get20:02:053
1977929019,51sleep319298-21apt-get20:02:053
1383729021,21sleep00-21swapper/021:26:570
1190629011,64sleep111913-21open_files23:02:231
570128915,21sleep10-21swapper/119:17:171
1802028734,35sleep20-21swapper/219:57:112
1719528721,49sleep20-21swapper/221:37:082
1062328719,47sleep310755-21timerwakeupswit19:32:253
985728616,53sleep39895-21/usr/sbin/munin21:12:233
811228648,24sleep28137-21kernelversion21:07:172
543028616,52sleep15494-21fw_packets22:43:081
257328615,54sleep02629-21if_eth000:17:160
257328615,54sleep02629-21if_eth000:17:160
117228621,49sleep01207-21ntp_kernel_pll_00:12:200
464528522,39sleep04786-21sed00:22:250
2857828533,35sleep20-21swapper/223:57:072
48228415,50sleep3528-21unin-run00:11:543
3692848,57sleep00-21swapper/022:27:130
2659528435,33sleep20-21swapper/222:07:122
1559428410,59sleep015613-21apt-get21:32:060
827528320,38sleep38298-21munin-run00:36:543
709928319,44sleep17116-21uniq00:32:111
2481628312,56sleep324818-21sshd23:42:393
2449028314,58sleep124513-21grep20:17:091
159928326,37sleep31591-21apt-get20:47:123
159928326,37sleep31591-21apt-get20:47:123
373928211,51sleep33825-21/usr/sbin/munin20:52:253
373928211,51sleep33825-21/usr/sbin/munin20:52:253
315952829,26sleep231171-21/usr/sbin/munin22:22:182
1987528225,39sleep319993-21grep23:27:243
527328123,31sleep30-21swapper/300:26:563
2674728122,27sleep30-21swapper/323:51:553
183528012,50sleep11840-21df_inode19:07:111
1037628019,33sleep30-21swapper/322:57:243
3259927917,43sleep332603-21acpi19:02:053
2990027910,32sleep30-21swapper/320:32:263
2744127916,16sleep10-21swapper/123:52:221
1152127915,25sleep30-21swapper/321:17:233
1074027916,47sleep010839-21latency_hist21:16:560
269872789,14sleep20-21swapper/223:52:112
2309827820,39sleep123131-21munin-run21:56:531
143427834,28sleep11502-21systemd-run19:06:551
1336427818,41sleep313330-21apt-get19:42:263
1336427818,41sleep313330-21apt-get19:42:263
109732787,15sleep20-21swapper/221:17:062
368127726,21sleep30-21swapper/300:21:563
368127726,21sleep30-21swapper/300:21:563
2439127718,30sleep00-21swapper/023:42:230
2148627716,39sleep321333-21/usr/sbin/munin20:07:143
1227927718,38sleep312419-21grep19:37:263
516427630,11sleep20-21swapper/200:26:432
3096827617,42sleep330980-21/usr/sbin/munin20:37:143
3096827617,42sleep330980-21/usr/sbin/munin20:37:143
2660327616,41sleep326493-21/usr/sbin/munin22:07:133
2492027630,11sleep20-21swapper/223:43:592
1620427613,48sleep016307-21cat19:51:560
742527527,12sleep20-21swapper/221:05:272
3079327522,24sleep00-21swapper/020:37:110
3079327522,24sleep00-21swapper/020:37:110
2037627518,13sleep20-21swapper/221:47:132
176027515,41sleep30-21swapper/319:07:093
65582747,55sleep00-21swapper/000:29:440
2546727430,12sleep20-21swapper/220:18:192
1588727412,46sleep116034-21latency_hist23:16:561
1453427430,11sleep20-21swapper/221:27:242
14142746,13sleep10-21swapper/122:30:081
1270327424,26sleep00-21swapper/023:06:540
296172728,51sleep011-21ksoftirqd/020:32:230
2131827224,24sleep10-21swapper/123:32:231
1334527212,45sleep013330-21apt-get19:42:110
1334527212,45sleep013330-21apt-get19:42:110
521927112,15sleep10-21swapper/100:26:541
334827124,22sleep00-21swapper/019:11:590
2738627117,36sleep327431-21grep23:52:203
2677527111,18sleep00-21swapper/023:51:560
215222710,7sleep30-21swapper/323:32:243
2094927116,24sleep00-21swapper/023:32:140
1231327125,23sleep00-21swapper/021:21:560
1016227126,26sleep10-21swapper/122:57:181
2242727017,37sleep322489-21fschecks_time23:37:133
160827021,27sleep30-21swapper/322:31:563
760526910,17sleep10-21swapper/121:06:561
481726922,29sleep04855-21grep20:57:180
36152698,46sleep13652-21ps20:52:231
36152698,46sleep13652-21ps20:52:231
3271826918,31sleep332760-21wc20:42:233
3246526915,38sleep032517-21cat20:42:160
2187726917,24sleep00-21swapper/020:07:230
1743426931,18sleep00-21swapper/023:21:560
88462689,12sleep30-21swapper/321:08:393
510026824,20sleep10-21swapper/119:16:561
2506026823,26sleep30-21swapper/320:17:243
1184926816,29sleep30-21swapper/323:02:213
1767126721,29sleep117734-21/usr/sbin/munin23:22:091
255582665,14sleep00-21swapper/020:19:310
2350026618,26sleep00-21swapper/021:57:130
1046626617,20sleep10-21swapper/119:32:231
2523926518,27sleep30-21swapper/322:02:173
583726416,33sleep15861-21netstat00:27:191
3117026413,27sleep00-21swapper/022:22:040
238592646,11sleep10-21swapper/123:42:001
2266726418,28sleep10-21swapper/120:11:561
2251826418,19sleep00-21swapper/021:52:240
222326413,38sleep12267-21ntpq22:32:181
106026420,28sleep11068-21modprobe20:46:531
685926318,18sleep00-21swapper/019:22:040
581326312,34sleep35887-21/usr/sbin/munin22:43:183
3019626316,30sleep330260-21switchtime22:17:233
194226311,12sleep00-21swapper/019:07:140
1854826310,39sleep318619-21sensors_temp19:57:253
1765726315,23sleep00-21swapper/021:37:230
1465826319,27sleep114631-21latency_hist19:46:571
1465826319,27sleep114631-21latency_hist19:46:571
472326217,22sleep10-21swapper/122:41:551
472326217,22sleep10-21swapper/122:41:551
1314526220,21sleep10-21swapper/121:22:261
1193326213,31sleep011972-21/usr/sbin/munin19:37:190
1097726218,23sleep10-21swapper/121:17:091
735026119,26sleep17411-21/usr/sbin/munin19:22:211
560826114,20sleep00-21swapper/019:17:160
3228626120,18sleep10-21swapper/100:07:241
2804926112,35sleep127564-21/usr/sbin/munin20:27:231
2804926112,35sleep127564-21/usr/sbin/munin20:27:231
1430126111,20sleep10-21swapper/123:11:561
831226017,21sleep00-21swapper/021:07:230
2904126016,28sleep029063-21/usr/sbin/munin23:57:210
2813526020,24sleep00-21swapper/022:12:110
2072826018,24sleep120744-21ntp_states21:47:211
1942426014,23sleep30-21swapper/321:42:263
1942426014,23sleep30-21swapper/321:42:263
263532596,11sleep10-21swapper/123:47:291
2475125922,20sleep30-21swapper/322:01:563
1341825924,20sleep30-21swapper/321:22:283
1308025915,24sleep00-21swapper/023:07:140
840125816,24sleep18491-21tune2fs22:52:131
533425819,25sleep35405-21/usr/sbin/munin20:57:293
2964725817,25sleep029641-21apt-get22:17:070
1777925819,18sleep30-21swapper/319:56:563
171382586,39sleep30-21swapper/323:18:483
1361725819,19sleep30-21swapper/323:07:253
3127325714,25sleep331280-21diskstats22:22:113
2884525717,17sleep00-21swapper/020:31:540
2884525717,17sleep00-21swapper/020:31:540
2756925714,20sleep00-21swapper/023:52:250
216725717,24sleep12315-21sed00:16:571
1998025715,25sleep00-21swapper/021:46:390
1998025715,25sleep00-21swapper/021:46:390
843625614,25sleep18430-21apt-get19:27:101
758925616,26sleep07676-21cat00:32:230
3047225619,18sleep10-21swapper/120:36:561
2920925614,18sleep30-21swapper/323:57:253
2415125617,22sleep324152-21sshd20:16:393
2415125617,22sleep324152-21sshd20:16:393
2191825617,24sleep321922-21diskstats21:52:113
1769025616,20sleep10-21swapper/121:37:241
151392569,32sleep015214-21find23:12:230
1287725613,26sleep112968-21systemd-run19:41:561
818625517,22sleep30-21swapper/319:26:563
659825513,23sleep36621-21munin-run19:21:553
468425510,19sleep00-21swapper/022:41:540
468425510,19sleep00-21swapper/022:41:540
3131825514,24sleep10-21swapper/122:22:131
3117425514,19sleep10-21swapper/120:37:201
3117425514,19sleep10-21swapper/120:37:201
2661525516,17sleep00-21swapper/022:07:140
184525512,26sleep31840-21apt-get22:32:083
1533625516,25sleep115380-21unin-run21:31:551
3013625413,26sleep030142-21grep00:02:100
791725316,19sleep10-21swapper/121:07:121
2964825318,19sleep10-21swapper/122:17:081
2005325312,20sleep10-21swapper/121:46:561
2005325312,20sleep10-21swapper/121:46:561
1588825316,23sleep315934-21sed21:32:173
676325213,22sleep16757-21apt-get22:47:081
521225211,24sleep04699-21apt-get22:42:100
3074025214,18sleep30-21swapper/300:02:253
2853025214,19sleep30-21swapper/322:12:223
18472527,16sleep00-21swapper/022:32:090
155262525,10sleep30-21swapper/323:13:273
1486825211,19sleep10-21swapper/119:47:051
929625111,21sleep39342-21users19:27:323
873025113,22sleep38764-21ntp_states22:52:193
2241225110,17sleep10-21swapper/123:37:111
1407325112,23sleep014068-21apt-get21:27:090
67602509,21sleep30-21swapper/322:47:063
554725013,22sleep35446-21/usr/sbin/munin00:27:133
28062509,16sleep10-21swapper/100:17:211
28062509,16sleep10-21swapper/100:17:211
2671925017,15sleep10-21swapper/122:07:151
2649625013,21sleep326502-21ntp_states20:22:243
2649625013,21sleep326502-21ntp_states20:22:243
2160225013,22sleep021749-21sed21:51:560
1921625012,22sleep119213-21apt-get23:27:061
1790825013,21sleep317941-21irqstats23:22:153
2496824913,20sleep124965-21apt-get22:02:061
2425024917,19sleep00-21swapper/020:16:560
2425024917,19sleep00-21swapper/020:16:560
1045924915,17sleep00-21swapper/022:57:250
2801124812,20sleep328040-21ntp_states20:27:213
2801124812,20sleep328040-21ntp_states20:27:213
1619824811,23sleep116205-21df_inode23:17:131
156222478,15sleep10-21swapper/121:32:111
883924011,16sleep00-21swapper/019:27:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional