You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 13:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Sep 02, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25542876644,8710sleep11063-21kworker/1:219:27:141
109402639313,6363sleep210458-21apt-get19:53:552
109402639313,6363sleep210458-21apt-get19:53:552
28341219919,1963sleep124704-21kworker/u8:120:48:551
28341219919,1963sleep124704-21kworker/u8:120:48:551
28320261813,530sleep024712-21kworker/u8:320:48:540
28320261813,530sleep024712-21kworker/u8:320:48:540
10443253911,516sleep110453-21find23:18:541
13607251916,485sleep013674-21/usr/sbin/munin21:44:160
20824235811,285sleep120890-21wc20:24:171
28306231515,126sleep128437-21unixbench_multi00:14:151
1017523055,288sleep37816-21kworker/3:019:50:163
29412226511,193sleep10-21swapper/119:09:171
29412226511,193sleep10-21swapper/119:09:171
195802255140,26sleep10-21swapper/122:04:071
195802255140,26sleep10-21swapper/122:04:071
30004221297,96sleep130097-21fschecks_time20:54:041
3066522100,5sleep10-21swapper/122:39:091
15658220616,171sleep215713-21sort20:09:002
72142189103,18sleep10-21swapper/121:24:071
1824221868,159sleep00-21swapper/023:43:590
379721788,98sleep13839-21fschecks_count21:14:061
379721788,98sleep13839-21fschecks_count21:14:061
282021786,160sleep12941-21cat21:09:131
282021786,160sleep12941-21cat21:09:131
26114217712,108sleep126148-21cut00:09:031
28511217054,101sleep128613-21latency_hist19:08:451
22105216655,97sleep122154-21unin-run22:13:431
1732921659,97sleep117182-21/usr/sbin/munin20:14:041
24934216310,82sleep10-21swapper/100:04:101
23482216249,94sleep123563-21switchtime23:59:181
19548215783,18sleep10-21swapper/123:48:441
1232021578,133sleep112469-21cat19:58:461
1232021578,133sleep112469-21cat19:58:461
31574215118,109sleep331590-21fschecks_count20:59:023
9644214912,124sleep19681-21modprobe21:33:431
1201214344,84sleep11350-21latency_hist00:33:451
23194214013,112sleep023236-21sed20:33:440
1072213910,76sleep11091-21apt-get22:48:551
14572213858,61sleep10-21swapper/120:04:111
27821213616,105sleep127835-21apt-get20:48:431
22773213665,15sleep10-21swapper/120:29:221
4839213018,97sleep14948-21timerandwakeup22:59:121
16385213054,64sleep116451-21unin-run23:38:431
16385213054,64sleep116451-21unin-run23:38:431
7346212720,92sleep07364-21awk19:43:430
2847212760,54sleep10-21swapper/122:54:031
3769212518,87sleep10-21swapper/119:29:201
8018212410,99sleep18178-21awk23:09:171
2097212416,92sleep22223-21unixbench_multi00:34:152
17924212214,89sleep1231-21systemd-journal23:43:241
23953211914,85sleep00-21swapper/022:18:550
17955211953,49sleep10-21swapper/121:59:061
17955211953,49sleep10-21swapper/121:59:061
24002211821,77sleep10-21swapper/120:34:141
10630211718,83sleep210780-21sed21:34:252
13675211618,20sleep10-21swapper/121:44:171
3421211514,84sleep03549-21uptime22:54:150
31697211417,74sleep10-21swapper/120:59:041
31488211215,80sleep331630-21sed00:24:153
27030211220,77sleep227070-21ps20:44:162
9181211133,17sleep10-21swapper/119:48:571
29903211120,65sleep330045-21grep00:19:143
28904211020,39sleep028943-21sendmail00:18:430
15631211010,85sleep130192-21munin-node20:08:551
1461221108,86sleep10-21swapper/121:48:571
14143210826,28sleep00-21swapper/023:29:140
8881210740,12sleep10-21swapper/123:13:571
8881210740,12sleep10-21swapper/123:13:571
32226210716,65sleep032366-21grep20:59:150
3121421077,90sleep00-21swapper/022:40:260
2973521065,11sleep10-21swapper/122:36:401
25990210611,74sleep128538-21cyclictest22:24:131
25990210611,74sleep128538-21cyclictest22:24:131
20340210672,17sleep10-21swapper/123:49:131
20340210672,17sleep10-21swapper/123:49:131
18000210617,69sleep218113-21cat23:43:452
14488210511,18sleep10-21swapper/123:29:231
572121048,80sleep05745-21ntp_states21:19:090
31408210418,63sleep331553-21unixbench_multi19:14:163
31311210411,80sleep131382-21swap00:24:131
31948210127,57sleep10-21swapper/122:44:061
31764210126,45sleep331757-21apt-get22:44:003
1976221005,40sleep10-21swapper/120:20:011
12669210018,30sleep10-21swapper/121:42:231
2259329822,49sleep222625-21munin-run23:58:422
2133629831,13sleep10-21swapper/123:53:571
1325529816,63sleep313039-21/usr/sbin/munin21:44:073
2502529715,33sleep10-21swapper/122:20:581
1238429619,58sleep10-21swapper/123:24:061
34429517,62sleep0432-21grep00:29:120
450029412,21sleep14502-21apt-get19:33:561
316912945,11sleep10-21swapper/119:14:591
1715829410,69sleep017203-21processes23:39:110
2929729320,19sleep10-21swapper/100:19:001
1119529327,43sleep011223-21modprobe21:38:430
459729219,53sleep24660-21fschecks_time19:34:052
2997229212,65sleep030113-21latency_hist22:38:440
171252925,32sleep10-21swapper/121:54:581
1498829211,64sleep015017-21/usr/sbin/munin21:49:120
1228429015,54sleep311991-21/usr/sbin/munin23:24:043
430328915,56sleep04310-21diskstats22:58:590
1905328912,63sleep319112-21cut22:03:443
1905328912,63sleep319112-21cut22:03:443
169028912,19sleep10-21swapper/100:34:061
58522880,6sleep30-21swapper/319:38:453
254462887,66sleep125470-21ntp_states20:39:101
1892028814,47sleep319068-21grep23:44:143
1774328711,60sleep017811-21sh21:59:010
1774328711,60sleep017811-21sh21:59:010
1455728720,52sleep314564-21/usr/sbin/munin20:04:093
970628610,60sleep09671-21latency_hist21:33:460
83428610,64sleep0367-21snmpd19:20:020
715328613,17sleep10-21swapper/119:41:141
316482868,45sleep00-21swapper/019:14:250
2843928631,34sleep328468-21munin-run19:08:433
1942828617,49sleep219272-21/usr/sbin/munin22:04:032
1942828617,49sleep219272-21/usr/sbin/munin22:04:032
68692845,47sleep10-21swapper/123:06:261
2736228432,27sleep20-21swapper/219:04:022
2389928414,53sleep323922-21/usr/sbin/munin20:34:123
109242845,48sleep10-21swapper/121:35:171
59228312,15sleep10-21swapper/121:03:571
192728322,32sleep00-21swapper/000:34:130
323002820,8sleep20-21swapper/222:44:152
275602829,18sleep10-21swapper/122:29:131
261728224,37sleep22649-21proc_pri21:09:102
261728224,37sleep22649-21proc_pri21:09:102
1331228212,18sleep20-21swapper/219:59:162
757828115,47sleep07724-21unixbench_singl21:24:140
70628121,37sleep0862-21gpgv22:48:460
402128121,42sleep34173-21latency_hist22:58:463
109428122,40sleep2231-21systemd-journal22:48:562
815028012,53sleep38117-21latency_hist21:28:453
896727923,29sleep20-21swapper/223:14:022
896727923,29sleep20-21swapper/223:14:022
2737927913,53sleep227408-21latency_hist00:13:442
1102127922,33sleep30-21swapper/323:19:203
2884927713,49sleep328869-21open_inodes20:49:303
2884927713,49sleep328869-21open_inodes20:49:303
18022776,14sleep00-21swapper/021:08:030
679627613,45sleep36943-21/usr/sbin/munin19:39:143
538927616,33sleep25486-21fschecks_time21:19:032
292192769,13sleep00-21swapper/019:09:120
292192769,13sleep00-21swapper/019:09:120
223962757,12sleep10-21swapper/122:13:591
2015327516,29sleep30-21swapper/323:49:073
2015327516,29sleep30-21swapper/323:49:073
1925327515,25sleep00-21swapper/020:19:110
1356527518,31sleep20-21swapper/223:28:562
1645327410,49sleep316577-21latency_hist23:38:453
1645327410,49sleep316577-21latency_hist23:38:453
3048227322,30sleep00-21swapper/000:23:430
280312739,40sleep028205-21gpgv20:48:460
2579127313,43sleep025894-21latency_hist00:08:450
2095927324,23sleep30-21swapper/320:24:193
3241527220,35sleep332521-21fschecks_time00:29:033
9582716,11sleep10-21swapper/119:21:371
85892716,10sleep10-21swapper/119:44:491
537727142,15sleep10-21swapper/121:19:021
2763427115,42sleep027642-21sh19:04:080
2579727123,23sleep30-21swapper/322:24:063
2579727123,23sleep30-21swapper/322:24:063
2108727115,40sleep021096-21systemd-run23:53:440
2108727115,40sleep021096-21systemd-run23:53:440
136692715,11sleep10-21swapper/120:01:491
1316127121,26sleep20-21swapper/221:44:062
859127019,35sleep38621-21munin-run23:13:433
2717227020,25sleep00-21swapper/022:29:040
2559327014,39sleep225663-21switchtime20:39:152
1623527016,25sleep00-21swapper/020:09:150
1557627016,30sleep00-21swapper/023:34:100
1557627016,30sleep00-21swapper/023:34:100
969526919,27sleep30-21swapper/321:33:453
2900026916,35sleep229097-21/usr/sbin/munin20:49:332
2900026916,35sleep229097-21/usr/sbin/munin20:49:332
427026814,38sleep34362-21latency_hist19:33:453
291012685,11sleep30-21swapper/322:34:113
1834326820,23sleep20-21swapper/221:59:152
1834326820,23sleep20-21swapper/221:59:152
1587226817,30sleep315900-21munin-run21:53:433
548726721,25sleep30-21swapper/321:19:043
2580626715,34sleep025921-21/usr/sbin/munin20:39:170
2563426735,17sleep30-21swapper/300:07:233
1960326723,19sleep20-21swapper/223:48:452
1960326723,19sleep20-21swapper/223:48:452
345826613,37sleep03467-21unin-run21:13:440
345826613,37sleep03467-21unin-run21:13:440
2567926625,23sleep30-21swapper/320:39:163
228572667,11sleep20-21swapper/222:14:132
213162668,41sleep30-21swapper/322:09:133
1718126612,32sleep00-21swapper/020:13:550
1230026618,25sleep012319-21wc19:58:450
1230026618,25sleep012319-21wc19:58:450
924926511,37sleep39280-21/usr/sbin/munin23:14:073
924926511,37sleep39280-21/usr/sbin/munin23:14:073
889226517,32sleep38921-21awk19:48:433
59326512,37sleep2594-21http21:03:582
2160026514,16sleep00-21swapper/020:28:440
784326418,20sleep00-21swapper/019:44:040
43662645,12sleep00-21swapper/019:33:460
356326414,20sleep20-21swapper/222:54:172
30826415,23sleep30-21swapper/319:19:123
274626413,38sleep32770-21munin-run00:38:443
1956426414,34sleep019579-21/usr/sbin/munin22:04:050
1956426414,34sleep019579-21/usr/sbin/munin22:04:050
169312647,13sleep00-21swapper/021:54:170
1613526414,36sleep316153-21apt21:53:553
1283826413,27sleep20-21swapper/221:43:452
1198926412,39sleep012009-21apt-get23:23:530
76062639,36sleep00-21swapper/023:09:080
3199226317,30sleep232019-21munin-run19:18:432
305226323,27sleep03136-21fschecks_time19:29:060
909126214,35sleep09143-21interrupts23:14:040
909126214,35sleep09143-21interrupts23:14:040
76322627,27sleep20-21swapper/219:43:582
414826213,30sleep04164-21ntp_states21:14:130
414826213,30sleep04164-21ntp_states21:14:130
2880126216,23sleep20-21swapper/200:17:232
2586526212,32sleep025896-21memory22:24:070
2586526212,32sleep025896-21memory22:24:070
2060726219,21sleep30-21swapper/322:08:453
2060726219,21sleep30-21swapper/322:08:453
34326120,17sleep20-21swapper/221:03:452
2898226118,25sleep329039-21irqstats19:09:073
2898226118,25sleep329039-21irqstats19:09:073
272552619,36sleep30-21swapper/300:13:043
188422618,12sleep20-21swapper/223:44:132
172732614,10sleep20-21swapper/221:56:492
626126013,22sleep00-21swapper/023:04:110
1879126015,28sleep318786-21apt-get20:18:583
1104826015,28sleep311054-21df_inode19:54:043
1104826015,28sleep311054-21df_inode19:54:043
3047025923,19sleep30-21swapper/320:54:143
2931425913,28sleep229442-21/usr/sbin/munin22:34:152
272025915,21sleep30-21swapper/319:28:453
2484625915,29sleep224870-21netstat00:04:072
241425916,18sleep30-21swapper/322:53:453
1568325918,18sleep20-21swapper/223:34:122
1568325918,18sleep20-21swapper/223:34:122
1419025916,21sleep00-21swapper/020:04:020
730725813,27sleep27308-21munin-node23:08:542
691125810,34sleep26920-21diskstats21:23:592
350025815,22sleep20-21swapper/219:29:172
3028125813,30sleep330290-21df_inode22:39:003
2213925811,31sleep022238-21users23:54:160
1254825813,28sleep012543-21apt-get19:58:570
1073025817,26sleep010742-21/usr/sbin/munin23:19:050
3240425716,26sleep232414-21/usr/sbin/munin00:29:022
2398125715,25sleep223973-21apt-get22:19:012
2152825711,31sleep221674-21unixbench_multi22:09:162
1737025713,25sleep317498-21/usr/sbin/munin23:39:143
156542579,24sleep30-21swapper/320:08:573
140625716,24sleep21403-21apt-get19:23:562
1354325710,27sleep30-21swapper/320:00:133
1050125720,20sleep00-21swapper/019:53:450
2967025618,22sleep229676-21ntp_states00:19:092
2477725616,20sleep00-21swapper/020:38:450
2406325617,22sleep20-21swapper/220:34:152
2201225615,17sleep30-21swapper/323:54:153
1022925612,18sleep30-21swapper/321:34:063
984125515,24sleep29965-21sort19:49:142
69012555,15sleep20-21swapper/223:06:522
48862554,39sleep20-21swapper/221:17:342
48862554,39sleep20-21swapper/221:17:342
378425516,23sleep33790-21df_inode21:14:043
378425516,23sleep33790-21df_inode21:14:043
3147325513,25sleep231479-21modprobe22:43:432
3098025514,23sleep031036-21/usr/sbin/munin00:24:040
2770025513,25sleep327709-21diskstats00:14:013
2750025514,25sleep227529-21ntp_states22:29:102
2304425517,14sleep30-21swapper/323:59:043
1277525512,16sleep00-21swapper/021:43:430
3039825414,17sleep20-21swapper/220:54:132
2288125417,18sleep30-21swapper/322:14:143
149525413,21sleep00-21swapper/019:24:020
575925312,24sleep35778-21apt-get23:03:543
2683925313,15sleep20-21swapper/222:28:462
2683925313,15sleep20-21swapper/222:28:462
2359325313,27sleep0367-21snmpd22:17:500
1932725314,21sleep218768-21/usr/sbin/munin20:19:132
1469725315,23sleep214699-21diskstats21:49:022
1250025316,18sleep20-21swapper/223:24:112
1057625310,20sleep20-21swapper/223:19:032
608625212,23sleep26080-21apt-get19:38:572
3085625211,27sleep230910-21grep00:24:022
2833425217,19sleep30-21swapper/322:33:433
2423925212,17sleep00-21swapper/000:03:450
2189025212,23sleep321885-21apt-get20:28:593
2138125212,19sleep00-21swapper/022:09:140
1200625213,20sleep30-21swapper/321:39:263
316425117,20sleep30-21swapper/321:10:283
316425117,20sleep30-21swapper/321:10:283
2659825113,18sleep00-21swapper/000:09:140
2292425116,18sleep20-21swapper/223:58:592
1463825112,24sleep214707-21/usr/sbin/munin20:04:122
110625115,20sleep00-21swapper/022:49:020
109325116,19sleep30-21swapper/321:04:273
3055725011,19sleep00-21swapper/020:54:150
2196525010,17sleep20-21swapper/220:29:022
176482508,14sleep30-21swapper/320:14:103
1720425011,21sleep217201-21apt-get20:13:572
112125011,25sleep21153-21/usr/sbin/munin19:23:432
2890324915,18sleep20-21swapper/219:09:062
2890324915,18sleep20-21swapper/219:09:062
2677624816,15sleep30-21swapper/320:44:093
2041324814,20sleep220420-21df_inode20:24:022
1364024812,21sleep313646-21df_inode23:29:013
422324711,19sleep30-21swapper/322:58:543
857224517,14sleep20-21swapper/221:29:042
42012459,18sleep20-21swapper/222:58:532
241042417,14sleep30-21swapper/322:19:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional