You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 12:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Jul 28, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
250862601214,5984sleep114353-21kworker/1:023:23:401
24623278722,748sleep224643-21/usr/sbin/munin23:23:252
10625247377,372sleep310628-21cron20:57:553
22742452122,200sleep32277-21cron23:57:553
17188237132,299sleep317324-21grep19:33:353
26712230913,280sleep026714-21sshd23:29:410
5933230816,277sleep25941-21ntpq22:23:242
1207023046,283sleep329233-1kworker/3:1H22:43:203
9840227347,165sleep39867-21munin-plugin-st22:37:553
2987225059,63sleep33122-21timerwakeupswit22:13:273
2987225059,63sleep33122-21timerwakeupswit22:13:273
2401922410,6sleep00-21swapper/023:22:580
15607223542,175sleep315692-21latency_hist21:12:583
239222231110,73sleep30-21swapper/323:22:583
16070221983,111sleep30-21swapper/319:32:583
18609221427,175sleep318762-21awk23:03:273
18609221427,175sleep318762-21awk23:03:273
2931422066,165sleep329392-21basename23:38:243
5237220430,138sleep35301-21latency_hist22:22:563
5237220430,138sleep35301-21latency_hist22:22:563
25182219669,108sleep30-21swapper/319:58:553
156721960,56sleep37821-21cyclictest22:08:263
680921949,126sleep36955-21cat22:27:573
5703219155,95sleep35787-21latency_hist00:07:563
1071821898,127sleep3364-21snmpd00:23:153
1071821898,127sleep3364-21snmpd00:23:153
25878218848,70sleep325940-21fschecks_time20:03:153
1847321889,114sleep30-21swapper/319:38:233
24955218612,156sleep224930-21latency_hist21:42:572
24955218612,156sleep224930-21latency_hist21:42:572
16742218116,149sleep216781-21ntpq22:58:212
10628218111,155sleep210719-21sed19:13:592
3905217452,73sleep30-21swapper/300:02:553
2314217217,140sleep02361-21unin-run23:57:560
27200216741,106sleep30-21swapper/321:48:243
30779216044,21sleep30-21swapper/323:43:253
7509215942,101sleep30-21swapper/322:28:263
28578215651,58sleep328594-21latency_hist23:37:563
14558215511,60sleep30-21swapper/322:52:573
13895215313,121sleep314024-21latency_hist21:07:573
1063215111,80sleep31066-21cron20:27:553
777215048,36sleep3905-21latency_hist23:52:583
31783214929,103sleep20-21swapper/220:22:572
11959214813,121sleep311999-21ntp_kernel_pll_19:18:203
6017214721,108sleep30-21swapper/320:43:093
29774214616,100sleep00-21swapper/021:57:580
29774214616,100sleep00-21swapper/021:57:580
28363214511,72sleep30-21swapper/321:53:163
28363214511,72sleep30-21swapper/321:53:163
6126214418,62sleep30-21swapper/300:08:173
12261214022,93sleep212291-21fschecks_count00:28:152
12261214022,93sleep212291-21fschecks_count00:28:152
2028213948,32sleep32150-21timerwakeupswit20:28:273
2801721388,91sleep328137-21cat20:08:283
1442021337,111sleep314500-21date19:27:573
18297213115,92sleep018328-21/usr/sbin/munin23:03:200
18297213115,92sleep018328-21/usr/sbin/munin23:03:200
25979212920,66sleep30-21swapper/323:28:193
19552212912,16sleep10-21swapper/123:08:141
27016212721,92sleep127068-21sort23:32:571
2713212619,90sleep02737-21netstat22:13:200
2713212619,90sleep02737-21netstat22:13:200
2344721238,78sleep323596-21date21:37:583
8590212036,23sleep38664-21switchtime19:08:273
16643212016,85sleep016708-21latency22:58:190
1011521205,11sleep30-21swapper/300:20:043
31872211953,27sleep331956-21apt-config20:22:583
2213321195,10sleep30-21swapper/323:15:513
32280211814,70sleep332374-21switchtime22:03:263
3063211811,88sleep03100-21ps23:58:240
4475211611,91sleep14479-21grep20:38:111
2012821155,11sleep30-21swapper/321:26:353
17222211515,60sleep30-21swapper/321:17:593
5190211414,86sleep25236-21unin-run22:22:552
5190211414,86sleep25236-21unin-run22:22:552
19567211444,30sleep319618-21fschecks_count23:08:163
19345211411,19sleep20-21swapper/221:23:232
1141821149,70sleep31-21systemd22:42:563
2359821130,7sleep00-21swapper/021:37:580
29285211249,51sleep329321-21ntpq20:13:213
24994211253,18sleep30-21swapper/321:42:583
14368211224,69sleep114397-21modprobe19:27:551
2158121115,47sleep30-21swapper/321:30:223
2158121115,47sleep30-21swapper/321:30:223
21199211117,78sleep121293-21fschecks_time19:48:151
31791211017,78sleep031801-21forks22:03:150
16417210861,24sleep00-21swapper/019:33:070
11310210814,16sleep10-21swapper/120:58:221
11310210814,16sleep10-21swapper/120:58:221
7712210634,48sleep07742-21munin-run19:07:550
26824210611,81sleep226853-21fschecks_count21:48:142
2266521049,76sleep30-21swapper/319:53:143
13718210413,70sleep00-21swapper/019:23:260
15273210114,23sleep30-21swapper/322:53:263
1451210117,63sleep01559-21fschecks_time20:28:160
1119321018,12sleep00-21swapper/019:17:120
7122210023,55sleep27223-21fschecks_time22:28:172
28888210018,25sleep20-21swapper/220:13:122
14787210012,64sleep114858-21mailstats21:08:321
992129917,21sleep20-21swapper/222:37:582
926329949,30sleep10-21swapper/122:33:291
467529918,59sleep20-21swapper/200:03:272
1402629818,18sleep00-21swapper/021:07:580
469829718,60sleep10-21swapper/122:18:261
469829718,60sleep10-21swapper/122:18:261
2527629713,69sleep125327-21fschecks_count21:43:131
2527629713,69sleep125327-21fschecks_count21:43:131
1763829735,46sleep317648-21modprobe23:02:553
3011229619,13sleep30-21swapper/321:58:013
298829511,62sleep33080-21fschecks_time20:33:163
710529412,62sleep07000-21/usr/sbin/munin22:28:150
2784229411,22sleep10-21swapper/120:08:261
2326329415,63sleep123327-21cat19:53:281
2266329427,43sleep222657-21apt-get19:53:132
1263729445,38sleep312671-21ps00:28:243
1263729445,38sleep312671-21ps00:28:243
816329315,61sleep28265-21timerwakeupswit00:13:252
376829321,54sleep10-21swapper/100:01:401
1611529315,61sleep015821-21/usr/sbin/munin21:13:190
929629211,65sleep09345-21systemd-run19:12:560
807029219,60sleep18162-21sed00:13:241
577629232,47sleep35795-21latency_hist20:42:573
3038429215,65sleep230440-21cpuspeed23:43:162
204622928,18sleep30-21swapper/319:43:283
3216729115,60sleep132199-21/usr/sbin/munin22:03:241
2469329110,16sleep00-21swapper/019:58:240
236229021,42sleep12348-40mandb23:57:571
876428916,54sleep0225-21systemd-journal00:17:550
1485028914,18sleep30-21swapper/300:36:153
254002878,16sleep30-21swapper/321:43:153
254002878,16sleep30-21swapper/321:43:153
158422868,23sleep20-21swapper/221:13:092
321572850,8sleep10-21swapper/120:22:591
1084028516,44sleep010988-21unixbench_multi22:38:280
917428416,50sleep09257-21tune2fs00:18:150
2805028416,18sleep00-21swapper/021:52:570
268128425,30sleep20-21swapper/220:32:592
1597528421,42sleep216068-21latency_hist19:32:572
1924628221,45sleep019300-21sed21:23:210
88522810,8sleep00-21swapper/022:33:180
746128123,40sleep30-21swapper/300:13:053
732328124,36sleep27326-21true20:47:572
1492928118,47sleep014954-21kernelversion19:28:190
138728119,48sleep21424-21ntp_kernel_err23:53:212
1062728123,31sleep00-21swapper/000:23:080
1062728123,31sleep00-21swapper/000:23:080
804628017,49sleep18089-21processes20:48:241
804628017,49sleep18089-21processes20:48:241
1450728022,38sleep114536-21modprobe22:52:551
1064128013,52sleep010703-21unin-run20:57:560
3046727919,43sleep130483-21sort23:43:191
3039227915,38sleep10-21swapper/121:58:231
2558227915,50sleep225731-21latency_hist20:02:582
2401227918,29sleep00-21swapper/021:38:190
2401227918,29sleep00-21swapper/021:38:190
209862790,5sleep20-21swapper/221:28:232
209862790,5sleep20-21swapper/221:28:232
1002527931,11sleep30-21swapper/319:13:443
2060827821,29sleep10-21swapper/121:28:141
2060827821,29sleep10-21swapper/121:28:141
592127710,53sleep15932-21ntp_states22:23:231
2391627724,14sleep30-21swapper/321:38:183
2229327623,31sleep122324-21modprobe23:17:551
174012768,52sleep30-21swapper/321:18:073
1591827639,21sleep10-21swapper/119:32:561
1873727525,30sleep10-21swapper/121:22:571
361327423,34sleep23662-21sendmail22:17:552
361327423,34sleep23662-21sendmail22:17:552
314092749,53sleep00-21swapper/023:45:030
2606127420,32sleep20-21swapper/220:03:182
242322745,12sleep30-21swapper/323:23:143
1584627414,47sleep115863-21cpu21:13:121
129527418,34sleep0771-21/usr/sbin/munin22:08:240
1137327420,15sleep30-21swapper/320:58:243
1137327420,15sleep30-21swapper/320:58:243
879427314,17sleep10-21swapper/119:08:291
2599727316,39sleep226089-21users21:43:262
2599727316,39sleep226089-21users21:43:262
1409027310,44sleep214135-21ntpq00:33:192
128512735,58sleep00-21swapper/022:46:320
423827223,27sleep00-21swapper/020:37:570
757027122,14sleep30-21swapper/320:48:103
757027122,14sleep30-21swapper/320:48:103
3052927116,21sleep00-21swapper/021:58:260
2919527116,25sleep20-21swapper/223:38:192
495627012,17sleep20-21swapper/220:38:242
2731227017,27sleep10-21swapper/121:48:261
2392027013,32sleep023947-21munin-run19:57:550
1343827016,37sleep213496-21if_eth022:48:162
1287227014,33sleep00-21swapper/019:22:580
1230527015,28sleep10-21swapper/122:43:261
1014627017,34sleep19340-21/usr/sbin/munin20:53:331
2725926914,38sleep127329-21df23:33:131
2710326925,20sleep10-21swapper/120:07:581
225226921,27sleep10-21swapper/120:28:291
215292697,47sleep30-21swapper/319:48:213
415926820,27sleep24226-21modprobe20:37:552
329926826,22sleep10-21swapper/120:33:221
2087626821,21sleep00-21swapper/019:47:570
976726717,30sleep29802-21/usr/sbin/munin20:53:262
79726717,30sleep2791-21apt-get22:08:092
653226718,34sleep16540-21df19:03:161
1486026721,23sleep20-21swapper/221:08:332
1222126722,23sleep10-21swapper/119:18:261
736126614,34sleep07503-21latency_hist20:47:590
2874526613,38sleep128751-21ntp_states21:53:261
2874526613,38sleep128751-21ntp_states21:53:261
2232626612,41sleep222372-21munin-run23:17:562
1842726613,36sleep218472-21ntp_states19:38:222
1188626625,26sleep20-21swapper/220:58:552
1188626625,26sleep20-21swapper/220:58:552
1041226622,23sleep20-21swapper/200:22:582
602426516,29sleep16019-21apt-get20:43:111
2851026515,34sleep228539-21munin-run23:37:552
2578426517,21sleep00-21swapper/021:43:240
2578426517,21sleep00-21swapper/021:43:240
1965726511,19sleep10-21swapper/121:23:301
1683326521,24sleep10-21swapper/122:58:241
1507326516,20sleep00-21swapper/021:08:350
1488826527,20sleep20-21swapper/222:53:162
1343226527,22sleep10-21swapper/100:32:541
1343226527,22sleep10-21swapper/100:32:541
283682646,14sleep00-21swapper/020:09:590
1700026414,33sleep117067-21awk19:33:331
72526322,23sleep00-21swapper/023:52:550
3008626319,25sleep130112-21munin-run23:42:541
565226213,35sleep15683-21awk00:07:541
205632629,40sleep20-21swapper/223:10:412
1848126216,30sleep018494-21/usr/sbin/munin19:38:240
157742627,11sleep00-21swapper/022:54:200
1433726217,27sleep114462-21sed00:33:271
1178826213,35sleep211818-21fschecks_count22:43:152
1125226217,22sleep20-21swapper/200:23:262
1125226217,22sleep20-21swapper/200:23:262
598726114,21sleep00-21swapper/022:23:260
477626120,21sleep00-21swapper/000:03:290
326572616,11sleep10-21swapper/120:23:411
3195826115,29sleep032156-21date20:22:590
2961626120,26sleep20-21swapper/221:57:562
2961626120,26sleep20-21swapper/221:57:562
2277626118,21sleep10-21swapper/123:18:151
1141326111,34sleep211415-21cron22:42:542
1132726117,25sleep10-21swapper/119:17:581
1099126111,34sleep111060-21/usr/sbin/munin22:38:291
719926018,27sleep17230-21munin-run00:12:551
579926021,21sleep00-21swapper/020:42:580
2676226011,37sleep2364-21snmpd23:30:202
2470026012,28sleep124722-21/usr/sbin/munin19:58:251
123842607,12sleep00-21swapper/000:28:170
123842607,12sleep00-21swapper/000:28:170
852325918,26sleep20-21swapper/219:08:262
765725913,33sleep07755-21perl20:48:150
765725913,33sleep07755-21perl20:48:150
591825916,26sleep25937-21apt-get00:08:082
3011125911,33sleep229615-21apt-get21:58:002
2742025915,24sleep227426-21diskstats20:08:152
2183325914,20sleep00-21swapper/021:32:580
2183325914,20sleep00-21swapper/021:32:580
1845625913,18sleep10-21swapper/123:03:251
1845625913,18sleep10-21swapper/123:03:251
896825817,19sleep20-21swapper/220:52:582
896825817,19sleep20-21swapper/220:52:582
654425816,23sleep36639-21perl19:03:183
2080525815,18sleep00-21swapper/023:12:570
142002588,39sleep00-21swapper/022:48:580
1166025819,20sleep20-21swapper/219:18:152
645825714,25sleep26452-21apt-get19:03:132
320992576,11sleep20-21swapper/222:03:212
2896425719,20sleep00-21swapper/020:13:140
2302725718,19sleep20-21swapper/223:18:202
184162578,37sleep00-21swapper/021:19:290
183892579,30sleep118426-21ntp_kernel_err19:38:211
1819725712,32sleep218255-21sed23:03:182
1819725712,32sleep218255-21sed23:03:182
1479925715,24sleep114793-21apt-get22:53:111
1038425720,17sleep10-21swapper/100:22:571
87425614,27sleep1881-21diskstats22:08:131
260025613,26sleep22594-21apt-get23:58:102
1706825615,17sleep20-21swapper/219:33:342
651725520,20sleep00-21swapper/000:08:260
3199425517,21sleep232000-21df_inode23:48:122
2886125515,20sleep10-21swapper/120:13:071
253025515,20sleep10-21swapper/122:13:171
253025515,20sleep10-21swapper/122:13:171
1380325514,24sleep013807-21diskstats00:33:130
1249625513,25sleep212492-21apt-get21:03:112
114625511,22sleep20-21swapper/220:27:582
1062825513,26sleep110623-21apt-get00:23:091
1062825513,26sleep110623-21apt-get00:23:091
2860225416,16sleep00-21swapper/023:37:570
2578925414,23sleep025785-21apt-get20:03:080
2370325416,24sleep123720-21df_inode21:38:121
2370325416,24sleep123720-21df_inode21:38:121
2207025416,17sleep10-21swapper/121:33:111
2128725417,18sleep00-21swapper/023:13:170
2002225415,24sleep120028-21ntp_states19:43:221
1139725412,16sleep00-21swapper/020:58:250
1139725412,16sleep00-21swapper/020:58:250
2457925313,24sleep024615-21ntp_states23:23:230
2242325312,27sleep022546-21wc19:52:580
2200725310,19sleep20-21swapper/219:48:282
1993225317,19sleep20-21swapper/219:43:192
2999025214,20sleep00-21swapper/023:41:400
1439825213,22sleep214419-21munin-run19:27:562
5912515,37sleep10-21swapper/123:51:141
2340325110,27sleep223446-21latency_hist21:37:572
1375225114,22sleep113795-21munin-run21:07:551
702525013,19sleep10-21swapper/122:28:111
1787925011,15sleep20-21swapper/221:18:222
438324813,17sleep00-21swapper/022:18:210
438324813,17sleep00-21swapper/022:18:210
3265224814,19sleep00-21swapper/020:23:400
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional