You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 18:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 03, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218492385719,3818sleep112944-21kworker/u8:107:08:211
1379727588,676sleep013844-21ntpq08:24:080
22784249918,466sleep322823-21ntpq10:33:253
20263237110,37sleep020407-21grep12:08:320
3591234916,315sleep23600-21/usr/sbin/munin07:53:202
22482231336,260sleep00-21swapper/010:33:180
6011227792,168sleep06076-21sendmail_mailqu11:23:290
31114227210,217sleep031160-21fschecks_time09:18:230
31114227210,217sleep031160-21fschecks_time09:18:230
19772226795,144sleep00-21swapper/008:43:220
777622565,167sleep00-21swapper/008:06:390
777622565,167sleep00-21swapper/008:06:390
28630223016,190sleep028788-21unixbench_multi09:09:390
28630223016,190sleep028788-21unixbench_multi09:09:390
15307222751,38sleep015468-21unixbench_multi10:09:030
21611222416,99sleep021660-21apt-key07:08:080
9365222321,185sleep09394-21idleruntime-cro09:53:000
10812221514,176sleep010891-21/usr/sbin/munin11:38:320
10812221514,176sleep010891-21/usr/sbin/munin11:38:320
20321221315,180sleep120410-21switchtime07:03:321
1849422025,13sleep00-21swapper/008:38:290
10417220021,156sleep010560-21unixbench_singl09:53:440
19952219711,166sleep00-21swapper/007:03:230
6382218934,19sleep00-21swapper/009:43:020
8663218811,157sleep08744-21cut08:08:290
21541217921,15sleep20-21swapper/210:28:442
8731217511,16sleep30-21swapper/311:33:223
26373217527,53sleep00-21swapper/007:23:020
269821719,149sleep02728-21kernelversion11:13:240
2478021719,52sleep024798-21munin-run07:18:010
11413217010,144sleep011456-21idleruntime09:58:210
30037216812,61sleep00-21swapper/009:13:380
30037216812,61sleep00-21swapper/009:13:380
1715216810,14sleep30-21swapper/309:28:193
504721667,81sleep05067-21apt-get07:58:110
2496521658,142sleep025048-21mailstats12:23:300
2775621637,13sleep30-21swapper/309:08:043
2775621637,13sleep30-21swapper/309:08:043
12547216321,11sleep00-21swapper/008:19:390
488521626,11sleep00-21swapper/011:18:330
4546215967,48sleep04643-21sort09:38:010
2418421575,11sleep00-21swapper/012:22:580
2137921569,86sleep0195-21jbd2/mmcblk2p2-12:13:160
1854221567,48sleep00-21swapper/012:03:310
31968215517,16sleep00-21swapper/007:38:320
31968215517,16sleep00-21swapper/007:38:320
1754621558,136sleep05666-21snmpd08:35:420
942215219,67sleep00-21swapper/007:43:290
1113921475,32sleep00-21swapper/008:16:280
2825121465,11sleep00-21swapper/010:52:080
2825121465,11sleep00-21swapper/010:52:080
1586721455,10sleep00-21swapper/011:54:180
23526214411,102sleep00-21swapper/008:53:310
180721435,10sleep00-21swapper/011:09:120
180721435,10sleep00-21swapper/011:09:120
961021425,10sleep00-21swapper/011:34:030
7075214217,44sleep00-21swapper/011:28:140
2573921425,10sleep00-21swapper/009:02:440
20260214023,90sleep320417-21unixbench_multi08:43:333
20828213911,48sleep00-21swapper/010:28:130
3194213826,88sleep1225-21systemd-journal07:52:591
24959213818,102sleep324967-21ntpq08:58:273
19271213818,101sleep00-21swapper/010:23:160
3266021376,13sleep30-21swapper/311:03:333
2974121369,55sleep029744-21grep07:33:150
3266721345,32sleep00-21swapper/011:03:390
2473213310,39sleep00-21swapper/007:48:300
451521318,48sleep05666-21snmpd07:54:360
26750213118,91sleep226884-21unixbench_singl12:28:332
30133213017,95sleep130265-21unixbench_multi09:13:391
30133213017,95sleep130265-21unixbench_multi09:13:391
17561212822,65sleep00-21swapper/012:01:130
14255212810,96sleep014408-21latency_hist10:08:020
14255212810,96sleep014408-21latency_hist10:08:020
258021256,39sleep00-21swapper/009:28:530
31052212225,20sleep30-21swapper/307:38:033
20113212217,92sleep320142-21nfs4_client07:03:263
7814212011,96sleep07844-21awk09:47:590
2548921208,36sleep00-21swapper/010:43:110
2548921208,36sleep00-21swapper/010:43:110
2144021207,31sleep00-21swapper/008:48:180
1471821168,32sleep00-21swapper/008:28:030
27693211335,27sleep00-21swapper/007:25:440
14564211321,74sleep214588-21munin-run11:52:592
2726521125,94sleep00-21swapper/012:32:430
1874321125,95sleep00-21swapper/010:19:570
17992211215,79sleep218059-21df12:03:172
14644211211,81sleep00-21swapper/011:53:010
25667211015,78sleep125725-21idleruntime10:43:201
25667211015,78sleep125725-21idleruntime10:43:201
21381211019,68sleep121382-21cpu12:13:171
1663421105,11sleep00-21swapper/010:13:270
12797211031,27sleep00-21swapper/011:45:100
5283210814,31sleep25339-21irqstats07:58:212
29196210830,28sleep00-21swapper/007:30:080
27071210823,11sleep00-21swapper/009:04:460
12228210722,70sleep112350-21sed08:18:311
30661210616,27sleep30-21swapper/310:58:293
25002210621,12sleep00-21swapper/010:40:120
25002210621,12sleep00-21swapper/010:40:120
22945210613,74sleep20-21swapper/208:53:132
1253210622,10sleep00-21swapper/009:26:470
1253210622,10sleep00-21swapper/009:26:470
28138210518,71sleep328139-21/usr/sbin/munin07:28:143
25763210513,79sleep225859-21timerwakeupswit07:18:522
11000210313,76sleep211036-21latency_hist09:58:022
3115721026,84sleep00-21swapper/010:59:100
7942210114,71sleep28034-21cat08:08:022
7942210114,71sleep28034-21cat08:08:022
3407210111,75sleep13408-21sshd11:14:221
18478210113,65sleep317992-21/usr/sbin/munin08:38:283
29764210028,27sleep00-21swapper/010:56:110
2995529718,61sleep130088-21latency_hist10:58:021
230222979,71sleep10-21swapper/110:33:311
666129613,65sleep20-21swapper/211:26:222
2314729611,65sleep10-21swapper/112:18:221
1841729516,59sleep117692-21/usr/sbin/munin10:18:451
524129313,60sleep15048-21/usr/sbin/munin07:58:201
549129114,60sleep15492-21ntp_states09:38:431
2745229022,33sleep10-21swapper/109:08:021
1124029010,63sleep10-21swapper/109:58:141
1812828923,28sleep10-21swapper/106:58:121
811228713,23sleep20-21swapper/209:48:172
2145628717,45sleep121540-21timerandwakeup10:28:431
193828420,44sleep21952-21cpu07:48:162
177128319,49sleep11621-21/usr/sbin/munin09:28:251
1623428216,50sleep216378-21latency_hist08:33:022
860628014,40sleep10-21swapper/108:08:261
3145028020,44sleep131478-21munin-plugin-st11:02:591
2732028020,43sleep327360-21http09:08:003
458727929,27sleep10-21swapper/111:18:301
2815227917,40sleep227825-21/usr/sbin/munin09:09:282
2815227917,40sleep227825-21/usr/sbin/munin09:09:282
2890327825,29sleep20-21swapper/210:53:232
1837327816,48sleep318415-21ntp_kernel_pll_12:03:263
99327717,15sleep30-21swapper/309:23:393
99327717,15sleep30-21swapper/309:23:393
2249527720,30sleep20-21swapper/210:33:202
819927413,32sleep30-21swapper/309:48:223
2843727418,41sleep128587-21cat10:53:011
2676527426,33sleep326787-21munin-run10:47:593
2676527426,33sleep326787-21munin-run10:47:593
2135127418,41sleep221486-21apt-config07:08:052
1941327421,33sleep219448-21/usr/sbin/munin07:03:012
1625327417,37sleep316278-21fschecks_count10:13:183
1278527419,31sleep30-21swapper/310:03:123
1278527419,31sleep30-21swapper/310:03:123
24162730,7sleep20-21swapper/211:13:132
2211327320,36sleep322258-21sed12:13:443
483427224,26sleep30-21swapper/307:58:023
3149627219,29sleep30-21swapper/309:18:343
3149627219,29sleep30-21swapper/309:18:343
3120327216,40sleep231206-21fw_packets09:18:252
3120327216,40sleep231206-21fw_packets09:18:252
2865427217,22sleep10-21swapper/107:28:321
2651227222,22sleep30-21swapper/312:28:303
1820527214,41sleep218240-21ps10:18:422
146427224,21sleep30-21swapper/311:08:343
146427224,21sleep30-21swapper/311:08:343
676527115,37sleep26818-21/usr/sbin/munin09:43:212
444327114,41sleep34511-21sshd11:18:283
2611227121,32sleep10-21swapper/112:28:201
2442527118,29sleep10-21swapper/110:38:281
2442527118,29sleep10-21swapper/110:38:281
1041127113,38sleep210461-21if_eth011:38:212
1041127113,38sleep210461-21if_eth011:38:212
3254827025,26sleep30-21swapper/307:43:003
3254827025,26sleep30-21swapper/307:43:003
2787227015,38sleep327963-21users10:48:313
2787227015,38sleep327963-21users10:48:313
218227024,24sleep10-21swapper/111:13:001
218227024,24sleep10-21swapper/111:13:001
2083527021,30sleep220860-21munin-run08:48:002
1777627020,20sleep10-21swapper/108:38:021
1772427026,26sleep317762-21modprobe08:38:003
169927023,25sleep10-21swapper/107:48:021
1404727018,32sleep114076-21awk10:07:591
1404727018,32sleep114076-21awk10:07:591
1376627019,37sleep313806-21ps11:48:283
1356427014,34sleep113578-21/usr/sbin/munin11:48:221
55026920,22sleep10-21swapper/111:08:001
3261126914,40sleep232620-21df_inode09:23:182
3261126914,40sleep232620-21df_inode09:23:182
1407826924,22sleep20-21swapper/210:08:002
1407826924,22sleep20-21swapper/210:08:002
776326817,26sleep10-21swapper/111:28:311
3181026819,24sleep20-21swapper/211:03:162
3149926819,28sleep20-21swapper/207:38:222
3149926819,28sleep20-21swapper/207:38:222
2982026812,12sleep20-21swapper/209:13:322
2982026812,12sleep20-21swapper/209:13:322
2791926816,29sleep30-21swapper/307:28:013
2417726819,18sleep30-21swapper/307:13:323
1963326814,41sleep319635-21entropy12:08:193
7662679,41sleep3809-21ntpq07:43:243
331426716,31sleep30-21swapper/309:33:203
2669926715,35sleep226746-21fschecks_count07:23:202
2131826721,32sleep321382-21ps10:28:403
867726619,15sleep20-21swapper/211:33:212
507126615,31sleep35066-21apt-get07:58:143
2263826615,37sleep222660-21modprobe12:17:592
213272667,43sleep30-21swapper/308:48:053
1132926617,29sleep30-21swapper/309:58:203
2578726520,28sleep325888-21latency_hist12:28:013
1623526513,37sleep116245-21df_inode10:13:161
2586426418,22sleep20-21swapper/209:03:022
2566526418,21sleep10-21swapper/107:18:511
2506526424,20sleep30-21swapper/307:18:163
2262526418,30sleep122658-21/usr/sbin/munin08:53:001
164626413,31sleep21674-21awk07:47:592
1471226414,34sleep214782-21grep10:08:492
1301426416,31sleep213017-21cron11:47:582
1069426416,27sleep30-21swapper/308:13:453
1008926425,20sleep10-21swapper/108:13:221
61722637,14sleep30-21swapper/309:41:483
598926314,28sleep36008-21sshd11:23:283
1992426316,22sleep10-21swapper/110:23:581
1796426323,25sleep10-21swapper/112:03:121
1700626320,20sleep20-21swapper/210:13:332
1691426312,30sleep20-21swapper/211:58:322
844726218,22sleep20-21swapper/208:08:232
841626218,20sleep30-21swapper/311:33:013
2631226217,27sleep226335-21irqstats09:03:232
2267726219,22sleep30-21swapper/307:09:243
1592726219,25sleep316043-21sed10:13:023
1285126217,28sleep312867-21apt-config08:23:013
467126120,20sleep20-21swapper/211:18:312
2669326118,25sleep126698-21diskstats07:23:191
2466026119,30sleep224709-21idleruntime12:23:202
2413426122,20sleep20-21swapper/210:38:212
2413426122,20sleep20-21swapper/210:38:212
1946526120,27sleep319482-21systemd-run07:03:023
1775126118,24sleep30-21swapper/312:03:023
1584726116,19sleep10-21swapper/110:13:001
103092615,14sleep30-21swapper/309:53:433
464426020,24sleep14846-21date09:38:021
370026022,19sleep10-21swapper/107:53:231
2419426018,26sleep324228-21irqstats10:38:223
2419426018,26sleep324228-21irqstats10:38:223
204822608,11sleep10-21swapper/108:43:351
1713626024,19sleep20-21swapper/208:33:302
1615126019,26sleep21-21systemd11:58:002
866125919,24sleep10-21swapper/111:33:201
517525912,22sleep20-21swapper/209:38:362
2992325917,21sleep10-21swapper/107:33:201
2490325919,21sleep10-21swapper/108:58:251
2097825917,25sleep121160-21gpgv12:13:011
1978025920,19sleep20-21swapper/210:23:552
194902597,15sleep00-21swapper/007:03:030
1404925914,31sleep214164-21/usr/sbin/munin08:24:142
1396825921,24sleep114048-21cat08:24:131
120122599,37sleep212052-21ps08:18:282
1151225914,22sleep20-21swapper/209:58:232
107392596,11sleep30-21swapper/311:38:313
107392596,11sleep30-21swapper/311:38:313
81525817,23sleep20-21swapper/211:08:132
81525817,23sleep20-21swapper/211:08:132
597525818,19sleep10-21swapper/111:23:261
380425818,21sleep20-21swapper/209:33:312
2732325817,22sleep10-21swapper/110:48:211
1535025814,31sleep315378-21proc_pri11:53:293
1507725821,21sleep10-21swapper/108:28:201
1354025818,22sleep30-21swapper/308:24:023
298052576,11sleep30-21swapper/310:56:433
2774325710,19sleep20-21swapper/210:48:302
2774325710,19sleep20-21swapper/210:48:302
231982578,20sleep20-21swapper/212:18:232
2302125718,23sleep323027-21df_inode08:53:183
1306825717,25sleep113221-21latency_hist11:48:011
967725615,24sleep19680-21apt-get09:53:131
840525616,24sleep28415-21munin-run11:33:002
2558325613,26sleep225589-21diskstats10:43:172
2558325613,26sleep225589-21diskstats10:43:172
1926925614,25sleep319264-21apt-get10:23:153
1151425618,20sleep10-21swapper/111:43:011
1151425618,20sleep10-21swapper/111:43:011
811125513,26sleep18106-21apt-get09:48:161
3080725514,26sleep230909-21timerandwakeup10:58:322
2356625514,26sleep123591-21/usr/sbin/munin07:13:171
2294925520,21sleep10-21swapper/108:53:161
786825412,25sleep38007-21latency_hist09:48:023
726625416,17sleep30-21swapper/308:03:313
726625416,17sleep30-21swapper/308:03:313
700325420,16sleep10-21swapper/108:03:251
700325420,16sleep10-21swapper/108:03:251
2936625410,33sleep20-21swapper/207:32:212
1712825421,17sleep10-21swapper/111:58:361
3128625318,21sleep131283-21apt-get07:38:131
3128625318,21sleep131283-21apt-get07:38:131
2134425316,20sleep20-21swapper/208:48:122
1843125315,23sleep118470-21ntp_states08:38:261
164725313,21sleep21714-21df09:28:182
1545725310,27sleep10-21swapper/111:53:311
2292925212,24sleep322924-21apt-get12:18:163
1896625210,29sleep30-21swapper/310:22:513
854325111,22sleep30-21swapper/308:08:243
2447825116,17sleep10-21swapper/112:23:151
190732515,34sleep20-21swapper/208:40:252
3015725016,16sleep20-21swapper/207:33:262
2840925017,19sleep10-21swapper/109:09:361
2840925017,19sleep10-21swapper/109:09:361
2001925012,19sleep220112-21memory07:03:252
3145724912,21sleep131458-21ntp_states09:18:321
3145724912,21sleep131458-21ntp_states09:18:321
279324913,20sleep30-21swapper/307:48:333
1997224911,20sleep220001-21ntp_states12:08:262
1488824911,21sleep314883-21apt-get08:28:143
2985224814,18sleep30-21swapper/309:13:333
2985224814,18sleep30-21swapper/309:13:333
2482824810,14sleep20-21swapper/207:18:032
654624512,17sleep10-21swapper/109:43:111
122162458,13sleep30-21swapper/311:43:313
264024410,15sleep30-21swapper/311:13:233
1126024313,18sleep20-21swapper/208:18:002
55124111,15sleep10-21swapper/109:23:331
55124111,15sleep10-21swapper/109:23:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional