You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 12:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 26, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
247852767713,7651sleep115021-21kworker/1:122:35:071
20905212706,1246sleep220981-21/usr/sbin/munin22:23:542
14088237921,340sleep014146-21/usr/sbin/munin23:43:530
14088237921,340sleep014146-21/usr/sbin/munin23:43:530
11289235614,213sleep211330-21grep21:53:552
38802307133,150sleep20-21swapper/219:48:452
17460229378,200sleep217527-21sort22:13:492
38062283112,48sleep23763-21latency_hist23:13:272
7991227515,244sleep38069-21/usr/sbin/munin21:43:503
136902256113,24sleep20-21swapper/223:43:432
136902256113,24sleep20-21swapper/223:43:432
13116221914,183sleep213269-21unixbench_singl21:58:592
18166221313,183sleep218372-21apt-config23:58:282
18166221313,183sleep218372-21apt-config23:58:282
2665822037,180sleep226820-21cut22:43:282
2179321957,177sleep221288-21apt-get19:03:312
2179321957,177sleep221288-21apt-get19:03:312
17077219513,162sleep217238-21latency_hist22:13:282
17077219513,162sleep217238-21latency_hist22:13:282
29090218810,158sleep228974-21/usr/sbin/munin21:08:452
30699218227,23sleep20-21swapper/219:28:592
26615218217,43sleep226770-21unixbench_singl20:58:572
23248217350,102sleep223241-21apt-get00:13:412
23248217350,102sleep223241-21apt-get00:13:412
29694216896,55sleep229709-21modprobe22:53:252
29694216896,55sleep229709-21modprobe22:53:252
25507216612,124sleep20-21swapper/200:18:582
247252166120,27sleep221505-21cyclictest20:53:522
1608021645,147sleep00-21swapper/023:49:010
5987216216,118sleep20-21swapper/219:53:572
18432160115,20sleep20-21swapper/221:23:592
145222159110,26sleep20-21swapper/220:23:282
6756215423,45sleep36839-21grep21:38:583
4210215213,122sleep24414-21gpgv21:33:282
12532215284,52sleep212041-21/usr/sbin/munin23:38:542
8581214919,114sleep28672-21fschecks_time20:03:502
18546214817,116sleep318539-21apt-get23:58:413
8644214715,115sleep28654-21modprobe23:28:252
1964021449,112sleep219634-21apt-get20:38:422
6138214274,52sleep26176-21latency23:19:122
1204421427,15sleep20-21swapper/220:13:512
1204421427,15sleep20-21swapper/220:13:512
3007521416,12sleep00-21swapper/019:28:440
1565221407,106sleep20-21swapper/223:48:552
2802921376,11sleep20-21swapper/222:46:552
2802921376,11sleep20-21swapper/222:46:552
20119213778,33sleep20-21swapper/222:23:252
5915213621,91sleep16002-21wc21:38:271
209921356,11sleep20-21swapper/223:07:302
500421315,12sleep20-21swapper/223:14:362
28557213114,18sleep20-21swapper/219:23:462
3185213021,85sleep13345-21sed23:08:581
2497021306,12sleep20-21swapper/219:12:192
2286021306,18sleep20-21swapper/200:12:322
21531212817,92sleep20-21swapper/200:08:292
21531212817,92sleep20-21swapper/200:08:292
1964721285,11sleep20-21swapper/200:01:232
19108212811,94sleep219134-21munin-run20:38:262
25099212711,18sleep20-21swapper/222:38:272
793921256,17sleep20-21swapper/220:00:152
1149221256,12sleep20-21swapper/223:34:502
30349212415,20sleep00-21swapper/021:13:280
670212113,93sleep0687-21apt19:38:370
31819212110,91sleep20-21swapper/222:58:462
30588211932,71sleep20-21swapper/221:13:422
16029211819,78sleep116079-21wc20:28:281
21664211682,14sleep20-21swapper/220:43:552
2952211424,64sleep30-21swapper/319:43:573
1053621149,14sleep20-21swapper/221:53:262
74021126,26sleep20-21swapper/221:20:522
21264211129,58sleep121316-21http19:03:261
21264211129,58sleep121316-21http19:03:261
17820211110,89sleep05666-21snmpd20:33:350
10589211115,81sleep310672-21latency_hist21:53:273
30674211024,29sleep30-21swapper/321:13:473
25054211016,77sleep325082-21munin-plugin-st19:13:253
154621109,14sleep30-21swapper/321:23:523
1499721108,91sleep25666-21snmpd22:05:572
1499721108,91sleep25666-21snmpd22:05:572
770221087,15sleep20-21swapper/221:43:412
1471821088,15sleep20-21swapper/220:23:402
12999210711,83sleep113115-21timerandwakeup21:58:581
2790121068,14sleep20-21swapper/221:03:532
2728221065,89sleep20-21swapper/200:24:192
2728221065,89sleep20-21swapper/200:24:192
1019421068,17sleep21-21systemd23:33:252
2766121057,13sleep20-21swapper/219:19:022
1750121056,88sleep20-21swapper/220:33:052
457121048,77sleep20-21swapper/221:33:362
329821047,85sleep20-21swapper/219:45:502
30468210410,83sleep230474-21ntp_states22:53:532
27296210418,68sleep327335-21ntpq19:18:543
3159721036,12sleep20-21swapper/219:33:392
2537321038,11sleep20-21swapper/219:13:412
23360210310,66sleep00-21swapper/019:08:300
2811321006,11sleep20-21swapper/200:28:512
71472996,10sleep20-21swapper/221:40:062
404629918,67sleep04120-21gpgv21:33:260
3210929810,67sleep332110-21/usr/sbin/munin21:18:353
2574329816,63sleep025774-21/usr/sbin/munin19:13:510
2916029621,21sleep00-21swapper/000:33:250
237072966,10sleep20-21swapper/220:49:362
230092966,10sleep20-21swapper/222:30:402
2070629535,33sleep020863-21sed18:58:580
2070629535,33sleep020863-21sed18:58:580
110702955,10sleep20-21swapper/220:10:322
90092938,59sleep00-21swapper/023:28:430
1448429313,20sleep30-21swapper/322:04:013
1448429313,20sleep30-21swapper/322:04:013
17672925,12sleep20-21swapper/219:41:022
1548429212,60sleep315211-21apt-get22:08:293
1548429212,60sleep315211-21apt-get22:08:293
586328914,52sleep35887-21idleruntime-cro21:38:253
2827228817,55sleep128299-21/usr/sbin/munin00:28:541
2038328823,34sleep10-21swapper/122:23:371
117062889,15sleep00-21swapper/020:13:450
117062889,15sleep00-21swapper/020:13:450
693028623,42sleep16980-21cpuspeed19:58:411
450928615,52sleep34023-21/usr/sbin/munin23:13:563
2324528520,35sleep00-21swapper/000:13:390
2324528520,35sleep00-21swapper/000:13:390
2311628522,46sleep123155-21ntp_kernel_err20:48:501
1214928524,30sleep00-21swapper/023:38:440
1124028511,18sleep30-21swapper/321:53:513
90192846,13sleep10-21swapper/123:28:441
3250228413,57sleep032531-21/usr/sbin/munin21:18:490
2998928412,45sleep30-21swapper/322:53:383
63182838,55sleep00-21swapper/023:19:180
2210428324,36sleep10-21swapper/119:03:511
2210428324,36sleep10-21swapper/119:03:511
1387928319,43sleep014027-21unixbench_singl20:18:580
90622828,60sleep00-21swapper/020:04:000
89412828,14sleep10-21swapper/120:03:571
3189528216,51sleep031902-21diskstats22:58:480
1130028222,40sleep111334-21munin-run20:13:261
249528127,35sleep02525-21munin-run21:28:250
2442428012,51sleep124533-21fschecks_time20:53:471
2346028014,51sleep323510-21/usr/sbin/munin00:13:513
2346028014,51sleep323510-21/usr/sbin/munin00:13:513
197492806,65sleep00-21swapper/000:02:380
1091428025,30sleep10-21swapper/121:53:451
2852327921,34sleep028538-21cut22:48:460
2852327921,34sleep028538-21cut22:48:460
2720727920,24sleep00-21swapper/021:03:270
534727816,32sleep30-21swapper/319:53:383
2263527819,46sleep022766-21sed22:28:560
1337527825,20sleep30-21swapper/323:43:273
2816927717,44sleep328185-21/usr/sbin/munin00:28:523
260602770,6sleep30-21swapper/320:58:443
1051027720,34sleep110533-21idleruntime-cro21:53:251
2139927616,30sleep30-21swapper/320:43:483
869527515,45sleep08779-21wc23:28:270
372727530,25sleep00-21swapper/023:13:250
2131727525,32sleep221393-21munin-run19:03:272
2131727525,32sleep221393-21munin-run19:03:272
1582827518,25sleep00-21swapper/022:08:580
1582827518,25sleep00-21swapper/022:08:580
3189927411,13sleep00-21swapper/019:33:490
2719727421,26sleep10-21swapper/119:18:521
2194427413,29sleep30-21swapper/300:08:583
1610927416,42sleep116133-21netstat22:09:031
1610927416,42sleep116133-21netstat22:09:031
807127318,30sleep00-21swapper/021:43:510
2094827315,38sleep322178-1kworker/3:2H20:43:273
1091027314,27sleep30-21swapper/323:33:513
3044427211,46sleep130467-21ntp_states22:53:521
2439127219,27sleep30-21swapper/322:34:123
2332527219,31sleep30-21swapper/322:33:273
1758327218,27sleep00-21swapper/022:13:510
2815827118,21sleep30-21swapper/321:03:583
2067127116,33sleep00-21swapper/000:03:590
2067127116,33sleep00-21swapper/000:03:590
189012715,10sleep10-21swapper/120:35:501
505427023,27sleep35081-21munin-run19:53:263
21702702,34sleep10-21swapper/123:08:251
1996527019,21sleep00-21swapper/022:21:300
1739827016,36sleep317525-21/usr/sbin/munin23:54:013
1739827016,36sleep317525-21/usr/sbin/munin23:54:013
55552698,49sleep00-21swapper/021:34:310
535826917,33sleep15442-21apt-config23:18:261
2436126921,27sleep10-21swapper/119:08:551
1434526923,29sleep30-21swapper/323:43:563
1434526923,29sleep30-21swapper/323:43:563
1364626918,33sleep013667-21munin-run22:03:250
400326817,24sleep00-21swapper/019:48:470
3128026818,33sleep131286-21modprobe19:33:261
2596326825,24sleep10-21swapper/120:58:431
2136326823,30sleep31-21systemd00:08:263
2136326823,30sleep31-21systemd00:08:263
125252686,48sleep10-21swapper/123:38:531
3220526723,21sleep30-21swapper/319:33:563
3178126712,39sleep131295-21apt-get22:58:361
2895626722,23sleep00-21swapper/019:23:560
2823526714,27sleep00-21swapper/019:23:280
2724926715,29sleep10-21swapper/122:43:471
2724926715,29sleep10-21swapper/122:43:471
2032226716,34sleep020459-21sort20:39:290
1677926721,30sleep016865-21mailstats20:28:590
56702657,12sleep30-21swapper/323:18:293
437926516,30sleep14471-21/usr/sbin/munin19:48:561
2749126521,25sleep00-21swapper/022:43:540
2749126521,25sleep00-21swapper/022:43:540
2205426520,29sleep10-21swapper/122:28:451
1519726517,30sleep015237-21http22:08:260
1519726517,30sleep015237-21http22:08:260
757226422,21sleep00-21swapper/019:58:560
2203926419,24sleep00-21swapper/000:08:590
2180226412,23sleep00-21swapper/019:03:370
2180226412,23sleep00-21swapper/019:03:370
153026419,21sleep00-21swapper/023:04:000
669326316,22sleep00-21swapper/019:58:270
285702637,14sleep30-21swapper/319:23:473
2141726320,21sleep30-21swapper/319:03:283
2141726320,21sleep30-21swapper/319:03:283
135392636,37sleep10-21swapper/120:18:511
9452626,13sleep10-21swapper/121:23:261
738626216,28sleep30-21swapper/323:23:413
287626225,20sleep10-21swapper/121:28:511
276326221,25sleep30-21swapper/321:28:403
2571826224,17sleep30-21swapper/300:19:073
2185626227,20sleep10-21swapper/100:08:561
2020026215,23sleep10-21swapper/122:23:281
106002626,12sleep10-21swapper/123:33:441
84126114,28sleep10-21swapper/119:38:461
770126115,28sleep17698-21apt-get21:43:401
3011826120,23sleep30-21swapper/319:28:473
2558226119,23sleep10-21swapper/119:13:481
1862326119,18sleep10-21swapper/122:18:281
177126113,25sleep10-21swapper/121:23:581
1533326115,20sleep30-21swapper/320:23:583
530626013,28sleep05356-21apt-config23:18:250
3133526022,20sleep00-21swapper/022:58:270
2634426019,23sleep00-21swapper/000:23:400
2634426019,23sleep00-21swapper/000:23:400
2503526016,19sleep00-21swapper/000:18:480
224226015,27sleep12239-21apt-get19:43:391
1802126021,18sleep00-21swapper/023:58:260
1802126021,18sleep00-21swapper/023:58:260
112892605,44sleep00-21swapper/020:13:170
750025919,18sleep10-21swapper/121:43:281
2778225916,21sleep10-21swapper/100:28:291
2778225916,21sleep10-21swapper/100:28:291
2554325916,27sleep125593-21if_eth022:38:461
2336025925,18sleep30-21swapper/320:48:563
2074625918,20sleep10-21swapper/100:04:001
2074625918,20sleep10-21swapper/100:04:001
179282595,18sleep10-21swapper/123:57:141
179282595,18sleep10-21swapper/123:57:141
1343825919,17sleep30-21swapper/320:18:493
1138025914,33sleep311492-21latency_hist20:13:283
946125815,17sleep00-21swapper/021:48:480
714725819,17sleep10-21swapper/123:23:271
36672585,17sleep30-21swapper/323:12:393
3160125817,21sleep10-21swapper/119:33:421
2904425813,31sleep329071-21df21:08:423
1816925814,26sleep318215-21memory20:33:513
1182725813,26sleep00-21swapper/023:38:280
1142125819,21sleep00-21swapper/021:53:570
932925717,25sleep39368-21ntpq23:28:503
745925718,23sleep00-21swapper/021:43:270
2375625710,22sleep00-21swapper/022:33:370
1980025713,19sleep10-21swapper/120:39:181
1664025720,19sleep10-21swapper/120:28:521
1286125720,20sleep30-21swapper/321:58:563
1225825717,20sleep30-21swapper/320:13:563
1225825717,20sleep30-21swapper/320:13:563
692725613,26sleep36923-21apt-get19:58:393
283342565,11sleep10-21swapper/119:23:291
2537425613,26sleep325367-21apt-get19:13:423
2190425618,17sleep30-21swapper/319:03:453
2190425618,17sleep30-21swapper/319:03:453
1654725620,16sleep30-21swapper/323:53:283
163062564,11sleep10-21swapper/123:51:531
154732568,11sleep00-21swapper/020:23:590
69425513,23sleep30-21swapper/319:38:403
3188725514,25sleep331893-21df_inode22:58:473
3059125512,17sleep00-21swapper/021:13:440
2347825521,17sleep00-21swapper/020:48:570
2108225520,19sleep00-21swapper/022:23:560
1867625515,21sleep10-21swapper/123:59:401
31812546,38sleep00-21swapper/019:44:190
2854125414,19sleep10-21swapper/122:48:471
2854125414,19sleep10-21swapper/122:48:471
2745125416,23sleep027518-21df21:03:430
2531025413,24sleep325329-21apt-get22:38:363
2507025411,29sleep124773-21/usr/sbin/munin00:18:491
2477225414,25sleep324782-21ntpq20:53:533
2899925312,24sleep128993-21apt-get21:08:401
2876025315,17sleep10-21swapper/121:08:281
2118925311,24sleep00-21swapper/020:43:410
90925213,22sleep3904-21apt-get23:03:393
867525212,18sleep30-21swapper/320:03:513
2888325218,19sleep30-21swapper/322:48:543
2888325218,19sleep30-21swapper/322:48:543
2391125215,18sleep10-21swapper/100:14:011
2391125215,18sleep10-21swapper/100:14:011
2147425215,23sleep121564-21memory20:43:501
3107325117,16sleep10-21swapper/121:13:571
2098225112,21sleep30-21swapper/322:23:553
1895525119,19sleep30-21swapper/322:18:463
1672325115,21sleep316729-21ntp_states20:28:543
1523825110,23sleep115326-21gpgv22:08:271
1523825110,23sleep115326-21gpgv22:08:271
780125010,15sleep10-21swapper/123:23:511
2579725013,16sleep00-21swapper/022:38:540
1277425013,22sleep112775-21sshd20:17:051
1277425013,22sleep112775-21sshd20:17:051
2478424917,18sleep00-21swapper/020:53:540
1860824921,15sleep30-21swapper/322:18:263
1368324912,22sleep113689-21df_inode23:43:421
1368324912,22sleep113689-21df_inode23:43:421
544524811,21sleep15452-21df_inode19:53:441
2765824511,19sleep327713-21latency_hist00:28:273
2765824511,19sleep327713-21latency_hist00:28:273
2741624513,16sleep30-21swapper/322:43:533
2741624513,16sleep30-21swapper/322:43:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional