You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 00:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sun May 31, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19363291966,9169sleep312534-21kworker/u8:312:18:583
12652261836,6164sleep08142-21kworker/0:008:30:590
98932520320,5167sleep05567-21kworker/0:210:03:580
4748241156,336sleep34758-21modprobe11:33:243
4748241156,336sleep34758-21modprobe11:33:243
20153231819,275sleep120296-21latency_hist10:38:271
1038229812,198sleep30-21swapper/307:53:513
2976022707,249sleep329786-21df11:08:413
3165022568,234sleep010235-1kworker/0:2H07:49:360
3165022568,234sleep010235-1kworker/0:2H07:49:360
3887224117,210sleep13923-21ntpq11:28:501
3887224117,210sleep13923-21ntpq11:28:501
1119222934,177sleep31133-21modprobe09:38:253
7300222114,191sleep17343-21ntpq08:13:521
7300222114,191sleep17343-21ntpq08:13:521
84862214110,34sleep30-21swapper/308:18:383
1545422089,153sleep315528-21latency_hist10:23:273
12269220872,76sleep312435-21tail08:29:003
23990219918,118sleep324006-21modprobe09:08:253
31347219714,123sleep331361-21fschecks_count11:13:463
31150219719,157sleep131304-21apt-key07:48:281
31150219719,157sleep131304-21apt-key07:48:281
285112190102,30sleep30-21swapper/307:38:493
28250218251,22sleep30-21swapper/311:03:453
1077421749,150sleep110767-21apt-get10:08:391
13123217119,18sleep00-21swapper/010:13:590
24382217049,105sleep324428-21fschecks_count09:08:453
8290216410,56sleep30-21swapper/309:58:563
14977216419,130sleep115043-21sed12:03:581
18072216329,116sleep30-21swapper/312:13:543
2370421579,12sleep20-21swapper/207:23:462
4536215318,112sleep14676-21grep08:03:581
26741215320,109sleep30-21swapper/307:33:443
1834215128,55sleep30-21swapper/309:38:493
9787214720,91sleep30-21swapper/311:48:423
9798214663,28sleep30-21swapper/310:03:573
20529214326,78sleep320618-21fschecks_time07:13:453
1565921429,120sleep315698-21modprobe12:08:243
30528214017,99sleep230651-21latency_hist09:28:262
462421389,24sleep00-21swapper/009:48:430
13465213464,48sleep30-21swapper/310:14:023
25688213210,62sleep30-21swapper/307:28:553
1821721329,105sleep30-21swapper/308:48:463
12533212820,88sleep112043-21apt-get10:13:361
3207721278,104sleep332215-21latency_hist09:33:283
7123212639,18sleep30-21swapper/311:38:533
7123212639,18sleep30-21swapper/311:38:533
20752212523,58sleep320358-21/usr/sbin/munin10:38:493
11742212416,51sleep10-21swapper/111:53:531
22896212347,63sleep322935-21wc12:28:513
31824212214,64sleep331826-21fschecks_time07:50:033
31824212214,64sleep331826-21fschecks_time07:50:033
26602212215,43sleep30-21swapper/310:58:403
23985212213,89sleep224064-21/usr/sbin/munin10:48:582
13328212165,40sleep30-21swapper/311:58:523
2435721206,11sleep10-21swapper/112:33:501
24878211922,73sleep024963-21latency_hist07:28:280
22146211616,86sleep022174-21idleruntime-cro12:28:240
26850211510,93sleep324073-21kworker/3:009:15:153
26850211510,93sleep324073-21kworker/3:009:15:153
20597211515,84sleep120614-21awk12:23:241
29306211421,73sleep129357-21fschecks_count09:23:451
15554211417,23sleep00-21swapper/008:38:580
7577211343,45sleep30-21swapper/308:13:583
7577211343,45sleep30-21swapper/308:13:583
11532211312,52sleep30-21swapper/311:53:473
1276211115,81sleep11381-21timerandwakeup07:53:571
31211211012,84sleep031222-21ntpq09:28:490
9474210810,19sleep30-21swapper/311:48:263
9474210810,19sleep30-21swapper/311:48:263
553321086,17sleep30-21swapper/309:50:003
15632210834,52sleep115799-21sed08:38:591
35321079,49sleep30-21swapper/311:18:343
23281210732,60sleep31-21systemd07:23:263
387721069,52sleep30-21swapper/309:43:583
17341210412,76sleep017428-21/usr/sbin/munin10:28:450
17341210412,76sleep017428-21/usr/sbin/munin10:28:450
8153210312,75sleep18214-21sendmail_mailst09:58:541
2045521039,47sleep30-21swapper/308:53:593
1859921028,74sleep00-21swapper/008:48:560
17210210213,54sleep31-21systemd12:13:253
76621010,14sleep20-21swapper/207:53:452
6069210119,38sleep30-21swapper/311:35:263
6069210119,38sleep30-21swapper/311:35:263
5811210017,58sleep25326-21/usr/sbin/munin08:08:522
775629824,47sleep07849-21fschecks_time09:58:450
639529812,70sleep16547-21latency_hist11:38:271
639529812,70sleep16547-21latency_hist11:38:271
2098829714,66sleep121064-21timerandwakeup10:38:561
2219929616,65sleep322173-21latency_hist12:28:273
1973629616,34sleep30-21swapper/310:34:133
182772966,13sleep10-21swapper/112:13:561
2353929512,68sleep123556-21/usr/sbin/munin07:23:361
233429519,54sleep12398-21fschecks_time07:58:451
1822829518,23sleep20-21swapper/207:08:272
145892957,75sleep00-21swapper/008:37:570
73212949,73sleep27322-21sendmail-mta09:58:062
307642948,16sleep10-21swapper/109:28:381
2896929327,17sleep30-21swapper/309:23:273
186772936,39sleep30-21swapper/310:33:283
186772936,39sleep30-21swapper/310:33:283
2243729215,58sleep30-21swapper/310:43:543
1624529220,59sleep216322-21switchtime10:23:542
1133729217,63sleep211365-21latency_hist08:28:262
1689929124,37sleep00-21swapper/007:03:410
2652829011,57sleep10-21swapper/109:13:561
2652829011,57sleep10-21swapper/109:13:561
307832896,38sleep30-21swapper/307:45:023
307832896,38sleep30-21swapper/307:45:023
162112896,12sleep10-21swapper/112:08:471
1331628921,47sleep212540-21/usr/sbin/munin10:14:012
120952890,6sleep00-21swapper/010:13:260
929828813,47sleep20-21swapper/210:03:432
729828817,44sleep17443-21grep11:38:551
729828817,44sleep17443-21grep11:38:551
3130728812,58sleep231473-21apt-config07:48:292
3130728812,58sleep231473-21apt-config07:48:292
3007828814,59sleep230105-21awk11:08:482
264728831,45sleep32682-21ntpq07:58:513
1462928811,30sleep314632-21cron08:38:253
2425728715,18sleep30-21swapper/312:33:483
2405628711,61sleep124123-21latency_hist09:08:271
1172828713,58sleep011741-21/usr/sbin/munin11:53:520
112662875,35sleep30-21swapper/308:28:073
3101728620,50sleep031077-21cat11:13:260
2576028621,44sleep025913-21unixbench_singl10:53:570
1686428640,34sleep30-21swapper/308:43:503
998528519,45sleep09688-21/usr/sbin/munin11:48:470
825028515,54sleep18262-21/usr/sbin/munin11:43:441
825028515,54sleep18262-21/usr/sbin/munin11:43:441
1879428523,43sleep218944-21grep08:48:582
2396728411,20sleep30-21swapper/307:23:513
596928320,32sleep10-21swapper/108:08:551
2929028337,32sleep30-21swapper/309:23:433
242862830,6sleep10-21swapper/110:49:001
2324028314,53sleep023337-21cat10:48:270
1168528336,11sleep30-21swapper/310:08:573
267128221,45sleep02817-21sed11:23:560
267128221,45sleep02817-21sed11:23:560
744328116,14sleep30-21swapper/309:58:273
553328110,53sleep30-21swapper/308:08:453
2644428115,18sleep00-21swapper/009:13:550
2644428115,18sleep00-21swapper/009:13:550
2178028120,28sleep10-21swapper/107:18:281
1519228131,34sleep30-21swapper/308:38:493
2947328021,31sleep10-21swapper/111:08:271
2791928011,55sleep127972-21latency_hist11:03:261
265962808,56sleep00-21swapper/010:58:370
2051828014,31sleep20-21swapper/207:13:442
48242796,30sleep30-21swapper/308:05:223
1738327822,38sleep117470-21mailstats07:03:551
1614927829,33sleep30-21swapper/310:23:513
2529727716,45sleep325386-21cat10:53:473
2247627720,31sleep10-21swapper/107:18:521
1848327720,13sleep30-21swapper/307:08:413
111482775,63sleep00-21swapper/008:26:340
577227622,17sleep00-21swapper/011:34:010
577227622,17sleep00-21swapper/011:34:010
31327622,42sleep0317-21ntp_states09:33:540
2995227614,49sleep230010-21idleruntime07:43:482
2995227614,49sleep230010-21idleruntime07:43:482
688127517,44sleep06899-21grep11:38:460
688127517,44sleep06899-21grep11:38:460
2910627524,35sleep129144-21users07:38:591
2562027523,35sleep10-21swapper/109:13:271
1440327519,25sleep20-21swapper/212:03:432
8242746,15sleep10-21swapper/111:18:481
70892745,57sleep00-21swapper/009:55:050
1986927417,41sleep019920-21sed08:53:480
1346727419,29sleep10-21swapper/111:58:541
2896327316,24sleep00-21swapper/007:38:580
1339727316,37sleep013465-21switchtime11:58:530
318527226,12sleep30-21swapper/311:28:153
318527226,12sleep30-21swapper/311:28:153
2463527228,11sleep30-21swapper/310:52:063
2093527215,39sleep220991-21latency_hist08:58:262
2082927212,26sleep00-21swapper/007:13:500
2826527129,24sleep00-21swapper/009:19:120
2797327110,16sleep20-21swapper/211:03:272
2738227120,30sleep127570-21apt-config09:18:281
2230127129,10sleep30-21swapper/309:01:383
2018827117,22sleep20-21swapper/212:20:272
1926827018,37sleep019292-21proc_pri10:33:540
187602706,13sleep20-21swapper/210:33:312
1415027020,31sleep114125-21/usr/sbin/munin10:18:411
1297427022,29sleep20-21swapper/208:33:272
974626922,24sleep10-21swapper/108:23:241
298626918,31sleep02981-21apt-get09:43:390
2067626919,27sleep20-21swapper/210:38:482
1725926924,21sleep00-21swapper/012:13:260
512626821,31sleep25134-21/usr/sbin/munin11:33:472
512626821,31sleep25134-21/usr/sbin/munin11:33:472
3176726829,22sleep10-21swapper/111:13:561
2711126814,32sleep227140-21apt-get09:18:252
2711126814,32sleep227140-21apt-get09:18:252
2327526821,22sleep20-21swapper/207:23:252
727026720,24sleep00-21swapper/008:13:510
727026720,24sleep00-21swapper/008:13:510
398526714,31sleep20-21swapper/208:03:472
1921626716,33sleep119353-21grep07:08:591
1617926717,34sleep116244-21sendmail_mailst10:23:531
1212126715,36sleep112191-21sensors08:28:581
31826613,36sleep1359-21wc09:33:551
2783626614,23sleep20-21swapper/209:19:022
2666526615,20sleep10-21swapper/107:33:421
2369826623,26sleep00-21swapper/012:33:250
1877826617,34sleep218815-21munin-run12:18:252
1742926622,24sleep10-21swapper/110:28:461
1742926622,24sleep10-21swapper/110:28:461
1710726613,30sleep10-21swapper/108:43:561
216352658,12sleep00-21swapper/008:58:530
1320726511,40sleep213225-21netstat11:58:472
698626427,23sleep20-21swapper/208:13:442
698626427,23sleep20-21swapper/208:13:442
348826415,22sleep00-21swapper/011:28:410
348826415,22sleep00-21swapper/011:28:410
3260626432,16sleep00-21swapper/011:18:270
298826412,38sleep23024-21cpu09:43:412
287812649,41sleep228849-21/usr/sbin/munin07:38:562
219342646,15sleep20-21swapper/210:43:412
2141526412,34sleep221444-21irqstats08:58:472
201726414,21sleep00-21swapper/009:38:540
1716826425,18sleep30-21swapper/307:03:483
15952646,11sleep00-21swapper/007:54:080
93912637,12sleep00-21swapper/011:47:590
93912637,12sleep00-21swapper/011:47:590
2666426318,28sleep026658-21apt-get07:33:410
1125726321,22sleep20-21swapper/210:08:522
2238726220,22sleep10-21swapper/110:43:521
472526114,29sleep24761-21/usr/sbin/munin09:48:462
3123026116,28sleep20-21swapper/209:28:512
1875626127,16sleep00-21swapper/007:08:500
1814426118,23sleep00-21swapper/012:13:550
1691126113,32sleep016948-21ntp_states08:43:510
153426121,21sleep20-21swapper/209:38:442
430326018,25sleep14265-21latency_hist09:48:281
2853326020,19sleep00-21swapper/011:03:500
2529726017,20sleep20-21swapper/207:28:462
2088526010,34sleep220882-21apt-get12:23:382
2072326010,36sleep020845-21sed12:23:270
2045526015,28sleep00-21swapper/010:38:420
1461126015,20sleep20-21swapper/210:18:542
1334026011,26sleep20-21swapper/208:33:352
821625921,21sleep20-21swapper/209:58:552
47825917,26sleep1615-21cut07:53:281
3185125911,28sleep231940-21awk11:13:572
2407525917,21sleep00-21swapper/012:33:450
150552595,13sleep00-21swapper/010:19:000
120125912,21sleep20-21swapper/209:38:282
934625820,19sleep10-21swapper/110:03:461
2967225814,21sleep00-21swapper/009:23:520
270925813,20sleep20-21swapper/207:58:542
2539725816,22sleep10-21swapper/110:53:491
1658225817,23sleep20-21swapper/208:43:452
1129425811,31sleep111336-21systemd-run08:28:251
105725816,18sleep20-21swapper/211:18:542
980525717,23sleep19896-21fschecks_time11:48:441
56152576,38sleep10-21swapper/109:51:031
323825715,26sleep13248-21munin-run11:28:251
323825715,26sleep13248-21munin-run11:28:251
3040425718,16sleep00-21swapper/007:43:580
3040425718,16sleep00-21swapper/007:43:580
2416525717,15sleep00-21swapper/010:49:000
2371825717,25sleep123747-21munin-run12:33:261
2243525715,26sleep222429-21apt-get12:28:382
2097825718,23sleep121070-21perl12:23:451
1629825718,19sleep20-21swapper/208:43:282
1489425712,29sleep014976-21mailstats12:03:570
975425611,32sleep29796-21unin-run08:23:252
21582568,28sleep20-21swapper/211:23:462
21582568,28sleep20-21swapper/211:23:462
1832325619,20sleep10-21swapper/108:48:481
143725614,25sleep11431-21apt-get09:38:391
608525515,24sleep16082-21apt-get09:53:371
306992555,38sleep10-21swapper/111:09:561
2535625519,17sleep20-21swapper/212:38:272
2502425513,25sleep225018-21apt-get10:53:382
2273025515,23sleep222726-21apt-get09:03:392
1567025512,17sleep00-21swapper/010:23:360
1147425513,24sleep211531-21irqstats11:53:462
863025419,21sleep20-21swapper/211:43:532
863025419,21sleep20-21swapper/211:43:532
2557725416,23sleep125089-21/usr/sbin/munin07:28:531
2352225414,22sleep023632-21users09:03:580
442025312,18sleep00-21swapper/008:03:570
251725313,27sleep02576-21/usr/sbin/munin07:58:480
2473825315,21sleep224749-21ntp_states09:08:522
1960025314,23sleep119592-21apt-get08:53:411
17642538,15sleep00-21swapper/011:23:280
534925212,23sleep05344-21apt-get08:08:380
2429225213,23sleep024289-21apt-get09:08:380
2332725116,18sleep00-21swapper/007:23:270
1594925110,19sleep20-21swapper/212:08:402
85672509,17sleep00-21swapper/008:18:430
35642509,16sleep20-21swapper/211:28:432
35642509,16sleep20-21swapper/211:28:432
1748525015,19sleep20-21swapper/210:28:472
1748525015,19sleep20-21swapper/210:28:472
2705824918,17sleep20-21swapper/210:58:512
2519924915,15sleep00-21swapper/007:28:440
2248224912,22sleep122492-21munin-run09:03:261
315722487,18sleep10-21swapper/107:48:401
1935624814,20sleep219389-21users07:09:002
1333424810,22sleep112857-21apt-get08:33:301
2673424710,21sleep226740-21df_inode07:33:432
1009024712,17sleep20-21swapper/211:48:492
661624514,16sleep20-21swapper/211:38:402
661624514,16sleep20-21swapper/211:38:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional