You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 03:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Tue Jun 16, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1589322033614,20303sleep013763-21kworker/u8:120:43:090
1589322033614,20303sleep013763-21kworker/u8:120:43:090
12917286815,840sleep37253-21kworker/3:022:13:523
23713272610,704sleep123765-21ntpq22:47:541
4893241916,386sleep24886-21apt-get23:32:452
2256240912,382sleep02261-21ntpq23:22:580
23157228572,137sleep10-21swapper/119:22:431
23157228572,137sleep10-21swapper/119:22:431
1091922859,265sleep110925-21cat20:27:401
314512284110,59sleep10-21swapper/121:32:321
11689227522,230sleep211839-21grep20:28:012
17566226916,202sleep117588-21fschecks_time20:47:551
17566226916,202sleep117588-21fschecks_time20:47:551
12027225516,143sleep112119-21fschecks_time22:12:491
12451225247,184sleep112520-21apt-key23:57:311
3157522317,212sleep131609-21ps23:12:571
27779222316,146sleep127807-21latency_hist23:02:321
12522222113,191sleep212634-21cat23:57:322
25968221918,178sleep225999-21awk19:32:292
9749221435,130sleep19540-21/usr/sbin/munin23:47:501
17722221017,138sleep10-21swapper/100:12:491
974621830,6sleep10-21swapper/122:03:031
974621830,6sleep10-21swapper/122:03:031
60321787,157sleep00-21swapper/023:17:530
10709217698,18sleep10-21swapper/120:27:321
786321759,124sleep17914-21switchtime21:57:591
4530217116,52sleep10-21swapper/121:47:581
15293217124,44sleep00-21swapper/022:22:410
16726216511,16sleep20-21swapper/219:02:502
2521821628,140sleep025315-21latency_hist21:12:320
2154216019,24sleep10-21swapper/121:42:331
597221589,133sleep16002-21kernelversion21:52:501
29649215818,117sleep10-21swapper/123:07:491
2008021587,35sleep10-21swapper/100:18:021
28119215514,111sleep10-21swapper/119:37:541
28119215514,111sleep10-21swapper/119:37:541
17309215520,118sleep117323-21modprobe00:12:291
17309215520,118sleep117323-21modprobe00:12:291
25421215314,119sleep225493-21/usr/sbin/munin22:53:002
613215072,59sleep10-21swapper/123:17:541
15814214828,106sleep115959-21latency_hist00:07:321
15814214828,106sleep115959-21latency_hist00:07:321
11000214610,118sleep111096-21timerandwakeup22:08:011
11000214610,118sleep111096-21timerandwakeup22:08:011
20513214418,110sleep220594-21grep19:12:592
20513214418,110sleep220594-21grep19:12:592
18176214114,17sleep10-21swapper/119:07:491
1199121418,12sleep20-21swapper/222:12:462
22429213916,62sleep10-21swapper/121:02:491
12933213922,45sleep012934-21apt-get23:57:420
20251213711,102sleep00-21swapper/000:19:540
20122213733,91sleep120224-21fschecks_time19:12:501
20122213733,91sleep120224-21fschecks_time19:12:501
21010213617,105sleep221031-21kernelversion00:22:512
20707213612,39sleep020729-21/usr/sbin/munin22:38:000
20707213612,39sleep020729-21/usr/sbin/munin22:38:000
20019213527,57sleep10-21swapper/122:37:321
15324213540,53sleep10-21swapper/122:22:501
4656213344,59sleep10-21swapper/123:32:321
4656213344,59sleep10-21swapper/123:32:321
3624213311,103sleep13642-21apt-get21:47:291
32241213110,22sleep10-21swapper/119:52:311
32241213110,22sleep10-21swapper/119:52:311
14210213123,91sleep214214-21modprobe00:02:282
21131213011,54sleep10-21swapper/100:22:541
2956421299,70sleep129613-21idleruntime19:42:501
2956421299,70sleep129613-21idleruntime19:42:501
24971212910,98sleep00-21swapper/022:52:500
14275212930,60sleep114433-21latency_hist00:02:321
23994212812,101sleep124071-21fschecks_time21:07:491
2360221289,105sleep123566-21latency_hist00:32:311
3562212632,59sleep13585-21ntpq20:02:571
3562212632,59sleep13585-21ntpq20:02:571
2960212617,75sleep10-21swapper/123:26:181
17099212651,31sleep117094-21apt-get22:27:431
15895212630,61sleep115847-21apt-get20:43:101
15895212630,61sleep115847-21apt-get20:43:101
21684212520,24sleep10-21swapper/119:17:471
15284212521,86sleep314798-21apt-get22:22:403
6561212411,96sleep16645-21fschecks_time23:37:491
1511821249,11sleep10-21swapper/120:39:471
4716212321,44sleep10-21swapper/120:07:451
4716212321,44sleep10-21swapper/120:07:451
53121209,27sleep1621-21mailstats19:53:001
28051211913,57sleep1363-21snmpd21:19:201
26260211712,89sleep326409-21latency_hist22:57:333
2176821178,28sleep10-21swapper/120:58:491
9348211613,87sleep09493-21latency_hist23:47:330
2972421167,11sleep10-21swapper/121:25:541
8814211519,73sleep28816-21/usr/sbin/munin22:02:412
8814211519,73sleep28816-21/usr/sbin/munin22:02:412
1725421125,97sleep00-21swapper/000:11:470
1725421125,97sleep00-21swapper/000:11:470
17824211022,63sleep017925-21latency_hist19:07:320
2603221097,66sleep126112-21cat21:13:021
2603221097,66sleep126112-21cat21:13:021
2734021075,54sleep10-21swapper/119:35:111
1476210755,37sleep10-21swapper/121:38:011
2327421065,90sleep00-21swapper/021:03:160
19125210617,68sleep318637-21/usr/sbin/munin22:32:583
28422210545,14sleep10-21swapper/123:02:561
1280210512,75sleep21304-21proc_pri21:37:582
1280210512,75sleep21304-21proc_pri21:37:582
9155210440,44sleep10-21swapper/120:22:311
311210411,27sleep20-21swapper/223:17:472
2384021018,51sleep10-21swapper/100:32:431
2184321010,9sleep00-21swapper/022:42:470
2184321010,9sleep00-21swapper/022:42:470
6969210010,14sleep00-21swapper/021:57:300
4992210015,62sleep25085-21memory20:07:542
4992210015,62sleep25085-21memory20:07:542
840829825,47sleep10-21swapper/123:42:531
2176229828,57sleep121781-21apt-get22:42:411
2176229828,57sleep121781-21apt-get22:42:411
3248329611,71sleep032456-21/usr/sbin/munin19:52:460
1936129630,16sleep10-21swapper/120:52:561
1415729514,65sleep014169-21/usr/sbin/munin20:37:480
2856229313,58sleep328570-21/usr/sbin/munin21:22:413
1391129315,61sleep213958-21awk22:17:552
2984929221,55sleep029899-21unin-run21:27:300
2063029210,21sleep120664-21ntpq22:37:571
2063029210,21sleep120664-21ntpq22:37:571
2479529123,14sleep10-21swapper/119:27:481
1641829113,62sleep216446-21/usr/sbin/munin00:07:532
1641829113,62sleep216446-21/usr/sbin/munin00:07:532
1011329112,20sleep20-21swapper/223:47:592
362829012,62sleep03141-21/usr/sbin/munin20:03:000
362829012,62sleep03141-21/usr/sbin/munin20:03:000
248802907,43sleep10-21swapper/122:52:431
1865528818,55sleep318744-21switchtime19:08:003
694628717,53sleep27096-21sed20:13:032
3168728734,38sleep10-21swapper/121:32:441
733228612,56sleep07361-21df21:57:460
986028513,55sleep09873-21/usr/sbin/munin20:22:590
41182859,15sleep00-21swapper/021:47:360
2407328517,52sleep224117-21fw_packets21:07:502
2032928512,59sleep320384-21fschecks_count22:37:513
2032928512,59sleep320384-21fschecks_count22:37:513
1289628522,19sleep10-21swapper/120:32:571
1289628522,19sleep10-21swapper/120:32:571
2129628420,42sleep021315-21latency_hist19:17:290
2129628420,42sleep021315-21latency_hist19:17:290
1578728314,50sleep215285-21/usr/sbin/munin22:23:042
737228214,52sleep27377-21/usr/sbin/munin21:57:482
62742829,48sleep10-21swapper/123:37:321
320592827,12sleep00-21swapper/021:32:550
2809128214,53sleep0225-21systemd-journal21:19:510
757028135,32sleep10-21swapper/120:17:301
511728126,26sleep00-21swapper/020:07:560
511728126,26sleep00-21swapper/020:07:560
248628124,26sleep20-21swapper/221:42:502
323602800,5sleep30-21swapper/321:33:013
3119828021,22sleep20-21swapper/219:47:522
3119828021,22sleep20-21swapper/219:47:522
2602028018,16sleep00-21swapper/019:32:320
1782028019,33sleep30-21swapper/320:48:013
1782028019,33sleep30-21swapper/320:48:013
1686928025,25sleep10-21swapper/119:02:531
247432790,6sleep00-21swapper/000:33:010
16832797,17sleep10-21swapper/119:57:481
31827823,35sleep3351-21fschecks_count23:17:483
2246427820,37sleep222544-21fschecks_time21:02:502
2179127813,26sleep30-21swapper/322:42:473
2179127813,26sleep30-21swapper/322:42:473
1606427813,54sleep216065-21sshd20:43:522
1606427813,54sleep216065-21sshd20:43:522
7832779,11sleep00-21swapper/021:37:350
7832779,11sleep00-21swapper/021:37:350
2820627711,51sleep227808-21/usr/sbin/munin19:37:552
2820627711,51sleep227808-21/usr/sbin/munin19:37:552
1952327721,27sleep20-21swapper/220:53:012
2050527634,25sleep220570-21cat00:22:312
2846727520,43sleep228478-21ntpq23:02:572
2333627516,41sleep023387-21cpuspeed22:47:450
1364027510,50sleep113726-21fschecks_time22:17:501
45932749,14sleep30-21swapper/321:48:003
2799127416,40sleep30-21swapper/323:02:423
2275927420,40sleep322814-21processes00:27:573
1359527411,46sleep313719-21timerwakeupswit23:58:103
1924627316,27sleep30-21swapper/320:52:533
1237627320,36sleep012450-21apt-get23:57:300
655527212,44sleep06560-21/usr/sbin/munin23:37:480
32732726,12sleep00-21swapper/021:43:040
192992724,32sleep20-21swapper/219:11:582
1759127215,45sleep217650-21idleruntime20:47:562
1759127215,45sleep217650-21idleruntime20:47:562
496727125,29sleep00-21swapper/023:32:470
2848727115,23sleep00-21swapper/023:02:590
2672127116,25sleep30-21swapper/319:32:593
1934027117,27sleep00-21swapper/020:52:550
932827016,28sleep00-21swapper/022:02:590
932827016,28sleep00-21swapper/022:02:590
3130127016,23sleep00-21swapper/019:47:540
3130127016,23sleep00-21swapper/019:47:540
3050527021,33sleep330536-21/usr/sbin/munin21:27:533
2810227018,24sleep00-21swapper/019:37:530
2810227018,24sleep00-21swapper/019:37:530
2384327013,24sleep20-21swapper/222:47:592
1493427020,24sleep30-21swapper/322:22:323
1010027017,37sleep310138-21modprobe22:07:303
1010027017,37sleep310138-21modprobe22:07:303
1874826919,27sleep00-21swapper/019:08:010
1640126918,32sleep316408-21/usr/sbin/munin20:44:013
1640126918,32sleep316408-21/usr/sbin/munin20:44:013
281822685,50sleep20-21swapper/221:20:582
3168926721,24sleep30-21swapper/323:13:003
2281926713,39sleep00-21swapper/000:27:580
1736026722,23sleep30-21swapper/300:12:313
1736026722,23sleep30-21swapper/300:12:313
155382675,12sleep30-21swapper/300:04:403
155382675,12sleep30-21swapper/300:04:403
1478526719,32sleep0225-21systemd-journal22:22:290
2079226612,21sleep20-21swapper/220:57:492
966726513,33sleep20-21swapper/220:22:532
617426515,31sleep36212-21/usr/sbin/munin21:52:573
3064226518,32sleep030644-21cron19:47:290
3064226518,32sleep030644-21cron19:47:290
2781026519,24sleep20-21swapper/223:02:332
728826424,19sleep30-21swapper/321:57:453
526326418,27sleep30-21swapper/323:32:543
3178726414,26sleep00-21swapper/023:13:010
295322646,11sleep20-21swapper/223:07:372
2714526419,23sleep00-21swapper/022:58:030
2611326414,36sleep226237-21ls21:13:032
2611326414,36sleep226237-21ls21:13:032
1648826415,30sleep316483-21apt-get19:02:433
2083126316,31sleep320873-21df20:57:543
124622635,14sleep00-21swapper/020:32:360
124622635,14sleep00-21swapper/020:32:360
666826214,29sleep06713-21/usr/sbin/munin20:12:570
3274226219,22sleep20-21swapper/219:52:522
172526220,27sleep31808-21tune2fs19:57:503
1115226216,24sleep30-21swapper/320:27:503
49126116,25sleep2495-21cron21:37:292
460426117,25sleep30-21swapper/323:32:303
460426117,25sleep30-21swapper/323:32:303
2691726115,29sleep227038-21timerwakeupswit19:33:022
2675426111,26sleep00-21swapper/019:33:000
2359026113,34sleep223614-21ntp_states19:22:562
2359026113,34sleep223614-21ntp_states19:22:562
2074426118,26sleep320736-21apt-get00:22:443
198662616,42sleep00-21swapper/022:36:470
258492607,13sleep00-21swapper/021:12:570
258492607,13sleep00-21swapper/021:12:570
2079926015,31sleep220875-21cat22:38:022
2079926015,31sleep220875-21cat22:38:022
1868126016,29sleep018731-21unin-run20:52:300
1868126016,29sleep018731-21unin-run20:52:300
735325912,22sleep20-21swapper/223:38:022
646825916,26sleep36463-21apt-get23:37:423
437225913,20sleep20-21swapper/221:47:552
377325924,19sleep30-21swapper/320:03:033
377325924,19sleep30-21swapper/320:03:033
342925914,20sleep00-21swapper/023:27:500
342925914,20sleep00-21swapper/023:27:500
2916325919,21sleep20-21swapper/221:23:002
2823825919,25sleep30-21swapper/319:37:563
2823825919,25sleep30-21swapper/319:37:563
2582325916,24sleep30-21swapper/321:12:563
2582325916,24sleep30-21swapper/321:12:563
252532595,12sleep30-21swapper/319:27:583
1576125918,26sleep215790-21munin-run00:07:292
1576125918,26sleep215790-21munin-run00:07:292
3131925816,26sleep20-21swapper/223:12:502
2010725814,27sleep320111-21diskstats19:12:483
2010725814,27sleep320111-21diskstats19:12:483
179125816,25sleep31786-21apt-get23:22:453
920025712,25sleep20-21swapper/220:22:322
73292575,15sleep30-21swapper/320:14:543
638725721,19sleep20-21swapper/221:53:002
337025715,24sleep23417-21df23:27:482
337025715,24sleep23417-21df23:27:482
3248125718,24sleep332476-21apt-get19:52:453
3093125715,24sleep330924-21apt-get19:47:453
3093125715,24sleep330924-21apt-get19:47:453
292525714,20sleep00-21swapper/020:02:310
2206925716,25sleep322075-21ntp_states19:17:573
859225614,21sleep20-21swapper/223:42:592
2998725614,24sleep330083-21sed19:43:003
2998725614,24sleep330083-21sed19:43:003
2382125615,21sleep30-21swapper/319:23:023
2382125615,21sleep30-21swapper/319:23:023
2316125615,25sleep023211-21/usr/sbin/munin19:22:460
2316125615,25sleep023211-21/usr/sbin/munin19:22:460
170742569,18sleep30-21swapper/322:27:413
157925616,20sleep20-21swapper/219:57:412
1546225614,26sleep315638-21date20:42:323
1316725618,20sleep30-21swapper/320:33:033
1316725618,20sleep30-21swapper/320:33:033
1175725613,18sleep00-21swapper/022:12:320
116042564,11sleep20-21swapper/222:11:532
116042564,11sleep20-21swapper/222:11:532
867725513,18sleep30-21swapper/323:43:003
3013625514,26sleep030132-21apt-get21:27:420
3006725521,19sleep30-21swapper/323:07:593
243625515,21sleep30-21swapper/321:42:473
2234625520,15sleep00-21swapper/019:18:020
1293125513,25sleep212953-21ntp_states20:32:582
1293125513,25sleep212953-21ntp_states20:32:582
1039025515,24sleep010385-21apt-get22:07:430
1039025515,24sleep010385-21apt-get22:07:430
310525417,19sleep20-21swapper/223:27:322
2241025414,23sleep322420-21df_inode21:02:473
1713225416,22sleep217094-21apt-get22:28:042
1111525417,24sleep311127-21grep23:52:453
483925310,17sleep30-21swapper/320:07:513
483925310,17sleep30-21swapper/320:07:513
3041125315,23sleep230504-21memory21:27:522
2982425319,18sleep00-21swapper/023:07:520
2706325318,22sleep327143-21awk22:58:013
2384125312,24sleep223835-21apt-get00:32:442
2333325313,24sleep323328-21apt-get22:47:443
2203625313,21sleep222068-21ntp_states19:17:562
2111225312,28sleep021130-21netstat00:22:530
1841325314,25sleep018423-21munin-run22:32:310
784025213,22sleep07843-21apt-get20:17:420
2390525213,24sleep323901-21apt-get21:07:433
2121625211,15sleep00-21swapper/020:58:030
1805125215,21sleep318062-21ntp_states00:12:553
2258925115,21sleep222658-21memory00:27:522
1757325114,22sleep017592-21apt-get00:12:400
1558925119,17sleep00-21swapper/000:05:170
1558925119,17sleep00-21swapper/000:05:170
1031125119,17sleep00-21swapper/023:48:010
1018225114,18sleep30-21swapper/323:48:003
324125014,20sleep23248-21diskstats20:02:502
324125014,20sleep23248-21diskstats20:02:502
2712925010,16sleep30-21swapper/321:17:483
1946525016,16sleep30-21swapper/300:17:523
1169825012,15sleep30-21swapper/322:12:313
1169825012,15sleep30-21swapper/322:12:313
3069324914,16sleep20-21swapper/219:47:312
3069324914,16sleep20-21swapper/219:47:312
2309124915,13sleep20-21swapper/222:47:312
2309124915,13sleep20-21swapper/222:47:312
1874724911,22sleep218772-21grep22:32:492
1767124911,23sleep217678-21df_inode00:12:462
1456824913,20sleep20-21swapper/220:37:592
110232499,16sleep00-21swapper/020:27:470
949124815,18sleep30-21swapper/320:22:493
238162467,15sleep30-21swapper/300:32:413
793324510,18sleep30-21swapper/320:17:493
8132429,17sleep30-21swapper/321:37:423
8132429,17sleep30-21swapper/321:37:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional