You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 11:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun May 17, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287762142414,1392sleep028751-21/usr/sbin/munin19:54:190
287762142414,1392sleep028751-21/usr/sbin/munin19:54:190
1984629985,983sleep210088-1kworker/2:2H19:24:572
12363265419,617sleep312416-21/usr/sbin/munin20:44:383
3187224376,396sleep131907-21unin-run20:04:181
885023837,340sleep18895-21ntpq22:14:421
18192231626,275sleep318204-21cut19:19:533
18192231626,275sleep318204-21cut19:19:533
157622281116,55sleep115868-21cat19:14:221
157622281116,55sleep115868-21cat19:14:221
11053226153,90sleep111091-21latency00:04:381
8536224819,213sleep38589-21fschecks_count22:14:363
27892224520,28sleep10-21swapper/121:34:201
73652242105,85sleep10-21swapper/120:29:341
2095822378,190sleep11-21systemd19:29:201
19604222213,194sleep019642-21cat19:24:230
19242222058,62sleep10-21swapper/119:24:191
19242222058,62sleep10-21swapper/119:24:191
2971722139,104sleep129864-21awk23:19:481
2971722139,104sleep129864-21awk23:19:481
1898221350,60sleep10-21swapper/123:34:501
19725220310,68sleep119865-21latency_hist21:09:201
22774220159,79sleep10-21swapper/121:19:171
13075218815,154sleep213205-21grep00:09:482
920021858,165sleep19210-21find23:59:251
920021858,165sleep19210-21find23:59:251
22633218445,56sleep10-21swapper/119:34:211
22633218445,56sleep10-21swapper/119:34:211
817921828,161sleep18212-21munin-run22:14:181
12241218251,117sleep112272-21fschecks_count20:44:361
28909217923,133sleep029039-21grep21:34:590
28909217923,133sleep029039-21grep21:34:590
5162217841,92sleep10-21swapper/122:04:201
1757421779,155sleep117631-21cat22:44:181
22416217417,134sleep122455-21awk22:59:161
14404217214,88sleep114426-21modprobe22:34:171
14612216512,96sleep114627-21awk20:49:491
14612216512,96sleep114627-21awk20:49:491
6766216312,131sleep16896-21grep23:49:491
376121638,14sleep00-21swapper/021:59:270
915216212,134sleep1972-21apt-key23:34:181
14497215611,131sleep114581-21timerandwakeup00:14:481
24933215594,14sleep10-21swapper/119:39:541
1066215428,111sleep11183-21sort21:49:461
7720215311,108sleep17770-21fschecks_count23:54:371
30274215313,119sleep130311-21awk19:59:171
30274215313,119sleep130311-21awk19:59:171
4241215225,107sleep24263-21grep23:44:162
20919215127,61sleep10-21swapper/122:54:201
11371215118,53sleep10-21swapper/122:24:191
23478215012,78sleep10-21swapper/119:34:541
23478215012,78sleep10-21swapper/119:34:541
10870214622,103sleep210928-21fschecks_count00:04:352
21289214512,108sleep121431-21latency_hist21:14:201
4112214419,13sleep10-21swapper/123:42:381
4401214116,19sleep10-21swapper/120:19:441
4401214116,19sleep10-21swapper/120:19:441
27396213919,105sleep027452-21cpuspeed23:14:380
5440213714,17sleep10-21swapper/122:04:361
8214213612,110sleep38320-21latency_hist22:14:203
2012321368,17sleep10-21swapper/100:34:311
18159213436,54sleep112-21rcu_sched21:04:191
12419212816,93sleep012435-21kernelversion20:44:390
1118521289,58sleep111273-21switchtime20:39:481
2874212715,95sleep00-21swapper/020:14:340
2874212715,95sleep00-21swapper/020:14:340
27683212623,86sleep227712-21idleruntime-cro21:34:162
19056212649,40sleep119115-21find00:29:471
19056212649,40sleep119115-21find00:29:471
3301212517,59sleep10-21swapper/120:14:521
3301212517,59sleep10-21swapper/120:14:521
2641521258,61sleep126482-21switchtime23:09:471
12950212415,91sleep10-21swapper/100:09:471
27387212324,62sleep127384-21apt-get23:14:311
20303212321,87sleep320457-21sed22:49:503
24050212215,93sleep224089-21latency_hist23:04:182
9129212111,68sleep19171-21fschecks_count20:34:431
9129212111,68sleep19171-21fschecks_count20:34:431
23451212110,97sleep123598-21unixbench_singl22:59:491
29438212010,40sleep10-21swapper/121:38:521
29438212010,40sleep10-21swapper/121:38:521
372121180,10sleep029213-21kworker/0:221:59:210
12891211731,57sleep212897-21fschecks_count19:04:392
23361211619,81sleep323439-21sed21:19:403
177621147,73sleep11851-21sensors20:09:471
27177211314,84sleep027228-21unin-run19:49:180
4510211132,20sleep10-21swapper/123:44:271
973211019,18sleep20-21swapper/223:34:192
2987721109,58sleep10-21swapper/121:39:371
2987721109,58sleep10-21swapper/121:39:371
21887211024,67sleep322023-21grep22:54:503
4196210914,81sleep24237-21ntpq21:59:452
13246210838,50sleep313308-21find19:04:483
2628621079,60sleep10-21swapper/119:44:431
2582221035,47sleep10-21swapper/121:28:181
2582221035,47sleep10-21swapper/121:28:181
21602210321,59sleep321690-21grep21:14:393
1754221035,46sleep10-21swapper/119:18:421
1686210313,71sleep31780-21memory23:34:443
18025210228,61sleep118038-21basename22:44:261
18025210228,61sleep118038-21basename22:44:261
16246210254,22sleep00-21swapper/022:39:290
2860221015,67sleep10-21swapper/119:52:431
2860221015,67sleep10-21swapper/119:52:431
5589210017,66sleep35623-21irqstats22:04:393
13751210015,70sleep013864-21timerandwakeup22:29:470
1893729814,12sleep20-21swapper/221:04:482
3263829624,60sleep11-21systemd21:49:181
1690629512,70sleep317028-21cat22:39:483
1071829422,51sleep310876-21sed22:19:493
1702429315,21sleep20-21swapper/220:59:392
79532907,25sleep10-21swapper/120:29:501
879528910,64sleep38856-21latency_hist20:34:203
1659128929,22sleep116599-21systemd-run20:59:181
90528825,13sleep10-21swapper/120:08:381
90528825,13sleep10-21swapper/120:08:381
1729928813,26sleep10-21swapper/100:24:421
1729928813,26sleep10-21swapper/100:24:421
2913928712,18sleep00-21swapper/019:54:390
2913928712,18sleep00-21swapper/019:54:390
1833828725,39sleep218314-21latency_hist00:29:192
1833828725,39sleep218314-21latency_hist00:29:192
816228616,53sleep08175-21idleruntime-cro22:14:170
2818328621,48sleep227699-21apt-get21:34:292
2818328621,48sleep227699-21apt-get21:34:292
2788028624,35sleep227364-21/usr/sbin/munin23:14:522
192828616,55sleep32068-21sed20:09:493
321882850,9sleep20-21swapper/221:44:502
3106828516,53sleep031132-21awk19:59:520
3099028416,15sleep10-21swapper/123:24:451
954728314,53sleep39643-21mailstats20:34:533
954728314,53sleep39643-21mailstats20:34:533
1469428317,27sleep00-21swapper/022:34:320
1073128315,52sleep310816-21fschecks_time20:39:373
289732828,15sleep30-21swapper/323:19:303
289732828,15sleep30-21swapper/323:19:303
243702828,56sleep10-21swapper/123:04:341
3109828116,49sleep231163-21latency_hist21:44:182
3109828116,49sleep231163-21latency_hist21:44:182
307228116,43sleep33130-21fschecks_count23:39:373
2650228113,55sleep326527-21cpu21:30:303
2008928116,16sleep30-21swapper/321:09:383
1603028122,39sleep215980-21latency_hist22:39:182
1166528114,51sleep311671-21diskstats22:24:373
93482809,51sleep30-21swapper/323:59:363
93482809,51sleep30-21swapper/323:59:363
584828013,43sleep05917-21latency_hist23:49:190
3086628013,52sleep230955-21/usr/sbin/munin19:59:422
2435528048,21sleep1284-21dbus-daemon21:24:181
1608228022,34sleep30-21swapper/300:19:473
1561928019,41sleep215641-21ls19:14:192
1561928019,41sleep215641-21ls19:14:192
1502528018,42sleep315006-21latency_hist20:54:193
1502528018,42sleep315006-21latency_hist20:54:193
355127917,47sleep23257-21apt-get21:59:202
2471827916,47sleep324725-21/usr/sbin/munin21:24:363
2471827916,47sleep324725-21/usr/sbin/munin21:24:363
2435127914,47sleep0662-21cron21:24:170
102512795,11sleep00-21swapper/000:01:100
102512795,11sleep00-21swapper/000:01:100
815227812,50sleep28153-21sshd22:14:112
2590027623,30sleep00-21swapper/021:29:180
2590027623,30sleep00-21swapper/021:29:180
2147927621,21sleep20-21swapper/221:14:302
1290127620,38sleep013028-21wc22:29:190
1041127615,20sleep20-21swapper/220:39:202
1041127615,20sleep20-21swapper/220:39:202
8602759,50sleep0884-21ntp_states21:49:410
323592757,12sleep00-21swapper/021:46:140
3183727518,27sleep30-21swapper/323:29:193
2429327516,29sleep30-21swapper/323:04:323
1983627519,38sleep31-21systemd00:34:173
1983627519,38sleep31-21systemd00:34:173
1763227522,25sleep20-21swapper/222:44:192
2584027419,28sleep30-21swapper/323:09:323
24902745,11sleep10-21swapper/121:54:471
1607427415,27sleep30-21swapper/322:39:193
979927325,20sleep20-21swapper/222:19:192
889127316,40sleep08936-21systemctl23:59:160
326182739,49sleep332625-21open_files20:04:523
326182739,49sleep332625-21open_files20:04:523
307092739,52sleep00-21swapper/021:39:560
307092739,52sleep00-21swapper/021:39:560
1837127319,40sleep318376-21sort21:04:293
429227220,30sleep00-21swapper/023:44:180
220272726,14sleep00-21swapper/022:54:510
1157027213,41sleep311683-21/usr/sbin/munin00:04:493
1150627214,26sleep00-21swapper/020:39:510
630827113,25sleep00-21swapper/020:24:440
2328027114,25sleep30-21swapper/322:59:473
45162700,7sleep20-21swapper/223:44:272
2958227023,27sleep30-21swapper/319:54:553
2958227023,27sleep30-21swapper/319:54:553
2733027014,38sleep027422-21users21:31:030
206527016,34sleep32123-21fschecks_count21:54:373
739326922,21sleep30-21swapper/323:54:193
3178326919,26sleep00-21swapper/023:29:160
123726916,24sleep00-21swapper/020:09:340
123726916,24sleep00-21swapper/020:09:340
263392689,14sleep00-21swapper/023:09:460
1717626813,38sleep317240-21/usr/sbin/munin00:24:403
1717626813,38sleep317240-21/usr/sbin/munin00:24:403
1595526821,30sleep10-21swapper/122:39:171
1547826811,18sleep00-21swapper/000:19:320
1526726816,26sleep30-21swapper/320:54:363
1427226819,26sleep20-21swapper/220:49:462
1427226819,26sleep20-21swapper/220:49:462
897726718,30sleep28960-40mandb23:59:182
283872679,12sleep30-21swapper/319:49:583
283872679,12sleep30-21swapper/319:49:583
2553526710,27sleep20-21swapper/221:24:522
2553526710,27sleep20-21swapper/221:24:522
2164426722,22sleep20-21swapper/219:29:582
2164426722,22sleep20-21swapper/219:29:582
1568226720,22sleep00-21swapper/019:14:210
1568226720,22sleep00-21swapper/019:14:210
142362678,47sleep114286-21latency_hist19:09:211
2298526611,28sleep00-21swapper/019:34:400
2298526611,28sleep00-21swapper/019:34:400
1935426621,19sleep20-21swapper/219:24:202
1935426621,19sleep20-21swapper/219:24:202
1564926610,38sleep10-21swapper/100:19:371
79426518,23sleep20-21swapper/221:49:392
784326517,30sleep37901-21/usr/sbin/munin23:54:393
337526519,23sleep20-21swapper/220:14:532
337526519,23sleep20-21swapper/220:14:532
2426526512,40sleep224278-21acpi23:04:262
1419126517,23sleep00-21swapper/019:09:200
1261926514,28sleep20-21swapper/220:44:452
2291426413,34sleep322978-21fschecks_time19:34:393
2291426413,34sleep322978-21fschecks_time19:34:393
2021526413,32sleep220302-21sort22:49:492
1804326422,22sleep00-21swapper/019:19:500
1445626413,32sleep314539-21latency_hist22:34:193
931226322,27sleep00-21swapper/020:34:460
931226322,27sleep00-21swapper/020:34:460
653126321,22sleep20-21swapper/220:24:502
562526322,26sleep00-21swapper/022:04:400
2657926314,34sleep226681-21sed19:44:522
255126320,25sleep00-21swapper/020:14:180
2058626313,34sleep320601-21/usr/sbin/munin00:34:453
1368226317,25sleep013701-21apt-get20:49:280
1368226317,25sleep013701-21apt-get20:49:280
2951126214,33sleep229580-21mailstats19:54:542
2951126214,33sleep229580-21mailstats19:54:542
2473826211,21sleep30-21swapper/319:39:443
128012625,26sleep10-21swapper/122:28:431
88026116,30sleep0906-21munin-run23:34:170
487526121,18sleep20-21swapper/220:19:582
487526121,18sleep20-21swapper/220:19:582
3032426117,26sleep330395-21sensors_temp21:39:473
3032426117,26sleep330395-21sensors_temp21:39:473
2915326121,18sleep20-21swapper/223:19:372
2915326121,18sleep20-21swapper/223:19:372
1678726117,21sleep20-21swapper/200:24:182
1535126114,32sleep215362-21awk20:54:392
799326010,31sleep28072-21systemctl20:29:512
55232606,42sleep00-21swapper/020:22:420
55232606,42sleep00-21swapper/020:22:420
344526020,22sleep30-21swapper/320:14:543
344526020,22sleep30-21swapper/320:14:543
2177026014,26sleep20-21swapper/222:54:492
2061526019,17sleep20-21swapper/221:09:492
1967926019,17sleep00-21swapper/022:49:340
1546926014,25sleep00-21swapper/020:54:410
1397026019,22sleep20-21swapper/200:14:372
3252525913,21sleep20-21swapper/220:04:472
3252525913,21sleep20-21swapper/220:04:472
2501425914,25sleep20-21swapper/219:39:552
213332598,38sleep00-21swapper/000:38:380
1987125911,26sleep00-21swapper/019:25:030
1780125817,25sleep317960-21apt-key22:44:203
1780125817,25sleep317960-21apt-key22:44:203
2897525714,25sleep028969-21apt-get23:19:310
2897525714,25sleep028969-21apt-get23:19:310
2650925720,21sleep20-21swapper/223:09:482
1134025721,20sleep00-21swapper/022:24:180
736225615,26sleep37358-21apt-get20:29:323
282192569,26sleep30-21swapper/321:34:383
282192569,26sleep30-21swapper/321:34:383
1904025619,20sleep00-21swapper/000:29:450
1904025619,20sleep00-21swapper/000:29:450
1803925618,25sleep217546-21apt-get22:44:272
1803925618,25sleep217546-21apt-get22:44:272
155262565,14sleep30-21swapper/319:13:113
155262565,14sleep30-21swapper/319:13:113
874425514,23sleep08752-21modprobe20:34:170
263725512,29sleep20-21swapper/220:14:202
2617625515,27sleep326189-21irqstats19:44:413
184902559,25sleep018578-21grep21:04:380
1741125516,24sleep317483-21sed20:59:493
167125515,23sleep21685-21irqstats23:34:432
1410625515,22sleep014161-21irqstats00:14:390
1232925516,23sleep312325-21apt-get00:09:293
472925413,18sleep30-21swapper/323:44:383
3141325414,23sleep331421-21diskstats21:44:353
2469225418,18sleep00-21swapper/023:04:410
2334325417,20sleep00-21swapper/022:59:480
2168725416,23sleep321713-21ntp_states19:29:593
2168725416,23sleep321713-21ntp_states19:29:593
215762548,16sleep00-21swapper/021:14:360
2104025419,15sleep30-21swapper/319:29:223
1828725414,23sleep018318-21munin-run00:29:170
1828725414,23sleep018318-21munin-run00:29:170
118232544,38sleep20-21swapper/220:43:442
583425310,20sleep20-21swapper/222:04:462
55362537,35sleep30-21swapper/320:22:533
55362537,35sleep30-21swapper/320:22:533
2746725314,23sleep227461-21apt-get19:49:332
2746725314,23sleep227461-21apt-get19:49:332
2440125318,18sleep20-21swapper/221:24:192
63325214,17sleep30-21swapper/321:49:363
1473225216,18sleep30-21swapper/300:14:503
738325113,23sleep27392-21/usr/sbin/munin23:54:182
3067125112,23sleep330675-21diskstats19:59:353
139725115,19sleep00-21swapper/023:34:290
845325010,15sleep00-21swapper/022:14:330
3215325013,23sleep332160-21diskstats23:29:353
3182525015,22sleep21-21systemd23:29:182
179652508,16sleep00-21swapper/022:44:210
179652508,16sleep00-21swapper/022:44:210
1333625010,28sleep0363-21snmpd00:11:000
2782424913,20sleep327848-21ntp_states23:14:493
2133224914,21sleep021341-21diskstats19:29:480
2133224914,21sleep021341-21diskstats19:29:480
1748524911,18sleep00-21swapper/020:59:500
1638524912,22sleep316405-21diskstats19:14:363
989624810,23sleep20-21swapper/223:59:472
989624810,23sleep20-21swapper/223:59:472
2360724811,20sleep223698-21users19:34:552
2360724811,20sleep223698-21users19:34:552
2057924811,22sleep220585-21ntp_states00:34:442
1899624810,21sleep219031-21ntp_states00:29:432
1899624810,21sleep219031-21ntp_states00:29:432
1003624810,20sleep20-21swapper/222:19:322
2486224714,18sleep00-21swapper/019:39:530
1700824714,18sleep20-21swapper/219:15:002
1320324711,22sleep213214-21df_inode22:29:332
42392457,14sleep30-21swapper/321:59:463
2105824514,16sleep00-21swapper/021:12:170
1683624513,17sleep00-21swapper/019:14:580
128482456,15sleep20-21swapper/222:29:172
1128324512,16sleep00-21swapper/000:04:460
616724412,16sleep00-21swapper/023:49:350
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional