You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 17:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Dec 08, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
148442104810,1027sleep221461-1kworker/2:1H07:13:142
8622282416,789sleep18696-21/usr/sbin/munin08:33:151
25078271018,675sleep225085-21/usr/sbin/munin09:28:022
29836266018,628sleep229837-21sshd11:22:522
5600257820,543sleep25645-21/usr/sbin/munin11:48:112
913625256,508sleep312173-1kworker/3:1H10:18:173
2539025156,496sleep025426-21ntpq09:28:080
4580251252,448sleep04618-21ntpq10:03:080
11131246870,385sleep011175-21fschecks_count07:03:020
11131246870,385sleep011175-21fschecks_count07:03:020
400824070,12sleep23620-21/usr/sbin/munin11:43:052
1564231510,288sleep00-21swapper/009:53:230
2239123018,61sleep00-21swapper/007:35:400
790622827,264sleep05666-21snmpd11:55:560
1746922560,21sleep010838-21cyclictest10:43:120
15350223716,206sleep315395-21/usr/sbin/munin08:57:433
1695222270,7sleep20-21swapper/207:18:262
8001222654,153sleep08156-21unixbench_multi10:13:140
15462222650,114sleep00-21swapper/012:19:010
19479221116,96sleep00-21swapper/007:27:450
2343220420,167sleep22390-21apt-key09:57:452
24752220115,171sleep124799-21latency_hist09:27:451
18743220019,127sleep00-21swapper/010:48:060
2867321988,178sleep028745-21expr09:38:130
2867321988,178sleep028745-21expr09:38:130
5593219614,170sleep0369-21sshd08:23:290
5593219614,170sleep0369-21sshd08:23:290
2304821956,172sleep00-21swapper/007:37:500
16663219214,159sleep30-21swapper/310:42:443
16033217417,130sleep00-21swapper/010:38:170
30631216518,94sleep00-21swapper/008:02:450
3940216320,96sleep00-21swapper/010:02:440
2460216213,135sleep32505-21awk11:38:073
29880215852,39sleep029900-21apt-get09:42:540
2463521559,19sleep00-21swapper/009:26:510
9590215314,36sleep09613-21idleruntime-cro12:02:420
22974215228,65sleep05666-21snmpd09:20:250
1534215210,83sleep00-21swapper/008:12:550
1534215210,83sleep00-21swapper/008:12:550
29081215141,24sleep00-21swapper/007:57:450
9050214719,35sleep09053-21cron08:37:420
8393214011,114sleep38471-21find08:33:133
1663321398,121sleep05666-21snmpd08:59:130
14576213987,26sleep00-21swapper/010:33:160
6861213825,87sleep16953-21tune2fs11:53:021
30459213817,105sleep330543-21find11:23:133
17512213884,15sleep00-21swapper/007:18:370
14173213813,69sleep00-21swapper/008:53:040
2463621366,20sleep00-21swapper/007:42:590
10653213610,111sleep210719-21cat08:42:442
518213398,18sleep00-21swapper/011:32:550
546421329,109sleep35174-21/usr/sbin/munin11:48:083
3615213289,23sleep00-21swapper/011:42:520
20329213270,49sleep020443-21timerandwakeup07:28:140
2361213013,104sleep12420-21latency11:38:051
9148212923,23sleep00-21swapper/010:18:230
15109212624,84sleep30-21swapper/310:37:433
32092212422,79sleep332226-21grep11:28:143
14309212158,48sleep014357-21apt-config07:12:430
10136212138,62sleep310293-21unixbench_multi06:58:133
2657021207,15sleep00-21swapper/009:32:560
26320212013,92sleep326330-21pluginstate09:32:433
20893212053,49sleep00-21swapper/009:13:140
330211921,80sleep1388-21gpgv09:52:441
30547211923,66sleep00-21swapper/011:23:140
9431211820,38sleep09449-21apt06:57:540
2178211813,86sleep22238-21/usr/sbin/munin11:38:022
6213211520,77sleep16186-21/usr/sbin/munin08:27:581
6213211520,77sleep16186-21/usr/sbin/munin08:27:581
5285211559,44sleep05327-21fschecks_count11:48:050
20447211423,74sleep120593-21unixbench_singl07:28:151
4736211312,85sleep14797-21switchtime10:03:141
10783211343,32sleep20-21swapper/207:02:442
1421921125,13sleep00-21swapper/012:16:470
9866211117,72sleep210012-21unixbench_singl10:18:412
24016211120,76sleep124150-21timerwakeupswit11:03:131
22806211119,72sleep322982-21latency_hist07:37:453
1833721116,16sleep00-21swapper/009:06:040
13761211114,76sleep313818-21tr08:52:423
1125121115,93sleep00-21swapper/010:23:150
672621095,11sleep00-21swapper/010:09:140
26650210719,72sleep126679-21proc_pri07:48:131
20437210545,42sleep00-21swapper/010:53:110
14291210514,77sleep114372-21mailstats10:33:131
9424210418,66sleep00-21swapper/008:38:020
16963210317,69sleep317019-21latency_hist09:02:443
26400210250,35sleep00-21swapper/011:12:450
6185210043,40sleep00-21swapper/008:27:530
6185210043,40sleep00-21swapper/008:27:530
19453210010,77sleep019601-21latency_hist12:32:460
310929940,15sleep00-21swapper/008:17:550
310929940,15sleep00-21swapper/008:17:550
2676329944,15sleep00-21swapper/011:12:570
2676329944,15sleep00-21swapper/011:12:570
230812999,17sleep20-21swapper/207:37:572
134712995,39sleep00-21swapper/010:32:320
2703429815,69sleep327058-21proc_pri09:33:123
3086529741,15sleep00-21swapper/008:02:540
3086529741,15sleep00-21swapper/008:02:540
1484329713,66sleep114811-21apt-get07:13:131
3240729642,13sleep00-21swapper/008:07:540
109052968,15sleep00-21swapper/008:42:580
2902329519,62sleep329035-21vmstat09:38:163
2902329519,62sleep329035-21vmstat09:38:163
1994729550,13sleep00-21swapper/009:11:450
1994729550,13sleep00-21swapper/009:11:450
102472956,12sleep30-21swapper/312:03:053
163712948,17sleep00-21swapper/012:22:570
126622945,78sleep00-21swapper/012:12:020
218632938,15sleep00-21swapper/010:58:190
218632938,15sleep00-21swapper/010:58:190
1260429311,57sleep00-21swapper/008:48:030
2776429217,20sleep20-21swapper/207:52:582
2164229247,23sleep30-21swapper/310:57:463
2164229247,23sleep30-21swapper/310:57:463
91329118,56sleep1952-21ntpq11:33:081
326252915,75sleep00-21swapper/011:32:020
219529121,50sleep12342-21apt-key09:57:441
1819429111,16sleep10-21swapper/110:47:431
1630829120,54sleep116431-21timerwakeupswit08:58:141
148732919,14sleep00-21swapper/007:13:360
2715229045,24sleep30-21swapper/311:13:243
2715229045,24sleep30-21swapper/311:13:243
1300929011,56sleep10-21swapper/112:12:561
3250828920,52sleep332539-21tune2fs08:08:013
3000728914,52sleep230164-21unixbench_multi07:58:152
108142895,11sleep20-21swapper/210:23:052
3216428813,48sleep00-21swapper/009:48:150
2991228826,44sleep124445-21kworker/u8:309:42:591
2355828815,61sleep123565-21ntpq07:38:241
294272875,62sleep00-21swapper/011:20:480
1290528713,56sleep112949-21awk08:48:081
2043728617,44sleep120547-21cut09:13:031
975728522,42sleep19865-21wc10:18:401
100902856,13sleep00-21swapper/012:03:020
507128416,47sleep25120-21/usr/sbin/munin08:23:102
507128416,47sleep25120-21/usr/sbin/munin08:23:102
21652846,14sleep00-21swapper/011:38:000
39128321,43sleep2614-21latency_hist09:52:452
1084828314,53sleep311005-21modprobe07:02:453
1084828314,53sleep311005-21modprobe07:02:453
233852827,59sleep00-21swapper/011:02:560
1573828217,45sleep215839-21fschecks_time08:58:022
427828120,44sleep34383-21fschecks_time10:03:033
2684028113,46sleep00-21swapper/007:48:160
2665828119,39sleep226712-21fschecks_count09:33:032
752728015,42sleep27295-21/usr/sbin/munin10:13:042
2714628011,48sleep127225-21sort09:33:141
2481928022,27sleep10-21swapper/107:43:041
1426427922,36sleep314293-21munin-run07:12:423
357227822,42sleep13609-21ps08:18:091
357227822,42sleep13609-21ps08:18:091
3192127811,52sleep132008-21mailstats11:28:121
1981927822,31sleep20-21swapper/210:52:442
402327720,36sleep30-21swapper/311:43:073
621327621,22sleep10-21swapper/110:08:141
1637327619,34sleep20-21swapper/212:22:592
2135427522,33sleep121446-21apt-config10:57:441
1271127517,41sleep312737-21munin-run12:12:423
2305127416,23sleep30-21swapper/307:37:533
593327325,23sleep10-21swapper/111:48:181
3157927318,30sleep231738-21grep08:03:122
3157927318,30sleep231738-21grep08:03:122
2541327320,36sleep325475-21sendmail_mailqu11:08:123
1661727318,39sleep316653-21grep07:17:593
145227314,44sleep31559-21grep09:53:223
888727218,42sleep18958-21switchtime11:58:151
3055727215,37sleep330685-21ls09:43:293
2529827221,24sleep30-21swapper/307:43:143
1234727236,23sleep11-21systemd07:07:431
1234727236,23sleep11-21systemd07:07:431
754927118,31sleep37673-21cat08:32:443
754927118,31sleep37673-21cat08:32:443
596327120,28sleep30-21swapper/308:27:433
596327120,28sleep30-21swapper/308:27:433
481827121,19sleep20-21swapper/210:03:152
1071727125,29sleep110780-21systemctl07:02:431
708227022,23sleep30-21swapper/310:12:443
2839127016,25sleep10-21swapper/111:18:031
964126914,40sleep29718-21cut12:02:442
2502526917,35sleep225105-21tune2fs11:08:022
2179226910,13sleep30-21swapper/309:17:453
1862426922,24sleep10-21swapper/107:23:101
1534226913,40sleep115348-21ntp_states12:18:581
1376326916,29sleep20-21swapper/207:08:132
1090626917,35sleep110914-21wc08:42:591
286926821,21sleep10-21swapper/108:17:441
286926821,21sleep10-21swapper/108:17:441
2319226814,39sleep123212-21ls09:22:441
1139326816,35sleep31-21systemd08:43:143
3275526717,22sleep20-21swapper/211:32:452
3004826717,23sleep20-21swapper/209:43:162
1937226724,21sleep20-21swapper/209:08:162
1937226724,21sleep20-21swapper/209:08:162
16452679,13sleep30-21swapper/308:13:023
16452679,13sleep30-21swapper/308:13:023
1581926718,35sleep115856-21ps10:38:141
1480926716,19sleep20-21swapper/212:17:542
1125526715,23sleep20-21swapper/207:03:042
1125526715,23sleep20-21swapper/207:03:042
2637126614,35sleep226434-21/usr/sbin/munin07:48:062
1986226617,29sleep219691-21/usr/sbin/munin07:28:032
1899826616,26sleep20-21swapper/207:23:152
1435826618,30sleep114513-21latency_hist07:12:441
131426616,23sleep10-21swapper/108:12:441
1279126617,35sleep112731-21latency_hist12:12:441
698526516,31sleep27115-21/usr/sbin/munin08:28:152
698526516,31sleep27115-21/usr/sbin/munin08:28:152
57326516,27sleep20-21swapper/208:08:132
330326515,20sleep10-21swapper/109:58:331
1935226519,23sleep30-21swapper/312:32:453
1608226519,27sleep316099-21apt-get07:17:433
1227526521,22sleep20-21swapper/208:47:452
983426417,30sleep39920-21mailstats08:38:133
2165226423,18sleep20-21swapper/209:17:442
1766726423,19sleep10-21swapper/109:03:101
1763126421,29sleep11-21systemd12:27:421
1651926413,21sleep30-21swapper/312:23:043
1408726423,21sleep10-21swapper/108:53:011
115232647,41sleep211526-21/usr/sbin/munin12:08:012
391826315,24sleep10-21swapper/111:43:051
2903326313,13sleep30-21swapper/311:18:213
2618526316,29sleep326181-21apt-get07:47:593
2130126313,21sleep30-21swapper/310:57:423
833526223,21sleep30-21swapper/311:58:013
327192628,12sleep30-21swapper/309:52:423
3147626215,27sleep10-21swapper/108:03:111
3147626215,27sleep10-21swapper/108:03:111
2959426219,21sleep10-21swapper/107:58:061
135726223,18sleep20-21swapper/208:12:452
135726223,18sleep20-21swapper/208:12:452
1116926218,22sleep30-21swapper/310:23:143
861526121,24sleep18650-21munin-run10:17:431
2969326120,17sleep30-21swapper/307:58:083
2200226118,20sleep10-21swapper/110:58:371
2200226118,20sleep10-21swapper/110:58:371
56112608,11sleep10-21swapper/108:23:421
56112608,11sleep10-21swapper/108:23:421
436726012,32sleep24392-21munin-run08:22:422
436726012,32sleep24392-21munin-run08:22:422
2874626019,17sleep10-21swapper/109:38:141
2874626019,17sleep10-21swapper/109:38:141
1922926014,25sleep30-21swapper/310:48:173
1691226017,31sleep116951-21latency07:18:251
800325916,19sleep20-21swapper/208:33:032
41242596,45sleep30-21swapper/308:19:323
41242596,45sleep30-21swapper/308:19:323
2407025915,24sleep30-21swapper/309:23:143
1538125912,22sleep30-21swapper/312:19:003
1309425917,24sleep213183-21threads08:48:132
75425816,23sleep2811-21irqstats11:33:052
69932585,12sleep30-21swapper/311:53:043
2998925819,21sleep10-21swapper/111:23:031
2258025818,22sleep10-21swapper/107:37:431
218652588,19sleep20-21swapper/210:58:212
218652588,19sleep20-21swapper/210:58:212
2012325821,22sleep20-21swapper/210:53:042
1821425817,22sleep20-21swapper/210:47:442
1448825813,28sleep20-21swapper/208:53:102
1349725817,26sleep213519-21/usr/sbin/munin12:13:142
1324225813,21sleep30-21swapper/312:13:063
2821825711,15sleep20-21swapper/209:38:032
2821825711,15sleep20-21swapper/209:38:032
2803325721,19sleep10-21swapper/107:53:051
2347325718,22sleep223568-21perl11:03:022
1851325718,25sleep30-21swapper/309:07:443
1830025715,22sleep20-21swapper/212:28:072
259925617,25sleep20-21swapper/209:57:542
2012425616,23sleep320169-21ntp_states07:28:083
1816225616,24sleep318158-21apt-get07:22:563
1290925619,18sleep20-21swapper/210:28:182
110902567,11sleep20-21swapper/212:06:562
2856525519,21sleep228591-21irqstats11:18:082
2265125511,20sleep20-21swapper/207:37:442
1113225516,22sleep111135-21ntp_states10:23:121
3218125415,15sleep30-21swapper/308:07:453
3218125415,15sleep30-21swapper/308:07:453
2010025417,15sleep30-21swapper/309:12:453
2010025417,15sleep30-21swapper/309:12:453
1883125416,18sleep10-21swapper/109:08:031
1883125416,18sleep10-21swapper/109:08:031
54225311,25sleep3538-21apt-get11:32:583
314425314,18sleep30-21swapper/309:58:313
142572538,17sleep20-21swapper/210:33:102
1356525315,26sleep113520-21latency_hist10:32:451
1014825319,16sleep10-21swapper/112:03:031
2676025213,23sleep226753-21apt-get11:12:552
2676025213,23sleep226753-21apt-get11:12:552
2661925213,21sleep126226-21apt-get11:12:461
2661925213,21sleep126226-21apt-get11:12:461
2196425213,24sleep221973-21df_inode09:18:012
627325115,20sleep25714-21/usr/sbin/munin10:08:152
2539225113,23sleep125398-21ntp_states11:08:091
1918025114,21sleep10-21swapper/112:32:431
980325013,22sleep19809-21ntp_states08:38:111
808225012,22sleep28091-21munin-run11:57:432
776625017,18sleep30-21swapper/310:13:093
2371125013,18sleep20-21swapper/209:23:052
1690225012,22sleep316898-21apt-get10:42:563
1577125013,20sleep215808-21ntp_states10:38:112
2506624913,22sleep225089-21ntp_states07:43:092
2386924812,20sleep323931-21sendmail_mailst11:03:113
1843824810,22sleep118450-21apt-get10:47:551
2552824710,21sleep125600-21sensors_temp09:28:131
1767324713,17sleep20-21swapper/209:03:112
134842469,23sleep313555-21systemd10:32:433
2888124415,15sleep30-21swapper/307:55:503
203402427,18sleep30-21swapper/309:12:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional