You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 08:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Sep 19, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31806267417,639sleep331848-21/usr/sbin/munin23:48:073
21754260518,569sleep221806-21/usr/sbin/munin21:33:052
504243851,354sleep1552-21awk23:52:581
27750240139,179sleep10-21swapper/120:08:111
14449236527,263sleep114592-21grep21:08:131
31103235019,316sleep26096-21kworker/u8:322:02:542
2922422697,224sleep128805-21/usr/sbin/munin20:13:081
2922422697,224sleep128805-21/usr/sbin/munin20:13:081
17077225562,177sleep1492-21cron23:02:421
3955224316,204sleep14040-21threads22:18:111
430922407,220sleep14130-21latency_hist20:37:521
21142223848,176sleep121226-21fschecks_time19:48:001
31535222630,183sleep131582-21ntp_offset22:03:091
1493222543,166sleep11496-21cron22:12:421
29025221520,176sleep329167-21unixbench_singl23:38:143
16150221139,115sleep10-21swapper/119:32:451
327422046,187sleep1367-21snmpd00:01:421
16584220411,144sleep113475-21kworker/u8:021:17:421
16584220411,144sleep113475-21kworker/u8:021:17:421
7522220112,120sleep10-21swapper/120:47:571
19002219711,42sleep019033-21fschecks_count23:08:020
22013219652,47sleep10-21swapper/121:33:121
17645219350,126sleep117691-21systemctl19:37:421
58721878,163sleep00-21swapper/023:53:010
11963218497,25sleep10-21swapper/121:02:431
829821839,128sleep18180-21/usr/sbin/munin19:08:021
27290218311,157sleep127358-21switchtime23:33:151
13302217811,126sleep113396-21fschecks_time00:33:031
11834217512,150sleep011905-21fschecks_time19:18:020
3186421738,153sleep131872-21ntpq23:48:091
2699521738,152sleep127044-21unin-run20:07:431
2070321699,145sleep219970-21/usr/sbin/munin21:28:132
1379821659,113sleep113270-21/usr/sbin/munin19:23:121
9682216227,124sleep1367-21snmpd00:20:521
9682216227,124sleep1367-21snmpd00:20:521
31050216243,51sleep10-21swapper/120:18:131
26288216221,119sleep30-21swapper/320:03:133
19055215772,26sleep10-21swapper/121:23:111
26362215315,28sleep00-21swapper/020:03:140
17735215240,52sleep10-21swapper/123:03:061
23946215011,124sleep124060-21latency_hist19:57:451
7950214615,113sleep08068-21sed19:07:440
3763214310,27sleep00-21swapper/022:18:060
3012921429,58sleep10-21swapper/123:43:041
3012921429,58sleep10-21swapper/123:43:041
18919214013,109sleep318980-21grep21:23:093
5817213712,66sleep10-21swapper/100:08:241
2459421365,118sleep00-21swapper/023:24:210
18894213618,11sleep10-21swapper/119:38:311
12016213623,94sleep312058-21sort19:18:053
10481213411,70sleep10-21swapper/100:23:061
30736213010,79sleep130916-21sed22:02:451
23762213034,17sleep10-21swapper/121:38:131
2565221297,86sleep125686-21ps23:28:101
4894212825,81sleep34918-21awk00:07:413
234121285,11sleep10-21swapper/120:29:521
234121285,11sleep10-21swapper/120:29:521
1266321289,38sleep10-21swapper/122:47:521
2128621270,4sleep30-21swapper/319:48:013
186721277,45sleep10-21swapper/122:13:021
12115212715,12sleep10-21swapper/122:44:041
26698212611,64sleep10-21swapper/121:48:111
1937821268,79sleep119419-21grep23:08:111
27996212520,85sleep028049-21kworker/u8:023:37:420
27791212212,85sleep20-21swapper/220:08:122
1086021219,75sleep11-21systemd22:42:431
921321185,11sleep10-21swapper/122:36:531
1536221185,12sleep10-21swapper/122:55:591
9597211717,74sleep09718-21latency_hist19:12:450
9446211516,83sleep29490-21ntpq20:53:072
692021157,74sleep16954-21ps22:28:091
9927211423,71sleep09999-21fschecks_time19:12:490
25342211447,57sleep1374-21sshd20:01:211
13758211317,27sleep00-21swapper/019:23:110
9347211120,78sleep09411-21latency20:53:050
11521211116,78sleep311560-21grep22:43:053
2962121098,40sleep10-21swapper/123:42:441
971921087,65sleep10-21swapper/119:12:461
3214221089,38sleep10-21swapper/120:23:011
22265210810,38sleep10-21swapper/123:18:091
32230210610,80sleep332321-21load20:23:033
2046221067,43sleep120456-21apt-get23:12:561
705121057,37sleep10-21swapper/100:13:011
26577210515,27sleep00-21swapper/021:48:060
9823210417,20sleep30-21swapper/319:12:483
22762210412,21sleep30-21swapper/323:18:193
811210357,28sleep2896-21switchtime22:08:152
490921038,34sleep10-21swapper/122:22:561
16263210313,74sleep016269-21ntp_states22:58:090
1304210223,60sleep31290-21apt-get20:28:003
1304210223,60sleep31290-21apt-get20:28:003
2985629920,39sleep30-21swapper/323:42:543
2985629920,39sleep30-21swapper/323:42:543
2046629916,68sleep00-21swapper/023:12:590
476029810,76sleep0367-21snmpd00:05:570
510329718,60sleep25249-21apt-key00:07:452
3046529710,74sleep330489-21df20:17:593
1199329715,66sleep212146-21date21:02:442
235642955,11sleep10-21swapper/123:22:491
209822945,11sleep10-21swapper/121:29:131
327202927,13sleep00-21swapper/020:23:130
3251129112,64sleep232588-21switchtime20:23:112
2858229014,62sleep228641-21/usr/sbin/munin23:38:042
2330429029,43sleep223314-21modprobe23:22:422
1142729016,61sleep211482-21sed00:27:432
539328922,49sleep24908-21apt-get00:07:532
410928824,44sleep04134-21munin-run20:37:430
2597328831,38sleep326011-21latency_hist21:47:443
636928715,48sleep05949-21/usr/sbin/munin20:43:080
636928715,48sleep05949-21/usr/sbin/munin20:43:080
519928723,47sleep10-21swapper/120:38:321
519928723,47sleep10-21swapper/120:38:321
962128613,55sleep39684-21/usr/sbin/munin20:53:123
60228617,55sleep0631-21nfs4_client22:08:090
300952866,56sleep00-21swapper/023:43:030
300952866,56sleep00-21swapper/023:43:030
1736828614,26sleep00-21swapper/023:02:570
1693328621,49sleep10-21swapper/121:17:581
1456228634,38sleep214494-21latency_hist00:37:452
322102850,6sleep00-21swapper/022:05:210
2476128526,21sleep124851-21grep21:43:021
1708028426,31sleep20-21swapper/223:02:432
2826928326,37sleep028342-21awk21:53:150
134862838,32sleep00-21swapper/000:33:060
1019028318,29sleep30-21swapper/300:23:003
916428213,33sleep10-21swapper/120:53:021
604728223,44sleep10-21swapper/120:43:011
604728223,44sleep10-21swapper/120:43:011
3177428216,50sleep231805-21/usr/sbin/munin23:48:062
2773228220,29sleep10-21swapper/121:52:511
951328121,30sleep30-21swapper/319:12:443
884028119,30sleep30-21swapper/320:52:443
815228113,45sleep08271-21/usr/sbin/munin20:48:130
457228118,11sleep10-21swapper/100:03:321
1276928118,11sleep10-21swapper/100:30:451
1958028020,46sleep10-21swapper/119:42:591
132328021,38sleep01349-21cpu20:28:130
132328021,38sleep01349-21cpu20:28:130
125832807,16sleep10-21swapper/119:18:161
122762807,16sleep00-21swapper/022:46:050
1951027917,17sleep00-21swapper/019:42:580
1340727918,11sleep10-21swapper/121:06:551
1317127913,50sleep313233-21/usr/sbin/munin22:48:103
103782799,55sleep00-21swapper/022:38:180
3028527810,55sleep230287-21modprobe21:58:422
1237127816,44sleep312511-21/usr/sbin/munin00:28:133
862227626,30sleep00-21swapper/022:33:130
2312127615,39sleep223096-21/usr/sbin/munin21:37:582
643427513,27sleep30-21swapper/320:43:113
643427513,27sleep30-21swapper/320:43:113
500227519,27sleep30-21swapper/320:38:303
500227519,27sleep30-21swapper/320:38:303
31082755,11sleep00-21swapper/023:59:350
2495127523,28sleep30-21swapper/323:27:443
1666227521,26sleep30-21swapper/321:17:443
1666227521,26sleep30-21swapper/321:17:443
2601527413,44sleep026160-21date21:47:450
2163227412,47sleep021729-21switchtime19:48:110
184627424,27sleep30-21swapper/323:57:443
153032747,12sleep00-21swapper/022:55:140
1065427421,34sleep30-21swapper/320:57:573
1927027316,39sleep219336-21latency_hist19:42:442
1639027316,43sleep316418-21cpu19:33:003
1074027319,36sleep010867-21timerwakeupswit00:23:130
1844527216,16sleep20-21swapper/221:22:562
571927126,19sleep30-21swapper/300:08:223
2685427113,43sleep326969-21cat21:48:133
716126827,24sleep20-21swapper/219:03:112
2527726812,41sleep025322-21fschecks_count23:28:010
1447326815,38sleep314518-21awk00:37:423
118092687,50sleep00-21swapper/021:01:170
771926715,25sleep20-21swapper/222:32:422
705826711,41sleep27082-21fschecks_count00:13:022
237832676,50sleep30-21swapper/319:56:383
1609826714,34sleep216117-21awk19:32:422
3121926630,21sleep30-21swapper/322:03:033
2994526621,21sleep20-21swapper/223:43:012
2994526621,21sleep20-21swapper/223:43:012
2386526617,33sleep223869-21cron19:57:422
2022626617,32sleep220268-21timerwakeupswit19:43:212
1918326618,32sleep30-21swapper/323:08:053
1397826613,36sleep314034-21/usr/sbin/munin21:08:033
462226521,24sleep30-21swapper/322:22:423
3096926514,28sleep00-21swapper/020:18:120
2649926522,23sleep30-21swapper/323:32:443
2094726516,19sleep20-21swapper/223:13:132
340226416,31sleep20-21swapper/200:02:442
312826413,23sleep20-21swapper/222:17:442
2436226420,19sleep20-21swapper/219:58:032
784926327,20sleep10-21swapper/119:07:211
734026319,22sleep30-21swapper/300:13:073
3010026317,30sleep330109-21modprobe20:17:423
3010026317,30sleep330109-21modprobe20:17:423
2073926321,22sleep30-21swapper/319:47:213
1549426314,26sleep215518-21munin-run22:57:422
688526223,18sleep30-21swapper/319:03:043
2929526214,24sleep00-21swapper/020:13:110
2929526214,24sleep00-21swapper/020:13:110
2442326215,27sleep324464-21latency_hist21:42:433
1736026223,23sleep00-21swapper/021:18:090
157642628,18sleep30-21swapper/319:28:233
723226123,23sleep20-21swapper/220:47:422
723226123,23sleep20-21swapper/220:47:422
527726120,19sleep30-21swapper/322:23:063
499526115,29sleep25051-21tune2fs22:23:012
463126120,19sleep20-21swapper/220:38:212
463126120,19sleep20-21swapper/220:38:212
3167426117,28sleep031690-21uname23:48:040
2943526110,23sleep30-21swapper/321:58:023
2725926115,31sleep027278-21apt-get20:07:540
2222626113,32sleep022363-21unixbench_multi21:33:150
1239226115,30sleep012445-21/usr/sbin/munin21:03:030
703226013,26sleep30-21swapper/322:28:113
31792609,10sleep30-21swapper/300:00:313
1851826015,23sleep00-21swapper/021:22:580
1531526016,27sleep015309-21apt-get21:12:560
1531526016,27sleep015309-21apt-get21:12:560
1403626013,29sleep00-21swapper/021:08:040
1248026015,28sleep20-21swapper/221:03:052
113826017,26sleep2788-21apt-get20:27:472
113826017,26sleep2788-21apt-get20:27:472
869025917,24sleep28774-21grep19:08:112
801325917,26sleep20-21swapper/220:48:112
352425914,28sleep33649-21timerwakeupswit20:33:423
283742595,13sleep30-21swapper/320:10:323
278922590,7sleep30-21swapper/321:53:043
1867925916,29sleep318756-21sed23:07:453
1665825918,24sleep216673-21kernelversion19:33:072
1403325919,17sleep20-21swapper/222:52:452
996425814,21sleep30-21swapper/322:38:083
3023825815,18sleep20-21swapper/220:17:452
2682425814,22sleep20-21swapper/223:33:042
2540125815,24sleep225458-21irqstats23:28:032
2199225826,18sleep30-21swapper/321:33:103
219625811,27sleep30-21swapper/322:13:093
1747125820,18sleep20-21swapper/221:18:112
1430925817,21sleep30-21swapper/322:52:593
968225718,18sleep20-21swapper/222:38:032
883625716,21sleep30-21swapper/300:18:033
883625716,21sleep30-21swapper/300:18:033
811425714,26sleep00-21swapper/000:15:580
508025722,18sleep00-21swapper/020:38:310
508025722,18sleep00-21swapper/020:38:310
2515925714,15sleep20-21swapper/221:43:112
2014025715,25sleep320204-21/usr/sbin/munin21:28:013
178125715,25sleep01775-21apt-get22:12:570
1150925715,17sleep20-21swapper/219:17:442
543325615,21sleep00-21swapper/000:08:030
382025612,30sleep23828-21ntpq22:18:082
268342569,31sleep30-21swapper/323:33:053
1820925614,27sleep318312-21latency_hist21:22:443
1796025614,25sleep317955-21apt-get19:37:563
1329525613,25sleep313290-21apt-get19:22:583
1252625613,27sleep312576-21memory21:03:063
1166325614,25sleep211657-21apt-get00:27:562
712625514,21sleep00-21swapper/022:28:120
2785025519,19sleep20-21swapper/221:53:042
2122925521,18sleep20-21swapper/219:48:012
1898325512,28sleep218993-21df_inode23:08:002
1638725514,25sleep016383-21apt-get19:32:570
2973425412,21sleep00-21swapper/021:58:070
183725416,21sleep20-21swapper/223:57:432
180912546,11sleep00-21swapper/019:38:020
1529425414,24sleep015300-21ntp_states19:28:080
143482548,35sleep20-21swapper/219:25:032
3268425318,19sleep30-21swapper/322:08:003
2678525315,23sleep20-21swapper/221:48:122
1744025314,24sleep217449-21diskstats23:02:592
877825214,24sleep28835-21irqstats00:18:022
877825214,24sleep28835-21irqstats00:18:022
808725215,20sleep30-21swapper/322:33:003
294052526,11sleep20-21swapper/220:13:132
294052526,11sleep20-21swapper/220:13:132
2868225218,19sleep00-21swapper/023:38:070
2446725217,16sleep00-21swapper/021:42:440
2406125214,18sleep20-21swapper/223:23:102
2328925213,18sleep00-21swapper/019:53:160
1248825213,23sleep212641-21latency_hist22:47:452
37982519,18sleep20-21swapper/220:33:452
1713025113,21sleep30-21swapper/323:02:443
1413725115,18sleep20-21swapper/221:08:062
1323525114,14sleep00-21swapper/022:48:110
801425011,23sleep28011-21apt-get22:32:542
2417925011,22sleep324175-21apt-get19:57:553
1663325011,23sleep216661-21munin-run21:17:432
1663325011,23sleep216661-21munin-run21:17:432
2622024910,25sleep226287-21systemctl20:03:122
2445124915,18sleep00-21swapper/019:58:050
2335524910,23sleep00-21swapper/023:22:440
2053924912,23sleep320545-21df_inode23:13:023
2040424911,20sleep020446-21ntp_states21:28:070
854024811,17sleep00-21swapper/000:17:520
854024811,17sleep00-21swapper/000:17:520
512024712,17sleep00-21swapper/022:23:030
125172459,19sleep012565-21users00:28:140
1182024511,18sleep20-21swapper/219:18:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional