You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 14:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Nov 18, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
221902155815,1527sleep032657-21kworker/u8:219:34:130
146372144013,1411sleep1195-21jbd2/mmcblk2p2-00:20:261
20563262131,578sleep220609-21ntpq22:54:202
1505125528,529sleep215092-21ntpq20:54:252
23741253723,502sleep223810-21df19:39:162
17953253322,493sleep017987-21/usr/sbin/munin00:29:210
927925229,463sleep29295-21latency_hist00:03:552
927925229,463sleep29295-21latency_hist00:03:552
3188624426,403sleep231925-21ntpq20:04:202
1775239812,372sleep31776-21sshd20:12:293
28505236717,206sleep20-21swapper/219:54:152
1884234046,214sleep21900-21modprobe20:13:532
14026232612,230sleep214116-21fschecks_time22:34:172
30760231549,193sleep230790-21munin-plugin-st23:28:532
13182226825,222sleep013248-21fschecks_time20:49:180
4907224980,132sleep24929-21fschecks_count23:49:132
4907224980,132sleep24929-21fschecks_count23:49:132
20247224322,202sleep020329-21fschecks_time22:54:140
14991223776,147sleep215060-21apt-key19:13:552
14991223776,147sleep215060-21apt-key19:13:552
12244223042,112sleep212389-21unixbench_multi20:44:272
30053222778,136sleep230074-21fschecks_count19:59:152
16177222080,91sleep216325-21unixbench_multi19:14:402
7826221614,181sleep20-21swapper/222:14:422
26887220970,56sleep20-21swapper/223:14:232
4531220059,64sleep24549-21modprobe23:48:522
2488221826,164sleep224993-21sed21:24:252
2488221826,164sleep224993-21sed21:24:252
11070217522,134sleep011107-21grep22:24:170
11481217231,116sleep20-21swapper/222:24:272
18285216512,138sleep218321-21http19:23:532
811421578,108sleep28155-21awk20:33:532
811421578,108sleep28155-21awk20:33:532
18171215526,19sleep20-21swapper/200:29:272
5512215413,129sleep25560-21if_eth020:24:152
13503215222,68sleep213071-21/usr/sbin/munin20:49:242
23491215047,45sleep20-21swapper/219:38:562
20075215027,66sleep220123-21systemd-run19:28:542
22974214946,54sleep222985-21modprobe23:03:532
2250521499,124sleep222654-21sed21:18:562
18953214844,85sleep20-21swapper/222:49:192
1080214531,102sleep21093-21awk23:34:312
11627214315,113sleep111639-21fschecks_count20:44:141
15951214110,79sleep20-21swapper/222:39:262
21700214052,46sleep221651-21munin-run19:33:552
21700214052,46sleep221651-21munin-run19:33:552
10778213830,90sleep110914-21unixbench_multi18:59:251
31047213628,43sleep30-21swapper/323:29:063
12348213610,86sleep21-21systemd00:13:522
16031213514,108sleep016101-21sensors_temp19:14:380
18430213252,40sleep20-21swapper/221:04:222
19017213151,64sleep119077-21cat19:24:291
6146212969,38sleep20-21swapper/222:09:212
2070621285,27sleep20-21swapper/221:12:102
26017212719,85sleep126152-21grep19:44:281
2287212615,94sleep20-21swapper/221:56:282
19579211514,84sleep119580-21cpu21:09:101
4084211325,76sleep24190-21latency_hist22:03:552
3036321127,90sleep00-21swapper/019:59:210
9339211126,50sleep29377-21grep22:19:142
3105221109,83sleep20-21swapper/223:29:092
1025121108,77sleep20-21swapper/220:39:182
28700210928,65sleep228770-21cut21:38:552
28700210928,65sleep228770-21cut21:38:552
3729210817,74sleep13252-21/usr/sbin/munin23:44:231
30714210816,73sleep30-21swapper/319:59:293
13618210814,77sleep30-21swapper/322:33:533
984210713,80sleep1995-21netstat20:09:201
9324210615,76sleep09331-21sh22:19:120
3466210619,69sleep13484-21modprobe20:18:531
25471210625,15sleep20-21swapper/223:09:262
25471210625,15sleep20-21swapper/223:09:262
5308210514,76sleep15378-21df20:24:101
30364210525,58sleep20-21swapper/221:43:562
7669210418,70sleep17164-21apt-get22:14:041
7145210419,68sleep17285-21/usr/sbin/munin23:54:451
7145210419,68sleep17285-21/usr/sbin/munin23:54:451
32164210424,16sleep20-21swapper/221:49:132
31518210417,73sleep131529-21grep20:04:111
2221210425,16sleep20-21swapper/220:14:222
8938210332,12sleep20-21swapper/220:34:252
8938210332,12sleep20-21swapper/220:34:252
18346210326,15sleep20-21swapper/222:48:552
15289210210,75sleep015397-21cat20:54:310
28891210112,78sleep2362-21snmpd23:19:482
2336021016,80sleep223457-21fschecks_time23:04:132
27639210018,15sleep20-21swapper/219:49:302
23970210019,15sleep20-21swapper/221:23:532
14544210015,69sleep114620-21timerandwakeup22:34:281
508429913,27sleep00-21swapper/022:04:260
508429913,27sleep00-21swapper/022:04:260
2955129911,71sleep329574-21sh23:24:093
1273829917,64sleep012746-21df_inode00:14:110
1442929810,41sleep20-21swapper/200:19:052
971429711,71sleep39737-21pluginstate22:19:233
164912976,77sleep21-21systemd00:24:222
164912976,77sleep21-21systemd00:24:222
853129613,70sleep08618-21/usr/sbin/munin20:34:150
853129613,70sleep08618-21/usr/sbin/munin20:34:150
399429623,15sleep20-21swapper/220:19:232
399429623,15sleep20-21swapper/220:19:232
2654029621,15sleep20-21swapper/219:48:532
1858729625,45sleep118581-21apt-get22:49:101
2939729512,68sleep329421-21/usr/sbin/munin21:39:213
1429029521,14sleep20-21swapper/200:18:572
121282958,15sleep20-21swapper/219:04:222
3205129413,64sleep332122-21sensors20:04:253
303529329,18sleep20-21swapper/223:43:562
271462938,20sleep20-21swapper/221:33:552
489029126,41sleep20-21swapper/222:04:242
489029126,41sleep20-21swapper/222:04:242
1643329116,57sleep015949-21apt-get20:59:030
136062916,50sleep20-21swapper/222:33:442
2147928933,28sleep10-21swapper/122:58:551
870628820,50sleep38412-21/usr/sbin/munin20:34:183
870628820,50sleep38412-21/usr/sbin/munin20:34:183
92532877,61sleep10-21swapper/122:19:091
369128711,58sleep00-21swapper/023:44:220
300092867,14sleep20-21swapper/221:41:302
289628616,55sleep33012-21timerwakeupswit20:14:433
134432869,22sleep10-21swapper/100:14:361
1200128615,53sleep011045-21/usr/sbin/munin00:09:420
692928412,16sleep10-21swapper/120:29:151
692928412,16sleep10-21swapper/120:29:151
2808928425,38sleep128112-21munin-run19:53:531
2771328319,47sleep127798-21latency_hist23:18:541
2521428017,47sleep025258-21awk23:09:200
2521428017,47sleep025258-21awk23:09:200
922327917,38sleep10-21swapper/120:34:281
922327917,38sleep10-21swapper/120:34:281
2241027918,42sleep322462-21if_eth019:34:413
170027810,53sleep00-21swapper/023:39:050
1576227815,46sleep115856-21memory22:39:201
759027716,42sleep07719-21/usr/sbin/munin20:29:310
759027716,42sleep07719-21/usr/sbin/munin20:29:310
72862776,32sleep20-21swapper/223:54:462
72862776,32sleep20-21swapper/223:54:462
2697427732,29sleep30-21swapper/323:14:243
1154727714,47sleep111558-21ntpq00:09:351
1573927611,14sleep00-21swapper/022:39:190
3126827513,47sleep131277-21munin-run20:03:551
2346027515,47sleep323516-21if_eth023:04:143
2008227517,44sleep320170-21grep21:09:243
1898527524,29sleep30-21swapper/322:49:203
1653727526,13sleep20-21swapper/220:59:132
440627416,43sleep34438-21tune2fs22:04:133
440627416,43sleep34438-21tune2fs22:04:133
2706927418,35sleep027096-21modprobe21:33:520
1848827422,24sleep00-21swapper/021:04:240
1840427425,28sleep00-21swapper/019:23:550
596427316,25sleep30-21swapper/322:09:183
385827319,38sleep03911-21fschecks_time20:19:210
385827319,38sleep03911-21fschecks_time20:19:210
3081327316,26sleep10-21swapper/123:28:561
2743627314,39sleep027511-21/usr/sbin/munin19:49:280
2110927317,31sleep021244-21head19:29:290
2110927317,31sleep021244-21head19:29:290
2084527322,34sleep020873-21munin-plugin-st21:13:530
3230927219,36sleep132333-21munin-run23:33:531
2884927114,37sleep327931-21/usr/sbin/munin23:19:293
253627119,26sleep30-21swapper/321:58:563
1576627117,28sleep10-21swapper/119:14:311
1141527120,31sleep311433-21munin-run19:03:553
96032706,12sleep20-21swapper/200:04:112
811227015,36sleep08146-21/usr/sbin/munin22:14:480
3123527018,28sleep10-21swapper/121:44:281
3062127018,21sleep00-21swapper/021:44:150
2589727011,43sleep325909-21fschecks_count21:29:153
1567627016,19sleep30-21swapper/319:14:293
1336427013,21sleep30-21swapper/320:49:213
1655826919,34sleep316626-21wc00:24:243
1655826919,34sleep316626-21wc00:24:243
1156626915,39sleep311598-21ps00:09:373
1129626918,36sleep311439-21latency_hist20:43:553
856226813,41sleep18634-21wc23:59:381
856226813,41sleep18634-21wc23:59:381
3194826820,22sleep00-21swapper/023:29:280
233126824,25sleep30-21swapper/323:39:243
2178826818,26sleep30-21swapper/319:33:563
2178826818,26sleep30-21swapper/319:33:563
1442626811,40sleep10-21swapper/120:53:551
1088726817,35sleep110937-21fschecks_count22:24:141
2180626719,20sleep10-21swapper/122:59:161
2074526719,28sleep120784-21/usr/sbin/munin19:29:211
2074526719,28sleep120784-21/usr/sbin/munin19:29:211
188326719,34sleep11946-21idleruntime23:39:141
1362426714,26sleep10-21swapper/119:09:061
1362426714,26sleep10-21swapper/119:09:061
364426619,23sleep30-21swapper/323:44:213
2280926621,23sleep10-21swapper/119:34:511
2230726615,37sleep322311-21sshd21:17:283
1851626614,23sleep10-21swapper/119:23:561
1505626620,31sleep315130-21/usr/sbin/munin00:20:403
3219026521,19sleep00-21swapper/021:49:150
2735726516,25sleep30-21swapper/319:49:273
248326519,31sleep02484-21cron21:58:530
2457826522,22sleep00-21swapper/023:08:560
2042926518,21sleep30-21swapper/322:54:173
1902926518,25sleep00-21swapper/022:49:210
1724326515,37sleep317290-21if_eth022:44:153
1313026518,28sleep113258-21/usr/sbin/munin22:29:271
1267926514,38sleep312682-21sshd20:47:233
585926420,20sleep30-21swapper/320:24:253
54982644,48sleep10-21swapper/122:07:461
54982644,48sleep10-21swapper/122:07:461
2278326418,27sleep00-21swapper/023:01:290
158992645,45sleep30-21swapper/320:58:253
628226313,34sleep16317-21latency_hist23:53:561
628226313,34sleep16317-21latency_hist23:53:561
608826313,28sleep10-21swapper/122:09:201
264482635,11sleep30-21swapper/319:47:413
2616326313,35sleep326304-21latency_hist23:13:563
2616326313,35sleep326304-21latency_hist23:13:563
2452926316,22sleep10-21swapper/121:24:161
2452926316,22sleep10-21swapper/121:24:161
1832426316,29sleep318399-21cat19:23:543
1399726320,21sleep30-21swapper/300:18:543
81682624,12sleep30-21swapper/323:59:273
81682624,12sleep30-21swapper/323:59:273
413426215,21sleep10-21swapper/120:19:261
413426215,21sleep10-21swapper/120:19:261
327482626,44sleep00-21swapper/020:08:300
317726211,23sleep00-21swapper/021:59:210
634526123,19sleep00-21swapper/022:09:270
3206126119,23sleep30-21swapper/321:49:063
1701126117,27sleep10-21swapper/122:44:051
1646626117,27sleep316459-21apt-get20:59:103
112922616,11sleep00-21swapper/019:02:590
1020126112,34sleep110250-21irqstats20:39:171
89526017,28sleep1935-21systemctl21:53:531
222132607,44sleep00-21swapper/021:16:160
1954526014,17sleep00-21swapper/019:24:400
1933926014,20sleep30-21swapper/321:08:553
130326018,26sleep11395-21perl21:54:131
3271625920,21sleep00-21swapper/023:34:160
2850825922,19sleep00-21swapper/023:19:260
2705225916,24sleep00-21swapper/023:14:250
2211025915,29sleep322146-21ntpq22:59:223
1588025911,25sleep30-21swapper/322:39:233
1539825914,23sleep10-21swapper/120:54:321
1446325916,20sleep00-21swapper/000:19:160
79922586,16sleep30-21swapper/320:32:203
79922586,16sleep30-21swapper/320:32:203
3053625817,25sleep330532-21apt-get21:44:093
2840825817,25sleep028403-21apt-get19:54:080
2462825814,19sleep30-21swapper/321:24:183
2462825814,19sleep30-21swapper/321:24:183
221825815,21sleep00-21swapper/020:14:200
1910725811,22sleep30-21swapper/319:24:313
1691925814,29sleep016920-21sshd00:24:290
1691925814,29sleep016920-21sshd00:24:290
1078025818,27sleep310805-21awk00:08:523
107312586,10sleep10-21swapper/100:08:141
105162586,40sleep00-21swapper/000:05:300
482425719,22sleep30-21swapper/323:49:103
482425719,22sleep30-21swapper/323:49:103
294122577,12sleep10-21swapper/119:55:311
2864925717,18sleep00-21swapper/021:38:530
2864925717,18sleep00-21swapper/021:38:530
249052574,11sleep10-21swapper/119:42:461
2373925716,25sleep023729-21apt-get19:39:140
2116625715,24sleep121233-21df21:14:111
1829125717,19sleep30-21swapper/300:29:283
1420525714,17sleep00-21swapper/019:09:250
1420525714,17sleep00-21swapper/019:09:250
2861325613,26sleep328665-21if_eth019:54:163
998025515,22sleep09976-21apt-get20:39:080
773725515,24sleep37688-21apt-get22:14:353
3029825523,19sleep10-21swapper/121:43:551
2696225512,29sleep1225-21systemd-journal21:32:321
2307625514,21sleep30-21swapper/321:19:223
1731025515,19sleep10-21swapper/100:28:541
1731025515,19sleep10-21swapper/100:28:541
1295025517,23sleep112982-21munin-run19:08:531
1087125510,31sleep310880-21df_inode22:24:123
482125412,24sleep04817-21apt-get23:49:070
482125412,24sleep04817-21apt-get23:49:070
2900725418,21sleep00-21swapper/021:39:100
2271425415,22sleep022710-21apt-get21:19:100
797025312,24sleep07964-21apt-get23:59:080
797025312,24sleep07964-21apt-get23:59:080
633525329,13sleep00-21swapper/020:25:590
507225317,19sleep30-21swapper/320:23:563
507225317,19sleep30-21swapper/320:23:563
3004625315,23sleep030113-21awk23:24:230
280125314,22sleep32848-21df21:59:123
2589625314,18sleep00-21swapper/019:44:270
2392225315,17sleep30-21swapper/319:39:213
1889225315,22sleep10-21swapper/100:33:551
1324725314,23sleep313296-21netstat19:09:003
1324725314,23sleep313296-21netstat19:09:003
59125215,23sleep0610-21apt-get20:09:070
3262925212,23sleep332625-21apt-get23:34:113
2481725211,24sleep124812-21apt-get23:09:091
2481725211,24sleep124812-21apt-get23:09:091
1393925215,22sleep013934-21apt-get22:34:110
1136325210,16sleep20-21swapper/219:03:542
685225113,22sleep36888-21ntp_states23:54:393
685225113,22sleep36888-21ntp_states23:54:393
2926225115,19sleep10-21swapper/123:23:551
2557225115,15sleep00-21swapper/021:28:560
2557225115,15sleep00-21swapper/021:28:560
1829625114,22sleep018326-21munin-run22:48:530
1400425112,22sleep314015-21df_inode22:34:143
1300025117,18sleep30-21swapper/319:08:553
1236125113,24sleep012355-21apt-get22:29:100
643425012,22sleep06428-21apt-get23:54:280
643425012,22sleep06428-21apt-get23:54:280
2738624912,23sleep327382-21apt-get21:34:083
2738624912,23sleep327382-21apt-get21:34:083
2326624811,22sleep123260-21apt-get23:04:081
176762487,29sleep10-21swapper/121:03:161
1169124812,21sleep011688-21apt-get19:04:080
1803324711,22sleep118100-21df21:04:131
143624714,18sleep10-21swapper/123:38:541
1162024713,19sleep011626-21diskstats20:44:130
2784824610,18sleep10-21swapper/121:34:221
2784824610,18sleep10-21swapper/121:34:221
2279024610,21sleep122799-21diskstats21:19:151
304624510,17sleep10-21swapper/121:59:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional