You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Apr 03, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
130192515714,5131sleep39992-21kworker/3:020:30:403
223822489914,4874sleep320644-21kworker/3:121:00:403
5593269016,656sleep35586-21apt-get23:32:163
20903228627,241sleep320917-21modprobe20:57:013
7186225810,49sleep00-21swapper/020:12:190
22396225681,154sleep013441-21snmpd19:17:190
32728224477,151sleep0376-21timerandwakeup21:32:310
22882222524,175sleep323019-21unixbench_multi22:42:333
14111222445,129sleep014149-21processes23:57:290
20518221733,146sleep020649-21timerwakeupswit00:17:320
11837221340,156sleep011990-21apt-key23:52:040
11837221340,156sleep011990-21apt-key23:52:040
20628221236,115sleep00-21swapper/022:37:130
10999221124,161sleep111141-21unixbench_multi20:22:341
1975622077,187sleep00-21swapper/022:32:410
7067220144,106sleep00-21swapper/021:56:020
7067220144,106sleep00-21swapper/021:56:020
28152218534,28sleep00-21swapper/019:36:130
10240218520,95sleep00-21swapper/023:47:110
664521835,167sleep00-21swapper/023:34:230
156692182122,45sleep015731-21uname00:02:300
12305218148,114sleep012339-21modprobe22:12:010
679721735,12sleep10-21swapper/121:52:351
679721735,12sleep10-21swapper/121:52:351
2065221707,149sleep020678-21latency19:12:100
1802217012,100sleep00-21swapper/019:54:510
1768721686,12sleep00-21swapper/022:27:280
9232216745,40sleep00-21swapper/022:02:120
2270121678,147sleep022726-21munin-run00:27:010
11710216616,135sleep211826-21timerwakeupswit22:07:392
17863216310,105sleep018014-21latency_hist20:47:040
18060216231,21sleep00-21swapper/000:12:030
13059216240,21sleep00-21swapper/022:12:290
32587216115,130sleep332715-21uptime23:12:333
10925216054,51sleep00-21swapper/020:22:330
2881921586,36sleep00-21swapper/023:02:210
25112215313,129sleep025190-21ls00:32:300
2410121529,22sleep00-21swapper/021:07:030
18926214617,52sleep218976-21fschecks_count22:32:222
15259214661,67sleep015277-21kernelversion20:37:230
3047214560,64sleep00-21swapper/023:22:320
25302214410,30sleep025409-21latency_hist22:52:030
1834721435,12sleep20-21swapper/200:12:172
32205214218,18sleep00-21swapper/019:47:320
32205214218,18sleep00-21swapper/019:47:320
4442214010,93sleep04482-21ntp_kernel_pll_23:27:240
3156921389,53sleep00-21swapper/023:12:020
28797213721,97sleep128815-21uname19:37:231
28797213721,97sleep128815-21uname19:37:231
1828221378,114sleep018339-21if_eth020:47:200
4315213616,60sleep00-21swapper/021:47:130
20926213610,26sleep00-21swapper/020:57:020
25168213514,106sleep325141-21latency_hist19:27:033
13814213551,60sleep00-21swapper/020:32:260
12546213518,100sleep212675-21unixbench_multi20:27:322
8810213434,88sleep00-21swapper/021:59:570
8810213434,88sleep00-21swapper/021:59:570
16990213429,84sleep217022-21munin-plugin-st22:27:012
21826213326,50sleep00-21swapper/020:57:300
24488213212,29sleep024554-21idleruntime21:07:240
8473212732,83sleep08503-21latency_hist23:42:020
1640212711,102sleep01686-21ntpq21:37:280
2974721265,11sleep00-21swapper/019:41:300
2974721265,11sleep00-21swapper/019:41:300
1988621255,108sleep00-21swapper/019:10:080
2471212422,71sleep12490-21apt-get23:22:131
27236212322,26sleep30-21swapper/321:17:043
9132212013,21sleep00-21swapper/020:17:240
1844212012,94sleep31856-21vmstat23:17:343
17222212017,88sleep317376-21unixbench_multi20:42:353
1492821199,94sleep015080-21sed22:17:340
22710211810,89sleep222755-21ntp_offset19:17:252
1945321185,10sleep00-21swapper/000:15:300
9182211618,76sleep38696-21/usr/sbin/munin23:42:303
10029211610,93sleep010085-21sed23:47:030
8832211414,84sleep20-21swapper/222:00:162
8832211414,84sleep20-21swapper/222:00:162
6733211453,48sleep021800-1kworker/0:1H20:09:570
30607211412,73sleep30-21swapper/323:07:253
917211334,27sleep00-21swapper/023:17:070
5326211334,67sleep00-21swapper/020:06:460
24793211313,77sleep224800-21/usr/sbin/munin21:07:302
22670211310,90sleep022711-21cut22:42:300
9020211024,70sleep39112-21latency_hist22:02:023
9020211024,70sleep39112-21latency_hist22:02:023
26943211034,28sleep00-21swapper/021:14:180
1307321105,10sleep00-21swapper/020:31:210
31577210934,28sleep00-21swapper/021:28:480
28215210916,39sleep00-21swapper/021:17:330
2553921098,50sleep025533-21apt-get22:52:150
13641210913,82sleep20-21swapper/223:57:152
28504210819,72sleep228521-21apt19:37:122
28504210819,72sleep228521-21apt19:37:122
25891210812,80sleep00-21swapper/019:27:290
18445210737,35sleep00-21swapper/019:07:010
19443210515,73sleep119520-21timerandwakeup22:32:331
2733521048,14sleep10-21swapper/122:57:211
22994210437,47sleep00-21swapper/000:27:170
17443210419,61sleep217593-21unixbench_multi00:07:332
355621038,47sleep03597-21sh20:02:010
26493210319,67sleep126576-21timerandwakeup21:12:331
18743210317,69sleep218863-21timerwakeupswit20:47:312
21273210221,66sleep321389-21sed22:37:363
19086210220,61sleep118272-21/usr/sbin/munin00:12:321
711210040,48sleep0709-21munin-run23:17:030
161042997,78sleep016140-21ntpq22:22:270
2948829738,41sleep00-21swapper/021:22:300
1887829432,41sleep318948-21switchtime00:12:303
80829312,68sleep2904-21fschecks_time19:52:222
530729317,62sleep15355-21unin-run23:32:021
3007529318,53sleep130050-21/usr/sbin/munin19:42:161
3007529318,53sleep130050-21/usr/sbin/munin19:42:161
3211429217,36sleep30-21swapper/321:32:143
2748529244,34sleep00-21swapper/022:57:240
689229119,52sleep26905-21cron20:12:012
289128919,56sleep12973-21grep19:57:301
1394428914,53sleep114027-21latency_hist22:17:031
1386528825,41sleep313891-21munin-plugin-st22:17:013
305272876,31sleep00-21swapper/019:42:270
305272876,31sleep00-21swapper/019:42:270
2326728711,61sleep123356-21/usr/sbin/munin00:27:261
3158028614,53sleep131726-21latency_hist23:12:031
1220428629,30sleep30-21swapper/323:52:433
1220428629,30sleep30-21swapper/323:52:433
884928525,28sleep30-21swapper/320:17:193
63028527,36sleep2653-21idleruntime-cro23:17:012
580428518,33sleep10-21swapper/123:32:221
450628517,46sleep34541-21grep23:27:283
3035128524,43sleep230499-21grep21:27:042
26212856,28sleep00-21swapper/021:42:040
2346028525,33sleep20-21swapper/222:47:002
1718528526,24sleep20-21swapper/219:02:152
3140428418,49sleep231406-21sshd23:10:262
298328416,46sleep32472-21/usr/sbin/munin23:22:313
2400328410,56sleep323974-21apt-get22:47:263
1490728425,32sleep30-21swapper/300:02:003
3003228326,37sleep20-21swapper/223:07:032
2564928317,20sleep30-21swapper/321:12:033
405528221,43sleep33572-21apt-get20:02:133
2720228217,46sleep327262-21/usr/sbin/munin22:57:193
181802820,5sleep30-21swapper/320:47:183
1746528228,32sleep10-21swapper/119:02:221
1152428014,50sleep111566-21gpgv23:52:011
2923227916,49sleep129289-21irqstats21:22:231
2010327915,31sleep20-21swapper/200:17:222
190422798,16sleep20-21swapper/219:07:242
2741827819,43sleep327455-21ps19:32:283
2372727814,43sleep223871-21grep00:27:362
2768027721,32sleep10-21swapper/121:17:221
2740827713,14sleep20-21swapper/219:32:272
245352777,49sleep00-21swapper/019:22:320
220627721,39sleep12235-21munin-plugin-st23:22:001
2152927727,22sleep10-21swapper/100:22:221
2915027624,31sleep10-21swapper/123:02:301
2453927611,51sleep224603-21/usr/sbin/munin00:32:162
2910727543,19sleep30-21swapper/321:22:213
820627412,45sleep10-21swapper/121:58:011
820627412,45sleep10-21swapper/121:58:011
80027416,25sleep10-21swapper/119:52:211
779827425,20sleep30-21swapper/320:12:353
698127418,42sleep26855-21cron23:37:042
2403527422,32sleep10-21swapper/122:47:481
2024827418,35sleep120323-21timerandwakeup20:52:301
1728327414,44sleep117277-21apt-get22:27:161
1156727420,31sleep20-21swapper/223:52:022
995227321,21sleep20-21swapper/222:02:342
37927319,36sleep1525-21grep21:32:321
3008127322,24sleep30-21swapper/323:07:043
2672027323,25sleep30-21swapper/319:32:043
104527323,26sleep20-21swapper/223:17:212
1408727216,15sleep20-21swapper/220:32:332
1165527219,23sleep30-21swapper/323:52:033
935627115,39sleep29439-21/usr/sbin/munin20:17:302
368627120,29sleep23866-21sed20:02:032
2156327121,20sleep20-21swapper/200:22:232
1877227115,39sleep318839-21/usr/sbin/munin19:07:163
1507827119,25sleep10-21swapper/120:37:201
3054927022,31sleep230611-21wc19:42:292
3054927022,31sleep230611-21wc19:42:292
209032708,14sleep20-21swapper/222:37:272
2065227018,16sleep10-21swapper/100:17:331
315942698,11sleep30-21swapper/319:47:103
315942698,11sleep30-21swapper/319:47:103
3073626929,18sleep30-21swapper/321:27:213
2254026922,21sleep10-21swapper/121:02:031
634526813,38sleep26358-21/usr/sbin/munin21:52:282
634526813,38sleep26358-21/usr/sbin/munin21:52:282
3259226817,19sleep20-21swapper/221:32:292
323912688,14sleep10-21swapper/123:12:311
304726814,26sleep20-21swapper/221:42:232
2591326822,24sleep10-21swapper/122:52:281
255026817,37sleep32617-21latency_hist21:42:033
850426720,22sleep10-21swapper/123:42:031
2978826721,29sleep329791-21cron19:42:013
2978826721,29sleep329791-21cron19:42:013
2744726719,34sleep327453-21cat21:17:123
211192675,18sleep30-21swapper/300:21:353
1859926724,26sleep30-21swapper/322:32:033
1588226717,25sleep30-21swapper/300:02:333
946626623,23sleep30-21swapper/322:02:233
2634926616,23sleep20-21swapper/221:12:302
2257126627,21sleep30-21swapper/319:17:223
1495926617,30sleep115044-21cat00:02:021
929426518,21sleep10-21swapper/123:42:321
535626511,24sleep20-21swapper/223:32:032
590126412,37sleep15610-21/usr/sbin/munin20:07:241
1920826425,19sleep30-21swapper/322:32:273
1104026412,19sleep10-21swapper/122:07:131
2319726315,17sleep30-21swapper/300:27:243
223822626,12sleep30-21swapper/300:22:563
2141026218,22sleep20-21swapper/219:12:322
1796426213,20sleep20-21swapper/222:27:342
1538926212,31sleep315423-21/usr/sbin/munin20:37:263
1205426211,28sleep20-21swapper/223:52:182
1205426211,28sleep20-21swapper/223:52:182
2499226118,26sleep124998-21ntp_states00:32:271
2348926112,20sleep10-21swapper/121:02:361
1680326117,16sleep20-21swapper/220:42:262
1427026113,31sleep314282-21fschecks_count22:17:213
1296426122,20sleep10-21swapper/120:29:571
264442605,18sleep30-21swapper/322:52:393
2540726016,27sleep225401-21apt-get19:27:142
2386126026,18sleep10-21swapper/119:22:171
1779726017,31sleep1588-21cron20:47:011
143472607,10sleep30-21swapper/320:32:503
1162926017,18sleep20-21swapper/220:27:032
539625916,20sleep10-21swapper/120:07:031
319902596,11sleep20-21swapper/219:47:262
319902596,11sleep20-21swapper/219:47:262
3137525919,20sleep30-21swapper/319:47:023
3137525919,20sleep30-21swapper/319:47:023
255442597,18sleep30-21swapper/300:33:463
1964225914,29sleep219644-21/usr/sbin/munin20:52:152
124025917,25sleep11233-21apt-get21:37:171
112822595,11sleep30-21swapper/320:23:173
559725816,24sleep30-21swapper/321:52:013
278625817,24sleep32779-21apt-get21:42:163
2738525816,26sleep227090-21/usr/sbin/munin22:57:222
2595425823,18sleep10-21swapper/119:27:301
2436425820,17sleep20-21swapper/219:22:302
2099725820,22sleep10-21swapper/119:12:231
1651125820,15sleep10-21swapper/100:07:031
1626925811,31sleep116339-21sensors_temp22:22:321
32072576,11sleep30-21swapper/319:57:403
2260425718,25sleep10-21swapper/122:42:271
1683125721,20sleep10-21swapper/100:07:191
1399025717,16sleep10-21swapper/120:32:321
1051825714,28sleep1281-21dbus-daemon22:07:011
755525616,24sleep17601-21/usr/sbin/munin20:12:281
316425619,17sleep10-21swapper/121:42:261
2722325611,29sleep127228-21kernelversion19:32:221
1331525620,20sleep10-21swapper/123:57:011
1331525620,20sleep10-21swapper/123:57:011
1120325613,18sleep30-21swapper/322:07:283
3162425514,25sleep131618-21apt-get19:47:161
3162425514,25sleep131618-21apt-get19:47:161
3065725518,16sleep20-21swapper/221:27:202
216502556,37sleep10-21swapper/122:38:391
191425515,16sleep20-21swapper/221:37:342
588125414,24sleep35877-21apt-get21:52:133
588125414,24sleep35877-21apt-get21:52:133
433725414,24sleep14334-21apt-get21:47:141
2878325411,22sleep20-21swapper/221:22:032
1570925414,24sleep315728-21apt-get22:22:143
1545525417,22sleep215490-21grep20:37:292
872025313,23sleep28714-21apt-get23:42:172
843225315,22sleep38464-21munin-run23:42:013
690725320,19sleep30-21swapper/320:12:023
464525323,17sleep20-21swapper/220:03:292
3084025320,19sleep10-21swapper/121:27:241
2158925316,17sleep20-21swapper/220:57:242
1528025315,23sleep215284-21diskstats00:02:202
725925211,16sleep20-21swapper/220:12:212
2450025220,15sleep20-21swapper/222:48:052
2319025213,21sleep323232-21ntp_states21:02:303
2154625212,15sleep30-21swapper/319:12:343
1759825212,24sleep317651-21users00:07:343
1723025222,17sleep30-21swapper/319:02:163
379725112,23sleep23807-21munin-run23:27:022
108425117,17sleep30-21swapper/319:52:273
926925016,17sleep10-21swapper/120:17:291
2484625010,18sleep10-21swapper/121:07:331
1508125016,19sleep10-21swapper/122:17:351
567424914,20sleep25679-21diskstats23:32:192
2278324914,20sleep222850-21df21:02:212
1877024914,18sleep10-21swapper/119:07:141
1834024810,16sleep10-21swapper/120:47:221
1418424811,20sleep214180-21apt-get22:17:162
223132478,14sleep10-21swapper/119:17:161
2561724410,20sleep225623-21df_inode22:52:212
130142428,15sleep20-21swapper/222:12:272
482523910,13sleep20-21swapper/221:47:322
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional