You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 22:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sat Jun 28, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26034295536,908sleep026070-21ntpq09:40:130
25049281023,771sleep125095-21/usr/sbin/munin07:55:121
3515247678,312sleep03581-21ls08:30:100
19917242110,84sleep019978-21latency_hist11:04:480
24050241724,357sleep00-21swapper/007:50:270
24050241724,357sleep00-21swapper/007:50:270
27457241120,375sleep127489-21irqstats09:45:151
29120234148,278sleep029085-21latency_hist08:09:500
1355423367,290sleep013699-21latency_hist07:19:490
2631823069,229sleep00-21swapper/008:00:120
8775230428,264sleep04092-1kworker/0:1H12:10:150
8775230428,264sleep04092-1kworker/0:1H12:10:150
1889225719,222sleep22013-21cat10:05:202
4329225320,216sleep00-21swapper/010:14:590
667922446,196sleep00-21swapper/010:20:290
20405223943,50sleep00-21swapper/011:05:080
15621223944,181sleep015336-21/usr/sbin/munin07:25:100
660022267,203sleep06671-21grep08:40:110
660022267,203sleep06671-21grep08:40:110
13764222620,89sleep00-21swapper/007:20:010
23816221920,173sleep323969-21unixbench_multi07:50:223
23816221920,173sleep323969-21unixbench_multi07:50:223
30613221733,145sleep030643-21munin-run08:14:460
30613221733,145sleep030643-21munin-run08:14:460
9609221115,179sleep09695-21latency_hist08:49:490
20678221012,98sleep020681-21cron09:24:460
16161221027,157sleep316290-21grep10:50:253
16161221027,157sleep316290-21grep10:50:253
474522007,157sleep0370-21snmpd11:56:460
15850219811,117sleep0370-21snmpd10:50:170
15850219811,117sleep0370-21snmpd10:50:170
29292219371,105sleep00-21swapper/009:50:190
31945218353,15sleep00-21swapper/011:40:140
247672181108,61sleep024851-21fschecks_time07:55:070
21342217344,115sleep021492-21latency_hist07:44:490
14609217212,139sleep014758-21tail10:45:250
14609217212,139sleep014758-21tail10:45:250
6916216218,130sleep36990-21nin-node06:59:483
690216023,81sleep00-21swapper/011:45:070
31464216018,99sleep030884-21/usr/sbin/munin08:15:190
12372215754,85sleep00-21swapper/008:55:590
805421567,12sleep00-21swapper/012:08:580
2501121530,7sleep30-21swapper/309:35:273
28480215010,95sleep028493-21apt-get09:49:470
1733214923,67sleep00-21swapper/010:05:180
17029214923,114sleep04092-1kworker/0:1H10:55:060
17029214923,114sleep04092-1kworker/0:1H10:55:060
26980214115,113sleep00-21swapper/011:25:110
6102213912,109sleep05592-21/usr/sbin/munin06:55:210
115821379,115sleep01169-21vmstat08:20:360
115821379,115sleep01169-21vmstat08:20:360
6874213650,67sleep26915-21unin-run06:59:472
3015921366,75sleep00-21swapper/011:35:130
8906213329,41sleep28917-21ls07:04:482
5548213339,69sleep00-21swapper/012:00:130
22047213127,27sleep00-21swapper/007:45:180
22047213127,27sleep00-21swapper/007:45:180
1330121319,76sleep00-21swapper/009:00:230
31866212715,86sleep00-21swapper/009:59:480
23586212721,86sleep323676-21grep11:15:093
1784421279,82sleep00-21swapper/009:14:570
7054212617,93sleep27123-21switchtime08:40:212
7054212617,93sleep27123-21switchtime08:40:212
13682212610,23sleep00-21swapper/010:44:480
22039212510,91sleep30-21swapper/307:45:173
22039212510,91sleep30-21swapper/307:45:173
17880212518,88sleep30-21swapper/309:15:083
20669212421,88sleep220794-21timerwakeupswit07:40:192
18424212428,85sleep0370-21snmpd07:34:560
26325212326,74sleep126420-21fschecks_time08:00:131
22546212321,86sleep222581-21cpu09:30:062
2137821209,100sleep021419-21ntpq09:25:150
1222121209,59sleep012004-21/usr/sbin/munin12:20:170
1222121209,59sleep012004-21/usr/sbin/munin12:20:170
868421190,7sleep30-21swapper/310:26:123
6493211923,78sleep35613-21apt-get10:20:153
2474221179,27sleep024861-21cat09:35:250
9610211531,65sleep09648-21grep07:05:190
9610211531,65sleep09648-21grep07:05:190
3247211416,22sleep00-21swapper/010:10:190
1732321149,51sleep00-21swapper/009:11:260
26694211219,75sleep20-21swapper/211:25:052
13585211213,84sleep113166-21/usr/sbin/munin09:00:281
7793211116,25sleep00-21swapper/010:25:090
23681211025,33sleep023740-21if_eth011:15:100
4402210921,73sleep14553-21unixbench_multi11:55:221
8211210820,63sleep18355-21apt-config12:09:481
25015210739,53sleep00-21swapper/011:20:020
1000210531,60sleep01144-21latency_hist10:04:490
32370210415,73sleep232558-21latency_hist08:19:492
20394210412,79sleep312066-21kworker/u8:311:05:033
5024210313,76sleep35071-21latency_hist11:59:483
17963210214,72sleep217973-21/usr/sbin/munin09:15:112
8613210019,62sleep28129-21apt-get12:09:572
8613210019,62sleep28129-21apt-get12:09:572
5906210010,19sleep10-21swapper/110:20:051
21461210015,69sleep321481-21/usr/sbin/munin09:25:183
817129915,69sleep18195-21proc_pri10:25:181
565929916,20sleep10-21swapper/110:19:491
1088929927,60sleep010910-21fschecks_count10:35:110
891929815,66sleep39065-21date07:04:493
1170129815,46sleep00-21swapper/012:19:480
58952977,54sleep0370-21snmpd08:36:030
31332965,35sleep00-21swapper/011:51:000
982529521,14sleep00-21swapper/008:50:010
982529521,14sleep00-21swapper/008:50:010
3090529517,40sleep30-21swapper/308:15:023
2749229516,43sleep20-21swapper/208:04:472
193029520,54sleep12018-21latency_hist11:49:481
1524129527,16sleep00-21swapper/009:05:160
2251829442,17sleep0370-21snmpd09:29:590
1843729413,32sleep118454-21apt-get07:35:051
1079129421,58sleep210870-21threads12:15:182
662129329,36sleep30-21swapper/312:04:493
106652937,39sleep00-21swapper/007:10:000
2499729214,19sleep20-21swapper/209:35:272
1833129227,30sleep10-21swapper/110:59:481
1833129227,30sleep10-21swapper/110:59:481
466329118,58sleep04810-21cat06:50:170
86652905,28sleep00-21swapper/007:02:400
713529018,47sleep37258-21wc08:40:223
713529018,47sleep37258-21wc08:40:223
420328928,28sleep20-21swapper/206:50:072
289162896,48sleep028992-21awk11:30:280
170028819,19sleep30-21swapper/310:05:173
140612887,56sleep20-21swapper/209:00:372
972128620,49sleep19722-21sshd12:12:261
972128620,49sleep19722-21sshd12:12:261
409028627,20sleep10-21swapper/110:14:481
2366828614,54sleep123111-21/usr/sbin/munin07:50:201
2366828614,54sleep123111-21/usr/sbin/munin07:50:201
2867228516,48sleep328879-21latency_hist09:49:503
83602830,10sleep20-21swapper/212:09:482
230432838,36sleep00-21swapper/011:12:430
1042228320,45sleep310505-21switchtime08:50:203
1042228320,45sleep310505-21switchtime08:50:203
187728212,54sleep31892-21idleruntime-cro11:49:463
1149728212,51sleep111506-21modprobe12:19:451
107628220,39sleep20-21swapper/211:45:172
307428124,35sleep10-21swapper/108:29:461
307428124,35sleep10-21swapper/108:29:461
1535828117,37sleep315355-21apt-get07:25:013
1253128117,29sleep20-21swapper/210:40:102
103592815,42sleep00-21swapper/010:32:490
468628018,46sleep34700-21/usr/sbin/munin10:15:103
1971028024,27sleep30-21swapper/307:39:473
169782807,16sleep00-21swapper/007:30:070
535927923,26sleep10-21swapper/112:00:101
1443227912,52sleep37166-21sshd09:04:263
1056227817,25sleep30-21swapper/310:34:493
3229127717,21sleep20-21swapper/210:00:112
1602427724,28sleep10-21swapper/107:25:191
829827621,35sleep38286-21apt-get08:45:113
3206327621,29sleep30-21swapper/311:40:173
2203227625,26sleep10-21swapper/111:10:101
1462827615,28sleep10-21swapper/107:20:231
1210927611,49sleep112207-21cat10:39:481
418727535,18sleep20-21swapper/211:55:192
3003427523,24sleep20-21swapper/211:35:112
2854927528,27sleep20-21swapper/209:49:492
1874127534,22sleep30-21swapper/311:00:143
92027423,33sleep10-21swapper/110:04:461
448227423,23sleep10-21swapper/106:50:141
2629327411,46sleep326357-21sendmail-msp11:24:463
2557327315,27sleep20-21swapper/211:20:212
2385427323,26sleep10-21swapper/109:34:491
1997927322,21sleep10-21swapper/111:04:491
1816227323,29sleep30-21swapper/307:34:463
922827221,35sleep20-21swapper/210:30:002
1918327220,32sleep219262-21timerandwakeup11:00:252
1821427217,41sleep118291-21latency_hist07:34:491
409727117,25sleep20-21swapper/208:30:242
3068527128,25sleep30-21swapper/311:35:243
304212717,51sleep10-21swapper/108:12:171
304212717,51sleep10-21swapper/108:12:171
2848727113,38sleep228491-21/usr/sbin/munin11:30:102
169082717,46sleep20-21swapper/207:30:052
1669427124,33sleep116708-21netstat09:10:131
1536227114,36sleep215425-21/usr/sbin/munin07:25:042
1118327115,12sleep10-21swapper/107:10:171
533227026,22sleep30-21swapper/306:54:483
305427013,39sleep33075-21sed10:10:143
2661127016,40sleep326612-21cpu11:25:023
2446527018,23sleep10-21swapper/109:35:171
225626914,31sleep10-21swapper/111:50:081
1543726917,25sleep30-21swapper/309:05:193
1116126915,36sleep211139-21latency_hist08:54:472
1116126915,36sleep211139-21latency_hist08:54:472
816426821,23sleep10-21swapper/107:00:191
2976426822,24sleep10-21swapper/109:50:281
2528726814,35sleep325393-21cat07:55:183
2522226810,21sleep10-21swapper/111:20:111
1087126816,29sleep310168-21/usr/sbin/munin12:15:193
3026526716,38sleep31-21systemd09:54:463
251226720,18sleep30-21swapper/308:25:283
251226720,18sleep30-21swapper/308:25:283
216102675,11sleep20-21swapper/211:09:142
318132667,17sleep10-21swapper/109:59:451
2981326619,32sleep329819-21ntp_states08:10:173
2981326619,32sleep329819-21ntp_states08:10:173
283622668,44sleep30-21swapper/311:29:553
2625426619,20sleep30-21swapper/309:40:203
866426515,32sleep28758-21/usr/sbin/munin08:46:192
28332655,48sleep00-21swapper/008:26:380
28332655,48sleep00-21swapper/008:26:380
229112656,44sleep30-21swapper/311:11:053
1517826519,20sleep10-21swapper/109:05:131
544326423,23sleep20-21swapper/208:35:292
441626419,22sleep24421-21entropy10:15:052
150822646,10sleep30-21swapper/310:48:273
150822646,10sleep30-21swapper/310:48:273
1361026420,26sleep31-21systemd10:44:473
1194726419,28sleep211986-21modprobe07:14:462
994726316,30sleep110050-21fschecks_time08:50:101
994726316,30sleep110050-21fschecks_time08:50:101
811526320,27sleep38141-21munin-run12:09:453
810326312,19sleep20-21swapper/210:25:152
616526313,30sleep16234-21/usr/sbin/munin06:55:221
52826323,19sleep20-21swapper/208:20:262
52826323,19sleep20-21swapper/208:20:262
518926312,13sleep30-21swapper/308:35:223
289842635,17sleep10-21swapper/108:08:411
1999626318,21sleep20-21swapper/209:20:202
160832638,13sleep20-21swapper/210:50:242
160832638,13sleep20-21swapper/210:50:242
1483126317,29sleep20-21swapper/210:45:272
1483126317,29sleep20-21swapper/210:45:272
1405726321,22sleep30-21swapper/307:20:123
1259726317,21sleep10-21swapper/112:20:261
1259726317,21sleep10-21swapper/112:20:261
1212326328,18sleep20-21swapper/208:55:202
2262726220,22sleep20-21swapper/211:10:232
163002626,17sleep30-21swapper/309:09:593
3149126120,23sleep131542-21cpuspeed11:40:031
3099226118,24sleep10-21swapper/108:15:081
2521926111,34sleep225286-21/usr/sbin/munin07:55:172
2289326124,19sleep20-21swapper/207:49:482
2289326124,19sleep20-21swapper/207:49:482
1828026113,27sleep30-21swapper/310:59:463
1828026113,27sleep30-21swapper/310:59:463
1307426118,18sleep10-21swapper/110:40:211
1232526119,25sleep30-21swapper/307:15:083
1133126116,20sleep30-21swapper/307:10:193
296372606,16sleep10-21swapper/111:34:331
2660926017,26sleep126604-21apt-get11:25:001
2406426018,20sleep10-21swapper/111:15:191
2057726019,27sleep120668-21sed07:40:181
153826022,21sleep30-21swapper/308:24:473
153826022,21sleep30-21swapper/308:24:473
97882598,13sleep10-21swapper/110:30:191
23122596,12sleep10-21swapper/110:07:251
1845225915,27sleep10-21swapper/109:15:221
117872597,10sleep30-21swapper/310:36:193
2349225816,25sleep223486-21apt-get11:15:032
1638025818,17sleep20-21swapper/209:10:062
913525716,25sleep30-21swapper/312:10:303
913525716,25sleep30-21swapper/312:10:303
575425719,22sleep25818-21find12:00:192
505425715,19sleep10-21swapper/108:35:201
2612425714,22sleep10-21swapper/109:40:181
2380125719,22sleep30-21swapper/309:34:473
1603525715,23sleep20-21swapper/209:09:462
1431625716,19sleep20-21swapper/207:20:202
1210925717,25sleep212113-21diskstats12:20:072
1210925717,25sleep212113-21diskstats12:20:072
942125613,18sleep30-21swapper/307:05:143
942125613,18sleep30-21swapper/307:05:143
896525619,16sleep10-21swapper/108:46:261
413125614,27sleep24059-21latency_hist10:14:492
382725617,23sleep33828-21ntp_states08:30:173
1942025616,24sleep319415-21apt-get09:20:053
914725513,25sleep19141-21apt-get07:05:041
914725513,25sleep19141-21apt-get07:05:041
3128025521,17sleep20-21swapper/209:55:222
3095325517,23sleep130955-21sshd11:36:351
2781525515,23sleep327811-21apt-get08:05:043
189725516,21sleep21903-21diskstats08:25:072
189725516,21sleep21903-21diskstats08:25:072
1429725511,31sleep114342-21awk10:45:181
1429725511,31sleep114342-21awk10:45:181
1968125418,21sleep219682-21sshd07:39:262
1186225415,24sleep111868-21ntp_states08:55:151
1140425415,22sleep311402-21apt-get08:55:013
647825313,20sleep10-21swapper/108:40:041
647825313,20sleep10-21swapper/108:40:041
389625315,22sleep10-21swapper/108:30:191
2901225314,23sleep229079-21df09:50:052
265525319,15sleep30-21swapper/311:50:193
2572225315,16sleep20-21swapper/209:40:072
2321625313,24sleep223220-21diskstats07:50:082
2321625313,24sleep223220-21diskstats07:50:082
214725312,17sleep20-21swapper/211:50:012
2107325316,17sleep10-21swapper/109:25:081
1702625317,20sleep217022-21apt-get10:55:042
1702625317,20sleep217022-21apt-get10:55:042
1212725317,20sleep312219-21perl12:20:163
1212725317,20sleep312219-21perl12:20:163
3062725214,23sleep330634-21df_inode09:55:073
2210225215,15sleep10-21swapper/107:45:191
2210225215,15sleep10-21swapper/107:45:191
1548025211,24sleep115475-21apt-get10:50:061
1548025211,24sleep115475-21apt-get10:50:061
2960925110,16sleep20-21swapper/208:10:122
2960925110,16sleep20-21swapper/208:10:122
2009625116,16sleep30-21swapper/307:40:083
2781925013,21sleep20-21swapper/208:05:072
2673525012,17sleep30-21swapper/308:00:233
2068925012,20sleep1286-21dbus-daemon09:24:471
2036025013,22sleep119886-21apt-get11:04:531
1080225014,18sleep20-21swapper/210:35:052
1045225013,15sleep20-21swapper/207:09:492
1045225013,15sleep20-21swapper/207:09:492
597824917,18sleep20-21swapper/210:20:062
126612498,16sleep20-21swapper/207:15:152
1243024917,21sleep112216-21/usr/sbin/munin07:15:101
326942489,16sleep10-21swapper/108:20:051
326942489,16sleep10-21swapper/108:20:051
3252924814,19sleep332561-21ntp_states10:00:163
3188624815,18sleep20-21swapper/211:40:122
2161424519,14sleep20-21swapper/209:25:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional