You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Jul 12, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2345222607,2238sleep01405-21/usr/sbin/munin22:29:320
2345222607,2238sleep01405-21/usr/sbin/munin22:29:320
4810210555,1039sleep031061-1kworker/0:1H22:39:460
1178276919,735sleep31210-21/usr/sbin/munin20:44:283
17610241315,377sleep017761-21unixbench_singl19:54:340
199812371227,23sleep00-21swapper/021:44:220
18575232512,234sleep018643-21ls21:39:250
2133523046,29sleep00-21swapper/020:06:500
258722303134,150sleep025811-21latency_hist22:04:000
18041228718,257sleep118113-21wc23:19:331
28048226611,239sleep228158-21cat20:29:012
28048226611,239sleep228158-21cat20:29:012
11314225882,30sleep00-21swapper/021:18:580
32242224038,187sleep032373-21timerwakeupswit20:39:290
9450223521,172sleep00-21swapper/022:54:160
1562222744,165sleep21637-21perl22:29:182
1562222744,165sleep21637-21perl22:29:182
28344221713,153sleep028444-21fschecks_time20:29:190
4400220135,49sleep24419-21apt-get19:14:102
2666922017,175sleep031061-1kworker/0:1H20:24:100
2666922017,175sleep031061-1kworker/0:1H20:24:100
28821219814,172sleep228904-21sed20:29:292
24357219374,105sleep024481-21timerwakeupswit23:39:310
586721929,169sleep05948-21mailstats20:59:320
78521876,19sleep00-21swapper/020:44:170
976321868,100sleep09799-21basename21:13:570
12790218111,149sleep012809-21apt-get23:04:090
2679321809,145sleep00-21swapper/023:49:080
19937218014,146sleep020003-21systemd-run20:03:590
9208217919,140sleep00-21swapper/022:54:000
3137721798,27sleep00-21swapper/020:39:000
3244021750,37sleep232444-21fschecks_count22:24:212
30227217517,18sleep00-21swapper/018:54:090
1255921716,13sleep00-21swapper/021:20:070
1255921716,13sleep00-21swapper/021:20:070
31738216883,17sleep00-21swapper/018:59:010
31738216883,17sleep00-21swapper/018:59:010
78821638,140sleep0832-21basename00:09:230
5127216213,129sleep00-21swapper/020:58:590
2185421560,6sleep20-21swapper/221:49:232
1472321567,73sleep014759-21latency_hist21:28:590
6852215419,18sleep20-21swapper/222:44:292
23337215434,108sleep023331-21apt-get20:14:130
4188215313,122sleep00-21swapper/019:13:590
76512151114,23sleep07671-21latency_hist22:48:590
1737421518,12sleep20-21swapper/221:34:432
1737421518,12sleep20-21swapper/221:34:432
21918215020,110sleep121993-21mailstats21:49:271
2335121495,69sleep00-21swapper/023:38:260
11263214615,76sleep011286-21kernelversion22:59:210
822221459,18sleep00-21swapper/019:24:340
822221459,18sleep00-21swapper/019:24:340
1995521435,64sleep00-21swapper/023:24:330
21064214011,52sleep021124-21find23:29:250
2890221399,48sleep07166-21sshd23:54:290
1795421399,40sleep00-21swapper/023:19:320
7237213816,99sleep27310-21unin-run19:23:592
2873921387,57sleep00-21swapper/022:11:390
2873921387,57sleep00-21swapper/022:11:390
21305213820,98sleep321445-21unixbench_singl23:29:283
450121367,16sleep00-21swapper/019:14:160
2988321366,57sleep00-21swapper/023:59:010
2988321366,57sleep00-21swapper/023:59:010
19922213630,90sleep319931-21kworker/u8:120:03:583
827021347,42sleep00-21swapper/021:09:000
423521348,110sleep04248-21apt-get22:38:570
119821338,44sleep01350-21date22:29:000
24645213217,93sleep324730-21cat20:19:003
1636021329,107sleep016403-21munin-run21:34:000
1636021329,107sleep016403-21munin-run21:34:000
1276921316,87sleep00-21swapper/019:39:320
30435213013,98sleep130009-21/usr/sbin/munin20:34:411
1195121308,66sleep011952-21munin-run19:38:590
6550212916,65sleep00-21swapper/019:19:290
2920621298,45sleep029198-21apt-get22:14:140
3551212810,40sleep00-21swapper/020:53:580
29678212822,79sleep20-21swapper/223:58:592
23100212872,41sleep023239-21latency_hist20:14:000
32004212526,79sleep132070-21sendmail_mailqu00:04:361
32004212526,79sleep132070-21sendmail_mailqu00:04:361
27434212415,94sleep027582-21cat22:09:010
22199212310,59sleep00-21swapper/021:49:300
1425221238,98sleep00-21swapper/019:44:280
1425221238,98sleep00-21swapper/019:44:280
30429212017,85sleep30-21swapper/323:59:293
30429212017,85sleep30-21swapper/323:59:293
25291212012,94sleep325332-21ntp_offset20:19:243
25291212012,94sleep325332-21ntp_offset20:19:243
2427921207,37sleep00-21swapper/021:58:590
3523211918,81sleep33546-21sh20:53:573
3085021196,31sleep00-21swapper/022:19:190
16579211913,86sleep031061-1kworker/0:1H19:53:570
24595211813,88sleep10-21swapper/120:18:581
9156211720,32sleep29185-21munin-run22:53:582
2496421176,33sleep00-21swapper/020:19:170
2496421176,33sleep00-21swapper/020:19:170
3177321160,38sleep231777-21fschecks_time20:39:172
159221167,29sleep00-21swapper/019:04:220
159221167,29sleep00-21swapper/019:04:220
29122211516,82sleep029128-21ntp_states18:49:250
7673211321,72sleep17813-21latency_hist22:49:001
368221136,29sleep00-21swapper/000:19:090
368221136,29sleep00-21swapper/000:19:090
20435211319,80sleep20-21swapper/220:04:252
957621127,28sleep00-21swapper/019:29:300
957621127,28sleep00-21swapper/019:29:300
10790211214,82sleep110855-21fw_packets19:34:201
1490021105,27sleep00-21swapper/023:09:260
2523321095,28sleep00-21swapper/023:44:030
1684621095,29sleep00-21swapper/023:14:480
3263921086,27sleep00-21swapper/000:07:170
3263921086,27sleep00-21swapper/000:07:170
24505210813,79sleep124524-21apt-get21:59:121
16974210845,51sleep017018-21cat21:34:320
16974210845,51sleep017018-21cat21:34:320
25267210010,17sleep20-21swapper/223:44:132
3216429925,56sleep332237-21switchtime20:39:283
163929917,68sleep31694-21interrupts22:29:193
163929917,68sleep31694-21interrupts22:29:193
495729817,68sleep25014-21cut22:40:132
2863829813,68sleep128697-21latency23:54:221
2992529627,43sleep129928-21cron18:53:571
511829417,64sleep35233-21latency_hist00:24:003
511829417,64sleep35233-21latency_hist00:24:003
331129418,28sleep00-21swapper/000:17:350
1533129414,66sleep215349-21netstat21:29:222
1533129414,66sleep215349-21netstat21:29:222
2823929219,20sleep20-21swapper/222:09:272
2823929219,20sleep20-21swapper/222:09:272
102429216,55sleep1421-21/usr/sbin/munin00:09:281
1663229113,66sleep316677-21latency_hist19:54:003
3267729018,15sleep30-21swapper/318:59:293
3267729018,15sleep30-21swapper/318:59:293
2929629024,38sleep329360-21fschecks_time22:14:213
1431929022,21sleep10-21swapper/119:44:291
1431929022,21sleep10-21swapper/119:44:291
108529018,56sleep21115-21latency_hist19:04:002
108529018,56sleep21115-21latency_hist19:04:002
297242890,20sleep112-21rcu_sched20:33:591
3155428719,16sleep20-21swapper/218:58:592
3012728615,57sleep230154-21fschecks_count20:34:182
503028514,39sleep10-21swapper/119:14:291
1497128515,56sleep115070-21sed23:09:271
1226228514,18sleep10-21swapper/123:03:581
2993028422,44sleep229995-21unin-run18:53:582
2178828312,50sleep10-21swapper/120:09:131
201028238,27sleep10-21swapper/119:04:311
201028238,27sleep10-21swapper/119:04:311
413428121,42sleep24136-21cron19:13:572
2948028120,48sleep229523-21cat22:14:242
277328114,55sleep12850-21mailstats20:49:271
2264228123,39sleep322731-21mailstats23:34:293
191212815,58sleep20-21swapper/219:59:292
515628017,16sleep10-21swapper/120:59:001
2699028013,52sleep327009-21kernelversion20:24:423
2699028013,52sleep327009-21kernelversion20:24:423
2119428020,28sleep20-21swapper/221:49:002
127428017,41sleep31293-21find00:09:343
1146328016,44sleep210970-21/usr/sbin/munin22:59:272
952727918,44sleep39425-21/usr/sbin/munin22:54:193
660727918,47sleep36627-21cut22:44:213
605927926,30sleep10-21swapper/119:19:171
2548427916,28sleep30-21swapper/323:44:263
2890527827,27sleep30-21swapper/320:29:303
2741627828,26sleep30-21swapper/322:09:003
2525327814,47sleep225290-21ntp_kernel_err20:19:232
2525327814,47sleep225290-21ntp_kernel_err20:19:232
2189127816,16sleep20-21swapper/223:34:002
1878227817,43sleep218840-21/usr/sbin/munin21:39:312
1043927820,42sleep210479-21ntp_kernel_pll_21:14:232
410127720,13sleep30-21swapper/320:54:223
859327610,19sleep20-21swapper/221:09:192
2992027626,23sleep10-21swapper/123:59:141
2992027626,23sleep10-21swapper/123:59:141
2493627618,41sleep224992-21tr23:43:572
1648827622,26sleep10-21swapper/123:14:281
1745227522,26sleep30-21swapper/321:34:443
1745227522,26sleep30-21swapper/321:34:443
1618727513,47sleep216280-21memory23:14:212
1228527513,39sleep112436-21unixbench_multi21:19:311
1228527513,39sleep112436-21unixbench_multi21:19:311
3085827419,28sleep330944-21ls18:54:283
2620827424,20sleep20-21swapper/222:04:182
1415727421,25sleep10-21swapper/121:24:521
1341927420,30sleep30-21swapper/321:24:143
907327324,29sleep39092-21apt-get19:29:133
907327324,29sleep39092-21apt-get19:29:133
88327319,33sleep2421-21/usr/sbin/munin00:09:252
2719227330,25sleep10-21swapper/120:24:481
2719227330,25sleep10-21swapper/120:24:481
215027311,43sleep20-21swapper/200:14:142
1577027319,27sleep10-21swapper/119:49:261
1483327311,48sleep314897-21ps23:09:253
3213927225,25sleep30-21swapper/300:04:383
3213927225,25sleep30-21swapper/300:04:383
2383127214,42sleep123892-21fw_packets23:39:201
192722727,12sleep10-21swapper/119:59:321
955327115,41sleep29559-21ntp_states19:29:282
955327115,41sleep29559-21ntp_states19:29:282
417027114,27sleep20-21swapper/200:19:272
417027114,27sleep20-21swapper/200:19:272
1599927121,26sleep20-21swapper/219:49:312
3042027012,42sleep130421-21sshd22:18:191
2619627011,12sleep10-21swapper/122:04:171
1416727013,25sleep30-21swapper/323:09:013
1930326929,20sleep20-21swapper/223:24:192
1879726920,35sleep318746-21latency_hist23:23:593
1826226914,39sleep218347-21apt-key19:59:002
1439226917,35sleep214510-21timerwakeupswit19:44:302
1439226917,35sleep214510-21timerwakeupswit19:44:302
1243926919,21sleep20-21swapper/221:19:322
1243926919,21sleep20-21swapper/221:19:322
646326821,30sleep36545-21mailstats19:19:283
47692687,39sleep30-21swapper/322:39:173
326552688,13sleep10-21swapper/118:59:271
326552688,13sleep10-21swapper/118:59:271
246232686,43sleep20-21swapper/220:18:592
884726715,32sleep28865-21grep19:29:002
884726715,32sleep28865-21grep19:29:002
434926723,21sleep20-21swapper/222:38:592
336626724,19sleep10-21swapper/119:09:281
30226719,32sleep2303-21cron00:08:572
30226719,32sleep2303-21cron00:08:572
2789326712,40sleep127894-21sshd23:51:351
1514626719,35sleep315209-21cat21:29:183
1514626719,35sleep315209-21cat21:29:183
140426719,27sleep10-21swapper/122:29:091
140426719,27sleep10-21swapper/122:29:091
1176426712,36sleep20-21swapper/219:37:122
85732666,51sleep10-21swapper/119:26:001
85732666,51sleep10-21swapper/119:26:001
821826621,23sleep18231-21idleruntime-cro21:08:571
3134426617,25sleep30-21swapper/320:38:593
1962526625,26sleep1286-21dbus-daemon21:43:581
1472526611,41sleep214738-21grep23:09:202
131022667,17sleep20-21swapper/223:04:362
1040526619,21sleep112-21rcu_sched19:33:591
1040526619,21sleep112-21rcu_sched19:33:591
664026523,26sleep26667-21munin-run21:03:582
489326515,27sleep30-21swapper/319:14:273
3209926521,29sleep132156-21cat22:24:001
2999626518,22sleep30-21swapper/318:53:593
2359926517,28sleep30-21swapper/321:54:303
2222126520,30sleep122255-21sshd23:34:191
787126420,23sleep10-21swapper/122:49:081
341826420,23sleep10-21swapper/100:18:581
2614126421,27sleep126168-21/usr/sbin/munin20:23:591
2614126421,27sleep126168-21/usr/sbin/munin20:23:591
244162645,45sleep20-21swapper/221:59:012
159422645,11sleep10-21swapper/121:30:221
159422645,11sleep10-21swapper/121:30:221
1508226419,29sleep315211-21latency_hist19:49:003
1508226419,29sleep315211-21latency_hist19:49:003
1181526420,28sleep311872-21/usr/sbin/munin21:19:213
1181526420,28sleep311872-21/usr/sbin/munin21:19:213
2160326316,20sleep10-21swapper/120:09:011
2080426322,19sleep30-21swapper/320:04:343
1817226314,32sleep118260-21systemd-run19:58:591
1106426318,18sleep30-21swapper/322:59:163
341326111,31sleep13437-21ntp_states22:34:261
2009026118,22sleep10-21swapper/121:44:231
142926116,27sleep21534-21fschecks_time19:04:202
142926116,27sleep21534-21fschecks_time19:04:202
98926020,18sleep20-21swapper/220:44:232
610426022,22sleep10-21swapper/122:43:591
278826018,19sleep30-21swapper/300:14:393
2739826016,21sleep30-21swapper/323:49:313
2387826014,19sleep10-21swapper/121:54:321
2353526015,18sleep20-21swapper/221:54:292
594925921,23sleep16022-21sed20:59:331
309192596,12sleep20-21swapper/200:00:332
309192596,12sleep20-21swapper/200:00:332
2268825917,22sleep30-21swapper/321:53:583
1070325918,25sleep110708-21modprobe22:58:571
1061625913,20sleep30-21swapper/321:14:293
538025819,25sleep35449-21df20:59:183
3254925816,27sleep332599-21/usr/sbin/munin22:24:233
3062325812,27sleep130656-21ntp_kernel_err18:54:221
280552589,33sleep128101-21ntpq22:09:221
280552589,33sleep128101-21ntpq22:09:221
1746625813,27sleep217459-21apt-get23:19:182
843225724,16sleep30-21swapper/322:49:263
2058525715,25sleep120580-21apt-get23:29:101
1903525714,24sleep319030-21apt-get23:24:113
1834825714,28sleep318555-21cut19:59:013
755225615,25sleep37548-21apt-get19:24:163
755225615,25sleep37548-21apt-get19:24:163
745125612,18sleep30-21swapper/321:04:313
315125614,27sleep33243-21load19:09:223
114725614,26sleep11170-21ntp_states20:44:261
438525517,18sleep10-21swapper/100:19:301
438525517,18sleep10-21swapper/100:19:301
3195825517,20sleep10-21swapper/120:39:221
3242725414,16sleep10-21swapper/122:24:211
223525420,20sleep30-21swapper/319:04:343
223525420,20sleep30-21swapper/319:04:343
2101625415,20sleep30-21swapper/321:47:193
2030025412,25sleep120341-21systemd-run23:28:581
1591425417,22sleep315981-21df23:14:143
1387525412,24sleep213899-21ntp_states21:24:452
1296525417,16sleep20-21swapper/219:39:342
706125310,29sleep27155-21fschecks_time21:04:222
2921925313,22sleep329272-21users23:54:323
2395125315,22sleep323953-21irqstats23:39:223
2308925312,24sleep323098-21munin-run20:13:593
1684125313,14sleep10-21swapper/121:34:211
1684125313,14sleep10-21swapper/121:34:211
1005625314,18sleep10-21swapper/121:14:141
850725212,24sleep18502-21apt-get21:09:141
3103125215,23sleep331110-21sed22:19:223
2460225213,22sleep324609-21df_inode21:59:193
1240225211,15sleep30-21swapper/319:39:233
1070925213,26sleep210753-21munin-run22:58:582
298325111,26sleep30-21swapper/322:34:163
2899425114,21sleep329146-21sed22:14:013
2899425114,21sleep329146-21sed22:14:013
2344525116,19sleep223418-21latency_hist23:39:002
1884125115,17sleep30-21swapper/321:39:323
117302515,34sleep30-21swapper/319:36:443
850925015,19sleep30-21swapper/321:09:163
2681625012,22sleep226813-21apt-get23:49:102
255592509,14sleep10-21swapper/123:44:291
2177125013,22sleep221782-21apt-get20:09:102
285124911,15sleep20-21swapper/220:49:282
1216824811,23sleep10-21swapper/119:39:121
425824612,17sleep20-21swapper/220:54:252
1323924310,14sleep10-21swapper/123:04:391
317962429,20sleep2370-21snmpd22:21:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional