You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 19:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Nov 22, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16448210187,9sleep30-21swapper/307:05:043
18485210087,9sleep10-21swapper/107:08:431
1846228169,8sleep00-21swapper/007:08:260
1842427868,6sleep20-21swapper/207:07:562
18664992825,2cyclictest13-21ksoftirqd/009:25:000
1867099270,3cyclictest0-21swapper/210:30:162
1866899270,2cyclictest0-21swapper/110:07:151
18674992625,1cyclictest0-21swapper/312:03:183
1866499235,14cyclictest13-21ksoftirqd/012:22:030
18674992221,1cyclictest0-21swapper/312:20:583
1867499211,19cyclictest26836-21kworker/u8:210:19:073
1867099210,0cyclictest0-21swapper/210:55:222
1867499202,1cyclictest0-21swapper/310:54:533
1867099209,10cyclictest20187-21sh09:45:182
1867099200,2cyclictest25113-21/usr/sbin/munin09:20:192
18668992018,0cyclictest141rcu_preempt11:31:331
1866499207,6cyclictest13466-21awk10:35:000
1866499206,8cyclictest13-21ksoftirqd/010:22:020
1866899192,1cyclictest141rcu_preempt10:47:481
1866899190,19cyclictest0-21swapper/111:54:311
1866899190,18cyclictest0-21swapper/112:17:291
18664991916,2cyclictest161rcuc/007:50:000
1867499180,15cyclictest0-21swapper/310:33:413
18670991817,1cyclictest1-21systemd07:55:242
18664991816,1cyclictest11725-21diskmemload10:15:340
18664991815,1cyclictest11725-21diskmemload12:33:360
18664991814,2cyclictest19686-21ssh09:45:000
1867499171,9cyclictest25092-21apt-get08:30:143
18674991716,1cyclictest0-21swapper/309:21:183
18670991716,1cyclictest37850irq/131-snd_hda12:10:192
18670991716,1cyclictest0-21swapper/210:35:292
1867099170,17cyclictest29145-21kworker/2:007:43:262
1867099170,16cyclictest0-21swapper/212:29:292
18668991713,4cyclictest221irq_work/109:28:421
18664991713,3cyclictest11725-21diskmemload09:16:170
18674991615,1cyclictest0-21swapper/311:49:583
1867099164,8cyclictest4141-21munin-plugin-st11:20:002
18670991615,1cyclictest12836-21kworker/2:208:05:152
18670991615,1cyclictest0-21swapper/210:10:322
1867099161,1cyclictest0-21swapper/209:26:372
1867099160,1cyclictest0-21swapper/211:35:512
1867099160,0cyclictest0-21swapper/209:51:452
1866899163,3cyclictest7494-1kworker/u9:210:31:541
1866899161,7cyclictest28419-21ssh12:30:201
1866499162,6cyclictest13-21ksoftirqd/012:39:220
18664991615,1cyclictest13-21ksoftirqd/011:10:040
18664991613,2cyclictest11725-21diskmemload11:18:420
18664991612,3cyclictest13-21ksoftirqd/012:05:040
1866499160,4cyclictest0-21swapper/008:25:150
1867099154,10cyclictest5469-21/usr/sbin/munin11:20:212
18670991513,1cyclictest28639-21ssh12:05:462
1867099151,14cyclictest29470-21kworker/2:108:49:182
1867099151,14cyclictest29470-21kworker/2:108:49:172
1867099151,14cyclictest10255-21kworker/2:212:18:492
18670991511,3cyclictest26598-21cat08:30:272
1867099150,15cyclictest0-21swapper/211:12:212
1866899158,5cyclictest25-21ksoftirqd/108:30:001
18668991510,3cyclictest1977-21idleruntime-cro08:50:001
18668991510,3cyclictest1977-21idleruntime-cro08:50:001
1866499153,4cyclictest1046-21kworker/u8:209:32:040
1866499152,7cyclictest28251-21idleruntime-cro07:30:010
18664991513,2cyclictest13-21ksoftirqd/011:43:330
1867499149,3cyclictest2251-21tr12:35:173
1867499149,3cyclictest13736-21cron09:40:003
18674991411,2cyclictest2917-21head10:25:183
18674991410,3cyclictest20749-21tr10:40:163
18674991410,3cyclictest14418-21cron11:55:003
18674991410,2cyclictest10389-21cut09:05:323
18670991411,2cyclictest24387-21grep08:25:282
18668991413,1cyclictest0-21swapper/112:25:131
18668991411,2cyclictest0-21swapper/109:25:011
18668991410,3cyclictest25-21ksoftirqd/108:00:001
18668991410,3cyclictest11076-21grep08:00:151
1866899140,1cyclictest0-21swapper/110:50:271
1866899140,12cyclictest2632-21ssh11:45:201
1866499149,4cyclictest13-21ksoftirqd/011:23:040
1866499149,3cyclictest28850-21wc11:40:000
1866499146,2cyclictest11725-21diskmemload09:58:470
1866499145,5cyclictest224-21systemd-journal10:15:000
1866499144,9cyclictest0-21swapper/009:47:150
1866499142,3cyclictest0-21swapper/008:46:450
1866499142,3cyclictest0-21swapper/008:46:450
18664991413,1cyclictest161rcuc/011:05:500
18664991410,3cyclictest3479-21grep08:50:220
1866499140,13cyclictest26690-21wc09:50:300
1866499140,0cyclictest0-21swapper/010:49:270
1867499139,3cyclictest2641-21ls09:30:003
1867499139,3cyclictest25148-21grep11:10:173
1867499139,2cyclictest5092-21cut11:20:183
1867499131,1cyclictest141rcu_preempt10:47:353
18674991310,2cyclictest6780-21grep10:00:283
18674991310,2cyclictest5952-21awk08:55:233
18674991310,2cyclictest30670-21sed09:55:003
18674991310,2cyclictest30623-21grep08:40:183
18674991310,2cyclictest2474-21grep07:40:283
18674991310,2cyclictest23976-21wc08:25:233
18674991310,2cyclictest20375-21grep11:05:233
1867499130,1cyclictest0-21swapper/312:25:183
1867499130,12cyclictest13675-21head11:00:173
1867099139,3cyclictest3331-21grep07:45:192
1867099139,2cyclictest27728-21cut10:45:202
1867099131,1cyclictest11725-21diskmemload10:40:592
1867099131,12cyclictest6707-21kworker/2:009:03:262
1867099131,12cyclictest23999-21cstates07:20:192
18670991310,2cyclictest3705-21tail11:45:252
18670991310,2cyclictest1465-21apt12:35:002
1866899138,3cyclictest26511-21cut12:05:151
1866899135,6cyclictest21008-21cron07:14:591
1866899134,4cyclictest224-21systemd-journal11:55:531
1866899131,1cyclictest141rcu_preempt10:14:421
1866899131,0cyclictest141rcu_preempt10:24:251
18668991310,2cyclictest20218-21grep08:20:001
18668991310,2cyclictest16048-21df08:10:181
1866899130,3cyclictest0-21swapper/109:54:351
1866499136,5cyclictest8340-21sh07:55:010
1866499133,7cyclictest11725-21diskmemload09:25:490
18664991311,1cyclictest11725-21diskmemload11:58:390
18664991310,3cyclictest13-21ksoftirqd/010:35:210
18664991310,2cyclictest31242-21wc08:40:240
18664991310,2cyclictest29091-21wc08:35:430
18664991310,2cyclictest16965-21ssh11:30:150
18664991310,2cyclictest11302-21tail09:05:410
1866499130,13cyclictest0-21swapper/010:42:370
1867499129,2cyclictest6564-21grep07:50:223
1867499129,2cyclictest31790-21cron12:09:593
1867499129,2cyclictest20965-21grep08:20:193
1867499129,2cyclictest20328-21idleruntime-cro12:00:003
1867499129,2cyclictest19634-21head07:10:193
1867499129,2cyclictest17968-21tr08:15:143
1867499129,2cyclictest11859-21wc08:00:203
1867499128,3cyclictest16029-21sed09:40:283
1867499120,2cyclictest224-21systemd-journal07:34:563
1867499120,11cyclictest4130-21cron11:20:003
1867099129,2cyclictest30190-21tr08:40:162
1867099129,2cyclictest2964-21grep08:50:172
1867099129,2cyclictest28218-21cron07:30:012
1867099129,2cyclictest20590-21sed07:10:282
1867099129,2cyclictest18791-21ssh09:15:192
1867099128,1cyclictest32101-21tr07:35:232
1867099120,12cyclictest0-21swapper/209:55:152
1867099120,11cyclictest18527-21grep11:30:282
1866899129,2cyclictest3893-21sed10:25:281
1866899129,2cyclictest3864-21head09:30:181
1866899129,2cyclictest26991-21uname07:25:201
1866899129,2cyclictest21887-21cat12:00:201
1866899129,2cyclictest20594-21cut08:20:171
1866899129,2cyclictest1990-21grep12:35:161
1866899129,2cyclictest12479-21grep09:10:141
1866899127,1cyclictest25-21ksoftirqd/110:02:211
1866899121,2cyclictest141rcu_preempt11:13:351
18668991211,1cyclictest0-21swapper/107:30:141
18668991210,1cyclictest25-21ksoftirqd/111:16:121
1866899120,11cyclictest13973-21grep12:20:151
1866899120,0cyclictest0-21swapper/110:38:181
1866499129,2cyclictest7741-21tail09:00:180
1866499129,2cyclictest6723-21cron09:00:000
1866499129,2cyclictest22553-21cat08:25:000
1866499129,2cyclictest12469-21tail08:00:270
1866499129,2cyclictest11123-21sort11:25:190
1866499129,2cyclictest10828-21sed11:50:280
1866499128,2cyclictest161rcuc/007:35:130
1866499128,2cyclictest13016-21cut09:10:170
1866499127,2cyclictest0-21swapper/010:28:250
1866499126,4cyclictest13-21ksoftirqd/012:29:310
1866499125,7cyclictest11725-21diskmemload10:08:260
1866499122,9cyclictest7146-21cstates10:55:160
18664991211,1cyclictest13-21ksoftirqd/012:03:080
18664991210,1cyclictest12885-21apache_volume11:00:130
1866499120,1cyclictest0-21swapper/012:11:090
1867499118,2cyclictest9321-21cat09:05:013
1867499118,2cyclictest650-21hddtemp_smartct08:45:183
1867499118,2cyclictest650-21hddtemp_smartct08:45:183
1867499118,2cyclictest32196-21ssh12:10:103
1867499118,2cyclictest31983-21uname07:35:203
1867499118,2cyclictest30067-21sensors10:20:243
1867499118,2cyclictest27876-21grep12:30:173
1867499118,2cyclictest25691-21ssh11:36:263
1867499118,2cyclictest24238-21grep07:20:193
1867499118,2cyclictest22463-21ssh09:18:123
1867499118,2cyclictest21504-21wc07:15:163
1867499118,2cyclictest2116-21ssh09:57:183
1867499118,2cyclictest20909-21grep09:45:223
1867499118,2cyclictest19491-21ssh10:11:323
1867499118,2cyclictest13790-21ssh10:35:093
1867499118,2cyclictest13010-21ssh10:05:303
1867499118,2cyclictest12081-21ssh09:10:083
1867499118,2cyclictest11287-21tail11:25:193
1867499116,4cyclictest9726-21ssh12:16:323
1867499110,3cyclictest0-21swapper/311:44:003
1867499110,10cyclictest4346-21taskset08:55:003
1867499110,10cyclictest19243-21ssh11:31:213
1867099119,1cyclictest25185-21taskset10:16:092
1867099118,2cyclictest931-21tr10:50:152
1867099118,2cyclictest7485-21sh10:01:042
1867099118,2cyclictest27974-21cat08:35:172
1867099118,2cyclictest2056-21cut12:35:162
1867099118,2cyclictest18968-21chrt11:05:132
1867099118,2cyclictest18284-21cat08:15:162
1867099118,2cyclictest16173-21ssh09:40:342
1867099118,2cyclictest15480-21rm11:01:172
1867099118,2cyclictest14827-21cat11:55:132
1867099118,2cyclictest14334-21fschecks_count12:20:172
1867099118,2cyclictest13855-21ssh11:27:142
1867099118,2cyclictest12180-21wc10:05:222
1867099118,2cyclictest12175-21sh09:10:122
1867099118,2cyclictest12039-21grep08:00:222
1867099118,2cyclictest10439-21sendmail_mailtr11:50:252
1867099118,2cyclictest10326-21sh09:35:382
1867099118,1cyclictest2297-21grep10:25:142
1867099117,3cyclictest22324-21taskset08:23:362
1867099110,7cyclictest224-21systemd-journal12:00:122
1867099110,1cyclictest11725-21diskmemload09:30:252
1866899119,1cyclictest11766-21ls11:25:251
1866899118,2cyclictest9922-21grep09:05:301
1866899118,2cyclictest8574-21tr09:35:181
1866899118,2cyclictest6712-21ssh11:20:551
1866899118,2cyclictest6589-21head07:50:221
1866899118,2cyclictest30938-21ssh09:55:111
1866899118,2cyclictest25851-21grep08:30:181
1866899118,2cyclictest25626-21ssh11:36:221
1866899118,2cyclictest25581-21taskset07:22:261
1866899118,2cyclictest2540-21cstates08:50:141
1866899118,2cyclictest25-21ksoftirqd/110:15:201
1866899118,2cyclictest2495-21sed07:40:281
1866899118,2cyclictest22949-21ssh09:47:031
1866899118,2cyclictest21835-21ssh09:17:301
1866899118,2cyclictest21291-21ssh11:05:501
1866899118,2cyclictest18712-21ssh09:43:521
1866899118,2cyclictest14514-21which08:05:251
1866899118,2cyclictest13168-21tr11:00:141
1866899117,3cyclictest8176-21wc10:55:231
1866899116,2cyclictest224-21systemd-journal09:00:001
1866899114,5cyclictest32058-21ssh12:10:031
1866899110,10cyclictest29805-21sed11:40:171
1866499118,2cyclictest8615-21grep09:35:170
1866499118,2cyclictest30590-21taskset07:34:560
1866499118,2cyclictest27028-21chrt08:32:560
1866499118,2cyclictest20206-21uname07:10:240
1866499118,2cyclictest18704-21sed08:15:180
1866499118,2cyclictest16800-21sed08:10:410
1866499118,2cyclictest14314-21ntpq08:05:230
1866499117,3cyclictest23979-21tr07:20:190
1866499117,3cyclictest1809-21grep10:50:210
1866499117,2cyclictest8236-21sshd11:49:420
1866499113,5cyclictest13-21ksoftirqd/010:00:050
1866499112,6cyclictest8076-21pmu-power12:15:250
1867499108,2cyclictest10301-21ssh10:57:163
1867499108,1cyclictest3703-21cat07:45:203
1867499108,1cyclictest15666-21grep08:10:163
1867499108,1cyclictest104592sleep307:59:333
1867499107,2cyclictest30682sleep309:30:133
1867499107,2cyclictest28262-21grep08:35:203
1867499107,2cyclictest14341-21ntpq08:05:223
1867099108,1cyclictest4808-21egrep08:55:152
1867099108,1cyclictest28542-21sh10:20:132
1867099108,1cyclictest21592-21cat07:15:162
1867099108,1cyclictest17694-21wc08:15:002
1867099108,1cyclictest1181-21sh11:44:062
1867099107,2cyclictest6491-21grep07:50:222
1867099107,2cyclictest30530-21taskset07:34:122
1866899108,1cyclictest31297-21expr07:35:161
1866899108,1cyclictest27805-21cut08:35:171
1866899108,1cyclictest23185-21taskset07:17:271
1866899108,1cyclictest20260-21ssh10:40:021
1866899107,2cyclictest8245-21grep09:00:241
1866899107,2cyclictest3462-21grep07:45:201
1866899107,2cyclictest31229-21grep08:40:231
1866499107,2cyclictest825-21cat07:40:130
1866499102,7cyclictest13-21ksoftirqd/007:15:150
186749997,1cyclictest27203-21grep07:25:213
186709997,1cyclictest9365-21gpgv09:05:072
186649991,5cyclictest0-21swapper/007:50:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional