You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 00:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Nov 24, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10375210591,9sleep10-21swapper/107:07:081
10516210375,9sleep30-21swapper/307:08:593
1058728366,13sleep20-21swapper/207:09:542
1036428269,9sleep00-21swapper/007:06:590
124992470,0sleep00-21swapper/012:00:580
10687992622,3cyclictest3564-21aten2_r7power_v12:20:152
10687992523,1cyclictest0-21swapper/211:45:142
10685992524,1cyclictest0-21swapper/110:18:531
1068599250,24cyclictest0-21swapper/111:57:401
1068799230,22cyclictest0-21swapper/212:13:532
1068799220,21cyclictest0-21swapper/209:34:132
1069099210,20cyclictest0-21swapper/310:11:333
1068099210,19cyclictest0-21swapper/010:31:260
1069099203,1cyclictest5221-21grep12:20:263
1069099200,19cyclictest0-21swapper/310:54:143
10685992012,6cyclictest8956-21idleruntime-cro11:05:001
1068099201,16cyclictest0-21swapper/011:57:450
1068799190,17cyclictest0-21swapper/211:05:162
1069099180,18cyclictest0-21swapper/311:21:013
1068799180,18cyclictest3581-21diskmemload10:01:442
1069099170,15cyclictest0-21swapper/310:21:383
1068799172,1cyclictest3581-21diskmemload11:18:242
10687991712,2cyclictest3577-21sh09:10:002
1068799170,16cyclictest0-21swapper/212:08:022
10690991616,0cyclictest0-21swapper/310:04:073
1068799160,16cyclictest0-21swapper/212:15:032
1068599163,12cyclictest1-21systemd09:29:191
10685991615,1cyclictest0-21swapper/109:52:461
1068599160,1cyclictest0-21swapper/108:44:541
1068099162,4cyclictest224-21systemd-journal10:15:000
1068099162,2cyclictest0-21swapper/012:25:140
1068099161,9cyclictest6458-21/usr/sbin/munin09:40:190
1069099152,12cyclictest0-21swapper/312:30:123
1069099151,3cyclictest141rcu_preempt09:27:533
1069099150,1cyclictest0-21swapper/311:25:153
1068799151,2cyclictest9415-21ssh12:25:122
1068799151,2cyclictest32-21ksoftirqd/209:25:222
1068799151,12cyclictest0-21swapper/210:50:152
1068599150,2cyclictest0-21swapper/112:38:471
1068099158,4cyclictest30287-21ssh11:50:170
1068099157,7cyclictest30198-21munin-run07:50:000
1068099154,2cyclictest224-21systemd-journal07:25:000
10680991512,2cyclictest9206-21cron09:15:010
1068099150,4cyclictest0-21swapper/011:00:410
1068099150,2cyclictest0-21swapper/011:14:480
1068099150,1cyclictest0-21swapper/012:13:530
1069099140,2cyclictest0-21swapper/310:27:073
1069099140,1cyclictest0-21swapper/309:31:033
1068799149,4cyclictest9376-21cut09:15:012
1068799144,9cyclictest22208-21munin-node11:28:172
1068799140,9cyclictest0-21swapper/211:33:572
1068799140,7cyclictest0-21swapper/210:14:592
1068799140,14cyclictest0-21swapper/210:27:232
1068599142,1cyclictest141rcu_preempt12:22:111
10685991410,3cyclictest15108-21wc07:15:281
10685991410,3cyclictest15108-21wc07:15:271
10685991410,2cyclictest7928-21ssh11:30:231
10685991410,2cyclictest31355-21cat09:00:001
1068599140,14cyclictest0-21swapper/110:47:481
1068099147,6cyclictest2912-21ssh12:19:590
1068099145,0cyclictest0-21swapper/011:42:050
1068099143,9cyclictest2344-21diskstats10:05:170
10680991411,2cyclictest29126-21wget08:55:150
1069099139,3cyclictest2932-21basename12:20:003
10690991310,2cyclictest9680-21cut09:15:143
10690991310,2cyclictest9680-21cut09:15:133
10690991310,2cyclictest6561-21tr10:35:223
10690991310,2cyclictest24671-21head11:45:203
10690991310,2cyclictest24225-21grep08:45:013
10690991310,2cyclictest19988-21cat11:15:013
10690991310,2cyclictest18933-21grep09:50:273
1068799130,2cyclictest0-21swapper/209:18:432
1068799130,2cyclictest0-21swapper/209:18:422
1068799130,1cyclictest0-21swapper/209:50:552
1068599139,3cyclictest21042-21wc08:35:271
10685991310,2cyclictest27322-21cut08:50:141
10685991310,2cyclictest1787-21grep09:35:281
10685991310,2cyclictest17646-21grep07:20:251
1068599130,12cyclictest10650-21ssh11:05:221
1068099133,3cyclictest224-21systemd-journal12:08:540
10680991310,2cyclictest26210-21head10:25:200
10680991310,2cyclictest11585-21cat09:45:180
1068099130,11cyclictest0-21swapper/011:34:580
1069099129,2cyclictest4843-21grep09:10:183
1069099129,2cyclictest4570-21head11:55:183
1069099129,2cyclictest3537-21chrt10:33:143
1069099129,2cyclictest25580-21cat07:40:003
1069099129,2cyclictest20298-21grep11:40:343
1069099129,2cyclictest17751-21ssh12:05:193
1069099128,3cyclictest4468-21ssh10:06:383
1069099128,3cyclictest23426-21tail09:55:183
1069099128,3cyclictest17442-21grep08:30:193
1069099128,3cyclictest16631-21ssh09:50:003
1069099128,2cyclictest17082-21tr07:20:193
1069099126,2cyclictest224-21systemd-journal10:20:003
1069099120,11cyclictest25966-21tr07:40:143
1069099120,0cyclictest0-21swapper/311:00:123
1068799129,2cyclictest919-21chrt09:01:362
1068799129,2cyclictest5902-21cut08:05:192
1068799129,2cyclictest2862-21sh10:32:232
1068799129,2cyclictest19739-21rm10:46:262
1068799129,2cyclictest18397-21head08:30:262
1068799129,2cyclictest1760-21chrt10:05:162
1068799129,2cyclictest16849-21head12:30:222
1068799129,2cyclictest16509-21head09:20:382
1068799129,2cyclictest16347-21sh09:49:392
1068799124,7cyclictest37850irq/131-snd_hda11:38:452
1068799123,8cyclictest9402-21kworker/2:211:10:162
1068799123,7cyclictest3581-21diskmemload12:02:412
10687991210,1cyclictest27932-21wc08:50:202
10687991210,1cyclictest24595-21tr08:45:162
10687991210,1cyclictest2039-21wc07:55:272
1068799120,9cyclictest0-21swapper/211:55:142
1068799120,7cyclictest0-21swapper/209:40:152
1068599129,2cyclictest8322-21grep08:10:161
1068599129,2cyclictest7624-21head09:40:291
1068599129,2cyclictest28842-21grep07:45:191
1068599129,2cyclictest24773-21grep12:10:301
1068599129,2cyclictest24437-21tail09:55:271
1068599129,2cyclictest23929-21grep07:35:161
1068599128,3cyclictest2826-21ssh10:05:211
1068599128,3cyclictest23434-21grep11:45:011
1068599128,3cyclictest11401-21grep09:45:181
1068599126,4cyclictest32305-21ssh10:30:171
1068599125,3cyclictest224-21systemd-journal09:20:001
1068599125,3cyclictest224-21systemd-journal09:20:001
1068099129,2cyclictest8601-21cat08:10:190
1068099129,2cyclictest4552-21cat12:20:200
1068099129,2cyclictest32621-21idleruntime-cro07:55:010
1068099129,2cyclictest30193-21ssh09:32:460
1068099129,2cyclictest28313-21ssh10:00:030
1068099129,2cyclictest27350-21taskset10:53:230
1068099129,2cyclictest26938-21tr08:50:120
1068099129,2cyclictest26760-21wc07:40:200
1068099129,2cyclictest25115-21ssh12:37:090
1068099129,2cyclictest25032-21head08:45:190
1068099129,2cyclictest24466-21sed09:55:270
1068099129,2cyclictest23490-21sh09:26:390
1068099129,2cyclictest23187-21ssh11:16:380
1068099129,2cyclictest19418-21ssh09:51:030
1068099129,2cyclictest19280-21rm10:45:590
1068099129,2cyclictest19163-21sort07:25:180
1068099129,2cyclictest18259-21sshd09:22:190
1068099129,2cyclictest17293-21grep12:30:270
1068099129,2cyclictest16179-21ssh11:37:490
1068099129,2cyclictest15951-21ssh10:15:440
1068099125,6cyclictest22830-21apt-get07:35:010
10680991210,1cyclictest5516-21grep08:05:160
10680991210,1cyclictest28073-21grep11:20:270
1068099120,8cyclictest0-21swapper/011:45:140
1068099120,11cyclictest995-21cat09:35:210
1068099120,11cyclictest15398-21cut08:25:180
1068099120,11cyclictest10670-21cat09:15:200
1068099120,11cyclictest10670-21cat09:15:190
1069099118,2cyclictest9514-21ssh12:25:153
1069099118,2cyclictest9202-21sed08:10:243
1069099118,2cyclictest7224-21fgrep09:40:243
1069099118,2cyclictest6352-21uname08:05:223
1069099118,2cyclictest4420-21ssh09:39:093
1069099118,2cyclictest3581-21diskmemload11:08:473
1069099118,2cyclictest31378-21taskset10:55:393
1069099118,2cyclictest30641-21cat08:55:313
1069099118,2cyclictest29791-21grep07:45:293
1069099118,2cyclictest26415-21idleruntime-cro08:49:593
1069099118,2cyclictest23718-21grep12:10:193
1069099118,2cyclictest23193-21ssh11:16:393
1069099118,2cyclictest19686-21ssh09:24:043
1069099118,2cyclictest18861-21ssh10:45:303
1069099118,2cyclictest14440-21switchtime11:35:273
1069099118,2cyclictest11593-21ssh12:00:213
1069099118,2cyclictest11236-21cat07:10:153
1069099114,4cyclictest31962-21unixbench_multi11:50:273
1069099110,11cyclictest0-21swapper/312:36:123
1069099110,10cyclictest20703-21cat07:30:123
1068799119,2cyclictest25356-21taskset07:39:242
1068799119,2cyclictest20234-21sed10:20:152
1068799119,2cyclictest14126-21ls10:15:142
1068799119,1cyclictest32561-21sh11:51:092
1068799119,1cyclictest314332sleep210:55:432
1068799119,1cyclictest30381-21ssh11:23:072
1068799119,1cyclictest30262-21sed08:55:272
1068799119,1cyclictest24300-21cat09:55:262
1068799119,1cyclictest21263-21sed07:30:172
1068799119,1cyclictest19685-21tr07:25:232
1068799119,1cyclictest13657-21cut07:15:162
1068799119,1cyclictest13657-21cut07:15:152
1068799118,2cyclictest7310-21ssh10:35:392
1068799118,2cyclictest687-21head09:35:192
1068799118,2cyclictest22581-21cut08:40:202
1068799118,2cyclictest22530-21tail12:35:182
1068799118,2cyclictest21634-21chrt11:42:052
1068799118,2cyclictest13453-21cat08:20:212
1068799118,2cyclictest13284-21seq10:41:082
1068799117,3cyclictest4828-21grep11:00:282
1068599118,2cyclictest5075-21taskset10:35:081
1068599118,2cyclictest4464-21grep08:00:261
1068599118,2cyclictest2880-21ssh11:54:291
1068599118,2cyclictest28374-21ssh11:20:411
1068599118,2cyclictest25918-21grep08:45:281
1068599118,2cyclictest23361-21grep10:50:141
1068599118,2cyclictest22367-21tail07:30:271
1068599118,2cyclictest21571-21head11:15:201
1068599118,2cyclictest18115-21rm11:11:221
1068599118,2cyclictest17718-21ssh12:05:191
1068599118,2cyclictest17449-21tr08:30:181
1068599118,2cyclictest16365-21ssh12:30:171
1068599118,2cyclictest15487-21grep08:25:181
1068599118,2cyclictest1547-21wc11:25:251
1068599118,2cyclictest15444-21ssh12:04:211
1068599118,2cyclictest1460-21wc10:05:001
1068599118,2cyclictest13625-21grep11:35:161
1068599118,2cyclictest13248-21ssh12:27:461
1068599118,2cyclictest10890-21ssh10:11:581
1068599118,2cyclictest1034-21rm10:58:221
1068599117,3cyclictest30031-21tail12:15:181
1068599117,3cyclictest1699-21cut09:05:171
1068599113,4cyclictest224-21systemd-journal10:20:001
1068599110,10cyclictest12331-21ssh10:40:241
1068099119,2cyclictest32562-21sh10:57:030
1068099119,1cyclictest745-21tr07:55:140
1068099119,1cyclictest5217-21apt10:35:140
1068099119,1cyclictest239432sleep008:43:540
1068099119,1cyclictest20974-21head10:20:210
1068099119,1cyclictest1827-21cat09:05:180
1068099119,1cyclictest12059-21sh10:40:220
1068099119,1cyclictest10718-21ssh11:05:240
1068099119,1cyclictest10259-21cut08:15:150
1068099118,2cyclictest13598-21grep07:15:150
1068099118,2cyclictest13598-21grep07:15:140
1068099118,2cyclictest12049-21ntp_kernel_pll_07:10:200
1068099113,2cyclictest32523-21apache_processe11:25:120
1068099110,10cyclictest32607-21cat09:00:260
1069099109,1cyclictest31944-21expr09:00:233
1069099108,1cyclictest6766-21cut11:30:143
1069099108,1cyclictest285892sleep308:52:323
1069099108,1cyclictest18659-21expr07:25:153
1069099108,1cyclictest12823-21cat08:20:163
1069099108,1cyclictest10620-21ssh10:40:033
1069099108,1cyclictest10198-21grep08:15:153
1069099107,2cyclictest427-21apache_volume07:55:133
1069099107,2cyclictest30739-21grep07:50:153
1069099107,2cyclictest15548-21grep08:25:183
1069099104,3cyclictest0-21swapper/307:15:243
1069099104,3cyclictest0-21swapper/307:15:233
1068799109,1cyclictest8747-21awk08:10:192
1068799109,1cyclictest11558-21grep07:10:162
1068799108,2cyclictest25812-21awk07:40:152
1068799108,1cyclictest3529-21cut08:00:162
1068799108,1cyclictest32337-21chrt07:51:282
1068799107,2cyclictest28290-21cat07:45:162
1068799107,2cyclictest21384-21taskset08:36:512
1068799107,2cyclictest17647-21head07:20:252
1068799107,2cyclictest11081-21cat08:15:202
1068599108,1cyclictest870-21taskset09:00:581
1068599108,1cyclictest5536-21ls08:05:171
1068599108,1cyclictest31224-21sh09:34:051
1068599108,1cyclictest304142sleep107:50:071
1068599108,1cyclictest26884-21ssh11:46:551
1068599108,1cyclictest2619-21sendmail-msp07:59:591
1068599108,1cyclictest13864-21sed08:20:271
1068599107,2cyclictest27574-21chrt07:42:061
1068599107,2cyclictest26879-21wc10:25:251
1068599107,2cyclictest19089-21tail07:25:181
1068599100,9cyclictest6073-21sh09:10:371
1068099109,1cyclictest18600-21switchtime08:30:290
1068099108,1cyclictest3604-21grep08:00:170
1068099107,2cyclictest19741-21cat08:35:160
1068099101,6cyclictest12401-21apache_processe08:20:130
106909997,1cyclictest2885-21basename08:00:133
106909997,1cyclictest19634-21wget08:35:143
106909996,2cyclictest3473-21seq09:08:473
106879997,1cyclictest15595-21sort08:25:192
106859996,2cyclictest16189-21cat09:20:201
106859996,2cyclictest12141-21sed07:10:221
106859996,2cyclictest11665-21tail08:15:271
106809998,1cyclictest23639-21expr07:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional