You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 02:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 06, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23348210576,9sleep30-21swapper/307:07:093
23348210576,9sleep30-21swapper/307:07:093
23501210378,9sleep10-21swapper/107:09:081
23501210378,9sleep10-21swapper/107:09:081
2196128466,13sleep00-21swapper/007:05:170
2196128466,13sleep00-21swapper/007:05:170
2349227868,6sleep20-21swapper/207:09:012
2349227868,6sleep20-21swapper/207:09:012
208342690,0sleep20-21swapper/211:28:412
78202620,0sleep10-21swapper/111:18:271
108272620,0sleep010828-21sshd11:45:540
65242600,1sleep10-21swapper/108:50:001
2366999330,32cyclictest0-21swapper/310:35:293
2366999320,31cyclictest0-21swapper/311:21:293
23662992521,3cyclictest94-21kswapd010:00:001
2366099250,1cyclictest0-21swapper/012:31:210
124692230,0sleep30-21swapper/310:28:283
23666992120,1cyclictest0-21swapper/212:19:502
23666992018,1cyclictest14425-21kworker/2:211:00:092
2366099200,3cyclictest0-21swapper/012:19:090
23662991917,2cyclictest25-21ksoftirqd/109:50:131
2366699187,11cyclictest22375-21kworker/2:008:46:272
2366699186,12cyclictest22375-21kworker/2:009:00:192
2366099187,6cyclictest10759-21munin-run10:55:000
2366099184,7cyclictest3202-21munin-run12:30:000
92612170,0sleep20-21swapper/212:34:012
2366999171,3cyclictest224-21systemd-journal10:31:083
2366999170,1cyclictest0-21swapper/309:20:443
2366699170,1cyclictest0-21swapper/207:20:152
2366099175,6cyclictest18005-21munin-run09:40:000
23666991613,2cyclictest37850irq/131-snd_hda08:35:232
23666991610,5cyclictest2477-21grep12:05:142
2366699160,13cyclictest0-21swapper/207:45:132
2366299166,3cyclictest25-21ksoftirqd/111:40:001
2366299162,13cyclictest0-21swapper/111:55:131
23662991615,1cyclictest0-21swapper/112:12:011
2366099160,4cyclictest0-21swapper/010:24:470
2366099160,2cyclictest0-21swapper/009:50:470
23669991514,1cyclictest0-21swapper/309:48:113
2366999151,1cyclictest0-21swapper/312:01:073
2366999151,0cyclictest0-21swapper/311:00:103
23669991510,3cyclictest30346-21tr08:30:173
2366699154,10cyclictest0-21swapper/210:05:162
23666991514,1cyclictest0-21swapper/212:02:162
23666991513,2cyclictest11060-21kworker/2:111:49:092
23666991512,3cyclictest32-21ksoftirqd/209:05:142
2366699150,13cyclictest0-21swapper/209:43:362
2366099153,6cyclictest22230-21perf10:10:000
2366099152,6cyclictest2822-21munin-run11:15:010
2366099150,2cyclictest0-21swapper/011:42:390
2366999146,4cyclictest224-21systemd-journal11:30:003
23669991411,2cyclictest17047-21cron12:40:003
2366999140,13cyclictest7859-21chrt12:09:383
2366699149,3cyclictest13853-21cat10:30:002
23666991412,2cyclictest24005-21kworker/2:112:10:292
23666991411,2cyclictest24690-21cat08:20:002
23666991410,4cyclictest37850irq/131-snd_hda12:38:412
23666991410,4cyclictest0-21swapper/211:40:432
23666991410,3cyclictest5886-21sed09:55:272
23666991410,3cyclictest24630-21grep07:10:192
23662991412,2cyclictest25-21ksoftirqd/109:49:071
23662991410,3cyclictest4379-21rm12:30:161
23662991410,2cyclictest0-21swapper/112:22:051
23660991411,2cyclictest19798-21cron11:55:000
23660991410,3cyclictest32765-21sed07:25:360
23660991410,2cyclictest6732-21tr09:30:150
23660991410,2cyclictest630-21grep10:45:160
23660991410,2cyclictest11603-21head11:20:290
2366999136,4cyclictest23423-21sh11:05:133
2366999135,7cyclictest30939-21runrttasks11:35:543
2366999133,5cyclictest224-21systemd-journal12:23:513
2366999133,3cyclictest5761-21/usr/sbin/munin10:50:153
23669991310,2cyclictest31565-21sort09:50:163
23669991310,2cyclictest30993-21grep11:10:223
23669991310,2cyclictest26414-21cut07:15:153
23669991310,2cyclictest1137-21grep10:45:183
2366699139,3cyclictest3796-21sed10:20:272
2366699139,3cyclictest26992-21basename11:59:592
2366699139,3cyclictest15732-21sh09:37:252
2366699139,1cyclictest29565-21sed10:15:382
2366699133,7cyclictest1378-21ntpd08:40:212
23666991312,1cyclictest15299-21apache210:38:042
23666991310,2cyclictest28259-21sed09:20:182
23666991310,2cyclictest16798-21tr09:10:192
2366699130,4cyclictest0-21swapper/211:21:382
2366299136,7cyclictest0-21swapper/111:44:581
23662991310,2cyclictest8896-21sed07:45:171
23662991310,2cyclictest3206-21sleep12:30:001
23662991310,2cyclictest23724-21tr08:15:201
23662991310,2cyclictest2274-21tr08:40:151
23662991310,2cyclictest1764-21sed07:30:161
23662991310,2cyclictest14277-21taskset10:56:531
2366299130,1cyclictest0-21swapper/112:15:271
2366099139,3cyclictest4439-21taskset11:15:210
2366099139,2cyclictest25675-21head08:20:170
2366099135,3cyclictest141rcu_preempt08:20:000
23660991310,2cyclictest29029-21wc12:00:210
23660991310,2cyclictest24092-21grep12:20:280
23660991310,2cyclictest10750-21grep07:50:130
2366099130,4cyclictest25929-21sshd07:14:130
2366099130,0cyclictest0-21swapper/010:12:460
2366999129,2cyclictest8084-21munin-plugin-st11:45:003
2366999129,2cyclictest25180-21cut08:20:143
2366999129,2cyclictest23606-21tr08:15:193
2366999129,2cyclictest19812-21apt11:55:003
2366999129,2cyclictest18539-21chrt12:16:033
2366999129,2cyclictest13752-21tr09:35:213
2366999129,2cyclictest10638-21basename10:00:143
2366999129,2cyclictest10271-21sed11:45:263
2366999128,3cyclictest31768-21grep07:25:273
2366999128,2cyclictest7343-21grep09:30:183
2366999128,2cyclictest4256-21sed11:15:203
2366999128,2cyclictest3255-21cat07:35:013
2366999124,6cyclictest30992-21grep12:25:253
2366999124,4cyclictest224-21systemd-journal10:45:003
2366999122,8cyclictest0-21swapper/310:15:133
2366699129,2cyclictest715-21cron07:30:002
2366699129,2cyclictest23054-21grep12:20:182
2366699129,2cyclictest22808-21df11:30:152
2366699128,3cyclictest24957-21cut08:20:142
2366699122,9cyclictest22684-21aten2_r7power_c10:10:152
23666991211,1cyclictest20590-21kworker/2:209:32:092
23666991211,1cyclictest12422-21kworker/2:009:49:012
2366699121,11cyclictest0-21swapper/209:51:262
23666991211,1cyclictest0-21swapper/209:27:542
23666991211,1cyclictest0-21swapper/208:06:102
2366699120,2cyclictest0-21swapper/211:37:192
2366699120,1cyclictest0-21swapper/210:49:552
2366699120,11cyclictest16030-21sort08:00:192
2366299129,2cyclictest30388-21grep08:30:171
2366299129,2cyclictest28251-21grep09:20:191
2366299129,2cyclictest27892-21sort08:25:161
2366299129,2cyclictest2340-21grep10:20:151
2366299129,2cyclictest16692-21grep11:25:171
23662991210,1cyclictest19874-21wc11:55:001
2366299120,11cyclictest30787-21dump-pmu-power07:25:001
2366099129,2cyclictest9421-21tr08:55:350
2366099129,2cyclictest9291-21sed12:10:150
2366099129,2cyclictest5839-21grep07:40:130
2366099129,2cyclictest22517-21uniq09:15:170
2366099129,2cyclictest19842-21ls08:10:000
2366099129,2cyclictest12480-21cut09:00:190
2366099129,2cyclictest11510-21sed12:35:180
2366099121,3cyclictest224-21systemd-journal11:06:170
2366999118,2cyclictest7869-21chrt12:32:423
2366999118,2cyclictest7386-21grep08:50:153
2366999118,2cyclictest4913-21df08:45:163
2366999118,2cyclictest4306-21date09:55:003
2366999118,2cyclictest3730-21ls07:35:153
2366999118,2cyclictest24871-21ssh09:16:433
2366999118,2cyclictest24494-21sed10:10:283
2366999118,2cyclictest22829-21ssh11:56:003
2366999118,2cyclictest22558-21tr11:30:143
2366999118,2cyclictest19059-21ssh10:06:003
2366999118,2cyclictest18817-21needreboot08:05:223
2366999118,2cyclictest18355-21ssh09:40:073
2366999118,2cyclictest18325-21grep09:10:313
2366999118,2cyclictest16071-21grep08:00:203
2366999118,2cyclictest11223-21wget10:55:153
2366999118,2cyclictest10599-21head12:10:263
2366999117,3cyclictest7883-21munin-plugin-st07:45:003
2366999117,3cyclictest16291-21munin-run09:10:003
2366999112,4cyclictest11410-21latency_hist09:00:003
2366999110,10cyclictest1709-21grep09:25:233
2366699119,1cyclictest0-21swapper/210:55:082
2366699118,2cyclictest5960-21cpuspeed_turbos07:40:142
2366699118,2cyclictest5739-21chrt10:50:102
2366699118,2cyclictest5492-21sh07:40:012
2366699118,2cyclictest30285-21expr11:10:162
2366699118,2cyclictest29985-21grep12:25:152
2366699118,2cyclictest29678-21chrt08:30:142
2366699118,2cyclictest29628-21ssh10:41:432
2366699118,2cyclictest26613-21expr07:15:152
2366699118,2cyclictest25298-21ssh09:17:242
2366699118,2cyclictest25176-21ssh11:05:282
2366699118,2cyclictest1741-21head07:30:162
2366699118,2cyclictest14987-21hwlatdetect10:30:162
2366699118,2cyclictest14795-21cut11:50:142
2366699118,2cyclictest12965-21tr07:55:122
2366699118,2cyclictest10254-21uname08:55:402
2366699117,2cyclictest3170-21sed11:15:002
2366699115,5cyclictest20142-21cut08:10:152
2366299119,1cyclictest8850-21sh08:55:001
2366299119,1cyclictest26995-21cron08:25:001
2366299119,1cyclictest21988-21cut09:15:141
2366299118,2cyclictest9858-21grep08:55:371
2366299118,2cyclictest7303-21grep09:30:181
2366299118,2cyclictest532-21ssh10:18:511
2366299118,2cyclictest3812-21chrt09:27:151
2366299118,2cyclictest32287-21grep07:25:321
2366299118,2cyclictest30146-21tr11:10:161
2366299118,2cyclictest28109-21expr12:00:151
2366299118,2cyclictest24935-21cat07:10:221
2366299118,2cyclictest24744-21ssh10:38:041
2366299118,2cyclictest23819-21expr11:05:161
2366299118,2cyclictest23312-21ssh10:10:181
2366299118,2cyclictest22623-21ssh11:30:151
2366299118,2cyclictest20550-21ssh09:40:401
2366299118,2cyclictest20417-21ssh10:07:381
2366299118,2cyclictest16823-21tr09:10:191
2366299118,2cyclictest16383-21expr08:00:221
2366299118,2cyclictest15451-21ssh10:30:191
2366299118,2cyclictest1355-21chrt08:35:421
2366299118,2cyclictest12494-21cut09:00:191
2366299118,2cyclictest12488-21ssh09:35:111
2366299118,2cyclictest12265-21taskset07:50:251
2366299118,2cyclictest10759-21ssh11:45:501
2366299118,2cyclictest10540-21head11:20:201
2366299118,2cyclictest10369-21apt-config10:00:011
2366299117,3cyclictest14595-21head07:55:251
2366299114,5cyclictest17293-21ssh11:00:111
2366299110,10cyclictest32651-21ssh10:45:061
2366099119,1cyclictest8893-21grep07:45:170
2366099118,2cyclictest8919-21chrt09:58:230
2366099118,2cyclictest8687-21ssh10:25:180
2366099118,2cyclictest7357-21cut08:50:150
2366099118,2cyclictest3407-21ssh12:05:200
2366099118,2cyclictest30675-21ssh11:35:350
2366099118,2cyclictest29690-21cut08:30:140
2366099118,2cyclictest29245-21cat10:15:230
2366099118,2cyclictest29231-21sed07:20:170
2366099118,2cyclictest28259-21ssh09:47:180
2366099118,2cyclictest27131-21ssh10:40:220
2366099118,2cyclictest26180-21ssh11:59:150
2366099118,2cyclictest26117-21ssh10:39:400
2366099118,2cyclictest21364-21wc08:10:230
2366099118,2cyclictest20834-21sh11:02:240
2366099118,2cyclictest19351-21ssh09:40:190
2366099118,2cyclictest16443-21cat11:25:160
2366099118,2cyclictest14402-21grep10:30:130
2366099118,2cyclictest14365-21tr09:05:170
2366099118,2cyclictest14291-21ssh10:56:530
2366099118,1cyclictest14516-21grep07:55:240
2366099117,3cyclictest24173-21grep11:30:260
2366999108,1cyclictest29107-21expr07:20:173
2366999108,1cyclictest21701-21cat08:10:273
2366999108,1cyclictest1820-21ssh10:20:013
2366999108,1cyclictest14367-21taskset07:55:243
2366999108,1cyclictest1374-21chrt08:35:563
2366999108,1cyclictest13060-21sendmail_mailst09:00:243
2366999107,2cyclictest2542-21sed08:40:163
2366999107,2cyclictest24680-21head07:10:203
2366999107,2cyclictest10634-21cut07:50:123
2366999101,5cyclictest224-21systemd-journal08:30:003
2366999100,9cyclictest9006-21if_enp2s007:45:183
2366699107,2cyclictest7360-21fschecks_count08:50:152
2366699107,2cyclictest10763-21tr07:50:132
2366699106,3cyclictest27301-21grep08:25:132
2366699100,9cyclictest10842-21cpuspeed_turbos10:00:162
2366299108,1cyclictest8618-21expr10:25:181
2366299108,1cyclictest5765-21basename10:50:121
2366299108,1cyclictest5420-21chrt07:39:201
2366299108,1cyclictest219012sleep108:11:061
2366299108,1cyclictest15016-21cat09:05:221
2366299108,1cyclictest10672-21ssh12:35:141
2366299107,2cyclictest28866-21ssh10:40:491
2366299107,2cyclictest26423-21grep07:15:151
2366299107,2cyclictest2011-21ssh12:05:041
2366299107,2cyclictest19488-21chrt08:05:371
2366299100,9cyclictest0-21swapper/107:40:131
2366099108,1cyclictest5407-21taskset07:39:110
2366099108,1cyclictest5240-21if_enp2s008:45:190
2366099108,1cyclictest28118-21ssh09:20:180
2366099108,1cyclictest18553-21ssh09:10:460
2366099108,1cyclictest1318-21expr07:30:140
2366099108,1cyclictest10267-21sed10:00:000
2366099107,2cyclictest28481-21ls08:25:210
2366099107,2cyclictest26957-21grep07:15:170
2366099107,2cyclictest2649-21sed09:25:400
2366099100,9cyclictest16747-21chrt08:00:270
236609997,1cyclictest1376-21chrt08:35:570
236609986,1cyclictest2181-21awk08:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional