You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 12:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Dec 02, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1776210390,9sleep30-21swapper/319:08:583
1854210288,9sleep10-21swapper/119:09:561
160128565,15sleep20-21swapper/219:06:452
179328370,9sleep00-21swapper/019:09:110
261762610,0sleep30-21swapper/322:15:373
1945993231,1cyclictest0-21swapper/322:02:383
194599321,2cyclictest14060-21gltestperf23:15:183
1945993129,1cyclictest0-21swapper/322:33:183
1945993029,1cyclictest0-21swapper/323:35:183
1945993028,1cyclictest9838-21ssh22:26:583
1945993028,1cyclictest0-21swapper/323:20:383
193299300,1cyclictest0-21swapper/021:20:000
194599290,29cyclictest0-21swapper/321:54:383
194399290,28cyclictest0-21swapper/221:51:452
1945992826,1cyclictest13747-21ssh22:07:383
1945992826,1cyclictest0-21swapper/322:13:583
193299281,25cyclictest1708-21sh21:15:000
193299280,27cyclictest0-21swapper/022:24:450
1945992726,1cyclictest0-21swapper/322:24:383
1945992725,1cyclictest0-21swapper/322:58:183
193299270,23cyclictest32756-21timerwakeupswit21:35:260
193299270,1cyclictest27216-21diskmemload23:18:400
193299270,1cyclictest0-21swapper/021:41:520
1945992625,1cyclictest0-21swapper/323:42:583
1945992625,1cyclictest0-21swapper/322:49:583
194399260,26cyclictest0-21swapper/222:55:552
193299261,24cyclictest27216-21diskmemload00:17:070
193299260,21cyclictest27216-21diskmemload21:30:080
1945992524,1cyclictest0-21swapper/323:49:383
1945992524,1cyclictest0-21swapper/323:07:583
1945992524,1cyclictest0-21swapper/300:13:583
194399250,25cyclictest0-21swapper/222:52:392
1945992423,1cyclictest0-21swapper/300:34:183
1945992423,1cyclictest0-21swapper/300:34:183
193899244,12cyclictest0-21swapper/100:10:151
193299246,17cyclictest136692sleep000:25:130
193299241,22cyclictest0-21swapper/022:25:190
193299240,23cyclictest27216-21diskmemload22:42:040
193299240,1cyclictest27216-21diskmemload22:15:530
193299240,1cyclictest0-21swapper/023:13:180
1945992322,1cyclictest0-21swapper/323:11:383
1945992322,1cyclictest0-21swapper/322:42:183
1945992322,1cyclictest0-21swapper/300:19:383
194399230,23cyclictest0-21swapper/222:15:482
193299232,20cyclictest27216-21diskmemload23:04:380
193299231,21cyclictest27216-21diskmemload23:35:510
193299231,21cyclictest0-21swapper/022:30:150
193299231,20cyclictest26056-21ssh00:10:080
193299230,22cyclictest27216-21diskmemload21:50:250
193299230,18cyclictest27216-21diskmemload23:48:450
1945992221,1cyclictest0-21swapper/323:58:183
1945992221,1cyclictest0-21swapper/323:54:583
1945992221,1cyclictest0-21swapper/321:10:383
1945992221,1cyclictest0-21swapper/300:22:383
194399222,19cyclictest24300-21apt-get23:00:162
194399220,22cyclictest0-21swapper/200:03:312
193299222,19cyclictest16496-21basename19:40:000
193299222,18cyclictest27216-21diskmemload21:24:420
193299221,20cyclictest26801-21aten2_r7power_v00:35:160
193299220,2cyclictest0-21swapper/023:20:120
193299220,1cyclictest0-21swapper/023:42:420
193299220,1cyclictest0-21swapper/023:06:460
193299220,0cyclictest0-21swapper/022:00:130
194399210,21cyclictest0-21swapper/221:17:102
1938992111,10cyclictest0-21swapper/122:18:161
193299213,17cyclictest27216-21diskmemload23:26:310
193299212,18cyclictest23207-21http21:30:000
193299212,15cyclictest0-21swapper/000:34:010
193299212,15cyclictest0-21swapper/000:34:010
193299211,18cyclictest3497-21latency23:30:210
193299211,18cyclictest27262-21cat21:10:000
193299210,20cyclictest0-21swapper/022:10:320
193299210,1cyclictest0-21swapper/022:55:180
193299210,1cyclictest0-21swapper/019:55:150
193299210,19cyclictest11155-21latency22:50:190
193299210,19cyclictest0-21swapper/023:57:380
193299210,0cyclictest0-21swapper/023:50:220
194399200,1cyclictest0-21swapper/223:33:122
193299202,17cyclictest0-21swapper/000:05:130
193299201,18cyclictest12996-21sh21:45:130
193299201,17cyclictest0-21swapper/000:00:050
193299201,15cyclictest0-21swapper/020:42:560
193299200,1cyclictest14466-21ssh22:08:190
193299200,18cyclictest0-21swapper/021:55:080
193299191,17cyclictest8571-21rm22:49:020
193299190,3cyclictest21144-21expr22:35:170
193299190,15cyclictest7978-21ssh00:20:140
32932181,1sleep23336-21ntp_states19:10:212
1945991816,1cyclictest0-21swapper/300:00:383
194599180,2cyclictest0-21swapper/300:37:083
194399181,16cyclictest29225-21sensors_fan21:10:232
193299181,16cyclictest11616-21date19:30:000
193299181,14cyclictest0-21swapper/020:55:170
193299180,1cyclictest0-21swapper/020:35:140
193299172,14cyclictest18735-21tail20:50:180
193299170,1cyclictest0-21swapper/019:40:280
193299170,1cyclictest0-21swapper/019:40:280
194399168,6cyclictest10041-21tune2fs21:20:172
194399163,4cyclictest31786-21munin-run00:40:002
1943991614,2cyclictest0-21swapper/200:08:242
1943991613,2cyclictest0-21swapper/221:40:002
194399161,14cyclictest13653-21apt-get00:25:132
194399160,5cyclictest0-21swapper/222:00:132
194399160,16cyclictest0-21swapper/223:26:102
194399160,16cyclictest0-21swapper/222:43:072
193899161,1cyclictest141rcu_preempt23:26:181
193899160,16cyclictest0-21swapper/121:39:301
193299160,1cyclictest0-21swapper/020:25:160
193299160,15cyclictest9019-21fschecks_time20:30:190
193299160,14cyclictest24046-21cut19:55:010
193299160,0cyclictest0-21swapper/019:45:130
194399155,4cyclictest0-21swapper/222:20:272
1943991514,1cyclictest0-21swapper/200:13:222
1943991513,1cyclictest0-21swapper/220:45:272
194399150,15cyclictest0-21swapper/223:38:242
194399150,15cyclictest0-21swapper/223:20:142
194399150,15cyclictest0-21swapper/223:09:292
193899150,0cyclictest0-21swapper/100:19:091
193299158,6cyclictest6918-21seq19:20:040
193299158,6cyclictest29454-21df20:05:200
193299158,6cyclictest17654-21chrt20:49:470
193299157,7cyclictest27586-21grep20:00:220
193299151,7cyclictest13606-21chrt19:32:220
193299150,3cyclictest6600-21sshd19:19:120
1945991410,3cyclictest3313-21cron20:19:593
1945991410,3cyclictest27832-21basename23:25:153
194599140,14cyclictest0-21swapper/321:40:013
193899145,1cyclictest224-21systemd-journal21:25:141
193299148,5cyclictest4353-21tail20:20:190
193299147,6cyclictest912-21chrt20:14:570
193299140,2cyclictest2920-21/usr/sbin/munin19:10:170
194599139,3cyclictest4279-21cron19:15:003
194599138,4cyclictest15552-21grep20:45:153
1945991310,2cyclictest4019-21grep21:15:263
1945991310,2cyclictest14905-21sort19:35:173
1945991310,2cyclictest13385-21grep20:40:153
1945991310,2cyclictest11432-21ls22:50:223
1945991310,1cyclictest15813-21apache223:32:533
194599130,1cyclictest0-21swapper/300:29:553
194599130,12cyclictest6768-21sed20:25:193
194399139,3cyclictest8086-21cron20:30:002
194399139,3cyclictest19694-21cron21:50:002
194399139,3cyclictest16652-21cut21:25:172
194399139,2cyclictest10075-21head20:30:282
1943991312,1cyclictest0-21swapper/219:30:132
1943991310,2cyclictest6411-21grep23:10:142
1943991310,2cyclictest5710-21cron20:25:002
1943991310,2cyclictest30570-21grep20:05:312
1943991310,2cyclictest14045-21grep23:15:182
194399130,1cyclictest0-21swapper/223:56:042
194399130,13cyclictest0-21swapper/221:58:442
194399130,12cyclictest23820-21cron19:55:012
193899139,3cyclictest20276-21basename22:35:001
193899136,3cyclictest0-21swapper/100:05:251
193899131,2cyclictest0-21swapper/121:18:031
1938991310,2cyclictest6704-21ssh21:40:191
1938991310,2cyclictest26363-21wc21:05:221
1938991310,2cyclictest25249-21wc21:30:221
1938991310,2cyclictest23765-21wc23:45:011
1938991310,2cyclictest21380-21head23:20:171
1938991310,2cyclictest10356-21tr22:50:151
193899130,2cyclictest1929-21cyclictest23:01:471
193899130,12cyclictest20569-21grep20:55:181
193899130,12cyclictest0-21swapper/120:10:141
193299130,12cyclictest0-21swapper/021:00:150
194599129,2cyclictest4830-21ls20:20:233
194599129,2cyclictest4593-21chrt21:39:193
194599129,2cyclictest24345-21cut23:00:163
194599129,2cyclictest23481-21head21:00:193
194599129,2cyclictest22588-21sed22:35:293
194599129,2cyclictest21115-21tail20:55:203
194599129,2cyclictest20707-21awk19:45:223
194599129,2cyclictest17740-21grep21:25:233
194599129,2cyclictest14380-21head21:45:243
194599129,2cyclictest10879-21tr20:35:153
194599128,3cyclictest27216-21diskmemload21:22:033
194399129,2cyclictest979-21seq23:50:392
194399129,2cyclictest6659-21grep21:40:192
194399129,2cyclictest31964-21grep20:10:182
194399129,2cyclictest2763-21grep00:15:272
194399129,2cyclictest20328-21tr00:30:132
194399129,2cyclictest20328-21tr00:30:132
194399129,2cyclictest20247-21cron22:35:002
194399129,2cyclictest17035-21tr22:10:142
194399128,3cyclictest23495-21head21:00:192
194399128,3cyclictest21583-21wc20:55:252
194399128,3cyclictest20581-21ssh23:41:532
194399127,4cyclictest6553-21taskset19:18:362
194399127,3cyclictest16580-21apt-key19:40:002
194399123,9cyclictest61850irq/126-enp2s0-21:31:222
194399123,5cyclictest224-21systemd-journal19:50:002
194399122,9cyclictest29856-21rm23:49:052
194399120,1cyclictest0-21swapper/222:07:042
193899129,2cyclictest865-21grep23:05:291
193899129,2cyclictest3813-21grep23:30:231
193899129,2cyclictest31172-21ls20:10:011
193899129,2cyclictest28600-21cron20:05:001
193899129,2cyclictest28185-21grep21:55:191
193899129,2cyclictest28047-21wget22:40:151
193899129,2cyclictest27115-21expr00:35:171
193899129,2cyclictest22269-21df19:50:161
193899129,2cyclictest21539-21tail22:35:201
193899129,2cyclictest21286-21tr00:30:191
193899129,2cyclictest21286-21tr00:30:191
193899129,2cyclictest19768-21cat00:30:001
193899129,2cyclictest14587-21grep19:35:151
193899129,2cyclictest13058-21wc20:40:001
193899129,2cyclictest10089-21grep23:35:171
193899127,4cyclictest7744-21snmpd22:12:201
1938991212,0cyclictest0-21swapper/121:20:091
1938991211,1cyclictest0-21swapper/121:50:181
1938991210,1cyclictest25-21ksoftirqd/100:22:251
193299120,3cyclictest224-21systemd-journal20:16:550
194599118,2cyclictest29398-21sed21:55:303
194599118,2cyclictest27244-21grep20:00:203
194599118,2cyclictest24845-21grep19:55:193
194599118,2cyclictest24644-21tail21:30:173
194599118,2cyclictest24558-21ssh00:08:533
194599118,2cyclictest22418-21cut19:50:173
194599118,2cyclictest10095-21sed19:25:193
194399118,2cyclictest7891-21ssh22:48:242
194399118,2cyclictest21172-21expr22:35:182
194399118,2cyclictest1876-21cut20:15:182
194399118,2cyclictest12601-21taskset20:37:002
194399118,2cyclictest11389-21taskset19:29:262
194399118,2cyclictest10390-21sh00:21:352
194399115,2cyclictest224-21systemd-journal19:40:012
194399115,2cyclictest224-21systemd-journal19:40:002
193899119,1cyclictest3707-21cut22:45:181
193899119,1cyclictest25293-21ls19:55:231
193899118,2cyclictest8624-21grep20:30:171
193899118,2cyclictest8367-21ssh22:25:341
193899118,2cyclictest6722-21grep20:25:191
193899118,2cyclictest6075-21cut23:55:191
193899118,2cyclictest6016-21taskset23:10:081
193899118,2cyclictest5347-21tail19:15:171
193899118,2cyclictest4339-21grep20:20:191
193899118,2cyclictest27455-21ssh21:10:011
193899118,2cyclictest24055-21ssh23:45:091
193899118,2cyclictest20605-21wc19:45:221
193899118,2cyclictest17251-21tr19:40:161
193899118,2cyclictest17251-21tr19:40:161
193899118,2cyclictest17053-21taskset22:55:051
193899118,2cyclictest16401-21basename20:45:201
193899118,2cyclictest14057-21grep23:15:181
193899118,2cyclictest13815-21tail21:45:191
193899118,2cyclictest12356-21ssh00:00:011
193899118,2cyclictest10081-21df22:05:151
193899113,1cyclictest0-21swapper/122:03:001
194599108,1cyclictest8576-21cat20:30:173
194599108,1cyclictest6111-21cat19:15:263
194599108,1cyclictest12382-21tune2fs19:30:183
194599107,2cyclictest31881-21grep20:10:183
194599107,2cyclictest29163-21tr20:05:193
194599107,2cyclictest16876-21sort19:40:013
194599107,2cyclictest16876-21sort19:40:013
194399108,1cyclictest6958-21chrt22:25:182
194399107,2cyclictest26722-21tr20:00:172
194399107,2cyclictest24718-21grep19:55:182
194399107,2cyclictest19779-21taskset20:51:362
194399107,2cyclictest13978-21if_enp0s31f620:40:182
194399105,4cyclictest26207-21head21:05:202
193899108,1cyclictest7439-21expr19:20:161
193899108,1cyclictest32082sleep120:18:391
193899108,1cyclictest31382-21ssh22:20:081
193899108,1cyclictest24360-21sed21:00:291
193899108,1cyclictest13466-21expr20:40:151
193899107,2cyclictest31870-21sh23:50:161
193899107,2cyclictest2924-21grep19:10:171
193899107,2cyclictest20036-21chrt20:54:541
193899107,2cyclictest14103-21gpgv19:35:001
19459997,1cyclictest25410-21cat21:05:163
19459997,1cyclictest17916-21taskset20:50:113
19459996,2cyclictest8289-21which19:20:233
19439997,1cyclictest88472sleep219:22:182
19389997,1cyclictest9296-21chrt19:25:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional