You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 09:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 03, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22333210592,9sleep30-21swapper/319:09:143
2215829481,9sleep10-21swapper/119:06:591
2226328564,6sleep00-21swapper/019:08:190
2222528366,13sleep20-21swapper/219:07:492
2248999310,31cyclictest0-21swapper/122:46:491
2248999310,31cyclictest0-21swapper/122:46:491
22485993028,1cyclictest13-21ksoftirqd/022:55:510
2248599242,2cyclictest141rcu_preempt23:32:530
22485992321,2cyclictest13-21ksoftirqd/022:40:010
22492992221,1cyclictest0-21swapper/223:00:152
22485992216,4cyclictest13-21ksoftirqd/022:28:280
2248599215,16cyclictest13-21ksoftirqd/019:40:150
2248599210,7cyclictest0-21swapper/023:10:140
2248599210,18cyclictest0-21swapper/000:06:250
303882200,1sleep218424-21kworker/2:221:20:032
2249299201,1cyclictest30627-21ssh23:11:532
2249299200,1cyclictest0-21swapper/200:02:582
2248599206,5cyclictest13-21ksoftirqd/023:49:250
2248599205,2cyclictest141rcu_preempt23:37:510
2248599205,1cyclictest141rcu_preempt22:33:410
22492991917,2cyclictest94-21kswapd021:45:122
2248599196,1cyclictest141rcu_preempt23:29:010
22485991916,2cyclictest13-21ksoftirqd/022:50:010
22485991916,2cyclictest13-21ksoftirqd/022:50:010
2248599190,1cyclictest141rcu_preempt22:41:350
22492991815,2cyclictest20278-21grep22:20:152
2249299180,0cyclictest0-21swapper/223:43:382
2248599187,6cyclictest13606-21perf21:05:000
2248599182,2cyclictest141rcu_preempt20:13:570
2248599182,2cyclictest141rcu_preempt00:00:390
2248599180,15cyclictest0-21swapper/021:38:310
2248599180,11cyclictest0-21swapper/021:51:260
2249599172,10cyclictest30967-21apt-get22:50:143
2249299170,2cyclictest0-21swapper/200:27:002
2248599178,1cyclictest141rcu_preempt21:16:280
2248599172,6cyclictest13-21ksoftirqd/000:31:300
22485991710,3cyclictest13-21ksoftirqd/022:20:000
2249599162,10cyclictest9566-21cron19:50:003
22495991613,2cyclictest29402-21sh00:20:273
22495991613,2cyclictest29402-21sh00:20:273
2249599160,16cyclictest0-21swapper/322:28:383
2249299160,16cyclictest0-21swapper/221:31:222
2248999160,2cyclictest0-21swapper/100:35:531
2248599162,1cyclictest141rcu_preempt00:27:220
22485991616,0cyclictest0-21swapper/021:49:540
22485991615,1cyclictest0-21swapper/021:11:270
22485991614,2cyclictest13-21ksoftirqd/022:04:280
2248599160,4cyclictest0-21swapper/021:34:130
2248599160,14cyclictest0-21swapper/019:35:160
22495991513,1cyclictest13007-31frcode00:10:183
2249599150,14cyclictest0-21swapper/321:40:573
22492991515,0cyclictest0-21swapper/223:29:402
22492991515,0cyclictest0-21swapper/222:49:062
22492991515,0cyclictest0-21swapper/222:49:062
2248999154,9cyclictest26956-21apt-get20:25:161
22489991510,3cyclictest9737-21tune2fs20:55:161
2248999150,14cyclictest0-21swapper/122:00:131
2248599152,6cyclictest13-21ksoftirqd/021:55:180
2248599152,10cyclictest13-21ksoftirqd/019:20:160
22485991514,1cyclictest13-21ksoftirqd/023:22:160
22485991511,2cyclictest13202-21wc19:55:190
22485991511,1cyclictest15977-21diskmemload22:50:140
22485991510,2cyclictest13-21ksoftirqd/021:25:430
2248599150,1cyclictest0-21swapper/023:57:360
2248599150,10cyclictest24100-21sendmail20:20:000
2249599149,4cyclictest0-21swapper/323:10:163
2249599144,9cyclictest0-21swapper/323:00:173
2249299143,10cyclictest19167-21ssh21:56:252
22492991413,1cyclictest0-21swapper/200:12:492
22492991412,1cyclictest0-21swapper/222:19:182
2249299140,14cyclictest0-21swapper/223:09:212
2248999148,4cyclictest12826-21aten2_r7power_v23:00:161
22489991413,1cyclictest28392-21apache223:29:231
22489991411,2cyclictest10765-21wc23:20:251
22489991410,3cyclictest2192-21grep23:15:141
22489991410,2cyclictest1766-21grep19:30:261
2248599142,7cyclictest15400-21/usr/sbin/munin00:10:190
2248599140,2cyclictest0-21swapper/023:54:480
2248599140,1cyclictest0-21swapper/022:07:590
115522140,0sleep30-21swapper/321:29:303
2249599139,3cyclictest8992-21cat00:05:223
2249599139,2cyclictest27040-21grep23:55:143
2249599135,5cyclictest0-21swapper/323:35:153
2249599131,11cyclictest558-21lldpd19:53:113
2249599131,0cyclictest0-21swapper/322:37:593
22495991310,2cyclictest5890-21grep23:40:153
22495991310,2cyclictest27711-21head20:25:203
22495991310,2cyclictest23507-21cut21:15:133
22495991310,2cyclictest22805-21cat00:15:273
22495991310,2cyclictest20534-21tune2fs21:35:173
22495991310,2cyclictest18660-21grep20:05:263
22495991310,2cyclictest18249-21tail23:25:253
22495991310,2cyclictest14492-21grep21:30:283
22495991310,2cyclictest14173-21wc22:15:243
2249299139,3cyclictest3080-21wc23:15:182
2249299139,3cyclictest10602-21grep19:50:192
22492991310,2cyclictest9873-21head21:50:202
22492991310,2cyclictest32727-21wc22:50:262
2249299130,12cyclictest17278-21grep22:40:182
2249299130,0cyclictest0-21swapper/223:45:282
2248999139,3cyclictest8617-21cat21:49:591
2248999139,3cyclictest26564-21sleep21:40:001
2248999139,2cyclictest14753-21tr00:10:151
2248999136,5cyclictest7744-21snmpd22:10:571
2248999133,9cyclictest5748-21/usr/sbin/munin22:55:151
2248999133,7cyclictest0-21swapper/119:17:191
22489991310,2cyclictest5909-21tr23:40:151
22489991310,2cyclictest27359-21taskset21:40:171
22489991310,2cyclictest22213-21sshd00:15:201
22489991310,2cyclictest20661-21grep23:50:191
22489991310,2cyclictest14102-21wc22:15:241
22489991310,2cyclictest11489-21grep19:50:291
22485991310,2cyclictest32378-21cut20:35:180
22485991310,2cyclictest27436-21sendmail19:20:000
22485991310,2cyclictest24840-21basename19:15:000
22485991310,2cyclictest22289-21wc00:15:200
2248599130,2cyclictest0-21swapper/023:00:440
2248599130,1cyclictest0-21swapper/019:50:160
2249599129,2cyclictest9715-21cut20:55:163
2249599129,2cyclictest9216-21tr23:20:133
2249599129,2cyclictest6520-21uname22:55:203
2249599129,2cyclictest6336-21wc19:40:233
2249599129,2cyclictest5210-21if_err_enp0s31f20:45:213
2249599129,2cyclictest4121-21ssh21:45:463
2249599129,2cyclictest3953-21date20:45:003
2249599129,2cyclictest31086-21sed20:30:263
2249599129,2cyclictest29006-21head19:20:243
2249599129,2cyclictest24124-21ls23:30:163
2249599129,2cyclictest23223-21cron22:45:003
2249599129,2cyclictest1999-21grep00:25:153
2249599129,2cyclictest18257-21grep21:10:273
2249599129,2cyclictest14277-21sh23:45:263
2249599129,2cyclictest14005-21chrt21:53:223
2249599128,3cyclictest22956-21seq23:51:363
2249599127,3cyclictest23525-21ssh22:45:063
2249599127,3cyclictest23525-21ssh22:45:063
2249599123,1cyclictest15977-21diskmemload22:20:153
2249599122,7cyclictest24863-21grep22:00:223
2249599120,12cyclictest0-21swapper/300:01:463
2249299129,3cyclictest37850irq/131-snd_hda00:30:182
2249299129,2cyclictest4308-21grep19:35:282
2249299129,2cyclictest2834-21grep20:40:192
2249299129,2cyclictest23194-21head20:15:232
2249299129,2cyclictest20154-21tr21:35:152
2249299129,2cyclictest16768-21cstates21:10:162
2249299129,2cyclictest16033-21systemctl22:40:002
2249299129,2cyclictest15320-21grep20:00:182
2249299129,2cyclictest13852-21sed19:55:272
2249299129,2cyclictest13139-21ssh00:09:052
2249299128,3cyclictest27325-21wc00:20:002
2249299126,2cyclictest6169-21cut21:25:192
22492991211,1cyclictest26025-21ssh22:01:032
2249299120,2cyclictest0-21swapper/222:12:412
2249299120,1cyclictest0-21swapper/221:16:202
2249299120,11cyclictest2214-21apt-get19:35:012
2248999129,2cyclictest8579-21tr19:45:201
2248999129,2cyclictest32480-21tail20:35:181
2248999129,2cyclictest31795-21sed19:25:271
2248999129,2cyclictest31322-21sort21:20:181
2248999129,2cyclictest27184-21tr23:10:151
2248999129,2cyclictest2321-21grep00:00:171
2248999129,2cyclictest20610-21grep23:05:181
2248999129,2cyclictest17585-21chrt20:05:171
2248999129,2cyclictest16977-21grep21:55:171
2248999128,3cyclictest24828-21wc23:30:201
2248999128,3cyclictest15073-21grep21:05:221
2248999128,2cyclictest14316-21cron19:59:591
2248999122,6cyclictest0-21swapper/121:50:071
2248999120,11cyclictest22783-21cut19:10:151
2248999120,10cyclictest0-21swapper/122:20:141
2248599129,2cyclictest7372-21cut20:50:170
2248599129,2cyclictest31807-21tail19:25:270
2248599129,2cyclictest27891-21cut21:40:190
2248599129,2cyclictest17710-21head20:05:180
2248599128,4cyclictest13-21ksoftirqd/023:19:540
2248599128,3cyclictest1590-21wc00:25:010
2248599128,3cyclictest1590-21wc00:25:010
2248599128,3cyclictest11209-21awk21:00:000
2248599124,4cyclictest13-21ksoftirqd/022:10:220
2248599120,11cyclictest2462-21tail20:40:170
2249599118,2cyclictest9707-21rm22:12:363
2249599118,2cyclictest7843-21taskset22:34:163
2249599118,2cyclictest6981-21tr20:50:163
2249599118,2cyclictest32522-21grep22:05:263
2249599118,2cyclictest32253-21cat21:20:273
2249599118,2cyclictest25577-21grep19:15:163
2249599118,2cyclictest22095-21ssh23:05:393
2249599118,2cyclictest2135-21wc23:15:133
2249599118,2cyclictest21158-21chrt20:11:083
2249599118,2cyclictest19997-21rm21:57:113
2249599118,2cyclictest16304-21grep20:00:273
2249599118,2cyclictest15230-21tr00:35:163
2249599118,2cyclictest1443-21cut19:30:233
2249599118,2cyclictest12231-21head21:00:173
2249599117,3cyclictest24946-21cat19:15:003
2249599112,6cyclictest0-21swapper/300:30:133
2249299119,1cyclictest29214-21cat19:20:282
2249299119,1cyclictest17696-21grep20:05:182
2249299118,2cyclictest7952-21ssh22:34:222
2249299118,2cyclictest7044-21cat20:50:172
2249299118,2cyclictest5842-21cat22:55:152
2249299118,2cyclictest31887-21sh22:05:192
2249299118,2cyclictest31685-21ssh21:42:592
2249299118,2cyclictest30770-21ssh23:35:062
2249299118,2cyclictest30293-21ssh22:26:322
2249299118,2cyclictest30267-21aten2_r7power_v19:25:152
2249299118,2cyclictest30032-21cut20:30:172
2249299118,2cyclictest29119-21sh20:30:002
2249299118,2cyclictest28262-21sh00:20:162
2249299118,2cyclictest28262-21sh00:20:152
2249299118,2cyclictest26702-21ssh23:55:022
2249299118,2cyclictest26011-21grep19:15:182
2249299118,2cyclictest17253-21ssh00:35:472
2249299118,2cyclictest11631-21sh23:20:592
2249299118,2cyclictest10694-21grep20:55:262
2249299115,4cyclictest20129-21cpuspeed_turbos23:50:172
2248999118,2cyclictest9781-21wc00:30:221
2248999118,2cyclictest9034-21ntpq00:05:221
2248999118,2cyclictest6651-21wc19:40:271
2248999118,2cyclictest6387-21sh22:32:501
2248999118,2cyclictest4546-21grep20:45:171
2248999118,2cyclictest30923-21chrt23:35:141
2248999118,2cyclictest29790-21ssh23:56:201
2248999118,2cyclictest29653-21grep20:30:151
2248999118,2cyclictest29529-21ssh22:25:461
2248999118,2cyclictest2863-21tr20:40:201
2248999118,2cyclictest28548-21sed19:20:201
2248999118,2cyclictest25735-21ssh21:15:531
2248999118,2cyclictest22713-21head20:15:191
2248999118,2cyclictest20140-21grep20:10:191
2248999118,2cyclictest19594-21ssh21:11:381
2248999118,2cyclictest1917-21ssh22:07:041
2248999118,2cyclictest17271-21grep22:40:171
2248999118,2cyclictest1468-21cron00:25:001
2248999118,2cyclictest1468-21cron00:24:591
2248999118,2cyclictest10202-21ssh21:28:131
2248999117,3cyclictest2824-21grep19:35:161
2248999117,3cyclictest16011-21cron22:40:001
2248999116,3cyclictest13407-21head21:30:171
2248999113,5cyclictest0-21swapper/123:45:261
2248999113,4cyclictest224-21systemd-journal20:54:191
2248999110,10cyclictest32550-21ssh22:50:251
2248599118,2cyclictest30631-21ssh21:20:150
2248599118,2cyclictest27232-21expr20:25:170
2248599118,2cyclictest25073-21tail20:20:180
2248599118,2cyclictest20641-21sort22:20:170
2248599118,2cyclictest19929-21ssh00:38:460
2248599114,5cyclictest13-21ksoftirqd/023:40:270
2248599112,3cyclictest13-21ksoftirqd/019:31:000
2248599111,1cyclictest141rcu_preempt23:05:490
2249599108,1cyclictest30442-21seq19:25:153
2249599108,1cyclictest25815-21sed20:20:263
2249599108,1cyclictest23029-21cat20:15:213
2249599108,1cyclictest13807-21seq21:05:023
2249599107,2cyclictest32526-21expr20:35:193
2249599100,9cyclictest3331-21fschecks_time19:35:193
2249599100,6cyclictest224-21systemd-journal19:55:203
2249299108,2cyclictest15832-21chrt21:08:292
2249299108,1cyclictest24677-21cat20:20:152
2249299108,1cyclictest115812sleep221:00:162
2249299107,2cyclictest9585-21awk19:50:002
2249299107,2cyclictest1014-21sed20:35:272
2249299101,2cyclictest224-21systemd-journal20:46:532
22492991010,0cyclictest0-21swapper/223:30:022
2249299100,9cyclictest7114-21seq19:44:122
2248999108,1cyclictest26131-21taskset20:20:491
2248999108,1cyclictest1942-21ssh00:25:141
2248999108,1cyclictest15703-21grep20:00:211
2248999108,1cyclictest13482-21chrt21:03:511
2248599108,1cyclictest7746-21cut19:45:140
2248599108,1cyclictest16531-21chrt20:02:440
2248599108,1cyclictest156402sleep021:06:020
2248599107,2cyclictest6089-21chrt20:45:470
224929997,1cyclictest22934-21grep19:10:152
224929993,3cyclictest224-21systemd-journal20:10:002
224859994,4cyclictest29388-21apt-get20:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional