You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 02:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Nov 26, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17841210382,10sleep30-21swapper/319:06:523
1808529986,9sleep10-21swapper/119:10:001
1776928169,8sleep00-21swapper/019:05:570
1599828069,6sleep20-21swapper/219:05:152
1817199300,1cyclictest10803-21diskmemload22:08:532
1817199261,12cyclictest10803-21diskmemload00:20:582
261442240,0sleep326145-21sleep22:35:013
1817199240,1cyclictest0-21swapper/223:02:282
253742230,0sleep00-21swapper/000:11:140
1817199230,8cyclictest0-21swapper/220:14:182
1816799230,1cyclictest0-21swapper/123:46:131
1816799230,18cyclictest0-21swapper/122:47:461
245262220,0sleep10-21swapper/122:09:101
1817599221,3cyclictest141rcu_preempt21:30:383
1817599200,11cyclictest0-21swapper/321:15:133
1816799200,20cyclictest0-21swapper/123:08:301
1816199202,17cyclictest0-21swapper/022:10:160
1817599190,2cyclictest0-21swapper/323:37:033
1817599190,2cyclictest0-21swapper/323:37:023
18171991911,7cyclictest22998-21sh23:20:372
1816799190,19cyclictest0-21swapper/100:27:351
18161991915,3cyclictest13-21ksoftirqd/021:57:220
18175991817,1cyclictest0-21swapper/319:45:133
1817199183,6cyclictest7744-21snmpd23:44:542
1816799180,1cyclictest0-21swapper/123:18:511
1816799180,0cyclictest0-21swapper/123:28:161
1816199181,13cyclictest0-21swapper/023:05:160
1817599176,8cyclictest846-21python21:50:283
1817599172,2cyclictest0-21swapper/321:49:273
18175991716,1cyclictest0-21swapper/323:08:303
1817599171,14cyclictest0-21swapper/323:28:163
1817199179,2cyclictest14560-21head22:25:212
1817199178,3cyclictest16183-21tr00:05:152
1817199177,5cyclictest24290-21ssh00:10:282
18171991716,1cyclictest37850irq/131-snd_hda22:19:512
1817199171,4cyclictest0-21swapper/222:20:162
18171991713,3cyclictest32-21ksoftirqd/223:30:132
1817199171,1cyclictest0-21swapper/220:40:132
1817199171,1cyclictest0-21swapper/220:40:132
18171991710,2cyclictest5411-21cat20:55:262
1816199175,6cyclictest26503-21perf20:35:010
18161991713,4cyclictest13-21ksoftirqd/022:05:230
1816199170,2cyclictest0-21swapper/023:20:590
1817199160,16cyclictest0-21swapper/222:12:472
18167991616,0cyclictest0-21swapper/121:49:271
18167991611,2cyclictest28647-21apache221:15:261
1816799160,16cyclictest0-21swapper/122:38:151
1816799160,16cyclictest0-21swapper/121:37:021
1816799160,12cyclictest0-21swapper/120:30:211
1816799160,12cyclictest0-21swapper/119:45:131
18175991513,1cyclictest0-21swapper/300:09:443
1817599150,1cyclictest0-21swapper/322:39:043
1817599150,15cyclictest0-21swapper/323:49:113
1817599150,14cyclictest0-21swapper/322:54:353
18171991513,1cyclictest0-21swapper/221:30:472
1817199150,1cyclictest0-21swapper/200:27:092
1816799150,1cyclictest0-21swapper/122:43:101
1816799150,12cyclictest0-21swapper/121:53:261
1816199151,10cyclictest0-21swapper/022:50:110
1816199150,0cyclictest0-21swapper/022:44:480
1817599145,7cyclictest10803-21diskmemload00:35:163
1817599142,4cyclictest0-21swapper/300:25:143
1817599141,12cyclictest14485-21apache_accesses00:30:113
18175991410,3cyclictest14023-21basename23:15:013
18171991410,3cyclictest8398-21tail19:55:162
18171991410,3cyclictest28847-21tr23:25:212
1817199140,1cyclictest0-21swapper/221:36:082
1816799140,13cyclictest0-21swapper/122:22:591
1816799140,13cyclictest0-21swapper/100:10:381
1816199144,9cyclictest26195-21python21:20:270
18161991410,3cyclictest7427-21sleep19:55:000
1816199140,12cyclictest0-21swapper/023:13:450
18175991312,1cyclictest0-21swapper/322:06:393
18175991310,2cyclictest30118-21basename19:34:593
18175991310,2cyclictest20811-21ls20:20:213
18175991310,2cyclictest18938-21grep20:15:273
18175991310,2cyclictest11787-21sed20:00:273
1817599130,12cyclictest0-21swapper/322:47:463
18171991310,2cyclictest7912-21grep21:00:272
18171991310,2cyclictest26445-21grep21:20:282
18171991310,2cyclictest25994-21grep20:30:272
18171991310,2cyclictest21834-21wc22:55:202
1817199130,12cyclictest14792-21tr00:30:152
1816799134,4cyclictest10789-21idleruntime-cro21:10:001
1816199139,3cyclictest22295-21sed23:45:170
1816199131,2cyclictest0-21swapper/023:34:510
18161991310,2cyclictest31326-21awk20:45:010
18161991310,2cyclictest31326-21awk20:45:000
18161991310,2cyclictest30132-21cat19:34:590
18161991310,2cyclictest1677-21awk19:40:200
18161991310,2cyclictest104372sleep000:00:240
1817599129,2cyclictest7948-21grep19:55:143
1817599129,2cyclictest7089-21grep22:20:213
1817599129,2cyclictest31514-21ssh00:15:563
1817599129,2cyclictest27504-21ssh23:00:163
1817599129,2cyclictest23912-21grep19:20:183
1817599129,2cyclictest15239-21tail23:15:183
1817599129,2cyclictest14465-21ntp_states22:25:203
1817599124,3cyclictest14847-21apt-get20:10:173
18175991210,1cyclictest3241-21grep23:55:173
18175991210,1cyclictest2246-21grep19:40:263
1817599120,5cyclictest0-21swapper/323:30:413
1817599120,11cyclictest29893-21grep20:40:163
1817599120,11cyclictest29893-21grep20:40:163
1817599120,11cyclictest0-21swapper/322:41:023
1817199129,2cyclictest999-21grep22:40:182
1817199129,2cyclictest8912-21tr21:05:142
1817199129,2cyclictest6562-21grep21:55:172
1817199129,2cyclictest29975-21sed00:15:182
1817199129,2cyclictest28727-21cut23:50:162
1817199129,2cyclictest27248-21grep19:25:282
1817199129,2cyclictest26753-21grep22:35:152
1817199129,2cyclictest23454-21chrt22:32:152
1817199129,2cyclictest21879-21grep19:15:182
1817199129,2cyclictest20536-21munin-setwarn19:15:012
1817199129,2cyclictest17891-21sort20:15:182
1817199129,2cyclictest1284-21wc20:50:002
1817199129,2cyclictest12022-21grep21:10:182
1817199128,3cyclictest8997-21ssh22:45:252
1817199128,3cyclictest27395-21cut20:35:182
1817199128,3cyclictest2342-21ssh21:27:062
1817199128,3cyclictest23311-21ssh23:45:272
1817199128,3cyclictest21559-21tr00:35:232
1817199120,12cyclictest0-21swapper/222:52:572
1816799129,2cyclictest7408-21cron19:55:011
1816799129,2cyclictest28601-21cut19:30:181
1816799129,2cyclictest2679-21basename19:45:001
1816799129,2cyclictest19347-21grep21:40:181
1816799129,2cyclictest13223-21chrt22:25:121
1816799129,2cyclictest10853-21wc00:00:301
1816799128,3cyclictest4111-21sort00:20:201
1816799128,3cyclictest29535-21ssh23:50:231
1816799128,3cyclictest27775-21ssh22:10:321
1816799128,3cyclictest22938-21taskset00:36:271
1816799128,3cyclictest13354-21ssh23:14:181
1816799125,2cyclictest224-21systemd-journal22:32:291
1816799121,6cyclictest49-21kcompactd020:27:441
18167991211,1cyclictest0-21swapper/123:03:461
18167991210,2cyclictest6885-21kworker/u8:200:06:221
1816799120,11cyclictest31545-21tr21:25:141
1816199129,2cyclictest7651-21head22:20:220
1816199129,2cyclictest4343-21cut00:20:220
1816199129,2cyclictest26520-21basename22:35:150
1816199129,2cyclictest26011-21cron22:35:000
1816199129,2cyclictest23963-21grep19:20:170
1816199129,2cyclictest21459-21wc00:35:210
1816199129,2cyclictest19337-21dump-pmu-power20:20:000
1816199129,2cyclictest10904-21cat21:10:010
1816199129,2cyclictest10305-21tr20:00:160
1816199128,3cyclictest8987-21grep23:35:180
1816199128,3cyclictest8987-21grep23:35:170
1816199123,5cyclictest493-21systemd-logind00:09:570
1816199122,6cyclictest19688-21tr20:20:150
1816199120,11cyclictest31811-21tr20:45:170
1817599119,2cyclictest32299-21dpkg22:15:003
1817599119,2cyclictest21380-21uniq19:15:143
1817599119,2cyclictest1821-21ssh21:26:353
1817599119,2cyclictest17504-21ssh21:39:373
1817599119,1cyclictest6689-21expr21:00:163
1817599119,1cyclictest6062-21sed19:50:173
1817599119,1cyclictest5814-21aten2_r7power_c21:55:133
1817599119,1cyclictest32315-21head20:45:183
1817599119,1cyclictest294472sleep323:50:223
1817599119,1cyclictest26858-21aten2_r7power_v20:35:163
1817599119,1cyclictest22877-21cpuspeed_turbos00:10:163
1817599119,1cyclictest22452-21expr20:25:153
1817599119,1cyclictest21921-21sh21:42:043
1817599119,1cyclictest19154-21ssh23:43:053
1817599119,1cyclictest18452-21basename19:10:153
1817599119,1cyclictest15432-21cron00:05:003
1817599118,3cyclictest5086-21taskset00:20:393
1817599118,2cyclictest26342-21grep19:25:193
1817599118,2cyclictest23406-21ssh23:21:053
1817599118,2cyclictest2296-21grep20:50:193
1817599118,2cyclictest13682-21grep20:05:213
1817599118,2cyclictest13483-21ssh22:00:203
1817599117,4cyclictest28647-21apache221:10:083
1817599115,3cyclictest0-21swapper/321:21:273
1817599110,10cyclictest26369-21df22:10:193
1817199118,2cyclictest9234-21head23:10:232
1817199118,2cyclictest5440-21basename21:55:012
1817199118,2cyclictest32475-21taskset19:37:222
1817199118,2cyclictest29151-21ntpq19:30:232
1817199118,2cyclictest24396-21grep19:20:222
1817199118,2cyclictest2205-21ssh23:55:032
1817199118,2cyclictest192942sleep221:15:212
1817199118,2cyclictest18877-21ssh23:18:142
1817199118,2cyclictest1268-21grep19:40:172
1817199118,2cyclictest11429-21ssh00:00:482
1817199111,3cyclictest21186-21sh21:41:122
1817199110,9cyclictest4786-21sshd19:46:552
1817199110,11cyclictest0-21swapper/222:03:322
1817199110,11cyclictest0-21swapper/221:46:042
1817199110,10cyclictest0-21swapper/223:08:322
1816799119,1cyclictest847-21sed20:45:271
1816799119,1cyclictest4186-21tr20:55:141
1816799119,1cyclictest27485-21tail20:35:181
1816799119,1cyclictest20968-21grep22:55:151
1816799119,1cyclictest11169-21grep20:00:221
1816799118,2cyclictest9367-21grep19:55:261
1816799118,2cyclictest83932sleep121:04:501
1816799118,2cyclictest7918-21ssh21:55:381
1816799118,2cyclictest7538-21sh21:30:391
1816799118,2cyclictest7488-21ssh23:34:421
1816799118,2cyclictest30225-21ssh00:15:191
1816799118,2cyclictest27352-21sh21:21:201
1816799118,2cyclictest2715-21tr23:55:141
1816799118,2cyclictest24267-21grep19:20:191
1816799118,2cyclictest21196-21tr23:20:171
1816799118,2cyclictest2111-21grep22:15:261
1816799118,2cyclictest17972-21tail20:15:181
1816799118,2cyclictest169362sleep123:40:421
1816799118,2cyclictest16852-21ssh00:31:051
1816799118,2cyclictest1530-21apache_volume20:50:131
1816799118,2cyclictest15291-21sed22:50:171
1816799118,2cyclictest14583-21ssh21:11:541
1816799117,3cyclictest12228-21ssh22:00:021
1816799116,4cyclictest8891-21fschecks_count23:35:171
1816799116,4cyclictest8891-21fschecks_count23:35:171
1816799112,5cyclictest1-21systemd20:20:401
1816199119,1cyclictest1513-21ls22:15:210
1816199118,2cyclictest8634-21head22:45:200
1816199118,2cyclictest7533-21grep21:00:220
1816199118,2cyclictest727-21ssh23:04:440
1816199118,2cyclictest6205-21tune2fs21:30:220
1816199118,2cyclictest5568-21sed20:55:270
1816199118,2cyclictest4684-21ssh23:55:480
1816199118,2cyclictest3997-21ssh21:53:290
1816199118,2cyclictest32321-21ssh23:28:340
1816199118,2cyclictest31312-21grep19:35:180
1816199118,2cyclictest30956-21grep00:15:260
1816199118,2cyclictest29275-21ssh23:50:200
1816199118,2cyclictest28951-21basename20:40:000
1816199118,2cyclictest24929-21ssh21:45:050
1816199118,2cyclictest21794-21kernelversion22:55:190
1816199118,2cyclictest21112-21ssh21:16:470
1816199118,2cyclictest18594-21taskset19:10:160
1816199118,2cyclictest1859-21expr20:50:150
1816199118,2cyclictest18399-21ssh22:04:420
1816199118,2cyclictest17668-21taskset23:16:530
1816199118,2cyclictest17208-21ssh00:31:370
1816199118,2cyclictest16996-21ssh23:40:480
1816199118,2cyclictest15322-21df20:10:200
1816199118,2cyclictest14682-21wc20:10:000
1816199118,2cyclictest1343-21taskset21:26:050
1816199118,2cyclictest13195-21ssh21:10:270
1816199118,2cyclictest12103-21cat21:35:180
1816199118,2cyclictest11547-21ssh00:26:420
1816199117,3cyclictest14027-21cut22:25:160
1816199114,3cyclictest224-21systemd-journal19:29:590
1817599109,1cyclictest3938-21wget20:55:143
1817599109,1cyclictest10482-21chrt21:06:123
1817599108,2cyclictest24836-21expr20:30:153
1817599108,2cyclictest22762-21sh22:55:293
1817199108,1cyclictest19892-21expr20:20:142
1817199107,2cyclictest8885-21cut23:35:162
1817199107,2cyclictest8885-21cut23:35:162
1817199107,2cyclictest13569-21head20:05:192
1817199107,2cyclictest10328-21grep20:00:152
1817199100,9cyclictest23947-21chrt20:28:092
1816799107,2cyclictest31798-21grep19:35:231
1816799107,2cyclictest25816-21cut19:25:161
1816799107,2cyclictest21550-21cut19:15:151
1816799107,2cyclictest20527-21basename19:15:001
1816799107,2cyclictest15310-21expr20:10:201
1816799107,2cyclictest13800-21tr20:05:221
1816199108,1cyclictest8078-21cat19:55:140
1816199108,1cyclictest22201-21cat20:25:150
1816199108,1cyclictest19650-21load21:40:200
1816199107,2cyclictest3508-21diskstats19:45:160
1816199107,2cyclictest21057-21tr19:15:130
181759998,1cyclictest31001-21cat19:35:163
181719997,1cyclictest2680-21ntp_states20:50:212
181719996,2cyclictest5348-21aten2_r7power_e19:50:152
181679992,4cyclictest0-21swapper/120:40:131
181679992,4cyclictest0-21swapper/120:40:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional