You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 04:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Nov 21, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21414210188,9sleep10-21swapper/119:06:101
1992529480,9sleep30-21swapper/319:05:173
2152428471,9sleep00-21swapper/019:07:340
150562840,0sleep10-21swapper/123:59:151
2159328366,13sleep20-21swapper/219:08:292
302122780,0sleep10-21swapper/121:20:171
21807993532,2cyclictest32-21ksoftirqd/221:50:002
2181099270,27cyclictest0-21swapper/322:53:353
21807992521,1cyclictest32-21ksoftirqd/221:57:062
2181099210,2cyclictest0-21swapper/323:16:373
21807992119,1cyclictest0-21swapper/221:00:142
21807992116,1cyclictest32-21ksoftirqd/221:25:132
2180699210,2cyclictest0-21swapper/122:27:011
2181099201,2cyclictest141rcu_preempt21:50:223
2181099200,18cyclictest0-21swapper/300:07:123
2181099190,19cyclictest0-21swapper/321:12:333
2181099190,18cyclictest11705-21sh00:21:013
2180799192,1cyclictest141rcu_preempt22:25:332
2180799191,17cyclictest4987-21sed00:15:302
2180799190,19cyclictest0-21swapper/223:44:332
2180699190,18cyclictest0-21swapper/122:46:411
21807991817,1cyclictest0-21swapper/223:25:142
21807991816,1cyclictest23916-21unixbench_multi19:10:272
21807991815,2cyclictest32-21ksoftirqd/221:20:142
2180799180,18cyclictest0-21swapper/219:55:272
2180399186,6cyclictest11979-21munin-run23:10:000
2180399185,7cyclictest29402-21munin-run21:45:010
2180399183,11cyclictest0-21swapper/022:30:140
2180399180,15cyclictest0-21swapper/022:55:140
2180799178,8cyclictest13092-21sh00:22:472
2180799176,2cyclictest4578-21tune2fs21:50:202
21807991716,1cyclictest0-21swapper/223:36:502
21807991715,1cyclictest0-21swapper/222:24:542
2180699173,1cyclictest141rcu_preempt23:52:421
2180399175,6cyclictest224-21systemd-journal22:50:540
2181099160,3cyclictest0-21swapper/300:14:443
2181099160,1cyclictest0-21swapper/323:52:563
2181099160,1cyclictest0-21swapper/322:04:283
2180799161,14cyclictest0-21swapper/223:50:142
2180799160,16cyclictest0-21swapper/221:16:252
21806991614,1cyclictest25-21ksoftirqd/100:25:141
21803991616,0cyclictest0-21swapper/021:11:360
21803991611,4cyclictest24276-21sh23:42:340
2181099151,7cyclictest12136-21idleruntime22:45:193
21810991514,1cyclictest0-21swapper/320:45:173
2180799158,3cyclictest32-21ksoftirqd/222:46:402
2180799152,1cyclictest141rcu_preempt21:44:102
21807991513,2cyclictest32-21ksoftirqd/200:40:002
21807991513,1cyclictest32-21ksoftirqd/200:25:122
21807991513,1cyclictest20300-21ls23:15:192
21807991512,1cyclictest141rcu_preempt22:35:432
21807991511,3cyclictest29534-21dump-pmu-power22:10:002
2180799150,2cyclictest0-21swapper/219:50:132
2180799150,14cyclictest0-21swapper/221:36:292
2180699154,8cyclictest0-21swapper/120:00:151
2180699153,0cyclictest141rcu_preempt22:53:591
2180699151,1cyclictest141rcu_preempt00:01:401
21806991511,4cyclictest10545-21apache223:25:561
2180699150,11cyclictest0-21swapper/122:16:341
22622140,2chrt2260-21grep00:40:000
21810991410,3cyclictest714-21grep23:00:263
21810991410,2cyclictest9501-21grep19:50:163
2181099140,2cyclictest0-21swapper/321:35:223
2180799148,2cyclictest473-21taskset23:00:252
2180799143,1cyclictest141rcu_preempt20:30:282
21807991411,1cyclictest311rcuc/219:20:132
2180799140,13cyclictest24731-21cat20:20:222
2180699148,6cyclictest10223-21munin-node22:42:111
21806991412,2cyclictest0-21swapper/122:14:461
21806991412,1cyclictest241rcuc/100:20:251
21806991410,3cyclictest32119-21fgrep20:35:221
21806991410,3cyclictest16340-21grep20:05:141
2180699140,1cyclictest0-21swapper/123:46:461
2180399147,6cyclictest3348-21perf21:25:010
2180399147,6cyclictest27497-21cut23:45:140
21803991410,2cyclictest11051-21sh19:55:000
2180399140,11cyclictest904-21wget20:40:170
2181099139,3cyclictest29397-21cut00:35:193
21810991313,0cyclictest0-21swapper/322:21:153
21810991312,1cyclictest39-21ksoftirqd/300:30:183
2181099131,1cyclictest0-21swapper/320:35:143
21810991310,2cyclictest5799-21grep23:05:173
21810991310,2cyclictest30171-21cat22:35:003
21810991310,2cyclictest1737-21cat20:40:223
21810991310,2cyclictest1462-21latency_hist19:35:013
21810991310,2cyclictest12077-21cat21:30:293
2181099130,1cyclictest141rcu_preempt23:38:273
2181099130,1cyclictest0-21swapper/300:29:033
2180799139,3cyclictest5661-21sort22:40:212
2180799138,4cyclictest32-21ksoftirqd/200:30:292
2180799135,4cyclictest224-21systemd-journal19:50:002
2180799131,8cyclictest11199-21hwlatdetect21:30:212
21807991312,1cyclictest32-21ksoftirqd/223:06:092
21807991310,2cyclictest21783-21grep20:15:192
2180799130,12cyclictest31138-21grep00:10:292
2180699139,3cyclictest30869-21cron23:00:011
2180699139,3cyclictest25185-21wc20:20:281
2180699139,3cyclictest20812-21wc23:15:261
2180699139,3cyclictest17489-21grep21:35:181
21806991310,2cyclictest9581-21grep20:55:341
21806991310,2cyclictest3960-21tr21:25:141
21806991310,2cyclictest23628-21sed21:15:171
21806991310,2cyclictest15067-21dump-pmu-power21:10:011
21806991310,2cyclictest11718-21sed21:55:261
21806991310,2cyclictest10546-21apache221:52:161
2180399139,3cyclictest9873-21sed21:30:000
2180399139,3cyclictest28851-21sleep00:10:000
2180399139,3cyclictest14487-21grep20:00:180
2180399139,2cyclictest17704-21grep20:05:250
2180399139,2cyclictest14344-21ssh23:35:250
21803991312,1cyclictest0-21swapper/022:25:510
21803991310,2cyclictest3375-21grep20:45:180
21803991310,2cyclictest24316-21tune2fs20:20:180
21803991310,2cyclictest23176-21turbostat20:20:000
21803991310,2cyclictest13213-21head23:10:170
21803991310,2cyclictest12749-21ssh22:20:290
21803991310,2cyclictest12086-21cut19:55:180
2180399130,13cyclictest0-21swapper/023:22:250
2181099129,2cyclictest4843-21grep19:40:143
2181099129,2cyclictest27984-21cat19:20:173
2181099129,2cyclictest27918-21grep23:20:313
2181099129,2cyclictest25374-21grep19:15:173
2181099129,2cyclictest22372-21sort00:05:003
2181099129,2cyclictest21288-21grep20:15:173
2181099129,2cyclictest21045-21grep23:40:173
2181099129,2cyclictest18203-21sh20:10:003
2181099129,2cyclictest14201-21df20:00:173
2181099128,3cyclictest1918-21grep23:25:263
21810991210,1cyclictest29291-21grep23:45:263
2181099120,8cyclictest0-21swapper/323:30:133
2181099120,11cyclictest0-21swapper/300:17:183
2180799129,2cyclictest30080-21apache_processe22:10:182
2180799129,2cyclictest27847-21grep23:20:312
2180799129,2cyclictest24540-21cut22:30:172
2180799129,2cyclictest19794-21unixbench_multi22:50:272
2180799129,2cyclictest18362-21cat20:10:002
2180799129,2cyclictest15828-21ls20:05:002
2180799128,3cyclictest23935-21tr00:05:202
2180799125,1cyclictest141rcu_preempt22:58:072
2180799124,5cyclictest32-21ksoftirqd/200:05:002
2180799123,1cyclictest15078-21diskmemload22:15:512
2180799122,7cyclictest32-21ksoftirqd/220:29:532
21807991211,1cyclictest32-21ksoftirqd/223:47:342
21807991210,2cyclictest32-21ksoftirqd/222:01:212
2180799120,10cyclictest4729-21fschecks_count19:40:132
2180699129,2cyclictest4616-21grep19:40:121
2180699129,2cyclictest32366-21head22:35:301
2180699129,2cyclictest31089-21grep00:10:291
2180699129,2cyclictest29201-21sed20:30:231
2180699129,2cyclictest28151-21cron20:29:591
2180699129,2cyclictest24494-21grep21:40:211
2180699129,2cyclictest2439-21grep19:35:181
2180699129,2cyclictest24241-21head00:30:281
2180699129,2cyclictest20373-21head20:10:311
2180699129,2cyclictest17737-21tail22:00:171
2180699129,2cyclictest16325-21sleep21:35:011
2180699129,2cyclictest12512-21wc21:05:001
2180699123,4cyclictest23374-21/usr/sbin/munin22:05:181
2180699122,8cyclictest32141-21sed23:00:171
2180699121,8cyclictest10945-21ssh22:20:161
2180699120,1cyclictest0-21swapper/123:05:081
2180699120,12cyclictest0-21swapper/122:34:581
2180399129,2cyclictest4347-21tr00:15:230
2180399129,2cyclictest30247-21ssh22:10:230
2180399129,2cyclictest23943-21head19:10:270
2180399129,2cyclictest18599-21wc22:00:250
2180399129,2cyclictest12374-21chrt23:56:330
2180399129,2cyclictest10647-21grep21:00:150
2180399129,2cyclictest1026-21grep19:30:260
2180399127,3cyclictest19862-21grep23:15:170
2180399124,6cyclictest22318-21idleruntime-cro00:05:000
2180399124,4cyclictest224-21systemd-journal00:30:000
21803991211,1cyclictest0-21swapper/020:50:140
2181099119,1cyclictest24362-21sh19:15:003
2181099119,1cyclictest14051-21grep21:05:203
2181099118,2cyclictest8293-21ssh22:17:563
2181099118,2cyclictest7303-21taskset20:50:543
2181099118,2cyclictest5865-21cut22:40:233
2181099118,2cyclictest3239-21sh22:39:063
2181099118,2cyclictest30102-21tr21:45:163
2181099118,2cyclictest28550-21ssh22:08:543
2181099118,2cyclictest28331-21ssh22:57:133
2181099118,2cyclictest1892-21rm21:23:303
2181099118,2cyclictest17394-21aten2_r7power_c22:25:163
2181099118,2cyclictest13220-21taskset19:57:313
2181099118,2cyclictest13026-21df23:10:153
2181099118,2cyclictest12575-21rm21:55:453
2181099118,2cyclictest11912-21sh23:56:023
2181099118,2cyclictest10938-21dnf21:00:173
2181099118,2cyclictest1080-21ssh22:12:133
2181099110,10cyclictest27510-21sh21:18:153
2180799118,2cyclictest8140-21cut23:30:222
2180799118,2cyclictest343-21ls19:30:212
2180799118,2cyclictest32669-21chrt20:37:032
2180799118,2cyclictest16224-21sort21:10:172
2180799118,2cyclictest10426-21cat23:55:212
2180799118,2cyclictest1033-21tr20:40:172
2180799117,3cyclictest35812sleep219:36:492
2180799110,1cyclictest0-21swapper/223:10:142
2180699118,2cyclictest761-21wc19:30:251
2180699118,2cyclictest6923-21ssh00:17:401
2180699118,2cyclictest5564-21tr20:50:141
2180699118,2cyclictest3617-21expr20:45:181
2180699118,2cyclictest3518-21ls21:50:001
2180699118,2cyclictest32269-21ssh00:37:261
2180699118,2cyclictest30624-21tr19:25:221
2180699118,2cyclictest27803-21cut19:20:171
2180699118,2cyclictest26838-21cut23:20:221
2180699118,2cyclictest21770-21grep20:15:191
2180699118,2cyclictest18333-21ssh21:11:191
2180699118,2cyclictest14447-21rm23:35:261
2180699118,2cyclictest12337-21memory19:55:201
2180699111,9cyclictest0-21swapper/100:05:121
2180699110,9cyclictest0-21swapper/119:15:131
2180699110,10cyclictest14479-21sh23:10:361
2180399118,2cyclictest8901-21sort20:55:260
2180399118,2cyclictest8607-21cron19:50:010
2180399118,2cyclictest8351-21ssh22:42:300
2180399118,2cyclictest6737-21ssh21:51:280
2180399118,2cyclictest4374-21tr22:15:170
2180399118,2cyclictest30623-21grep21:45:180
2180399118,2cyclictest30383-21ssh22:35:090
2180399118,2cyclictest28217-21taskset19:20:210
2180399118,2cyclictest2717-21ssh23:02:300
2180399118,2cyclictest26233-21ssh21:16:570
2180399118,2cyclictest24363-21ssh22:05:180
2180399118,2cyclictest2257-21ssh23:25:390
2180399118,2cyclictest14719-21ssh21:33:120
2180399118,2cyclictest14671-21sed21:05:260
2180399118,2cyclictest14529-21ssh21:57:510
2180399118,2cyclictest11105-21ssh22:45:050
2180399118,2cyclictest10162-21sed00:20:180
2180399117,3cyclictest30474-21tail00:10:230
2180399112,5cyclictest224-21systemd-journal20:15:010
2180399110,10cyclictest19105-21sh21:35:530
2181099108,1cyclictest8696-21head20:55:233
2181099108,1cyclictest4376-21grep21:25:163
2181099108,1cyclictest29456-21cut19:25:143
2181099108,1cyclictest25237-21vmstat21:40:273
2181099107,2cyclictest7126-21grep19:45:153
2181099107,2cyclictest3592-21chrt19:36:583
2181099107,2cyclictest29107-21grep20:30:233
2181099107,2cyclictest26308-21tr20:25:163
2181099107,2cyclictest24763-21grep20:20:213
2181099100,9cyclictest18600-21cat20:10:013
2180799108,1cyclictest9825-21taskset20:57:452
2180799108,1cyclictest29738-21cat19:25:152
2180799108,1cyclictest14492-21grep21:05:272
2180799107,2cyclictest3378-21cut20:45:172
2180799107,2cyclictest24916-21tr19:15:152
2180699108,1cyclictest9526-21cut19:50:161
2180699108,1cyclictest25882sleep120:41:231
2180699107,2cyclictest7607-21head19:45:201
2180699107,2cyclictest6904-21cut23:30:141
2180699107,2cyclictest22673-21ssh23:40:541
2180699100,9cyclictest24410-21cat19:15:001
2180399108,1cyclictest7144-21cat23:30:150
2180399108,1cyclictest32751-21chrt20:38:050
2180399108,1cyclictest280492sleep020:28:440
2180399108,1cyclictest24602-21basename19:15:120
2180399107,2cyclictest30503-21grep19:25:200
2180399107,2cyclictest28716-21tr20:30:200
2180399107,2cyclictest24560-21ssh00:30:480
2180399107,2cyclictest2204-21expr19:35:160
2180399100,2cyclictest0-21swapper/023:50:140
218079997,2cyclictest32-21ksoftirqd/220:10:182
218079992,6cyclictest5817-21tr20:50:152
218039997,1cyclictest4767-21tune2fs19:40:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional