You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-17 - 16:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Oct 17, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31302210288,9sleep10-21swapper/107:05:251
3165929755,38sleep30-21swapper/307:09:503
3148428467,13sleep20-21swapper/207:07:322
3155428270,8sleep00-21swapper/007:08:260
3176799250,4cyclictest0-21swapper/010:40:430
81762210,1sleep20-21swapper/212:29:422
3176799219,2cyclictest141rcu_preempt11:05:170
3177499200,20cyclictest0-21swapper/211:12:102
3177799191,16cyclictest0-21swapper/310:30:143
3177099190,3cyclictest0-21swapper/111:34:301
3176799190,1cyclictest141rcu_preempt12:01:020
3177799180,15cyclictest0-21swapper/310:18:243
3177499180,3cyclictest0-21swapper/207:15:142
3177099181,9cyclictest0-21swapper/108:40:131
31767991816,1cyclictest13-21ksoftirqd/010:46:410
3176799180,0cyclictest141rcu_preempt10:31:500
3177799170,17cyclictest0-21swapper/311:48:473
31774991711,4cyclictest21695-21grep07:55:162
3177499170,17cyclictest0-21swapper/211:48:482
3177099170,15cyclictest0-21swapper/112:33:361
31767991715,1cyclictest13-21ksoftirqd/012:12:420
31767991715,1cyclictest13-21ksoftirqd/010:55:420
3177499163,10cyclictest24390-21diskmemload09:29:062
31774991616,0cyclictest0-21swapper/209:47:562
3176799163,7cyclictest5129-21munin-run08:30:000
31767991611,3cyclictest22613-21sh12:40:010
3176799160,14cyclictest0-21swapper/010:29:370
3176799160,0cyclictest141rcu_preempt11:41:480
3176799155,3cyclictest24390-21diskmemload12:20:270
3176799154,6cyclictest7495-21idleruntime-cro08:35:000
31767991513,1cyclictest13-21ksoftirqd/009:30:380
31767991513,1cyclictest13-21ksoftirqd/009:30:370
3177799149,3cyclictest13260-21grep08:45:153
31777991410,3cyclictest20011-21grep09:00:193
3177499148,5cyclictest18508-21grep10:00:162
31774991412,2cyclictest32-21ksoftirqd/212:10:122
31774991412,1cyclictest9660-21dpkg11:40:012
31774991410,3cyclictest21987-21tr10:55:252
31774991410,3cyclictest1257-21sort08:20:162
3177499140,2cyclictest31433-21kworker/u8:009:19:532
3177099146,1cyclictest224-21systemd-journal11:20:011
31770991412,2cyclictest25-21ksoftirqd/109:21:231
31770991410,2cyclictest2210-21cat08:20:251
31770991410,2cyclictest18210-21grep11:45:281
3176799141,7cyclictest13571-21awk12:09:590
31767991412,1cyclictest13-21ksoftirqd/009:20:410
3176799141,0cyclictest141rcu_preempt11:46:400
31767991410,3cyclictest10014-21head12:30:180
31767991410,2cyclictest28794-21sed11:55:190
3176799140,0cyclictest141rcu_preempt11:13:480
3177799139,2cyclictest6157-21head08:30:163
3177799139,2cyclictest5682-21ssh09:50:013
31777991310,2cyclictest31974-21ssh09:15:293
31777991310,2cyclictest30316-21ssh11:30:013
31777991310,2cyclictest30125-21ssh10:35:193
31777991310,2cyclictest21107-21grep07:55:153
3177799130,12cyclictest11267-21tr10:20:203
3177499139,3cyclictest32-21ksoftirqd/208:45:192
3177499132,8cyclictest10008-21apache_accesses11:40:112
3177499132,4cyclictest224-21systemd-journal07:14:592
31774991310,2cyclictest3575-21ssh11:35:002
31774991310,2cyclictest24868-21tr10:05:212
31774991310,2cyclictest20376-21head11:20:262
31774991310,2cyclictest1202-21cron11:59:592
3177099139,3cyclictest3713-21sed10:40:191
3177099139,3cyclictest3688-21grep07:15:281
3177099139,3cyclictest25276-21head11:25:171
3177099139,3cyclictest22813-21grep12:15:311
3177099135,6cyclictest21886-21ntp_states09:35:191
31770991312,1cyclictest61750irq/125-enp2s0-12:13:221
31770991310,2cyclictest5419-21grep10:15:271
31770991310,2cyclictest25977-21basename08:05:151
31770991310,2cyclictest22442-21sh12:40:001
3176799139,3cyclictest6514-21cron07:25:000
3176799139,3cyclictest24380-21cron09:10:000
3176799139,3cyclictest17931-21cut10:00:140
3176799139,2cyclictest24390-21diskmemload10:16:400
3176799139,1cyclictest141rcu_preempt11:04:270
3176799135,7cyclictest2419-21munin-run10:39:590
31767991313,0cyclictest0-21swapper/009:38:480
31767991310,2cyclictest3593-21taskset11:35:000
31767991310,2cyclictest26138-21grep08:05:150
3177799129,2cyclictest9813-21cut10:45:213
3177799129,2cyclictest3679-21cut10:40:193
3177799129,2cyclictest32760-21grep07:10:173
3177799129,2cyclictest27291-21tail09:40:173
3177799129,2cyclictest20872-21cat07:55:013
3177799129,2cyclictest18041-21head07:45:283
3177799129,2cyclictest18030-21tr10:00:133
3177799129,2cyclictest12714-21grep09:55:193
3177799129,2cyclictest10293-21grep11:40:143
3177799129,2cyclictest10031-21sed12:30:183
3177799128,3cyclictest13602-21sed11:15:223
3177799125,3cyclictest224-21systemd-journal07:30:003
3177799121,9cyclictest24058-21ssh11:25:003
3177799120,1cyclictest0-21swapper/312:39:553
3177499129,2cyclictest9088-21wc08:35:232
3177499129,2cyclictest5778-21cron11:10:002
3177499129,2cyclictest5122-21head07:20:172
3177499129,2cyclictest31088-21sed10:35:272
3177499129,2cyclictest28825-21ls12:20:262
3177499129,2cyclictest24144-21grep08:00:172
3177499129,2cyclictest17378-21awk10:00:012
3177499129,2cyclictest16812-21df07:45:162
3177499129,2cyclictest10384-21ls07:30:202
3177499129,2cyclictest10373-21cat12:30:202
3177499128,3cyclictest27521-21awk11:55:002
3177499128,3cyclictest15862-21grep10:50:182
3177499128,3cyclictest13541-21cron12:10:002
3177499128,2cyclictest17670-21ntp_offset10:25:232
3177499128,1cyclictest8408-21ssh09:24:532
3177499121,1cyclictest24390-21diskmemload10:45:342
3177499120,11cyclictest11653-21tr07:35:162
3177499120,11cyclictest0-21swapper/212:38:532
3177099129,2cyclictest8493-21grep08:35:171
3177099129,2cyclictest7704-21grep09:50:281
3177099129,2cyclictest3628-21tune2fs08:25:141
3177099129,2cyclictest3544-21grep12:00:281
3177099129,2cyclictest31892-21grep09:15:271
3177099129,2cyclictest31061-21grep10:35:261
3177099129,2cyclictest25573-21cron08:05:011
3177099129,2cyclictest21117-21basename07:55:141
3177099129,2cyclictest18563-21grep10:00:171
3177099129,2cyclictest18445-21awk07:50:001
3177099128,3cyclictest6084-21tune2fs08:30:171
3177099128,3cyclictest18616-21cut11:20:141
3177099128,3cyclictest14910-21ssh09:30:161
3177099128,3cyclictest14910-21ssh09:30:151
3177099126,4cyclictest24405-21apt09:10:001
3177099124,4cyclictest24390-21diskmemload10:46:111
3177099122,8cyclictest0-21swapper/111:05:171
3176799129,2cyclictest4294-21ssh11:35:190
3176799129,2cyclictest31274-21cut08:15:240
3176799129,2cyclictest25285-21grep11:25:170
3176799129,2cyclictest20687-21cut09:00:210
3176799129,2cyclictest1735-21cron07:15:000
3176799129,2cyclictest14044-21sh09:29:590
3176799128,3cyclictest29218-21cron10:09:590
3176799128,3cyclictest17095-21wc08:54:590
3176799124,4cyclictest24390-21diskmemload11:19:330
3176799121,8cyclictest16070-21awk07:45:010
3176799121,8cyclictest16070-21awk07:45:000
31767991210,1cyclictest30214-21basename09:15:130
3177799118,2cyclictest9748-21taskset08:38:303
3177799118,2cyclictest964-21head11:05:203
3177799118,2cyclictest6177-21ssh12:27:373
3177799118,2cyclictest6021-21ssh11:10:013
3177799118,2cyclictest4860-21cat09:20:273
3177799118,2cyclictest4019-21cut11:35:173
3177799118,2cyclictest31970-21tr11:30:233
3177799118,2cyclictest31376-21ssh12:22:323
3177799118,2cyclictest28765-21grep11:55:193
3177799118,2cyclictest28251-21basename08:10:133
3177799118,2cyclictest27875-21sendmail_mailtr11:00:233
3177799118,2cyclictest25122-21grep09:10:143
3177799118,2cyclictest24649-21uname10:05:203
3177799118,2cyclictest24359-21cat08:00:193
3177799118,2cyclictest24016-21ssh09:37:363
3177799118,2cyclictest23017-21grep09:05:173
3177799118,2cyclictest22843-21ssh11:50:163
3177799118,2cyclictest22750-21ssh10:55:443
3177799118,2cyclictest21876-21sh12:15:233
3177799118,2cyclictest1887-21cat12:00:173
3177799118,2cyclictest18690-21taskset08:55:233
3177799118,2cyclictest17746-21taskset10:51:273
3177799118,2cyclictest17617-21ssh09:31:453
3177799118,2cyclictest17617-21ssh09:31:443
3177799118,2cyclictest17257-21head10:25:193
3177799118,2cyclictest16884-21taskset08:53:203
3177799118,2cyclictest13851-21ssh12:10:043
3177799118,2cyclictest1293-21ssh10:12:503
3177799118,2cyclictest12094-21chrt08:42:483
3177799118,2cyclictest11472-21pluginstate09:55:003
3177799118,2cyclictest10238-21ssh12:06:363
3177799112,4cyclictest8972-21apache_volume09:25:163
3177499118,2cyclictest8695-21chrt10:18:572
3177499118,2cyclictest7994-21cut07:25:222
3177499118,2cyclictest6348-21expr09:50:172
3177499118,2cyclictest3590-21sort10:40:182
3177499118,2cyclictest3243-21tr08:25:122
3177499118,2cyclictest32039-21ssh10:10:592
3177499118,2cyclictest27183-21grep09:40:182
3177499118,2cyclictest26005-21mailstats11:25:232
3177499118,2cyclictest23369-21grep09:05:202
3177499118,2cyclictest23131-21grep10:30:162
3177499118,2cyclictest20883-21grep09:35:112
3177499118,2cyclictest14201-21sh10:23:052
3177499118,2cyclictest13749-21grep11:15:242
3177499116,4cyclictest18954-21expr07:50:172
3177499111,9cyclictest350-21sendmail-msp08:20:002
3177499110,4cyclictest224-21systemd-journal08:57:232
3177499110,10cyclictest0-21swapper/211:00:232
3177099118,2cyclictest9912-21ssh11:40:031
3177099118,2cyclictest7525-21grep12:05:141
3177099118,2cyclictest6578-21tr11:10:201
3177099118,2cyclictest4622-21taskset12:26:061
3177099118,2cyclictest30987-21sh11:56:431
3177099118,2cyclictest30108-21chrt08:12:181
3177099118,2cyclictest29046-21chrt12:20:301
3177099118,2cyclictest27938-21ssh11:00:261
3177099118,2cyclictest25532-21grep10:05:261
3177099118,2cyclictest23522-21tail11:50:191
3177099118,2cyclictest2099-21ssh09:45:501
3177099118,2cyclictest19477-21head07:50:191
3177099118,2cyclictest19437-21awk09:00:001
3177099118,2cyclictest18613-21ssh10:25:501
3177099118,2cyclictest18534-21ssh10:52:361
3177099118,2cyclictest1429-21ssh10:13:031
3177099118,2cyclictest13251-21head08:45:151
3177099118,2cyclictest12246-21ssh10:20:441
3177099118,2cyclictest11732-21sh09:55:031
3177099116,2cyclictest224-21systemd-journal09:15:001
3177099115,4cyclictest32763-21cron08:20:001
3177099115,2cyclictest141rcu_preempt07:40:011
3177099113,4cyclictest224-21systemd-journal08:54:591
3177099112,4cyclictest8925-21sshd09:25:131
3177099110,4cyclictest0-21swapper/110:55:171
3177099110,10cyclictest22851-21sh10:30:131
3176799118,2cyclictest7193-21ssh12:28:430
3176799118,2cyclictest23307-21cat08:00:010
3176799118,2cyclictest20041-21python11:20:260
3176799118,2cyclictest1991-21ssh09:45:440
3176799118,2cyclictest18816-21ssh10:52:580
3176799118,2cyclictest17919-21sort08:55:170
3176799118,2cyclictest17063-21ssh09:59:400
3176799118,2cyclictest13328-21head08:45:150
3176799118,2cyclictest10232-21ssh09:53:320
3176799118,1cyclictest141rcu_preempt12:16:120
3176799117,3cyclictest16412-21aten2_r7power_p07:45:160
31767991110,1cyclictest161rcuc/008:40:140
3177799108,1cyclictest5611-21cut07:20:213
3177799108,1cyclictest3201-21egrep08:25:123
3177799108,1cyclictest2866-21cat07:15:203
3177799108,1cyclictest14196-21expr07:40:153
3177799108,1cyclictest14196-21expr07:40:153
3177799108,1cyclictest10099-21cut07:30:183
3177799107,2cyclictest32104-21expr08:15:363
3177799107,2cyclictest25856-21apt-get08:05:153
3177799107,2cyclictest1178-21df08:20:153
3177799107,2cyclictest11638-21aten2_r7power_p07:35:163
3177499108,1cyclictest7744-21snmpd09:10:052
3177499108,1cyclictest5681-21expr08:30:142
3177499108,1cyclictest3414-21expr12:00:282
3177499108,1cyclictest216902sleep209:00:592
3177499108,1cyclictest14255-21cstates07:40:152
3177499108,1cyclictest14255-21cstates07:40:142
3177499107,2cyclictest28572-21expr08:10:142
3177499107,2cyclictest24982-21ssh12:18:052
3177499107,2cyclictest15762-21sh09:30:202
3177499107,2cyclictest15762-21sh09:30:202
3177499107,2cyclictest15203-21grep08:50:172
3177499107,2cyclictest10613-21expr08:40:152
3177499100,10cyclictest37850irq/131-snd_hda08:05:152
3177099108,1cyclictest9477-21cstates07:30:141
3177099108,1cyclictest7270-21df07:25:161
3177099108,1cyclictest405-21wc07:10:181
3177099108,1cyclictest26128-21dpkg09:40:011
3177099107,2cyclictest4744-21expr07:20:151
3177099107,2cyclictest4266-21tr11:35:191
3177099107,2cyclictest14718-21head07:40:171
3177099107,2cyclictest14718-21head07:40:161
3177099100,9cyclictest20239-21expr09:00:191
3176799108,1cyclictest9929-21ssh10:20:030
3176799108,1cyclictest29192-21cat08:10:200
3176799108,1cyclictest284282sleep009:40:280
3176799108,1cyclictest27035-21sh09:11:050
3176799108,1cyclictest23657-21egrep08:00:160
3176799108,1cyclictest226392sleep011:50:110
3176799108,1cyclictest2223-21cat07:15:150
3176799108,1cyclictest10973-21taskset07:31:150
3176799107,2cyclictest6932-21grep07:25:140
3176799107,2cyclictest30673-21ssh10:10:220
3176799107,2cyclictest19459-21grep07:50:190
3176799107,2cyclictest1364-21head08:20:170
3176799107,2cyclictest13255-21grep07:35:290
317679996,2cyclictest8976-21cut08:35:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional