You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 13:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Nov 24, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1813621320,1sleep232-21ksoftirqd/222:29:232
710929986,9sleep30-21swapper/319:09:393
697729683,9sleep10-21swapper/119:07:561
712328666,16sleep20-21swapper/219:09:502
679528168,8sleep00-21swapper/019:05:340
721899240,1cyclictest0-21swapper/019:48:130
722999231,1cyclictest141rcu_preempt21:18:433
722799231,9cyclictest0-21swapper/222:20:142
722799221,20cyclictest32264-21diskmemload22:15:032
722299220,1cyclictest0-21swapper/120:25:131
722999210,20cyclictest0-21swapper/300:31:123
722799203,11cyclictest28584-21unixbench_singl23:20:312
722799201,1cyclictest32264-21diskmemload22:56:332
722799200,1cyclictest0-21swapper/221:18:342
722299200,2cyclictest0-21swapper/121:17:281
7218992015,5cyclictest13-21ksoftirqd/023:45:320
722799190,2cyclictest0-21swapper/222:48:582
722799190,18cyclictest0-21swapper/222:52:082
7222991918,1cyclictest0-21swapper/100:01:091
7218991918,1cyclictest0-21swapper/022:30:360
722799187,6cyclictest13396-21kworker/2:223:01:282
722799180,16cyclictest0-21swapper/221:52:142
7229991717,0cyclictest0-21swapper/321:49:193
7229991714,2cyclictest32264-21diskmemload00:25:333
7229991714,2cyclictest32264-21diskmemload00:25:333
722999170,1cyclictest0-21swapper/322:05:043
722799174,13cyclictest0-21swapper/223:25:152
722799172,3cyclictest0-21swapper/221:56:522
7227991715,1cyclictest0-21swapper/220:15:162
722799170,1cyclictest32264-21diskmemload23:10:262
722799170,15cyclictest0-21swapper/221:31:312
722299172,15cyclictest0-21swapper/122:35:191
722299170,10cyclictest0-21swapper/119:25:151
721899176,10cyclictest26671-21munin-node21:27:050
722999165,9cyclictest32264-21diskmemload23:57:103
722799161,3cyclictest26948-21ssh22:12:582
722799160,14cyclictest0-21swapper/221:39:002
722799160,13cyclictest0-21swapper/200:08:172
218912160,0sleep20-21swapper/223:16:002
722999155,6cyclictest30570-21apt-get23:45:123
7229991512,2cyclictest17276-21wc23:35:183
722999150,12cyclictest0-21swapper/319:45:143
7227991512,2cyclictest2996-21sh22:40:252
7227991512,2cyclictest2996-21sh22:40:242
722799150,2cyclictest0-21swapper/200:31:012
722799150,10cyclictest0-21swapper/223:09:162
722299150,15cyclictest0-21swapper/123:56:041
722299150,15cyclictest0-21swapper/123:03:371
721899155,9cyclictest32264-21diskmemload00:11:560
721899155,3cyclictest224-21systemd-journal23:21:230
722999146,7cyclictest32264-21diskmemload21:42:593
722999142,2cyclictest0-21swapper/323:30:153
7229991410,2cyclictest31357-21grep20:00:143
722799147,6cyclictest19217-21apt-get22:30:132
722799147,2cyclictest2239-21tr00:10:202
7227991410,2cyclictest9493-21sshd19:14:122
722299147,4cyclictest0-21swapper/121:50:201
7222991412,2cyclictest25-21ksoftirqd/100:33:231
7222991410,3cyclictest31368-21sed23:45:171
722299140,14cyclictest0-21swapper/123:27:161
721899145,3cyclictest27521-21perf21:00:010
721899145,3cyclictest27521-21perf21:00:000
721899142,5cyclictest8968-21ssh21:15:270
7218991411,2cyclictest15719-21sh23:34:590
721899141,12cyclictest0-21swapper/000:16:330
722999139,3cyclictest8205-21cron20:20:003
722999139,3cyclictest31599-21head23:00:223
722999139,3cyclictest30397-21grep21:05:163
722999139,3cyclictest24247-21wc20:50:223
722999139,3cyclictest21082-21grep22:30:273
722999139,2cyclictest27250-21grep19:50:183
7229991310,2cyclictest8470-21sed23:30:003
7229991310,2cyclictest7359-21turbostat21:59:593
7229991310,2cyclictest14926-21ssh21:20:183
7229991310,2cyclictest10974-21sort20:25:003
7229991310,2cyclictest10493-21cat21:39:593
722999130,1cyclictest0-21swapper/319:55:123
722999130,13cyclictest0-21swapper/300:15:343
722999130,12cyclictest19955-21grep20:40:263
722799132,2cyclictest32264-21diskmemload00:29:532
722799132,2cyclictest32264-21diskmemload00:29:522
7227991310,2cyclictest23877-21grep19:45:002
7227991310,2cyclictest14179-21tail20:30:172
7227991310,2cyclictest12510-21wc20:25:262
722799130,2cyclictest0-21swapper/200:37:472
722299139,3cyclictest1442-21basename21:55:191
7222991310,2cyclictest8059-21cut19:10:181
7222991310,2cyclictest5664-21grep23:50:151
7222991310,2cyclictest2903-21ssh22:40:221
7222991310,2cyclictest2903-21ssh22:40:221
7222991310,2cyclictest28997-21grep21:00:231
7222991310,2cyclictest25764-21grep19:45:281
7222991310,2cyclictest23698-21grep23:40:151
7222991310,2cyclictest23506-21wc22:10:201
7222991310,2cyclictest17539-21grep19:30:161
7222991310,2cyclictest16993-21tail23:35:161
7222991310,2cyclictest11072-21grep19:15:211
722299130,13cyclictest0-21swapper/121:27:031
722299130,13cyclictest0-21swapper/121:14:241
722299130,12cyclictest16975-21tr20:35:211
721899134,7cyclictest6263-21ntp_states23:05:220
721899133,9cyclictest6537-21grep22:20:270
7218991312,1cyclictest0-21swapper/021:21:550
7218991310,2cyclictest32049-21grep20:00:180
7218991310,2cyclictest3021-21munin-plugin-st21:34:590
7218991310,2cyclictest16184-21cat19:25:280
7218991310,2cyclictest11977-21cat21:40:190
7218991310,2cyclictest11067-21grep19:15:210
722999129,2cyclictest9547-21grep22:45:163
722999129,2cyclictest8128-21grep22:00:173
722999129,2cyclictest2907-21grep20:05:273
722999129,2cyclictest2636-21grep22:40:203
722999129,2cyclictest2636-21grep22:40:193
722999129,2cyclictest23136-21tail22:10:193
722999129,2cyclictest21487-21cat19:40:003
722999129,2cyclictest16881-21grep00:20:153
722999129,2cyclictest15947-21grep20:35:163
722999129,2cyclictest13846-21grep19:20:293
722999129,2cyclictest12804-21tail22:25:193
722999129,2cyclictest11487-21cron22:25:003
722999128,3cyclictest29942-21ssh00:06:273
722999128,3cyclictest24826-21wc23:40:233
722999128,3cyclictest246472sleep323:18:373
722999128,3cyclictest1048-21cat23:25:003
722999128,2cyclictest4739-21grep00:35:173
722999127,4cyclictest9273-21ssh23:52:203
722999126,2cyclictest18026-21head22:50:273
722999123,6cyclictest2042-21/usr/sbin/munin00:10:193
722999121,6cyclictest0-21swapper/320:55:133
722999121,6cyclictest0-21swapper/320:55:133
7229991210,2cyclictest39-21ksoftirqd/322:55:163
722799129,2cyclictest30093-21latency_hist23:45:002
722799129,2cyclictest13699-21grep23:55:202
722799129,2cyclictest10602-21cut19:15:162
722799127,4cyclictest2098-21sed21:10:252
722799126,5cyclictest11705-21rm21:40:182
722799123,6cyclictest26671-21munin-node22:04:272
722799121,9cyclictest0-21swapper/223:35:412
7227991210,1cyclictest15718-21cron23:34:592
722799120,11cyclictest6065-21grep23:50:182
722799120,11cyclictest28051-21tr21:00:162
722299129,2cyclictest8922-21chrt22:45:141
722299129,2cyclictest30411-21tr21:05:161
722299129,2cyclictest27628-21grep00:05:211
722299129,2cyclictest26933-21tr23:20:181
722299129,2cyclictest17637-21sort21:44:591
722299129,2cyclictest14378-21ssh21:20:141
722299129,2cyclictest12918-21sort19:20:171
722299128,3cyclictest26065-21cron23:20:001
722299120,1cyclictest0-21swapper/121:37:051
721899129,2cyclictest8956-21grep20:20:170
721899129,2cyclictest5428-21basename23:50:140
721899129,2cyclictest24936-21sed00:25:290
721899129,2cyclictest24936-21sed00:25:290
721899129,2cyclictest24109-21grep22:10:270
721899129,2cyclictest20898-21tr20:45:140
721899129,2cyclictest16503-21tr23:35:140
721899129,2cyclictest10113-21grep22:45:200
721899128,3cyclictest23061-21tr20:50:150
721899128,3cyclictest20455-21ssh00:22:460
721899128,3cyclictest14565-21rm22:04:560
721899125,6cyclictest29765-21munin-run23:00:010
721899124,2cyclictest19371-21apt-get19:35:140
721899121,10cyclictest32264-21diskmemload22:28:260
7218991210,1cyclictest3333-21grep22:40:260
7218991210,1cyclictest3333-21grep22:40:250
721899120,11cyclictest13200-21sh20:29:590
722999118,2cyclictest9795-21chrt19:15:033
722999118,2cyclictest9047-21ssh23:07:293
722999118,2cyclictest31607-21ssh21:54:133
722999118,2cyclictest29964-21date21:05:013
722999118,2cyclictest28508-21sh21:30:063
722999118,2cyclictest23483-21ssh00:02:213
722999118,2cyclictest22682-21taskset20:49:383
722999118,2cyclictest22443-21cat21:25:203
722999118,2cyclictest21964-21tr19:40:173
722999118,2cyclictest17470-21sh19:30:163
722999118,2cyclictest14826-21ssh23:11:113
722999110,10cyclictest1075-21head21:10:173
722799119,1cyclictest9419-21grep20:20:192
722799119,1cyclictest30436-21wc19:55:262
722799119,1cyclictest207382sleep222:08:562
722799119,1cyclictest20318-21sort00:00:172
722799119,1cyclictest10269-21ssh00:15:232
722799118,2cyclictest3904-21ssh23:49:082
722799118,2cyclictest31174-21wc21:05:212
722799118,2cyclictest28210-21head19:50:272
722799118,2cyclictest25806-21tail19:45:282
722799118,2cyclictest24327-21tr20:50:242
722799118,2cyclictest21271-21chrt19:37:302
722799118,2cyclictest20015-21chrt20:40:532
722799118,2cyclictest16337-21ssh00:20:122
722799111,2cyclictest224-21systemd-journal20:15:002
722799110,10cyclictest23651-21ssh21:25:392
722799110,10cyclictest17685-21sh21:21:512
722299119,1cyclictest23714-21grep20:50:181
722299118,2cyclictest6043-21tr22:20:231
722299118,2cyclictest5832-21sed00:35:271
722299118,2cyclictest3913-21ssh22:19:511
722299118,2cyclictest29376-21df19:55:161
722299118,2cyclictest23421-21ssh22:32:421
722299118,2cyclictest22727-21expr19:40:201
722299118,2cyclictest1845-21ssh00:10:131
722299118,2cyclictest18068-21rm21:45:051
722299118,2cyclictest17275-21ssh22:05:391
722299118,2cyclictest16762-21ssh23:12:591
722299118,2cyclictest16405-21ssh00:20:131
722299118,2cyclictest15998-21sh22:50:081
722299118,2cyclictest13568-21grep20:30:141
722299118,2cyclictest11916-21ssh22:25:101
722299118,2cyclictest10082-21ssh22:00:431
722299114,6cyclictest20346-21cron20:45:001
722299111,8cyclictest6512-21sensors23:05:241
722299110,10cyclictest1979-21ssh21:34:011
722299110,10cyclictest0-21swapper/122:56:381
721899119,1cyclictest726-21ssh21:55:160
721899119,1cyclictest30472-21grep23:00:140
721899119,1cyclictest29484-21head21:00:280
721899119,1cyclictest25315-21cat23:40:330
721899119,1cyclictest19901-21grep20:40:270
721899119,1cyclictest16101-21grep20:35:150
721899119,1cyclictest1067-21grep21:10:160
721899118,2cyclictest6326-21grep20:15:160
721899118,2cyclictest5264-21ssh00:35:220
721899118,2cyclictest4638-21sed21:35:210
721899118,2cyclictest31874-21vmstat21:05:280
721899118,2cyclictest26551-21date00:05:010
721899118,2cyclictest23687-21chrt19:42:440
721899118,2cyclictest20144-21sed00:00:150
721899118,2cyclictest1128-21ssh23:25:010
721899118,1cyclictest17386-21expr19:30:150
721899115,3cyclictest0-21swapper/021:47:230
721899112,7cyclictest14082-21sed23:10:270
721899112,4cyclictest22314-21sshd23:16:240
722999108,1cyclictest30057-21expr22:15:153
722999107,2cyclictest4093-21expr20:10:163
722999107,2cyclictest26180-21ssh22:35:033
722999107,2cyclictest16861-21cat19:30:013
722999106,3cyclictest13604-21grep20:30:133
722799108,1cyclictest3076-21chrt20:06:352
722799108,1cyclictest21365-21tail20:45:182
722799108,1cyclictest18509-21tr21:45:152
722799108,1cyclictest176382sleep220:36:082
722799108,1cyclictest13896-21grep19:20:282
722799107,2cyclictest17950-21wc19:30:202
722799107,2cyclictest14870-21tr19:25:172
722799101,3cyclictest224-21systemd-journal20:56:032
722799101,3cyclictest224-21systemd-journal20:56:022
722799100,9cyclictest26883-21expr22:35:162
722299108,1cyclictest9149-21expr20:20:171
722299108,1cyclictest8906-21ssh00:15:001
722299108,1cyclictest32114-21basename20:00:191
722299108,1cyclictest22985-21ssh00:25:121
722299108,1cyclictest22985-21ssh00:25:111
722299107,2cyclictest26750-21cut19:50:151
722299107,2cyclictest2039-21smartctl20:05:171
722299107,2cyclictest19810-21expr19:35:151
722299106,3cyclictest7702-21sed20:15:271
722299100,9cyclictest8724-21ssh23:30:031
722299100,9cyclictest20929-21expr20:45:141
721899108,2cyclictest26237-21sh22:35:070
721899108,1cyclictest7462-21chrt19:10:140
721899108,1cyclictest30549-21tail22:15:180
721899108,1cyclictest29354-21expr19:55:160
721899108,1cyclictest285622sleep019:54:180
721899108,1cyclictest14875-21ssh23:55:510
721899108,1cyclictest12628-21expr19:20:150
721899107,2cyclictest30228-21df00:30:150
721899107,2cyclictest25440-21ssh21:50:010
721899107,2cyclictest17263-21cat22:50:210
721899107,2cyclictest14971-21ssh22:05:070
72299996,2cyclictest9465-21taskset19:13:523
72279990,4cyclictest31303-21aten2_r7power_c20:00:152
72229998,1cyclictest26931-21timerwakeupswit20:55:251
72229998,1cyclictest26931-21timerwakeupswit20:55:241
72229997,1cyclictest4143-21sed20:10:171
72189998,1cyclictest2172-21cut20:05:190
72189997,1cyclictest3822-21cat20:10:160
72189996,2cyclictest13567-21wget20:30:140
11201290,1sleep30-21swapper/320:25:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional