You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-23 - 09:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Oct 23, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31144210780,9sleep30-21swapper/319:07:303
3101129668,9sleep00-21swapper/019:05:460
2914729682,9sleep10-21swapper/119:05:111
3106128564,16sleep20-21swapper/219:06:242
3143099251,24cyclictest0-21swapper/323:17:293
3142699230,22cyclictest0-21swapper/222:20:192
3141699228,7cyclictest9946-21tr20:39:590
3143099210,1cyclictest0-21swapper/300:39:013
3142699210,19cyclictest0-21swapper/221:20:502
3142699210,18cyclictest0-21swapper/221:57:212
3142699200,19cyclictest0-21swapper/223:48:082
31421991919,0cyclictest0-21swapper/100:27:441
203372190,0sleep10-21swapper/123:32:381
3143099180,18cyclictest0-21swapper/322:49:233
3143099180,18cyclictest0-21swapper/322:11:283
3142699180,15cyclictest0-21swapper/221:35:132
3142199180,18cyclictest0-21swapper/121:17:441
31416991816,1cyclictest0-21swapper/022:23:360
31430991717,0cyclictest0-21swapper/300:03:513
31430991714,1cyclictest8830-21apt-get23:00:123
3143099170,13cyclictest0-21swapper/321:55:043
31426991715,1cyclictest0-21swapper/221:53:322
31426991713,3cyclictest5861-21fschecks_time20:30:192
3142699170,1cyclictest0-21swapper/200:05:212
3143099168,3cyclictest0-21swapper/321:00:003
3143099160,2cyclictest0-21swapper/323:36:133
3143099160,1cyclictest0-21swapper/321:32:333
3141699162,7cyclictest14977-21idleruntime-cro23:05:010
31416991613,3cyclictest13-21ksoftirqd/022:41:080
3141699160,1cyclictest0-21swapper/022:25:120
3143099152,8cyclictest32353-21cat21:40:013
31430991515,0cyclictest0-21swapper/322:26:433
31430991513,2cyclictest94-21kswapd021:50:003
3142699157,3cyclictest224-21systemd-journal23:55:002
3142699150,14cyclictest0-21swapper/219:48:222
3142699150,10cyclictest0-21swapper/219:17:202
3142699150,0cyclictest0-21swapper/222:59:012
3142199150,14cyclictest0-21swapper/122:15:471
3141699154,7cyclictest24223-21diskmemload22:15:140
31430991413,1cyclictest61750irq/125-enp2s0-00:15:223
31430991411,2cyclictest10477-21grep23:25:183
3142699148,6cyclictest30033-21apache223:34:442
31426991414,0cyclictest0-21swapper/223:22:392
31426991412,2cyclictest0-21swapper/200:33:162
31426991411,2cyclictest29043-21grep20:10:242
3142699140,14cyclictest0-21swapper/223:39:192
3142199143,10cyclictest27814-21apt-get22:50:141
31421991411,2cyclictest710-21cron22:29:591
31421991411,2cyclictest2680-21cat20:25:001
31421991410,3cyclictest30641-21grep23:15:261
3141699149,3cyclictest22869-21sed21:05:150
3141699141,8cyclictest1686-21wget19:15:150
31416991410,3cyclictest19880-21ssh00:20:200
3143099139,3cyclictest1706-21cat22:55:013
3143099139,3cyclictest15742-21uname22:40:213
3143099139,3cyclictest14407-21grep21:25:233
31430991310,2cyclictest8571-21cat19:30:003
31430991310,2cyclictest8451-21cut20:35:173
31430991310,2cyclictest5132-21cat20:29:593
31430991310,2cyclictest32638-21wc00:30:273
31430991310,2cyclictest29547-21sed20:10:293
31430991310,2cyclictest23087-21cat21:05:193
31430991310,2cyclictest21147-21sh22:20:173
3143099130,12cyclictest1582-21grep22:05:203
3143099130,0cyclictest0-21swapper/322:18:143
3142699139,3cyclictest7672-21rm00:10:362
3142699139,3cyclictest7672-21rm00:10:362
3142699139,3cyclictest23743-21taskset22:46:192
3142699139,2cyclictest15172-21grep20:50:152
31426991310,2cyclictest6988-21grep21:45:112
31426991310,2cyclictest28606-21ls22:25:262
31426991310,2cyclictest28092-21chrt21:11:582
31426991310,2cyclictest23335-21awk23:10:262
3142199139,3cyclictest8485-21cat22:10:201
3142199139,3cyclictest4819-21tail23:20:261
3142199139,3cyclictest4662-21grep19:20:161
3142199139,2cyclictest20088-21sed21:30:231
31421991310,2cyclictest7282-21ls21:20:261
31421991310,2cyclictest5852-21grep23:45:271
31421991310,2cyclictest31366-21basename00:05:171
31421991310,2cyclictest25070-21cut00:00:191
3142199130,11cyclictest0-21swapper/119:30:131
3141699139,3cyclictest27328-21cron22:50:010
31416991310,2cyclictest9242-21cron23:25:000
31416991310,2cyclictest28149-21wc23:15:000
31416991310,2cyclictest12855-21basename21:49:590
31416991310,2cyclictest10010-21grep23:25:150
3143099129,2cyclictest6875-21grep20:30:283
3143099129,2cyclictest3899-21grep19:20:003
3143099129,2cyclictest30406-21grep23:40:173
3143099129,2cyclictest25749-21ssh21:10:213
3143099129,2cyclictest25149-21cut00:25:133
3143099129,2cyclictest22814-21sed23:10:213
3143099129,2cyclictest19974-21grep19:50:293
3143099129,2cyclictest19448-21grep23:55:293
3143099129,2cyclictest13824-21grep19:40:163
3143099129,2cyclictest11600-21head23:50:163
31430991211,1cyclictest0-21swapper/321:44:493
31430991210,1cyclictest16408-21head23:05:183
3143099120,0cyclictest0-21swapper/323:24:263
3142699129,2cyclictest540-21fgrep19:10:232
3142699129,2cyclictest540-21fgrep19:10:222
3142699129,2cyclictest18879-21tr23:55:232
3142699129,2cyclictest12402-21grep19:35:242
3142699128,3cyclictest606-21tr21:40:192
3142699128,3cyclictest16087-21ls23:30:002
3142699128,3cyclictest14505-21sed21:25:232
3142699127,5cyclictest29488-21munin-node23:04:102
3142699127,4cyclictest1908-21ssh23:43:152
3142699123,6cyclictest1258-21apache_volume22:30:172
3142699123,6cyclictest1258-21apache_volume22:30:172
3142699122,9cyclictest24964-21kworker/2:020:05:172
31426991211,1cyclictest0-21swapper/221:34:422
3142699120,1cyclictest0-21swapper/222:12:402
3142699120,12cyclictest0-21swapper/221:15:472
3142699120,11cyclictest7013-21grep19:25:152
3142199129,2cyclictest6637-21wc00:35:271
3142199129,2cyclictest3719-21cron19:20:011
3142199129,2cyclictest3596-21cut20:25:201
3142199129,2cyclictest32217-21cron21:40:001
3142199129,2cyclictest25466-21cut21:10:191
3142199129,2cyclictest23641-21wc21:05:241
3142199129,2cyclictest2208-21wc22:05:251
3142199129,2cyclictest21808-21sed22:45:181
3142199129,2cyclictest21318-21head21:55:271
3142199129,2cyclictest19490-21ssh00:20:181
3142199129,2cyclictest18416-21tail23:55:181
3142199129,2cyclictest16204-21wc20:50:221
3142199129,2cyclictest13348-21basename21:50:141
3142199128,3cyclictest21215-21grep19:55:151
3142199128,3cyclictest14959-21wc19:40:261
3142199128,2cyclictest2825-21cut22:55:161
3142199125,3cyclictest224-21systemd-journal22:03:351
3142199122,4cyclictest2057-21grep22:30:211
3142199122,4cyclictest2057-21grep22:30:201
31421991211,1cyclictest25-21ksoftirqd/121:04:381
3141699129,2cyclictest668-21grep00:05:300
3141699129,2cyclictest5762-21df20:30:180
3141699129,2cyclictest4830-21head23:45:180
3141699129,2cyclictest32568-21cat21:15:200
3141699129,2cyclictest29920-21ssh23:40:160
3141699129,2cyclictest29482-21tail20:10:270
3141699129,2cyclictest26501-21cat20:05:200
3141699129,2cyclictest26353-21wc00:25:210
3141699129,2cyclictest25178-21idleruntime-cro20:04:590
3141699129,2cyclictest21174-21grep21:30:320
3141699129,2cyclictest18533-21sleep00:20:010
3141699129,2cyclictest16423-21ssh23:05:170
3141699129,2cyclictest11486-21ssh00:14:280
3141699129,2cyclictest11486-21ssh00:14:270
3141699129,2cyclictest0-21swapper/021:35:140
3141699128,3cyclictest14294-21expr19:40:170
3141699128,2cyclictest14387-21grep21:25:230
3141699126,6cyclictest14919-21/usr/sbin/munin20:50:200
3141699125,3cyclictest224-21systemd-journal19:40:010
3141699124,4cyclictest224-21systemd-journal20:00:000
3141699123,4cyclictest224-21systemd-journal23:55:000
3143099118,2cyclictest7445-21cat22:35:013
3143099118,2cyclictest7445-21cat22:35:003
3143099118,2cyclictest5814-21ssh21:20:103
3143099118,2cyclictest4833-21grep23:45:193
3143099118,2cyclictest4244-21tr19:20:143
3143099118,2cyclictest31960-21ssh00:05:213
3143099118,2cyclictest31262-21ssh21:15:093
3143099118,2cyclictest30730-21tr20:15:153
3143099118,2cyclictest27576-21wc22:00:193
3143099118,2cyclictest25649-21grep20:05:143
3143099118,2cyclictest23270-21grep20:00:153
3143099118,2cyclictest20736-21cut19:55:123
3143099118,2cyclictest20667-21ssh23:32:563
3143099118,2cyclictest19223-21tr00:20:173
3143099118,2cyclictest17577-21sed19:45:283
3143099118,2cyclictest16272-21fgrep20:50:223
3143099118,2cyclictest14358-21chrt20:45:583
3143099118,2cyclictest13193-21ssh21:50:063
3143099118,2cyclictest1192-21head20:20:183
3143099118,2cyclictest10872-21ssh22:36:463
3143099110,8cyclictest0-21swapper/319:15:013
3143099110,8cyclictest0-21swapper/319:15:003
3143099110,1cyclictest141rcu_preempt22:56:133
3142699118,2cyclictest863-21expr20:20:162
3142699118,2cyclictest7942-21cut22:35:142
3142699118,2cyclictest771-21seq22:05:132
3142699118,2cyclictest7583-21ssh00:36:102
3142699118,2cyclictest31967-21ssh22:04:092
3142699118,2cyclictest30563-21sh22:51:152
3142699118,2cyclictest25809-21python00:00:272
3142699118,2cyclictest24981-21apache_volume00:25:102
3142699118,2cyclictest23024-21taskset20:00:052
3142699118,2cyclictest21821-21ntpq19:55:192
3142699118,2cyclictest21537-21basename23:10:002
3142699118,2cyclictest19287-21taskset20:58:132
3142699118,2cyclictest1761-21ssh23:19:112
3142699118,2cyclictest15560-21chrt19:43:592
3142699118,2cyclictest15080-21expr22:40:162
3142699118,1cyclictest19856-21cat00:20:202
3142699117,3cyclictest14000-21ssh22:15:192
3142699112,4cyclictest6134-21idleruntime-cro19:25:012
3142699111,1cyclictest24223-21diskmemload00:16:572
3142199119,1cyclictest24223-21diskmemload22:22:071
3142199118,2cyclictest9967-21sort20:39:591
3142199118,2cyclictest7259-21cat19:25:161
3142199118,2cyclictest4086-21ssh21:42:331
3142199118,2cyclictest3587-21ssh00:33:531
3142199118,2cyclictest31888-21tr19:10:151
3142199118,2cyclictest31888-21tr19:10:151
3142199118,2cyclictest28578-21tail20:10:191
3142199118,2cyclictest26192-21ssh23:36:471
3142199118,2cyclictest23889-21sort20:00:191
3142199118,2cyclictest19618-21ssh23:07:511
3142199118,2cyclictest19008-21tail19:50:201
3142199118,2cyclictest18823-21ssh22:43:071
3142199118,2cyclictest14680-21ssh00:15:451
3142199118,2cyclictest13728-21ssh23:51:301
3142199118,2cyclictest12659-21ssh22:38:421
3142199118,2cyclictest12509-21cat20:45:011
3142199118,2cyclictest11695-21ssh23:01:261
3142199118,2cyclictest1158-21ssh23:42:291
3142199116,3cyclictest0-21swapper/121:25:161
3142199110,10cyclictest23946-21ssh23:10:331
3141699118,2cyclictest9393-21ssh00:38:180
3141699118,2cyclictest7990-21grep22:10:160
3141699118,2cyclictest5944-21taskset19:22:470
3141699118,2cyclictest5429-21ssh21:44:000
3141699118,2cyclictest4216-21ssh22:55:350
3141699118,2cyclictest32014-21perl00:30:220
3141699118,2cyclictest29318-21ssh22:50:240
3141699118,2cyclictest29057-21ssh22:01:090
3141699118,2cyclictest28849-21grep23:15:130
3141699118,2cyclictest28091-21ssh21:11:580
3141699118,2cyclictest27857-21ssh23:38:350
3141699118,2cyclictest24861-21ssh00:00:180
3141699118,2cyclictest18582-21grep19:50:160
3141699118,2cyclictest18081-21grep20:55:180
3141699118,2cyclictest17739-21cut23:55:150
3141699118,2cyclictest17738-21ls23:30:220
3141699118,2cyclictest16412-21grep19:45:180
3141699118,2cyclictest11599-21chrt21:24:220
3141699118,2cyclictest1020-21grep22:05:170
3141699118,2cyclictest10120-21ssh22:36:010
3141699117,3cyclictest21322-21grep21:55:260
3141699117,2cyclictest10338-21tr20:40:150
3143099109,1cyclictest0-21swapper/300:10:093
3143099109,1cyclictest0-21swapper/300:10:083
3143099108,1cyclictest9583-21basename19:30:173
3143099108,1cyclictest11445-21seq19:35:163
3143099107,2cyclictest20496-21expr21:00:183
3142699108,1cyclictest8768-21cat20:35:192
3142699108,1cyclictest8766-21apache_processe19:30:122
3142699108,1cyclictest32387-21taskset20:17:262
3142699108,1cyclictest22211-21aten2_r7power_v21:05:122
3142699108,1cyclictest19702-21cat21:00:122
3142699107,2cyclictest3940-21grep20:25:222
3142699107,2cyclictest13990-21cat20:45:252
3142699107,2cyclictest11985-21taskset20:41:002
3142199108,1cyclictest8085-21ssh00:11:001
3142199108,1cyclictest8085-21ssh00:11:001
3142199108,1cyclictest31951-21sed20:15:251
3142199108,1cyclictest12616-21sed19:35:281
3142199107,2cyclictest7241-21sed21:45:121
3142199107,2cyclictest17907-21dnf20:55:161
3142199107,2cyclictest12610-21grep20:45:131
3142199107,2cyclictest10687-21ssh23:25:181
3142199100,0cyclictest0-21swapper/120:30:161
3141699108,1cyclictest322962sleep020:16:180
3141699108,1cyclictest31983-21cat19:10:160
3141699108,1cyclictest31983-21cat19:10:160
3141699108,1cyclictest2483-21ssh22:30:240
3141699108,1cyclictest2483-21ssh22:30:230
3141699108,1cyclictest12678-21wget20:45:140
3141699107,2cyclictest6985-21awk19:25:150
3141699107,2cyclictest3882-21sed20:25:220
3141699107,2cyclictest19952-21tr21:00:150
3141699107,2cyclictest1331-21if_enp2s020:20:190
3141699107,2cyclictest13181-21ssh21:50:060
3141699107,2cyclictest10756-21chrt19:33:340
314309997,1cyclictest101532sleep320:40:013
314269997,1cyclictest18891-21fschecks_count19:50:192
314219997,1cyclictest271982sleep120:05:331
314219997,1cyclictest16566-21fschecks_time19:45:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional