You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 14:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jan 01, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2225929480,9sleep00-21swapper/007:06:550
2249529266,8sleep30-21swapper/307:09:583
2218829279,9sleep20-21swapper/207:06:002
2245128272,6sleep10-21swapper/107:09:251
27622650,0sleep00-21swapper/008:40:250
22591992524,1cyclictest0-21swapper/207:53:402
2258899232,14cyclictest0-21swapper/110:45:141
2258899224,16cyclictest9398-21awk10:05:001
2258899221,3cyclictest0-21swapper/107:25:131
2258899211,17cyclictest0-21swapper/107:35:131
22586992019,1cyclictest0-21swapper/007:51:580
2258699176,6cyclictest25143-21perf11:44:590
2258699174,7cyclictest9395-21munin-run10:05:000
22594991611,1cyclictest39-21ksoftirqd/310:55:263
2258899169,6cyclictest1732-21grep12:00:291
22586991611,3cyclictest8583-21ls07:45:260
2259499151,2cyclictest141rcu_preempt11:10:143
22594991511,3cyclictest381rcuc/311:35:003
2259499150,1cyclictest0-21swapper/311:45:143
22591991512,3cyclictest32-21ksoftirqd/209:50:152
22594991411,2cyclictest21358-21munin-run08:15:003
22594991410,3cyclictest21201-21grep09:20:163
2259199149,3cyclictest12236-21cat12:25:002
2259199144,6cyclictest0-21swapper/209:55:122
22591991411,2cyclictest23089-21tr07:10:142
22591991410,3cyclictest9391-21munin-plugin-st10:05:002
2258899149,3cyclictest15196-21grep08:00:161
2258699147,6cyclictest15387-21cron09:10:000
22586991410,2cyclictest26499-21grep08:25:160
2259499139,3cyclictest18351-21grep12:35:223
2259499139,3cyclictest18351-21grep12:35:213
22594991310,2cyclictest7172-21grep07:45:133
22594991310,2cyclictest6829-21sed08:50:173
22594991310,2cyclictest31393-21ls09:40:223
22594991310,2cyclictest27117-21grep10:40:193
22594991310,2cyclictest27117-21grep10:40:183
22594991310,2cyclictest14997-21head09:05:263
22594991310,2cyclictest14019-21head11:20:183
2259199139,3cyclictest8812-21ls12:15:272
2259199139,3cyclictest18358-21tr09:15:122
2259199134,4cyclictest225-21systemd-journal07:55:012
22591991310,2cyclictest15432-21wc09:10:002
22591991310,2cyclictest11886-21cat10:10:012
2258899139,3cyclictest7139-21wc08:50:201
2258899139,3cyclictest29610-21sort08:30:171
2258899139,3cyclictest14908-21grep12:30:141
2258899139,3cyclictest10983-21cat07:50:251
2258899139,2cyclictest11135-21tr09:00:161
22588991311,1cyclictest0-21swapper/111:35:151
22588991310,2cyclictest25-21ksoftirqd/111:10:231
22588991310,2cyclictest17615-21tail08:05:151
22588991310,2cyclictest17115-21tr09:10:221
22588991310,2cyclictest10202-21tr12:20:151
2258699138,3cyclictest9209-21cut11:10:180
22586991310,2cyclictest8332sleep010:55:000
22586991310,2cyclictest7496-21tail11:05:270
22586991310,2cyclictest30016-21wc08:30:220
22586991310,2cyclictest24036-21sed07:10:220
22586991310,2cyclictest19394-21basename12:40:000
22586991310,2cyclictest19394-21basename12:40:000
2258699130,0cyclictest0-21swapper/009:50:140
2259499129,2cyclictest6964-21cron12:15:013
2259499129,2cyclictest5663-21cut09:55:163
2259499129,2cyclictest4101-21tune2fs11:00:183
2259499129,2cyclictest322-21grep09:45:133
2259499129,2cyclictest31559-21ls11:55:263
2259499129,2cyclictest30706-21head07:25:183
2259499129,2cyclictest29284-21sed07:20:263
2259499129,2cyclictest2029-21cron07:34:593
2259499129,2cyclictest1433-21grep08:40:133
2259499129,2cyclictest13301-21ls10:10:213
2259499128,3cyclictest26414-21ls07:15:223
2259499128,3cyclictest25674-21sed10:35:263
2259499128,3cyclictest21078-21grep11:35:173
2259499128,3cyclictest17280-21grep11:25:273
2259499128,3cyclictest12628-21sed09:00:293
2259499127,3cyclictest26316-21wc08:25:013
2259499123,1cyclictest141rcu_preempt12:05:123
2259199129,2cyclictest9188-21turbostat.cron07:49:592
2259199129,2cyclictest7524-21cat08:50:262
2259199129,2cyclictest3914-21grep07:35:272
2259199129,2cyclictest32104-21grep08:35:192
2259199129,2cyclictest31745-21grep11:55:272
2259199129,2cyclictest31375-21grep10:50:152
2259199129,2cyclictest27995-21grep10:40:292
2259199129,2cyclictest27995-21grep10:40:282
2259199129,2cyclictest24057-21cut10:35:132
2259199129,2cyclictest23054-21grep11:40:152
2259199129,2cyclictest21843-21grep10:30:162
2259199129,2cyclictest18654-21sort11:30:162
2259199129,2cyclictest17959-21cat08:05:192
2259199123,7cyclictest31573-21python09:40:252
22591991211,1cyclictest0-21swapper/211:05:232
2258899129,2cyclictest9344-21cron12:20:011
2258899129,2cyclictest5605-21grep12:10:191
2258899129,2cyclictest353-21tr09:45:131
2258899129,2cyclictest28489-21grep09:35:171
2258899129,2cyclictest28215-21cut07:20:161
2258899129,2cyclictest28057-21tr11:50:171
2258899129,2cyclictest2587-21wc08:40:221
2258899129,2cyclictest25707-21head10:35:261
2258899129,2cyclictest23791-21grep09:25:181
2258899129,2cyclictest19371-21basename12:40:011
2258899129,2cyclictest19371-21basename12:40:001
2258899129,2cyclictest15937-21tr11:25:151
2258899129,2cyclictest12574-21df10:10:141
2258899128,3cyclictest1309-21tr10:55:151
2258899122,2cyclictest225-21systemd-journal09:59:591
22588991210,1cyclictest6574-21grep11:05:171
2258699129,2cyclictest8297-21wc12:15:210
2258699129,2cyclictest6794-21cron07:44:590
2258699129,2cyclictest4467-21grep08:45:190
2258699129,2cyclictest4467-21grep08:45:180
2258699129,2cyclictest4400-21cron07:39:590
2258699129,2cyclictest3853-21cat12:05:240
2258699129,2cyclictest32081-21grep08:35:190
2258699129,2cyclictest29912-21sleep09:39:590
2258699129,2cyclictest27104-21grep10:40:180
2258699129,2cyclictest27104-21grep10:40:180
2258699129,2cyclictest27063-21tail09:30:280
2258699129,2cyclictest25692-21sed10:35:270
2258699129,2cyclictest23909-21wc10:35:000
2258699129,2cyclictest22071-21wc09:20:260
2258699129,2cyclictest19522-21grep08:10:150
2258699129,2cyclictest17578-21grep08:05:150
2258699129,2cyclictest14196-21ls10:15:000
2258699129,2cyclictest12817-21head07:55:140
2258699129,2cyclictest10334-21sort10:05:160
2258699128,2cyclictest30238-21tr07:25:150
2258699128,2cyclictest10873-21wc12:20:200
2258699124,7cyclictest6987-21munin-run12:15:000
2259499119,1cyclictest8938-21sed12:15:273
2259499119,1cyclictest6946-21tail11:05:213
2259499119,1cyclictest6667-21taskset07:43:243
2259499119,1cyclictest28394-21sort11:50:193
2259499118,2cyclictest5398-21grep08:45:293
2259499118,2cyclictest5398-21grep08:45:283
2259499118,2cyclictest39-21ksoftirqd/312:25:133
2259499118,2cyclictest3216-21tail09:50:193
2259499118,2cyclictest28465-21taskset08:29:413
2259499118,2cyclictest25901-21sed09:30:163
2259499118,2cyclictest24522-21grep07:10:263
2259499118,2cyclictest23739-21latency_hist08:20:003
2259499118,2cyclictest22811-21ls10:30:243
2259499118,2cyclictest19974-21sed09:15:253
2259499118,2cyclictest17193-21sendmail_mailqu09:10:223
2259499118,2cyclictest1236-21ntpq12:00:233
2259499117,3cyclictest9979-21tr10:05:143
2259499115,2cyclictest225-21systemd-journal10:05:003
22594991110,1cyclictest19964-21kworker/u8:111:42:393
2259499110,10cyclictest4051-21taskset07:35:303
2259499110,10cyclictest14972-21sh08:00:153
2259199118,2cyclictest8813-21expr08:55:152
2259199118,2cyclictest5585-21sh11:05:002
2259199118,2cyclictest3214-21perf11:00:012
2259199118,2cyclictest31497-21cat07:25:272
2259199118,2cyclictest29244-21df_abs10:45:162
2259199118,2cyclictest28555-21head11:50:202
2259199118,2cyclictest28143-21df07:20:142
2259199118,2cyclictest27586-21pluginstate08:25:232
2259199118,2cyclictest25865-21taskset08:21:462
2259199118,2cyclictest19942-21head10:25:172
2259199118,2cyclictest19013-21taskset12:35:452
2259199118,2cyclictest19013-21taskset12:35:442
2259199118,2cyclictest1728-21grep12:00:292
2259199118,2cyclictest1609-21tr08:40:142
2259199118,2cyclictest15324-21expr12:30:152
2259199118,2cyclictest15275-21chrt11:23:442
2259199118,2cyclictest12767-21chrt11:17:142
2259199118,2cyclictest12163-21aten2_r7power_p10:10:132
2259199118,2cyclictest11642-21grep09:00:172
2259199117,3cyclictest31097-21idleruntime-cro08:35:002
2259199110,8cyclictest0-21swapper/211:25:142
2259199110,0cyclictest0-21swapper/210:20:172
2258899118,2cyclictest9986-21tr10:05:141
2258899118,2cyclictest7153-21hostname07:45:141
2258899118,2cyclictest641-21tail07:30:211
2258899118,2cyclictest5086-21grep11:00:281
2258899118,2cyclictest4666-21wc08:45:211
2258899118,2cyclictest4666-21wc08:45:201
2258899118,2cyclictest3279-21tail09:50:191
2258899118,2cyclictest32463-21head08:35:231
2258899118,2cyclictest30404-21grep11:55:151
2258899118,2cyclictest28320-21chrt08:27:481
2258899118,2cyclictest27566-21cat11:50:001
2258899118,2cyclictest25637-21grep08:20:251
2258899118,2cyclictest23273-21sed08:15:281
2258899118,2cyclictest21874-21tr10:30:151
2258899118,2cyclictest18125-21wc10:20:261
2258899118,2cyclictest14312-21chrt12:26:211
2258899118,2cyclictest11544-21tune2fs11:15:151
2258899116,4cyclictest574-21taskset10:52:081
2258899110,10cyclictest32204-21taskset09:44:091
2258699118,2cyclictest8489-21cut08:55:130
2258699118,2cyclictest7190-21uname08:50:210
2258699118,2cyclictest4241-21cut11:00:190
2258699118,2cyclictest29569-21chrt07:23:240
2258699118,2cyclictest28986-21sed11:50:260
2258699118,2cyclictest2725-21grep10:55:280
2258699118,2cyclictest26139-21grep11:45:170
2258699118,2cyclictest25798-21cut07:15:150
2258699118,2cyclictest2090-21date07:35:000
2258699118,2cyclictest20041-21chrt09:16:000
2258699118,2cyclictest16375-21taskset10:17:380
2258699118,2cyclictest16367-21grep11:25:170
2258699118,2cyclictest11598-21grep11:15:160
2258699117,3cyclictest30794-21tune2fs09:40:170
2258699113,4cyclictest225-21systemd-journal12:30:000
2258699111,6cyclictest17487-21grep10:20:200
2258699110,10cyclictest21468-21sed11:35:220
2258699110,10cyclictest19881-21cut10:25:170
2259499108,1cyclictest9174-21head08:55:183
2259499108,1cyclictest31069-21taskset08:34:543
2259499108,1cyclictest30334-21cat10:45:283
2259499108,1cyclictest28180-21expr09:35:163
2259499108,1cyclictest21218-21chrt10:28:353
2259499108,1cyclictest14646-21egrep10:15:153
2259499108,1cyclictest11923-21readlink07:55:013
2259499108,1cyclictest11173-21cat11:15:133
2259499108,1cyclictest10975-21cat12:20:203
2259499107,2cyclictest32084-21head08:35:193
2259499107,2cyclictest15349-21expr12:30:153
2259499103,6cyclictest18101-21munin-node10:22:023
2259199108,2cyclictest21234-21dnf09:20:162
2259199108,1cyclictest28523-21expr09:35:182
2259199108,1cyclictest27252-21taskset07:19:272
2259199108,1cyclictest27222-21chrt11:46:362
2259199108,1cyclictest25411-21apache_volume09:30:132
2259199108,1cyclictest1784-21taskset07:31:562
2259199108,1cyclictest15797-21gzip08:00:222
2259199107,2cyclictest937-21basename09:45:162
2259199107,2cyclictest8400-21dpkg11:10:022
2259199107,2cyclictest6417-21sed07:40:272
2259199107,2cyclictest3495-21cat12:05:202
2259199107,2cyclictest23231-21wc09:25:152
2259199107,2cyclictest21907-21cut08:15:142
2259199107,2cyclictest16403-21taskset10:18:012
2259199107,2cyclictest13146-21grep12:25:172
2259199106,3cyclictest19498-21grep08:10:152
2259199100,9cyclictest0-21swapper/209:10:122
2258899108,1cyclictest9264-21grep08:55:171
2258899108,1cyclictest6652-21seq07:43:121
2258899108,1cyclictest27823-21sed10:40:281
2258899108,1cyclictest27823-21sed10:40:271
2258899108,1cyclictest25290-21cat11:45:001
2258899108,1cyclictest247182sleep107:12:271
2258899108,1cyclictest18374-21cut09:15:121
2258899108,1cyclictest13073-21wc07:55:181
2258899107,2cyclictest25849-21cat09:30:161
2258899107,2cyclictest20903-21cut09:20:141
2258899107,2cyclictest15041-21chrt11:20:421
2258899107,2cyclictest14799-21expr10:15:141
2258899106,3cyclictest20297-21cat08:10:201
2258899100,9cyclictest25759-21sed07:15:151
2258899100,9cyclictest179892sleep111:30:071
2258899100,9cyclictest13995-21expr09:05:161
2258699108,1cyclictest19983-21taskset11:33:100
2258699108,1cyclictest16717-21if_enp0s31f609:10:170
2258699108,1cyclictest11164-21grep09:00:160
2258699107,2cyclictest6671-21taskset09:56:060
2258699107,2cyclictest369-21tr09:45:130
2258699107,2cyclictest32037-21chrt11:57:270
2258699107,2cyclictest29023-21grep10:45:160
2258699107,2cyclictest25795-21chrt08:20:500
2258699107,2cyclictest23119-21cut09:25:150
2258699107,2cyclictest1831-21sed12:00:280
2258699107,2cyclictest16801-21taskset12:32:520
2258699106,3cyclictest14893-21grep11:20:280
2258699100,9cyclictest23217-21sed08:15:280
2258699100,9cyclictest16242-21taskset08:00:340
225949997,1cyclictest9889-21expr07:50:133
225949997,1cyclictest168602sleep308:05:103
225949991,4cyclictest225-21systemd-journal10:50:013
225919997,1cyclictest66392sleep212:10:512
225919996,2cyclictest20375-21cat11:35:122
225919993,3cyclictest225-21systemd-journal08:45:102
225919993,3cyclictest225-21systemd-journal08:45:092
225889997,1cyclictest20364-21grep10:25:211
225889992,3cyclictest225-21systemd-journal12:05:081
225949985,2cyclictest24145-21uname09:25:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional