You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 14:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 16, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1308929268,18sleep10-21swapper/107:05:171
1447728774,9sleep00-21swapper/007:09:540
1421128768,15sleep30-21swapper/307:06:283
1415828661,9sleep20-21swapper/207:05:482
14597993617,18cyclictest31828-21kworker/3:211:21:453
1459099346,21cyclictest14825-21grep12:25:241
14590992319,3cyclictest8762-21df_abs09:10:151
1459799220,20cyclictest0-21swapper/312:28:053
1459799220,1cyclictest0-21swapper/312:39:323
1459799220,0cyclictest0-21swapper/309:43:123
1458999222,9cyclictest0-21swapper/011:35:150
1459799211,2cyclictest141rcu_preempt12:10:443
1459099212,2cyclictest0-21swapper/111:15:141
14590992120,1cyclictest7624-21diskmemload11:03:221
1459799200,2cyclictest0-21swapper/310:32:103
1459799200,19cyclictest0-21swapper/310:55:043
1459599203,16cyclictest0-21swapper/211:32:412
1459599200,19cyclictest0-21swapper/209:45:522
1459099190,1cyclictest0-21swapper/111:10:501
1459799180,2cyclictest0-21swapper/310:36:123
1459599170,16cyclictest0-21swapper/209:27:272
14590991711,6cyclictest25-21ksoftirqd/108:55:121
1459099170,1cyclictest0-21swapper/111:26:271
1458999170,16cyclictest18351-21cron08:25:000
1459799160,14cyclictest0-21swapper/309:49:193
1459599160,1cyclictest0-21swapper/212:01:242
1459099163,12cyclictest7624-21diskmemload09:44:221
1459099162,4cyclictest1932-21apache_processe11:40:131
14590991614,1cyclictest7624-21diskmemload12:35:461
1459099160,15cyclictest0-21swapper/110:05:561
1458999164,6cyclictest2861-21munin-run09:00:000
1458999160,1cyclictest0-21swapper/008:45:130
1459599153,4cyclictest11482-21ssh10:28:462
14595991512,2cyclictest27861-21ssh09:40:212
14595991511,3cyclictest10334-21ntp_kernel_pll_11:25:212
1458999150,4cyclictest0-21swapper/011:55:130
112362150,0sleep30-21swapper/310:28:353
14597991410,3cyclictest29165-21cut08:45:163
14597991410,3cyclictest1582-21grep09:25:143
14595991412,1cyclictest0-21swapper/209:32:542
1459599141,13cyclictest10018-21kworker/u8:110:50:002
14595991411,2cyclictest23688-21tr10:55:162
1459599140,13cyclictest0-21swapper/211:57:002
1459099144,9cyclictest7624-21diskmemload10:48:011
14590991412,2cyclictest10550irq/122-ahci[0008:10:001
14590991410,3cyclictest3493-21grep12:00:151
14590991410,3cyclictest18412-21date08:25:011
14590991410,3cyclictest12455-21ls12:25:001
14590991410,3cyclictest12455-21ls12:25:001
1459099140,2cyclictest141rcu_preempt09:57:151
1459099140,13cyclictest22689-21grep10:15:261
1458999148,5cyclictest23769-21apache212:09:220
1458999141,6cyclictest225-21systemd-journal08:05:000
14589991411,2cyclictest20054-21sh11:11:540
14589991410,3cyclictest26630-21turbostat.cron11:55:000
1458999140,13cyclictest32590-21grep11:00:170
1459799139,3cyclictest25975-21sh09:20:173
1459799139,2cyclictest9624-21grep08:05:183
1459799136,5cyclictest0-21swapper/309:40:003
1459799133,9cyclictest7624-21diskmemload10:45:523
14597991310,2cyclictest7622-21sh09:10:003
14597991310,2cyclictest31930-21grep11:00:153
14597991310,2cyclictest25872-21cut11:15:213
14597991310,2cyclictest24498-21cut07:30:133
14597991310,2cyclictest23642-21grep08:35:143
14597991310,2cyclictest19857-21tr07:20:133
14597991310,2cyclictest19680-21tr09:55:143
14597991310,2cyclictest0-21swapper/312:01:523
1459799130,3cyclictest0-21swapper/312:15:343
1459799130,13cyclictest0-21swapper/312:31:413
1459799130,12cyclictest29078-21grep11:55:263
1459599139,3cyclictest6785-21tr08:00:132
1459599139,3cyclictest4011-21cat12:20:002
1459599139,3cyclictest27261-21grep08:40:222
1459599139,3cyclictest23955-21wc10:35:262
1459599139,2cyclictest17320-21grep07:15:142
1459599138,3cyclictest21219-21tail07:20:252
1459599134,4cyclictest0-21swapper/210:18:582
1459599132,2cyclictest0-21swapper/209:17:122
14595991310,2cyclictest6214-21cron08:00:002
14595991310,2cyclictest6214-21cron08:00:002
14595991310,2cyclictest5802-21tr09:05:152
14595991310,2cyclictest3448-21wc10:05:012
14595991310,2cyclictest30058-21grep12:35:152
14595991310,2cyclictest29682-21tr07:40:132
14595991310,2cyclictest27265-21grep07:35:152
14595991310,2cyclictest22883-21ssh10:55:002
14595991310,2cyclictest16451-21grep07:10:252
14595991310,2cyclictest1458-21grep11:20:182
14595991310,2cyclictest12983-21cut10:10:162
14595991310,2cyclictest12902-21grep12:25:122
14595991310,2cyclictest11700-21grep08:10:142
1459099139,3cyclictest27997-21head11:55:171
1459099139,3cyclictest22934-21sed08:30:271
1459099138,4cyclictest29637-21expr10:20:131
1459099131,1cyclictest0-21swapper/107:15:141
14590991311,1cyclictest0-21swapper/109:16:321
14590991310,2cyclictest9805-21tr09:30:131
14590991310,2cyclictest28665-21grep07:35:271
14590991310,2cyclictest26052-21grep07:30:261
14590991310,2cyclictest25302-21grep10:55:271
14590991310,2cyclictest21797-21cat07:25:001
14590991310,2cyclictest20635-21grep12:10:151
14590991310,2cyclictest11075-21rm09:50:121
1459099130,1cyclictest7624-21diskmemload12:19:281
1459099130,0cyclictest1308-21kworker/1:207:50:241
1458999136,4cyclictest0-21swapper/009:10:150
14589991310,2cyclictest9008-21basename09:30:000
14589991310,2cyclictest6389-21cat08:00:010
14589991310,2cyclictest6389-21cat08:00:000
14589991310,2cyclictest5237-21cron12:40:000
14589991310,2cyclictest23763-21apache209:50:030
14589991310,2cyclictest23466-21cat08:35:000
14589991310,2cyclictest23206-21sed12:30:260
14589991310,2cyclictest14226-21ls12:25:190
1459799129,2cyclictest9734-21grep11:25:173
1459799129,2cyclictest5722-21grep07:55:273
1459799129,2cyclictest5722-21grep07:55:273
1459799129,2cyclictest5256-21ssh10:05:233
1459799129,2cyclictest4295-21ntpq09:00:193
1459799129,2cyclictest31536-21sed08:50:173
1459799129,2cyclictest30009-21tail10:20:153
1459799129,2cyclictest27129-21grep08:40:203
1459799129,2cyclictest22028-21grep07:25:123
1459799129,2cyclictest16634-21cut09:15:143
1459799120,12cyclictest0-21swapper/310:00:363
1459799120,11cyclictest0-21swapper/311:09:083
1459799120,0cyclictest0-21swapper/311:11:223
1459599129,2cyclictest630-21grep11:00:202
1459599129,2cyclictest32005-21sed08:50:222
1459599129,2cyclictest31465-21grep10:40:172
1459599129,2cyclictest29482-21tr10:20:132
1459599129,2cyclictest27180-21taskset11:35:212
1459599129,2cyclictest2523-21cut07:50:162
1459599129,2cyclictest23227-21grep07:25:202
1459599129,2cyclictest17212-21wc08:20:182
1459599129,2cyclictest15090-21tr08:15:222
1459599129,2cyclictest12398-21sleep12:25:002
1459599129,2cyclictest12398-21sleep12:25:002
1459599128,3cyclictest5651-21wc10:05:292
1459599128,3cyclictest11681-21cut09:50:142
1459599126,5cyclictest23769-21apache211:10:122
1459099129,2cyclictest8876-21wc11:05:201
1459099129,2cyclictest8210-21grep08:00:251
1459099129,2cyclictest30922-21tr10:40:141
1459099129,2cyclictest29402-21cat08:45:181
1459099129,2cyclictest16426-21sed10:50:221
1459099129,2cyclictest15084-21tr07:10:131
1459099129,2cyclictest1490-21basename07:50:001
1459099128,3cyclictest19373-21sed11:30:261
1459099128,3cyclictest15223-21head10:30:221
1459099128,3cyclictest12099-21sort08:10:161
1459099126,6cyclictest23763-21apache209:39:211
1459099126,5cyclictest28179-21wc08:45:001
14590991211,1cyclictest1413-21latency_hist11:40:011
1459099120,12cyclictest0-21swapper/110:12:411
1458999129,2cyclictest5256-21apt09:05:000
1458999129,2cyclictest4895-21cat12:00:250
1458999129,2cyclictest31485-21cut10:40:170
1458999129,2cyclictest31463-21grep08:50:170
1458999129,2cyclictest30027-21grep10:20:150
1458999129,2cyclictest26690-21sort08:40:160
1458999129,2cyclictest2598-21aten2_r7power_e09:45:130
1458999129,2cyclictest21803-21wc10:35:000
1458999129,2cyclictest20906-21seq08:29:390
1458999129,2cyclictest2081-21tr11:40:150
1458999129,2cyclictest18174-21grep11:30:160
1458999129,2cyclictest14052-21cat10:10:250
1458999129,2cyclictest13159-21sed08:10:260
1458999129,2cyclictest10186-21grep09:30:150
1458999127,4cyclictest23763-21apache211:28:440
1458999123,6cyclictest475-21in:imuxsock10:15:000
1458999123,5cyclictest27373-21sendmail10:00:000
1458999121,6cyclictest0-21swapper/011:45:120
1459799118,2cyclictest9813-21expr09:10:253
1459799118,2cyclictest7851-21tr08:00:213
1459799118,2cyclictest4033-21cat07:55:003
1459799118,2cyclictest3476-21grep11:40:233
1459799118,2cyclictest31860-21ssh11:38:293
1459799118,2cyclictest31202-21taskset07:41:033
1459799118,2cyclictest20603-21sed11:50:263
1459799118,2cyclictest18326-21grep11:30:173
1459799118,2cyclictest17873-21ssh10:51:053
1459799118,2cyclictest16949-21wc07:15:003
1459799118,2cyclictest14833-21ssh10:10:513
1459799118,2cyclictest13916-21grep08:15:143
1459799118,2cyclictest12160-21wc11:45:263
1459799118,2cyclictest12098-21df12:05:153
1459799118,2cyclictest11754-21grep09:50:153
1459799118,2cyclictest11623-21ssh09:30:303
1459799117,3cyclictest31759-21cat10:40:193
1459799117,2cyclictest31889-21grep07:45:133
1459799117,2cyclictest17765-21ls08:20:253
1459799117,2cyclictest11576-21grep08:10:143
1459799115,2cyclictest225-21systemd-journal08:25:013
1459799115,2cyclictest225-21systemd-journal07:35:013
14597991110,1cyclictest0-21swapper/312:20:103
14597991110,1cyclictest0-21swapper/312:20:103
1459799110,10cyclictest21181-21expr10:15:153
1459599118,2cyclictest9837-21cut08:05:202
1459599118,2cyclictest9387-21strings09:10:192
1459599118,2cyclictest3270-21ntpq11:40:222
1459599118,2cyclictest31933-21grep07:45:132
1459599118,2cyclictest28649-21ssh12:34:322
1459599118,2cyclictest27934-21ssh09:21:062
1459599118,2cyclictest24549-21cat08:35:182
1459599118,2cyclictest24538-21sh11:15:102
1459599118,2cyclictest21439-21ssh11:51:002
1459599118,2cyclictest21235-21date09:55:252
1459599118,2cyclictest20216-21expr12:10:132
1459599118,2cyclictest19839-21taskset10:33:342
1459599118,2cyclictest18813-21tail09:35:172
1459599118,2cyclictest16834-21sh12:08:032
1459599118,2cyclictest11305-21ssh11:45:182
1459599117,3cyclictest18942-21grep08:25:142
1459599114,5cyclictest21473-21cpuspeed_turbos08:30:152
1459599110,10cyclictest0-21swapper/211:06:292
1459099118,2cyclictest9967-21ssh10:27:361
1459099118,2cyclictest897-21ssh09:25:061
1459099118,2cyclictest4794-21uniq07:55:171
1459099118,2cyclictest4794-21uniq07:55:171
1459099118,2cyclictest4348-21ls09:00:191
1459099118,2cyclictest31285-21chrt07:42:081
1459099118,2cyclictest28735-21sed10:00:161
1459099118,2cyclictest26335-21ssh09:20:191
1459099118,2cyclictest2294-21ssh09:45:031
1459099118,2cyclictest22358-21wget10:35:151
1459099118,2cyclictest11364-21ssh12:05:041
1459099118,1cyclictest22880-21ssh12:30:221
1459099110,1cyclictest0-21swapper/111:21:591
1458999118,2cyclictest9212-21sh10:45:520
1458999118,2cyclictest8058-21grep11:05:150
1458999118,2cyclictest5299-21cut10:05:240
1458999118,2cyclictest4378-21rm12:20:090
1458999118,2cyclictest4378-21rm12:20:090
1458999118,2cyclictest31536-21sleep07:45:000
1458999118,2cyclictest31273-21ssh12:16:140
1458999118,2cyclictest31150-21ssh11:18:580
1458999118,2cyclictest30530-21ssh09:23:060
1458999118,2cyclictest28594-21ssh09:40:290
1458999118,2cyclictest25659-21tr07:30:210
1458999118,2cyclictest23563-21basename08:35:110
1458999118,2cyclictest23492-21cut10:55:150
1458999118,2cyclictest23349-21cat10:35:200
1458999118,2cyclictest2047-21grep11:20:230
1458999118,2cyclictest20369-21ssh09:35:430
1458999118,2cyclictest20143-21grep12:10:130
1458999118,2cyclictest19821-21tr07:20:130
1458999118,2cyclictest1845-21tr07:50:130
1458999118,2cyclictest16583-21ssh09:15:130
1458999118,2cyclictest10976-21chrt08:09:560
1458999117,3cyclictest28536-21grep10:00:150
1458999110,6cyclictest0-21swapper/010:55:000
1459799108,1cyclictest21266-21cut08:30:143
1459799108,1cyclictest189552sleep307:15:493
1459599108,1cyclictest3868-21grep09:00:142
1459599108,1cyclictest25100-21cut07:30:152
1459099108,1cyclictest20754-21ssh11:50:271
1459099108,1cyclictest14939-21uname08:15:201
1459099108,1cyclictest10451-21hostname11:45:141
1459099100,9cyclictest30998-21taskset08:50:151
1459099100,9cyclictest18880-21awk08:25:131
1458999108,1cyclictest28165-21cat07:35:210
1458999108,1cyclictest16551-21taskset07:10:310
1458999108,1cyclictest14194-21expr08:15:150
1458999107,2cyclictest5546-21wget09:05:140
1458999107,2cyclictest5477-21tr10:25:140
1458999107,2cyclictest24094-21taskset07:29:180
1458999107,2cyclictest17454-21grep07:15:150
1458999100,9cyclictest31761-21cat07:45:100
145979990,5cyclictest225-21systemd-journal09:00:003
145959997,1cyclictest29813-21python08:45:242
145909997,1cyclictest6448-21tr09:05:191
145909997,1cyclictest22021-21cut07:25:121
145909990,9cyclictest0-21swapper/108:35:041
145959986,1cyclictest2583-21chrt08:56:422
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional