You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-12 - 12:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jun 12, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
174721200,0sleep30-21swapper/323:50:183
2988329065,9sleep00-21swapper/019:07:340
3002928774,9sleep20-21swapper/219:09:292
2928328470,9sleep30-21swapper/319:05:243
2982528366,13sleep10-21swapper/119:06:511
252062700,0sleep20-21swapper/222:46:582
157062620,1sleep013-21ksoftirqd/021:25:010
30165992827,1cyclictest0-21swapper/123:09:231
3016599280,0cyclictest0-21swapper/123:19:021
3016599270,1cyclictest0-21swapper/123:47:481
3016799260,26cyclictest0-21swapper/222:50:392
3016599260,1cyclictest0-21swapper/100:00:131
3016599250,2cyclictest0-21swapper/122:31:151
3016599250,2cyclictest0-21swapper/122:31:151
3016599250,1cyclictest0-21swapper/121:37:141
30167992421,2cyclictest23215-21apache_volume21:10:132
3016599240,1cyclictest0-21swapper/122:02:241
3016599230,1cyclictest0-21swapper/122:08:361
3016599230,0cyclictest0-21swapper/122:27:271
3016599230,0cyclictest0-21swapper/122:27:271
30160992320,2cyclictest1480-21/usr/sbin/munin21:55:210
3016799220,19cyclictest0-21swapper/200:30:392
30165992220,1cyclictest9350-21df_inode21:40:151
3016599220,0cyclictest0-21swapper/123:53:051
30160992220,1cyclictest4417-21apache_volume22:35:120
30170992118,2cyclictest30119-21python22:30:253
30170992118,2cyclictest30119-21python22:30:253
3017099210,19cyclictest0-21swapper/323:06:483
30167992119,1cyclictest9894-21/usr/sbin/munin22:00:262
30165992119,1cyclictest0-21swapper/121:05:111
3016599210,1cyclictest0-21swapper/123:29:021
30160992119,1cyclictest23046-21munin-run23:25:000
30160992118,2cyclictest598-21irqstats23:30:190
3016099210,1cyclictest141rcu_preempt00:10:110
30170992018,1cyclictest6047-21python22:35:263
30170992017,2cyclictest24257-21/usr/sbin/munin21:30:113
30167992018,1cyclictest13388-21df22:40:162
30167992017,2cyclictest18271-21/usr/sbin/munin22:05:182
3016799200,15cyclictest0-21swapper/221:45:132
3016799200,0cyclictest0-21swapper/223:09:562
30165992014,3cyclictest0-21swapper/121:51:261
3016599200,1cyclictest0-21swapper/122:57:211
3016099208,1cyclictest22722-21diskmemload21:36:210
3016099200,1cyclictest0-21swapper/022:03:400
313962190,0chrt0-21swapper/022:31:100
313962190,0chrt0-21swapper/022:31:090
30170991919,0cyclictest0-21swapper/323:40:433
3017099190,18cyclictest0-21swapper/323:36:053
3016799190,19cyclictest0-21swapper/223:32:162
3016799190,19cyclictest0-21swapper/221:22:472
3016599192,3cyclictest0-21swapper/100:12:501
30165991919,0cyclictest0-21swapper/100:23:251
30165991916,2cyclictest14674-21diskstats23:00:151
3016599190,2cyclictest0-21swapper/122:48:441
3016599190,1cyclictest0-21swapper/100:07:311
30170991816,1cyclictest32564-21hddtemp_smartct21:15:193
30167991815,2cyclictest2854-21ntp_states21:55:212
30165991817,1cyclictest38150irq/131-snd_hda23:59:341
3016099186,10cyclictest13-21ksoftirqd/000:05:550
30167991715,1cyclictest3600-21df_inode22:15:162
30165991713,4cyclictest26502-21apache222:42:241
3016599170,1cyclictest0-21swapper/122:18:141
3016599170,15cyclictest0-21swapper/123:39:161
30160991714,2cyclictest11414-21/usr/sbin/munin00:35:200
3017099160,14cyclictest0-21swapper/319:25:113
3016799169,2cyclictest8723-21grep19:30:202
30165991614,1cyclictest17975-21users21:25:271
30165991614,1cyclictest12264-21hddtemp_smartct22:20:181
3016599160,6cyclictest25238-21if_enp2s020:05:181
3016599160,2cyclictest0-21swapper/100:33:581
3016099163,6cyclictest1142-21runrttasks22:49:220
30170991513,1cyclictest28036-21hddtemp_smartct00:25:163
3016799153,11cyclictest489-21cron20:55:002
30167991515,0cyclictest0-21swapper/222:30:352
30167991515,0cyclictest0-21swapper/222:30:352
3016799150,15cyclictest0-21swapper/223:48:422
30165991514,1cyclictest0-21swapper/119:15:171
30165991513,1cyclictest29663-21apache_accesses22:50:131
3016599150,5cyclictest1102-21kworker/1:200:16:281
3016599150,2cyclictest0-21swapper/121:33:591
30160991514,1cyclictest0-21swapper/021:15:140
3016099150,4cyclictest0-21swapper/023:00:190
233072150,0sleep30-21swapper/323:25:023
3017099142,1cyclictest141rcu_preempt00:15:313
30170991411,2cyclictest3399-21grep21:55:263
30170991411,2cyclictest14006-21apache_accesses23:00:123
30170991410,3cyclictest9892-21head23:55:173
30170991410,3cyclictest27496-21ssh21:50:263
30170991410,3cyclictest21002-21ssh21:46:543
30167991412,1cyclictest0-21swapper/223:42:262
30167991411,2cyclictest25083-21perl21:30:172
30167991410,3cyclictest15970-21grep23:20:162
3016799140,14cyclictest0-21swapper/221:37:042
3016799140,14cyclictest0-21swapper/200:16:342
30165991413,1cyclictest0-21swapper/122:13:221
3016599140,11cyclictest0-21swapper/121:55:581
3016099145,3cyclictest225-21systemd-journal21:05:000
3016099141,1cyclictest141rcu_preempt00:00:210
30160991410,3cyclictest7836-21ssh21:20:150
3017099139,3cyclictest19334-21uname21:00:193
3017099139,3cyclictest17423-21wc21:25:223
3017099139,2cyclictest31103-21grep19:10:163
30170991312,0cyclictest0-21swapper/323:20:053
30170991311,1cyclictest2075-21/usr/sbin/munin00:10:183
30170991310,2cyclictest30327-21head20:15:203
30170991310,2cyclictest25116-21head20:05:173
30170991310,2cyclictest19542-21cut00:20:163
3017099130,12cyclictest0-21swapper/300:32:313
3016799139,3cyclictest21519-21awk20:00:002
3016799139,3cyclictest14911-21tr19:45:142
30167991310,2cyclictest8417-21cron20:40:002
30167991310,2cyclictest27741-21sh22:30:002
30167991310,2cyclictest27741-21sh22:30:002
30167991310,2cyclictest18772-21apache_processe00:20:112
30167991310,2cyclictest14647-21sh23:20:002
3016799130,7cyclictest0-21swapper/220:10:122
3016799130,1cyclictest0-21swapper/223:03:302
3016599138,4cyclictest15473-21ssh00:37:011
30165991310,2cyclictest5782-21ssh21:18:591
30165991310,2cyclictest28494-21grep20:10:251
30165991310,2cyclictest17603-21grep23:40:211
30165991310,2cyclictest1290-21sh23:11:051
30165991310,2cyclictest11525-21grep19:35:261
3016599130,13cyclictest0-21swapper/100:26:281
3016099139,3cyclictest17937-21sh21:00:010
30160991311,2cyclictest13-21ksoftirqd/022:59:340
30160991311,1cyclictest27130-21apache_accesses00:25:100
30160991310,2cyclictest8755-21grep23:15:270
30160991310,2cyclictest7215-21cron19:29:590
30160991310,2cyclictest5142-21munin-run22:55:000
30160991310,2cyclictest32134-21tr23:10:190
30160991310,2cyclictest23664-21grep23:25:130
3016099130,7cyclictest22722-21diskmemload21:15:010
3016099130,3cyclictest0-21swapper/022:14:310
3016099130,12cyclictest4915-21wc00:30:250
3016099130,12cyclictest26232-21sleep22:10:000
3017099129,2cyclictest9933-21head20:40:213
3017099129,2cyclictest24655-21wc21:10:223
3017099129,2cyclictest18811-21ls00:00:223
3017099129,2cyclictest17675-21sort19:50:153
3017099129,2cyclictest17267-21ssh00:38:253
3017099129,2cyclictest11095-21wc22:00:203
30170991210,1cyclictest0-21swapper/323:49:023
3017099120,1cyclictest0-21swapper/322:05:043
3017099120,11cyclictest2444-21cron22:15:003
3017099120,11cyclictest14233-21sed20:50:163
3016799129,2cyclictest9966-21cut19:35:142
3016799129,2cyclictest604-21grep19:15:142
3016799129,2cyclictest4848-21sh19:24:592
3016799129,2cyclictest31656-21tr20:20:122
3016799129,2cyclictest30862-21sed20:15:272
3016799129,2cyclictest27477-21tr00:25:132
3016799129,2cyclictest25140-21grep20:05:172
3016799129,2cyclictest1802-21tr20:25:122
3016799129,2cyclictest16300-21cstates21:25:142
3016799129,2cyclictest15901-21cut20:55:132
3016799128,3cyclictest32184-21ssh23:10:202
3016799128,3cyclictest19120-21cat00:00:262
3016799126,6cyclictest26520-21apache221:42:272
30167991210,1cyclictest13144-21python00:35:262
3016599129,2cyclictest942-21grep23:30:231
3016599129,2cyclictest7640-21wc20:35:211
3016599129,2cyclictest5153-21sort22:35:161
3016599129,2cyclictest32331-21grep20:20:151
3016599129,2cyclictest30642-21grep19:10:131
3016599129,2cyclictest24081-21grep21:10:171
3016599129,2cyclictest17764-21head19:50:161
3016599129,2cyclictest17494-21grep20:55:261
3016599128,3cyclictest20331-21cron21:05:001
3016099129,2cyclictest9088-21grep23:55:140
3016099129,2cyclictest8233-21tr23:35:140
3016099129,2cyclictest26297-21ssh21:50:160
3016099129,2cyclictest22991-21grep20:00:210
3016099129,2cyclictest19111-21grep00:20:140
3016099129,2cyclictest17261-21grep19:50:130
3016099129,2cyclictest1516-21cut23:50:160
3016099129,2cyclictest13516-21wc19:40:220
3016099128,3cyclictest375-21sed20:20:210
3016099128,2cyclictest12636-21head22:20:200
3016099124,7cyclictest4872-21idleruntime-cro19:24:590
3016099124,4cyclictest225-21systemd-journal19:20:000
3016099124,4cyclictest13-21ksoftirqd/022:18:420
3016099120,9cyclictest161rcuc/022:25:170
3016099120,9cyclictest161rcuc/022:25:170
3016099120,8cyclictest0-21swapper/019:55:120
3016099120,1cyclictest0-21swapper/023:47:240
3017099119,1cyclictest21130-21cat22:25:263
3017099119,1cyclictest21130-21cat22:25:253
3017099118,2cyclictest8543-21ssh22:18:133
3017099118,2cyclictest8308-21chrt19:30:173
3017099118,2cyclictest8150-21ssh21:20:153
3017099118,2cyclictest6605-21ssh23:15:043
3017099118,2cyclictest5529-21ssh22:55:093
3017099118,2cyclictest31631-21cut20:20:123
3017099118,2cyclictest2931-21egrep19:20:143
3017099118,2cyclictest28754-21taskset20:12:373
3017099118,2cyclictest25570-21ssh00:05:073
3017099118,2cyclictest2451-21cron19:20:003
3017099118,2cyclictest23890-21sh20:05:013
3017099118,2cyclictest23873-21ssh22:45:563
3017099118,2cyclictest21093-21head19:55:253
3017099118,2cyclictest16208-21ssh22:22:373
3017099118,2cyclictest15025-21taskset22:40:353
3017099118,2cyclictest13400-21ssh21:42:303
3017099118,2cyclictest11846-21sed20:45:163
3017099118,2cyclictest10089-21grep19:35:153
3017099110,10cyclictest32626-21sed23:10:253
3017099110,10cyclictest0-21swapper/321:35:183
3016799119,1cyclictest4023-21grep00:10:262
3016799118,2cyclictest9866-21sh20:40:212
3016799118,2cyclictest7630-21sed22:55:272
3016799118,2cyclictest6212-21sh23:53:112
3016799118,2cyclictest6103-21cat20:35:002
3016799118,2cyclictest6050-21python22:35:262
3016799118,2cyclictest32050-21sed19:10:252
3016799118,2cyclictest29818-21ssh22:11:042
3016799118,2cyclictest27728-21ssh00:05:272
3016799118,2cyclictest26174-21ssh23:25:552
3016799118,2cyclictest25910-21expr21:50:142
3016799118,2cyclictest19120-21munin-run19:55:012
3016799118,2cyclictest18682-21expr21:00:142
3016799118,2cyclictest1655-21ssh21:15:452
3016799118,2cyclictest12188-21grep20:45:192
3016799118,2cyclictest11279-21ssh23:55:372
3016799118,2cyclictest10963-21ssh23:36:142
3016799110,10cyclictest13743-21ssh22:20:432
3016599118,2cyclictest6787-21tail19:25:261
3016599118,2cyclictest4182-21tr20:30:151
3016599118,2cyclictest3672-21cron20:30:001
3016599118,2cyclictest29650-21expr20:15:151
3016599118,2cyclictest2848-21tr19:20:131
3016599118,2cyclictest22503-21head20:00:161
3016599118,2cyclictest21364-21ssh21:47:121
3016599118,2cyclictest19734-21expr19:55:131
3016599118,2cyclictest16797-21date19:50:001
3016599118,2cyclictest15495-21grep23:20:141
3016599118,2cyclictest10644-21sh21:21:151
3016099118,2cyclictest7721-21grep19:30:130
3016099118,2cyclictest4715-21grep20:30:170
3016099118,2cyclictest22691-21tr23:05:150
3016099118,2cyclictest21318-21expr21:05:140
3016099118,2cyclictest19455-21ssh21:45:440
3016099118,2cyclictest16229-21grep19:45:260
3016099118,2cyclictest15141-21head20:50:250
3016099118,2cyclictest10864-21ssh21:40:320
3016099118,2cyclictest10625-21cut00:15:140
3016099118,2cyclictest10077-21wc19:35:140
3016099115,3cyclictest13-21ksoftirqd/023:44:200
3016099114,6cyclictest12802-21/usr/sbin/munin22:40:160
30160991111,0cyclictest0-21swapper/021:31:550
30160991110,1cyclictest0-21swapper/020:35:120
3016099110,10cyclictest31202-21taskset20:18:260
3017099108,1cyclictest57412sleep320:31:073
3017099108,1cyclictest30444-21cut22:50:183
3017099108,1cyclictest2315-21tr20:25:153
3017099107,2cyclictest6986-21cut20:35:153
3017099107,2cyclictest21305-21tr21:05:153
3017099107,2cyclictest13622-21tr19:40:223
3017099106,3cyclictest32438-21expr23:30:143
3016799108,1cyclictest22761-21cat20:00:182
3016799108,1cyclictest21308-21grep21:05:152
3016799107,2cyclictest6658-21wc19:25:252
3016799107,2cyclictest12562-21cat19:40:142
3016599108,1cyclictest10861-21date20:45:001
3016599107,2cyclictest8268-21expr19:30:161
3016599107,2cyclictest14259-21taskset19:43:501
3016599107,2cyclictest13562-21tr20:50:131
3016599107,2cyclictest11527-21cat20:45:141
3016099108,1cyclictest324302sleep019:14:190
3016099108,1cyclictest26129-21chrt20:05:450
3016099108,1cyclictest129102sleep020:46:310
3016099107,2cyclictest26796-21apache_volume20:10:120
3016099107,2cyclictest2305-21expr20:25:150
3016099107,2cyclictest10431-21sed20:40:250
301709997,1cyclictest165532sleep319:47:483
301709996,2cyclictest15877-21cut20:55:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional