You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 19:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 21, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23213210085,10sleep20-21swapper/207:05:172
2452229377,11sleep00-21swapper/007:06:200
2462828774,9sleep30-21swapper/307:07:433
2477228666,16sleep10-21swapper/107:09:341
36772460,0sleep00-21swapper/010:38:090
2488399290,2cyclictest0-21swapper/211:04:502
2488899280,1cyclictest0-21swapper/312:13:323
2488899270,0cyclictest0-21swapper/309:53:493
2488899260,1cyclictest0-21swapper/311:36:013
2488899260,1cyclictest0-21swapper/310:55:173
2487999260,22cyclictest0-21swapper/109:54:021
217912260,1sleep021790-21cron10:29:590
24888992423,1cyclictest0-21swapper/310:07:363
2488399230,22cyclictest0-21swapper/210:54:542
2487899231,5cyclictest0-21swapper/009:30:370
2488899220,1cyclictest0-21swapper/311:24:203
2488899220,1cyclictest0-21swapper/309:28:473
24879992220,2cyclictest25-21ksoftirqd/110:09:201
2488399211,1cyclictest0-21swapper/211:45:132
2487999210,2cyclictest0-21swapper/110:00:141
24888992018,2cyclictest39-21ksoftirqd/310:34:003
2487999203,17cyclictest17476-21diskmemload11:44:161
2487999202,2cyclictest141rcu_preempt11:10:221
24879992020,0cyclictest0-21swapper/110:46:401
2487999200,20cyclictest0-21swapper/111:45:131
2487999200,19cyclictest0-21swapper/110:20:401
2487999200,19cyclictest0-21swapper/108:10:131
2488899190,2cyclictest828-21ssh11:34:543
2488899190,2cyclictest0-21swapper/310:24:173
2488399190,18cyclictest0-21swapper/210:59:312
24878991919,0cyclictest0-21swapper/011:57:520
2488899180,17cyclictest0-21swapper/312:39:523
2488399185,12cyclictest1277-21cat07:25:282
24883991818,0cyclictest0-21swapper/211:39:282
24879991814,2cyclictest0-21swapper/109:15:281
2487999180,2cyclictest0-21swapper/109:56:281
280302170,0chrt0-21swapper/311:11:473
24888991716,1cyclictest7834-21kworker/3:012:02:233
24888991716,1cyclictest7834-21kworker/3:010:01:133
2488399170,16cyclictest0-21swapper/211:32:052
24879991711,4cyclictest16112-21sh12:02:511
2487899172,14cyclictest7561-21apt-get11:00:130
24878991716,1cyclictest0-21swapper/009:26:040
24878991715,2cyclictest0-21swapper/010:01:090
2488899169,6cyclictest12849-21apache210:29:553
24888991615,1cyclictest7834-21kworker/3:011:15:033
24888991615,1cyclictest7834-21kworker/3:011:04:033
2488899160,3cyclictest22457-21kworker/u8:209:46:113
2488399162,13cyclictest0-21swapper/211:12:472
2488399160,1cyclictest0-21swapper/212:38:472
2487999163,9cyclictest38150irq/131-snd_hda07:50:111
2487999160,16cyclictest0-21swapper/111:16:061
24878991615,1cyclictest0-21swapper/010:45:530
2487899160,2cyclictest17476-21diskmemload10:20:230
2487899160,1cyclictest0-21swapper/011:43:360
24888991514,1cyclictest7834-21kworker/3:012:27:033
24888991514,1cyclictest7834-21kworker/3:012:27:023
2488399153,1cyclictest141rcu_preempt12:00:062
2488399152,4cyclictest225-21systemd-journal08:40:012
24883991514,1cyclictest0-21swapper/211:57:212
24883991513,2cyclictest32-21ksoftirqd/210:15:502
2488399151,1cyclictest16928-21ssh09:28:512
2488399150,15cyclictest0-21swapper/209:32:332
24879991512,3cyclictest0-21swapper/109:27:341
2487999150,15cyclictest0-21swapper/110:18:011
24878991510,3cyclictest18892-21sh08:05:000
2487899150,13cyclictest0-21swapper/009:11:380
2488899147,4cyclictest3124-21idleruntime-cro08:40:013
24888991413,1cyclictest7834-21kworker/3:010:42:133
24888991410,3cyclictest20875-21grep09:30:283
2488899140,1cyclictest0-21swapper/311:56:343
2488899140,13cyclictest19971-21sort11:45:253
2488399141,12cyclictest10452-21wc08:55:012
24879991412,2cyclictest25-21ksoftirqd/107:19:591
24879991411,3cyclictest55650irq/127-enp2s0-12:21:001
2487899149,3cyclictest22637-21grep08:10:200
2487899146,7cyclictest29614-21munin-run07:20:000
2487899144,5cyclictest475-21in:imuxsock08:50:000
24878991410,3cyclictest16656-21cut11:05:180
2487899140,1cyclictest0-21swapper/011:48:000
2488899139,3cyclictest14494-21cut07:55:163
2488899138,4cyclictest28016-21sed08:20:283
24888991312,1cyclictest7834-21kworker/3:012:07:523
24888991312,1cyclictest7834-21kworker/3:011:06:023
24888991310,2cyclictest5729-21cat12:15:273
24888991310,2cyclictest30431-21sed09:55:293
24888991310,2cyclictest29439-21sed08:25:193
24888991310,2cyclictest18276-21tr09:10:143
2488899130,4cyclictest0-21swapper/307:40:153
2488899130,11cyclictest0-21swapper/311:40:483
2488899130,0cyclictest0-21swapper/312:22:273
2488399139,3cyclictest30423-21sed09:55:292
2488399139,2cyclictest8348-21wc10:40:262
2488399132,1cyclictest0-21swapper/207:50:112
24883991312,1cyclictest0-21swapper/209:46:062
24883991311,2cyclictest32-21ksoftirqd/210:23:212
24883991310,2cyclictest3410-21dpkg12:15:002
24883991310,2cyclictest32-21ksoftirqd/212:30:512
24883991310,2cyclictest32009-21sh07:25:002
24883991310,2cyclictest28984-21cat07:15:262
24883991310,2cyclictest27265-21awk12:10:002
24883991310,2cyclictest19388-21tr08:05:172
24883991310,2cyclictest17938-21grep11:25:192
2488399130,5cyclictest0-21swapper/210:39:492
2487999139,2cyclictest4490-21cat09:40:191
2487999138,4cyclictest27963-21grep12:10:151
24879991311,1cyclictest25-21ksoftirqd/110:55:161
24879991310,2cyclictest32558-21tr07:25:171
24879991310,2cyclictest25818-21awk09:14:591
24879991310,2cyclictest17479-21turbostat.cron09:10:001
24879991310,2cyclictest16815-21grep11:05:181
24879991310,2cyclictest13680-21sh10:44:241
24879991310,1cyclictest0-21swapper/111:35:491
2487999130,2cyclictest0-21swapper/110:37:121
2487999130,2cyclictest0-21swapper/110:31:521
2487999130,13cyclictest0-21swapper/108:51:001
2487999130,10cyclictest0-21swapper/108:30:161
2487899139,3cyclictest8199-21ssh09:23:340
2487899139,3cyclictest27534-21ls08:20:240
2487899139,3cyclictest23161-21head10:30:180
2487899139,3cyclictest14398-21grep12:20:260
2487899137,4cyclictest13-21ksoftirqd/010:40:030
2487899131,6cyclictest26031-21sendmail08:20:000
24878991310,2cyclictest28229-21grep09:35:150
2487899130,11cyclictest0-21swapper/009:50:140
2488899129,2cyclictest3645-21grep09:20:213
2488899129,2cyclictest32311-21tr07:25:153
2488899129,2cyclictest31778-21grep08:30:193
2488899129,2cyclictest30460-21awk12:30:233
2488899129,2cyclictest2774-21df07:30:173
2488899129,2cyclictest21786-21grep08:10:133
2488899125,3cyclictest225-21systemd-journal08:50:003
2488899122,5cyclictest225-21systemd-journal08:55:013
24888991211,1cyclictest7834-21kworker/3:011:26:433
24888991211,1cyclictest7834-21kworker/3:010:48:523
2488899121,10cyclictest30625-21apache_volume10:35:123
2488399129,3cyclictest0-21swapper/210:48:162
2488399129,2cyclictest6719-21sendmail07:39:592
2488399129,2cyclictest3495-21cut09:40:142
2488399129,2cyclictest31845-21head08:30:202
2488399129,2cyclictest2912-21uniq09:20:162
2488399129,2cyclictest27106-21grep09:15:192
2488399129,2cyclictest17756-21cron11:45:002
2488399129,2cyclictest13272-21grep09:00:142
2488399129,2cyclictest11858-21basename12:20:002
2488399128,3cyclictest28105-21grep09:35:142
2488399128,3cyclictest18898-21sh08:05:002
2488399121,6cyclictest30791-21cron08:30:012
24883991211,1cyclictest12838-21apache210:33:122
24883991210,1cyclictest6572-21ssh11:18:492
24883991210,1cyclictest17918-21ssh11:05:282
2488399120,1cyclictest0-21swapper/210:29:042
2488399120,11cyclictest12863-21wc09:00:002
2487999129,2cyclictest7896-21sh08:50:001
2487999129,2cyclictest6246-21grep07:35:281
2487999129,2cyclictest27306-21wc07:15:001
2487999129,2cyclictest24406-21uname10:50:191
2487999129,2cyclictest2332-21grep09:20:141
2487999129,2cyclictest20869-21ssh12:25:141
2487999129,2cyclictest20869-21ssh12:25:141
2487999129,2cyclictest17458-21tr11:25:171
2487999129,2cyclictest14713-21sort10:25:181
2487999129,2cyclictest13018-21cut09:00:131
2487999128,3cyclictest5789-21sort12:35:171
2487999128,3cyclictest27614-21ssh11:30:321
2487999128,3cyclictest24639-21ssh10:11:391
2487999128,3cyclictest1693-21sort08:35:171
2487999128,3cyclictest15482-21tail07:55:221
2487999126,5cyclictest0-21swapper/112:30:281
2487999121,9cyclictest18871-21/usr/sbin/munin09:30:231
2487999121,2cyclictest3809-21ssh12:15:081
24879991212,0cyclictest0-21swapper/111:22:021
2487899129,2cyclictest990-21cat10:55:240
2487899129,2cyclictest9316-21turbostat11:39:590
2487899129,2cyclictest8083-21grep07:40:200
2487899129,2cyclictest21918-21grep10:10:170
2487899129,2cyclictest19792-21grep08:05:200
2487899129,2cyclictest18882-21sleep12:04:590
2487899129,2cyclictest1563-21df08:35:160
2487899129,2cyclictest1161-21ssh11:54:180
2487899128,4cyclictest17715-21munin-node11:21:050
2487899128,3cyclictest15611-21grep09:05:140
2487899128,2cyclictest1692-21sed11:15:240
2487899127,4cyclictest537-21snmpd11:27:590
2487899126,5cyclictest9373-21apt-get07:45:150
2487899125,7cyclictest12838-21apache210:19:280
2487899125,3cyclictest225-21systemd-journal09:45:000
2487899125,3cyclictest0-21swapper/009:45:280
2487899124,1cyclictest3220-21apt-get12:15:030
2487899123,8cyclictest0-21swapper/010:52:220
2487899121,2cyclictest0-21swapper/012:10:220
2487899121,1cyclictest141rcu_preempt12:38:450
24878991210,1cyclictest13-21ksoftirqd/012:30:090
2488899119,1cyclictest17124-21cat08:00:163
2488899118,2cyclictest8060-21ssh09:42:333
2488899118,2cyclictest6697-21cron07:39:593
2488899118,2cyclictest4534-21tr08:40:223
2488899118,2cyclictest30299-21expr07:20:163
2488899118,2cyclictest29790-21tr10:15:153
2488899118,2cyclictest27628-21chrt09:35:123
2488899118,2cyclictest26792-21aten2_r7power_p11:50:153
2488899118,2cyclictest26335-21cut07:10:223
2488899118,2cyclictest23149-21ssh10:50:053
2488899118,2cyclictest21410-21tr10:10:163
2488899118,2cyclictest16069-21head09:05:173
2488899118,2cyclictest13017-21grep07:50:203
2488899118,2cyclictest12638-21turbostat.cron08:59:593
2488899118,2cyclictest10606-21grep07:45:243
2488899117,3cyclictest25584-21tail08:15:273
2488899110,10cyclictest32207-21chrt09:18:303
2488399118,2cyclictest9092-21cron07:45:002
2488399118,2cyclictest5137-21cut10:00:182
2488399118,2cyclictest28195-21systemctl11:50:242
2488399118,2cyclictest27003-21chrt07:12:152
2488399118,2cyclictest26164-21ssh10:12:502
2488399118,2cyclictest17745-21ssh09:10:022
2488399118,2cyclictest15984-21sort09:05:172
2488399118,2cyclictest14992-21sort07:55:182
2488399118,2cyclictest12859-21wget10:05:152
2488399118,2cyclictest12308-21basename12:20:102
2488399118,2cyclictest10725-21ssh11:20:292
2488399116,1cyclictest225-21systemd-journal08:44:192
2487999118,2cyclictest8394-21taskset07:40:251
2487999118,2cyclictest3835-21sed07:30:281
2487999118,2cyclictest2984-21expr11:55:171
2487999118,2cyclictest28102-21grep11:50:221
2487999118,2cyclictest25830-21ssh12:08:551
2487999118,2cyclictest24657-21grep08:15:181
2487999118,2cyclictest13747-21ssh11:03:401
2487999118,2cyclictest13671-21ssh09:45:281
2487999118,2cyclictest12561-21chrt08:59:131
2487999117,3cyclictest29067-21ssh09:35:231
2487899118,2cyclictest6096-21cat07:35:270
2487899118,2cyclictest32011-21apt07:24:590
2487899118,2cyclictest29477-21head08:25:180
2487899118,2cyclictest28594-21ssh11:31:180
2487899118,2cyclictest28434-21rm09:55:130
2487899118,2cyclictest26014-21cat07:10:190
2487899118,2cyclictest2467-21tr07:30:150
2487899118,2cyclictest24377-21cut11:10:140
2487899118,2cyclictest22854-21ssh12:25:290
2487899118,2cyclictest22854-21ssh12:25:280
2487899118,2cyclictest19433-21ssh12:05:140
2487899118,2cyclictest16964-21ssh10:07:110
2487899118,2cyclictest10293-21cat08:55:010
2487899117,3cyclictest15015-21grep07:55:180
2487899116,4cyclictest27157-21ssh09:15:190
2487899111,6cyclictest1971-21ls07:30:000
2488899108,1cyclictest19609-21cat08:05:183
2488899102,4cyclictest0-21swapper/309:00:143
2488399108,1cyclictest23900-21cat08:15:122
2488399108,1cyclictest21848-21expr08:10:142
2488399108,1cyclictest20789-21apache_accesses12:25:122
2488399108,1cyclictest20789-21apache_accesses12:25:112
2488399107,2cyclictest9754-21expr07:45:172
2488399107,2cyclictest6994-21cut08:45:222
2488399107,2cyclictest27078-21head08:20:182
2488399107,2cyclictest19895-21ssh09:50:042
2487999108,2cyclictest38150irq/131-snd_hda07:45:161
2487999108,1cyclictest317702sleep107:22:081
2487999107,2cyclictest26354-21tr08:20:161
2487999107,2cyclictest18889-21cron08:05:001
2487999105,3cyclictest5059-21grep08:40:281
2487899108,1cyclictest31401-21expr08:30:180
2487899108,1cyclictest11196-21tune2fs08:55:180
2487899107,2cyclictest3893-21df_abs08:40:160
2487899105,2cyclictest225-21systemd-journal09:02:030
248889997,1cyclictest27714-21cpuspeed_turbos07:15:143
248839996,2cyclictest2482-21expr07:30:162
248799997,1cyclictest19366-21cat08:05:171
248789996,2cyclictest12015-21tr07:50:120
248799986,1cyclictest28901-21cpuspeed_turbos08:25:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional