You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 16:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat May 16, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5781210483,6sleep00-21swapper/007:09:380
562428370,9sleep30-21swapper/307:07:363
368028366,13sleep10-21swapper/107:05:051
546928165,12sleep20-21swapper/207:05:352
589699290,29cyclictest0-21swapper/210:48:022
589699270,27cyclictest0-21swapper/211:29:002
589099270,2cyclictest0-21swapper/009:10:140
589699261,5cyclictest11174-21grep11:50:282
589699260,25cyclictest0-21swapper/210:59:142
589699253,5cyclictest2761-21tail12:05:172
590199241,22cyclictest0-21swapper/311:00:133
589699243,3cyclictest7721-21apt-get11:30:132
589699243,2cyclictest0-21swapper/210:29:552
589699240,5cyclictest0-21swapper/210:00:322
590199234,1cyclictest6436-21apt-get11:10:113
589699237,15cyclictest8446-21apache211:37:132
589699232,3cyclictest1590-21acpi09:50:122
589399234,18cyclictest31113-21diskmemload11:57:231
275352230,0sleep20-21swapper/212:39:542
589699221,2cyclictest17964-21cpuspeed_turbos11:55:142
589699220,20cyclictest0-21swapper/212:34:372
589699220,1cyclictest0-21swapper/212:28:062
589699220,1cyclictest0-21swapper/209:18:142
589699220,18cyclictest0-21swapper/210:32:052
589699220,16cyclictest0-21swapper/210:23:272
589399221,1cyclictest0-21swapper/107:30:141
589699214,5cyclictest0-21swapper/209:37:032
589699210,21cyclictest0-21swapper/211:42:412
589699210,21cyclictest0-21swapper/211:17:372
589699210,16cyclictest0-21swapper/210:10:532
589699210,16cyclictest0-21swapper/210:10:532
589099210,21cyclictest0-21swapper/011:36:470
589699201,7cyclictest8446-21apache210:08:552
589399202,1cyclictest31113-21diskmemload12:13:341
589699195,2cyclictest3833-21ssh11:08:322
589699190,8cyclictest0-21swapper/211:12:042
589699190,3cyclictest0-21swapper/212:19:562
589699190,2cyclictest0-21swapper/212:14:192
589699190,19cyclictest0-21swapper/210:44:582
589399191,1cyclictest0-21swapper/110:15:131
589399190,0cyclictest0-21swapper/110:21:221
589099190,19cyclictest0-21swapper/010:06:500
589099190,19cyclictest0-21swapper/009:31:410
590199181,10cyclictest0-21swapper/312:35:133
589699187,1cyclictest0-21swapper/209:13:072
5893991814,1cyclictest0-21swapper/111:29:141
589099181,1cyclictest141rcu_preempt12:19:490
590199173,1cyclictest26781-21apt-get10:05:153
590199172,1cyclictest0-21swapper/309:19:243
5901991714,2cyclictest39-21ksoftirqd/312:26:473
590199171,15cyclictest0-21swapper/310:49:473
590199170,16cyclictest0-21swapper/309:54:343
589699178,5cyclictest31113-21diskmemload09:10:002
5896991711,5cyclictest8446-21apache207:40:112
5901991612,4cyclictest39-21ksoftirqd/311:09:273
589699167,8cyclictest27623-21grep09:45:272
589699164,2cyclictest31113-21diskmemload09:28:032
5896991611,4cyclictest32727-21ssh09:30:142
589699160,16cyclictest0-21swapper/212:24:272
589399165,10cyclictest1142-21runrttasks11:00:421
589399162,9cyclictest25221-21/usr/sbin/munin09:25:181
589399160,16cyclictest0-21swapper/110:54:431
5890991615,1cyclictest0-21swapper/009:51:030
5890991615,1cyclictest0-21swapper/009:21:120
589099161,1cyclictest141rcu_preempt11:58:480
590199151,9cyclictest31124-21munin-run09:10:003
589699158,6cyclictest7085-21wc07:10:192
589699158,6cyclictest23605-21sort11:20:152
589699158,6cyclictest19776-21head07:35:282
589699153,2cyclictest31113-21diskmemload09:40:162
589399159,5cyclictest27526-21grep10:05:191
589099158,6cyclictest181irq_work/010:15:010
589099158,6cyclictest181irq_work/010:15:000
589099154,9cyclictest24539-21apt-get11:40:130
590199149,3cyclictest18340-21uniq11:55:163
590199141,4cyclictest141rcu_preempt10:59:083
5901991411,2cyclictest26215-21munin-run10:05:003
5901991411,2cyclictest1418-21grep11:45:163
5901991410,2cyclictest16235-21cut11:35:153
589699148,5cyclictest7793-21chrt10:50:582
589699148,2cyclictest13145-21ssh09:56:162
589699147,2cyclictest24757-21grep08:55:182
589699147,2cyclictest17280-21cut09:20:212
589699147,2cyclictest13890-21grep07:25:152
589699144,4cyclictest31113-21diskmemload10:18:522
589699143,1cyclictest31113-21diskmemload11:01:052
589699142,2cyclictest0-21swapper/208:20:172
5896991410,3cyclictest12414-21wc07:20:252
5896991410,2cyclictest22860-21sed08:50:222
589399148,3cyclictest14827-21cstates08:35:161
589399143,1cyclictest0-21swapper/110:37:011
589399141,9cyclictest141rcu_preempt12:04:131
5893991414,0cyclictest6391-21kworker/1:209:16:181
5893991413,1cyclictest38150irq/131-snd_hda12:28:041
5893991411,2cyclictest22309-21tune2fs08:50:161
5893991410,3cyclictest31159-21head11:05:171
5893991410,3cyclictest28638-21cut10:25:161
5893991410,3cyclictest18848-21cut07:35:191
589099143,5cyclictest225-21systemd-journal10:48:230
589099140,12cyclictest14572-21grep08:35:140
590199139,3cyclictest14737-21sh12:31:263
590199135,4cyclictest19394-21aten2_r7power_v08:45:143
590199134,7cyclictest28616-21perf10:45:003
590199134,5cyclictest0-21swapper/310:20:153
5901991310,2cyclictest9664-21sed11:30:253
5901991310,2cyclictest32405-21grep11:25:163
5901991310,2cyclictest1991-21seq12:05:143
5901991310,2cyclictest19567-21grep12:15:163
5901991310,2cyclictest18229-21grep07:35:153
5901991310,2cyclictest13156-21ssh10:15:283
5901991310,2cyclictest13065-21cron07:25:003
5901991310,2cyclictest12914-21head08:30:173
5901991310,2cyclictest11451-21grep09:55:203
590199130,12cyclictest32518-21sed11:45:013
589699138,4cyclictest19842sleep208:09:102
589699137,1cyclictest3653-21find08:10:242
589699136,2cyclictest7648-21sh11:49:202
589699136,2cyclictest16212-21head08:35:282
589699136,1cyclictest26573-21tr07:50:222
589699136,1cyclictest139672sleep208:31:412
589699134,3cyclictest17136-21wc07:30:262
5896991310,2cyclictest5803-21grep08:15:212
5896991310,2cyclictest27885-21wc09:00:232
5896991310,2cyclictest26263-21tr12:00:132
589699130,4cyclictest225-21systemd-journal08:42:582
589699130,4cyclictest225-21systemd-journal08:42:582
589399139,3cyclictest9656-21cat09:55:011
589399139,3cyclictest16950-21ssh09:40:001
589399139,2cyclictest7249-21head11:10:151
589399138,4cyclictest2590-21ssh12:05:161
589399132,8cyclictest0-21swapper/110:30:141
5893991311,2cyclictest0-21swapper/110:58:251
5893991310,2cyclictest29709-21sh08:00:001
5893991310,2cyclictest218702sleep112:35:281
5893991310,2cyclictest13566-21tr07:25:141
589099139,3cyclictest9701-21tr07:15:210
589099139,3cyclictest19137-21grep10:00:180
589099139,2cyclictest12356-21grep08:30:140
589099136,1cyclictest28760-21idleruntime-cro09:05:000
589099132,6cyclictest17835-21munin-run07:34:590
5890991312,1cyclictest0-21swapper/010:15:180
5890991311,1cyclictest0-21swapper/008:45:130
5890991310,2cyclictest4774-21cut08:15:150
5890991310,2cyclictest3041-21grep08:10:170
5890991310,2cyclictest29567-21cat10:25:250
5890991310,2cyclictest25084-21head09:25:160
5890991310,2cyclictest24219-21cut11:20:190
5890991310,2cyclictest22928-21grep07:45:150
5890991310,2cyclictest17671-21sed08:40:170
5890991310,2cyclictest17671-21sed08:40:160
5890991310,2cyclictest16056-21taskset09:39:370
5890991310,2cyclictest15809-21cat11:35:000
5890991310,2cyclictest15502-21sed10:55:270
589099130,2cyclictest0-21swapper/010:51:570
590199129,2cyclictest8842-21tr07:15:153
590199129,2cyclictest8768-21grep08:20:263
590199129,2cyclictest6838-21head07:10:163
590199129,2cyclictest6160-21ls08:15:263
590199129,2cyclictest5356-21tr10:50:143
590199129,2cyclictest490-21cut09:30:143
590199129,2cyclictest3946-21sed08:10:283
590199129,2cyclictest31876-21chrt09:48:443
590199129,2cyclictest30227-21tr08:00:153
590199129,2cyclictest20224-21cron10:39:593
590199129,2cyclictest18351-21ls12:15:003
590199129,2cyclictest1117-21tail09:10:263
590199128,3cyclictest16171-21cat11:15:203
589699129,2cyclictest9220-21sort07:15:172
589699129,2cyclictest31104-21grep08:00:202
589699129,2cyclictest21447-21cat08:50:002
589699127,1cyclictest12195-21taskset10:35:052
589699126,2cyclictest23402-21df_abs07:45:172
589699124,2cyclictest11399-21sed08:25:262
589399129,2cyclictest9437-21sh09:35:181
589399129,2cyclictest9150-21grep11:30:221
589399129,2cyclictest729-21grep08:05:181
589399129,2cyclictest6812-21cron08:20:001
589399129,2cyclictest30196-21ls09:05:231
589399129,2cyclictest26501-21grep11:40:271
589399129,2cyclictest26122-21sh09:00:001
589399129,2cyclictest10588-21cron10:15:001
589399129,2cyclictest10588-21cron10:15:001
589399129,2cyclictest0-21swapper/109:32:441
589399128,4cyclictest0-21swapper/107:20:001
589399128,2cyclictest19546-21sed12:15:161
589399122,5cyclictest28609-21perf10:45:001
5893991212,0cyclictest0-21swapper/109:10:071
589399121,11cyclictest38150irq/131-snd_hda11:45:151
5893991210,1cyclictest0-21swapper/112:20:401
589399120,11cyclictest23965-21cut07:45:211
589399120,11cyclictest0-21swapper/110:49:101
589099129,2cyclictest8620-21cat11:50:000
589099129,2cyclictest29091-21grep12:20:270
589099129,2cyclictest25130-21perf09:45:000
589099129,2cyclictest21728-21grep12:35:260
589099129,2cyclictest11674-21grep07:20:170
589099129,2cyclictest10176-21tail11:50:180
589099128,3cyclictest7780-21grep08:20:150
589099128,3cyclictest11200-21head12:10:160
589099124,4cyclictest225-21systemd-journal12:35:000
589099122,9cyclictest30304-21apt-get11:05:130
589099121,9cyclictest15114-21grep11:15:140
590199119,1cyclictest23109-21ssh11:20:033
590199118,2cyclictest9591-21tr11:50:163
590199118,2cyclictest8360-21ssh09:35:033
590199118,2cyclictest6734-21ssh10:12:033
590199118,2cyclictest6734-21ssh10:12:023
590199118,2cyclictest4056-21cut10:30:163
590199118,2cyclictest31830-21ssh12:22:343
590199118,2cyclictest30842-21sh09:29:013
590199118,2cyclictest29852-21ssh12:01:523
590199118,2cyclictest28184-21tr10:25:143
590199118,2cyclictest21866-21tr08:50:143
590199118,2cyclictest17155-21sh09:40:083
590199117,2cyclictest24051-21grep08:55:143
590199115,2cyclictest225-21systemd-journal07:34:593
590199110,3cyclictest16043-21aten2_r7power_p09:20:143
589699110,3cyclictest225-21systemd-journal07:57:492
589399118,2cyclictest6847-21grep07:10:171
589399118,2cyclictest26419-21rm11:21:161
589399118,2cyclictest2592-21grep08:10:151
589399118,2cyclictest21810-21which07:40:201
589399118,2cyclictest20464-21ls08:45:211
589399118,2cyclictest19202-21head10:00:181
589399118,2cyclictest18741-21chrt08:41:471
589399118,2cyclictest18741-21chrt08:41:461
589399118,2cyclictest18335-21ssh09:20:461
589399118,2cyclictest18110-21sed11:35:271
589399118,2cyclictest17865-21ssh12:33:541
589399118,2cyclictest17297-21ssh11:15:431
589399118,2cyclictest12382-21ssh09:55:411
589399118,2cyclictest11402-21ssh11:50:311
589399118,2cyclictest10712-21cat08:25:191
589399116,4cyclictest31113-21diskmemload09:48:301
589099118,2cyclictest723-21sed08:05:180
589099118,2cyclictest6184-21ssh11:10:010
589099118,2cyclictest6006-21ssh12:26:120
589099118,2cyclictest3072-21ssh12:05:200
589099118,2cyclictest28338-21ssh12:00:410
589099118,2cyclictest25836-21grep09:45:150
589099118,2cyclictest25330-21gzip08:55:230
589099118,2cyclictest25060-21ssh10:42:130
589099118,2cyclictest22731-21grep11:00:170
589099118,2cyclictest20012-21expr10:20:150
589099118,2cyclictest1867-21ssh11:25:520
589099118,2cyclictest14998-21sed07:25:270
589099118,2cyclictest13937-21ssh10:35:240
589099115,2cyclictest225-21systemd-journal08:30:010
589099114,3cyclictest10018-21apache_processe09:55:100
589099113,7cyclictest7505-21apt09:15:140
589099111,2cyclictest141rcu_preempt10:31:510
590199108,1cyclictest25555-21expr07:50:153
590199108,1cyclictest224122sleep307:42:503
590199108,1cyclictest14819-21ls08:35:153
590199107,2cyclictest9715-21taskset08:25:063
590199107,2cyclictest27679-21tr07:55:153
590199107,2cyclictest22997-21cut07:45:153
590199107,2cyclictest16974-21cut08:40:133
590199107,2cyclictest16974-21cut08:40:133
590199107,2cyclictest14125-21grep07:25:173
590199100,9cyclictest27437-21wc09:00:193
589399108,1cyclictest31786-21chrt08:01:001
589399108,1cyclictest272152sleep107:53:341
589399108,1cyclictest109692sleep107:20:121
589399107,2cyclictest7833-21head08:20:161
589399107,2cyclictest28735-21sh09:05:001
589399107,2cyclictest12876-21sed08:30:171
589099108,1cyclictest293512sleep007:55:290
589099108,1cyclictest27120-21seq07:52:220
589099107,2cyclictest7286-21ntpq07:10:200
589099107,2cyclictest30674-21grep08:00:170
589099107,2cyclictest30591-21cat09:05:270
589099107,2cyclictest22442-21hostname08:50:170
589099107,2cyclictest22439-21taskset07:43:110
59019997,1cyclictest17022sleep308:05:303
58909997,1cyclictest18480-21ls07:35:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional