You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 17:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Nov 21, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23742210678,9sleep10-21swapper/107:06:231
23989210491,9sleep30-21swapper/307:09:343
2390529470,8sleep00-21swapper/007:08:310
178742880,1sleep017209-21diskmemload12:35:110
2193628564,15sleep20-21swapper/207:05:152
2410999280,28cyclictest0-21swapper/211:51:412
2410599213,2cyclictest141rcu_preempt11:53:181
2411499191,14cyclictest6915-21/usr/sbin/munin10:20:173
2410999190,17cyclictest0-21swapper/212:34:572
24100991918,1cyclictest0-21swapper/008:25:130
2410099190,2cyclictest0-21swapper/011:50:240
2410999181,2cyclictest32-21ksoftirqd/212:05:272
2410999181,2cyclictest32-21ksoftirqd/212:05:272
293512178,2sleep229379-21sh09:20:012
2411499179,7cyclictest39-21ksoftirqd/311:54:523
24109991714,2cyclictest0-21swapper/211:45:202
24109991712,3cyclictest0-21swapper/209:25:132
2410099173,8cyclictest26468-21munin-run07:15:000
2410099172,8cyclictest27079-21/usr/sbin/munin09:45:200
24114991614,2cyclictest39-21ksoftirqd/310:03:333
2410999160,16cyclictest0-21swapper/209:51:402
2410099167,6cyclictest3194-21apt-get08:40:150
245332150,0chrt0-21swapper/012:16:320
2411499151,11cyclictest21076-21ssh09:40:153
2410999159,5cyclictest22111-21cat09:40:212
24109991513,1cyclictest37850irq/131-snd_hda07:45:172
24109991513,1cyclictest0-21swapper/208:25:132
24109991512,1cyclictest0-21swapper/212:38:462
2410999150,15cyclictest0-21swapper/210:43:562
2410599150,14cyclictest0-21swapper/112:03:121
24100991513,1cyclictest17209-21diskmemload09:14:020
311152140,2sleep232-21ksoftirqd/212:21:032
2411499149,4cyclictest19194-21grep08:05:143
2411499144,6cyclictest29185-21apt-get07:20:123
24114991412,1cyclictest19304-21ssh12:13:193
24114991410,3cyclictest25723-21grep10:35:183
24109991410,3cyclictest8202-21cpuspeed_turbos11:15:142
2410599143,4cyclictest28231-21awk12:20:001
24105991411,2cyclictest1578-21grep07:30:121
24105991410,3cyclictest27976-21cron08:25:001
24105991410,3cyclictest14345-21sed10:25:271
24105991410,2cyclictest15505-21grep10:00:251
76102130,1sleep10-21swapper/110:46:541
2411499139,3cyclictest8829-21grep12:05:223
2411499139,3cyclictest8829-21grep12:05:223
2411499138,4cyclictest14377-21sh09:35:003
24114991313,0cyclictest0-21swapper/309:56:083
24114991312,1cyclictest0-21swapper/309:26:143
24114991310,2cyclictest25127-21head08:15:283
24114991310,2cyclictest23404-21sed11:00:263
24114991310,2cyclictest18391-21wc10:55:223
2410999139,4cyclictest24095-21cyclictest09:55:132
2410999138,4cyclictest32222-21cut07:25:142
2410999138,4cyclictest28667-21tr09:45:232
2410999137,3cyclictest17209-21diskmemload12:14:412
2410999134,8cyclictest7744-21snmpd09:33:462
2410999132,2cyclictest224-21systemd-journal08:08:302
2410999130,13cyclictest0-21swapper/212:00:242
2410599139,3cyclictest5817-21grep08:45:161
2410599139,3cyclictest4405-21sed09:25:211
24105991310,3cyclictest25-21ksoftirqd/110:20:311
24105991310,2cyclictest9196-21tr07:45:141
24105991310,2cyclictest30886-21wc09:20:191
24105991310,2cyclictest27130-21grep11:55:201
24105991310,2cyclictest2393-21grep11:35:271
2410099136,4cyclictest17209-21diskmemload09:41:460
2410099135,1cyclictest6053-21apt-get07:40:000
24100991310,2cyclictest8674-21grep08:50:170
24100991310,2cyclictest8036-21cat11:40:200
2410099130,8cyclictest11044-21perf07:50:000
2410099130,13cyclictest0-21swapper/011:58:060
281022120,1sleep10-21swapper/110:37:081
2411499129,2cyclictest775-21cron09:50:003
2411499129,2cyclictest6029-21cron07:40:003
2411499129,2cyclictest3333-21tr12:25:143
2411499129,2cyclictest31007-21cut09:20:203
2411499129,2cyclictest30147-21grep08:25:263
2411499129,2cyclictest30036-21grep12:20:243
2411499129,2cyclictest23811-21tr09:15:143
2411499129,2cyclictest18190-21grep12:35:133
2411499129,2cyclictest16320-21head09:05:253
2411499129,2cyclictest15796-21ls08:00:003
2411499129,2cyclictest14140-21ssh11:20:003
2411499129,2cyclictest1258-21grep12:00:173
2411499129,2cyclictest10803-21df08:55:173
2411499129,2cyclictest1005-21tr08:35:153
2411499128,3cyclictest31567-21head08:30:183
2411499128,3cyclictest31567-21head08:30:183
2411499128,2cyclictest15372-21grep11:20:193
2411499121,5cyclictest0-21swapper/308:40:143
24114991210,1cyclictest22394-21head12:15:183
2411499120,11cyclictest19620-21sort10:30:193
2410999129,2cyclictest8841-21grep11:40:282
2410999129,2cyclictest7860-21grep08:50:002
2410999129,2cyclictest21752-21sed11:25:172
2410999129,2cyclictest16716-21sort08:00:172
2410999129,2cyclictest16321-21tr09:05:262
2410999129,2cyclictest16291-21wc11:20:272
2410999129,2cyclictest10862-21grep08:55:162
2410999128,3cyclictest32-21ksoftirqd/207:30:122
2410999128,3cyclictest29526-21ssh11:57:042
2410999128,3cyclictest14287-21wc09:00:252
2410999128,3cyclictest10581-21rm10:23:232
2410999127,2cyclictest2282-21sh09:24:342
2410999126,5cyclictest27202-21apache209:39:302
24109991211,1cyclictest0-21swapper/210:39:222
24109991211,1cyclictest0-21swapper/210:30:172
24109991210,1cyclictest1831-21wc10:15:262
24109991210,1cyclictest14294-21grep10:25:262
2410999120,3cyclictest0-21swapper/210:10:172
2410999120,11cyclictest28307-21tr11:05:162
2410999120,11cyclictest1903-21uname08:35:202
2410599129,2cyclictest8808-21cut09:30:221
2410599129,2cyclictest8119-21head09:55:171
2410599129,2cyclictest6913-21cron09:55:001
2410599129,2cyclictest4658-21sed07:35:181
2410599129,2cyclictest31087-21ssh10:40:161
2410599129,2cyclictest28166-21sed11:30:211
2410599129,2cyclictest25071-21grep08:15:281
2410599129,2cyclictest22897-21grep11:00:241
2410599129,2cyclictest17919-21tr09:10:131
2410599129,2cyclictest12929-21seq11:45:161
2410599128,3cyclictest820-21sshd11:09:211
2410599128,2cyclictest8714-21grep11:15:161
2410599127,4cyclictest14794-21taskset12:32:491
2410599127,3cyclictest16122-21wc12:10:261
2410599127,1cyclictest29217-21sh09:19:501
2410599125,4cyclictest27993-21apt-get08:25:021
2410099129,2cyclictest9827-21sed11:15:260
2410099129,2cyclictest8236-21sed07:40:300
2410099129,2cyclictest8177-21tail09:55:180
2410099129,2cyclictest32488-21sed08:30:270
2410099129,2cyclictest32488-21sed08:30:270
2410099129,2cyclictest19677-21grep10:30:200
2410099129,2cyclictest1517-21grep08:35:170
2410099129,2cyclictest1237-21ls07:30:000
2410099128,3cyclictest875-21ssh12:23:090
2410099128,2cyclictest2060-21ssh10:15:290
2410099125,6cyclictest26030-21ssh09:16:100
2410099123,6cyclictest0-21swapper/010:10:180
2410099122,3cyclictest6944-21perf09:55:000
24100991210,1cyclictest15907-21ssh11:20:230
24100991210,1cyclictest15833-21head09:05:210
24100991210,1cyclictest11944-21grep08:55:260
2410099120,11cyclictest6474-21grep10:20:150
115822120,0chrt0-21swapper/112:07:431
115822120,0chrt0-21swapper/112:07:431
2411499118,2cyclictest9164-21grep07:45:133
2411499118,2cyclictest903-21taskset07:25:563
2411499118,2cyclictest8670-21tail08:50:173
2411499118,2cyclictest7292-21grep11:40:153
2411499118,2cyclictest7062-21ssh10:46:053
2411499118,2cyclictest6034-21expr08:45:163
2411499118,2cyclictest5719-21turbostat.cron10:20:003
2411499118,2cyclictest326-21ssh11:35:043
2411499118,2cyclictest28038-21grep11:30:193
2411499118,2cyclictest27810-21ssh11:05:003
2411499118,2cyclictest26141-21ssh10:10:043
2411499118,2cyclictest20957-21ssh11:25:133
2411499118,2cyclictest19803-21date10:05:003
2411499118,2cyclictest19672-21ssh09:10:343
2411499118,2cyclictest1544-21grep07:30:123
2411499118,2cyclictest14985-21ssh10:52:473
2411499118,2cyclictest14931-21date09:05:013
2411499118,2cyclictest14782-21ssh09:35:133
2411499118,2cyclictest13259-21grep11:45:183
2411499118,2cyclictest12439-21chrt10:25:113
2411499118,2cyclictest1017-21ssh09:50:033
2411499117,1cyclictest16012-21ssh12:33:503
2411499110,10cyclictest769-21ssh10:40:453
2410999119,1cyclictest884-21expr11:35:152
2410999119,1cyclictest6180-21cut10:45:232
2410999119,1cyclictest22528-21taskset08:10:242
2410999119,1cyclictest18087-21cat10:55:192
2410999119,1cyclictest175352sleep209:10:092
2410999118,2cyclictest8227-21tr08:50:152
2410999118,2cyclictest3172-21grep12:25:142
2410999118,2cyclictest24585-21ssh12:16:342
2410999118,2cyclictest11342-21cut07:50:172
2410999117,3cyclictest32-21ksoftirqd/210:00:112
2410999115,5cyclictest31123-21ssh11:33:162
2410999112,3cyclictest311rcuc/210:05:282
2410999110,9cyclictest0-21swapper/211:02:342
2410999110,11cyclictest0-21swapper/210:51:152
2410999110,10cyclictest4227-21expr07:35:152
2410599118,2cyclictest8465-21tr11:40:231
2410599118,2cyclictest8084-21egrep08:50:161
2410599118,2cyclictest746-21grep10:15:171
2410599118,2cyclictest31801-21tr07:25:121
2410599118,2cyclictest3141-21ssh12:25:131
2410599118,2cyclictest29257-21ssh09:45:361
2410599118,2cyclictest20963-21ssh09:40:111
2410599118,2cyclictest20832-21sed10:30:291
2410599118,2cyclictest20035-21taskset09:39:191
2410599118,2cyclictest19744-21sed12:35:251
2410599118,2cyclictest1878-21needreboot08:35:201
2410599118,2cyclictest17318-21grep10:55:151
2410599118,2cyclictest15771-21cron08:00:001
2410599118,2cyclictest15751-21uniq09:05:201
2410599118,2cyclictest14826-21ssh11:20:161
2410599118,2cyclictest11417-21expr10:50:171
2410599118,2cyclictest11024-21head08:55:181
2410599117,3cyclictest5307-21turbostat.cron08:45:001
2410599117,3cyclictest27482-21wc10:10:181
2410599117,3cyclictest20845-21cut10:05:191
2410599114,3cyclictest224-21systemd-journal07:19:591
2410099119,2cyclictest0-21swapper/010:54:340
2410099119,1cyclictest6242-21sort08:45:180
2410099119,1cyclictest27254-21basename08:20:240
2410099119,1cyclictest17842-21sed10:55:170
2410099119,1cyclictest166982sleep011:48:030
2410099119,1cyclictest15789-21munin-plugin-st08:00:000
2410099118,3cyclictest17348-21ssh12:11:240
2410099118,2cyclictest8907-21taskset12:05:240
2410099118,2cyclictest8907-21taskset12:05:230
2410099118,2cyclictest5716-21rm09:26:180
2410099118,2cyclictest32651-21ssh09:21:350
2410099118,2cyclictest28866-21head11:05:190
2410099118,2cyclictest27892-21cat11:30:180
2410099118,2cyclictest22875-21taskset08:11:460
2410099118,2cyclictest17691-21grep08:00:270
2410099118,2cyclictest17155-21ssh09:35:570
2410099118,2cyclictest12151-21ssh09:32:130
2410099117,3cyclictest30996-21ssh10:40:120
2410099116,4cyclictest24676-21ps08:15:230
2410099116,3cyclictest6501-21ssh11:13:160
2410099110,9cyclictest0-21swapper/011:35:470
2410099110,10cyclictest30450-21ssh10:39:500
2410099110,10cyclictest21171-21cat11:25:150
2411499108,1cyclictest6961-21expr07:40:183
2411499108,1cyclictest27849-21sh11:55:303
2411499108,1cyclictest26692-21chrt07:15:103
2411499108,1cyclictest180532sleep308:03:343
2411499108,1cyclictest1766-21ssh11:10:033
2411499108,1cyclictest13426-21sleep07:55:013
2411499107,2cyclictest27517-21tail08:20:283
2411499107,2cyclictest25071-21grep07:10:183
2411499106,4cyclictest39-21ksoftirqd/308:10:023
2410999108,1cyclictest26987-21expr07:15:152
2410999108,1cyclictest156722sleep207:58:492
2410999107,2cyclictest7238-21cut07:40:202
2410999107,2cyclictest3828-21grep08:40:172
2410999107,2cyclictest32513-21tail08:30:272
2410999107,2cyclictest32513-21tail08:30:272
2410999107,2cyclictest29308-21cut07:20:122
2410999107,2cyclictest26079-21tr08:20:162
2410999100,6cyclictest224-21systemd-journal11:10:202
2410999100,4cyclictest224-21systemd-journal07:12:202
2410599108,1cyclictest311292sleep107:23:051
2410599108,1cyclictest17365-21processes08:00:231
2410599108,1cyclictest146322sleep109:02:381
2410599107,2cyclictest7318-21grep07:40:201
2410599107,2cyclictest3730-21grep11:10:251
2410599107,2cyclictest29690-21uname12:20:211
2410599107,2cyclictest21714-21grep11:25:171
2410599107,2cyclictest21671-21tune2fs08:10:151
2410599107,2cyclictest18498-21basename08:05:111
2410599107,2cyclictest11244-21taskset07:50:011
2410599103,6cyclictest540-21latency_hist08:35:011
2410599103,6cyclictest540-21latency_hist08:35:001
2410599101,5cyclictest224-21systemd-journal07:12:201
2410099108,2cyclictest24315-21ssh11:00:360
2410099108,1cyclictest30415-21tr07:20:210
2410099108,1cyclictest2912-21sh12:25:030
2410099108,1cyclictest14156-21expr10:00:140
2410099108,1cyclictest12670-21chrt09:00:010
2410099108,1cyclictest12463-21apt-get10:25:130
2410099108,1cyclictest100122sleep012:30:100
2410099107,2cyclictest4576-21ssh10:45:080
2410099107,2cyclictest2725-21grep07:30:200
2410099107,2cyclictest26762-21tr07:15:140
2410099107,2cyclictest19827-21ssh10:05:030
2410099107,2cyclictest18901-21ls08:05:120
2410099107,2cyclictest1766-21cut12:00:210
2410099107,2cyclictest13004-21sed07:50:360
241099997,1cyclictest251732sleep208:15:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional