You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 00:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Dec 13, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1698929963,31sleep00-21swapper/007:06:300
1697729851,43sleep30-21swapper/307:06:203
1699129279,9sleep20-21swapper/207:06:322
1708729069,6sleep10-21swapper/107:07:481
1734499211,18cyclictest30691-21latency_hist11:00:011
1734499180,1cyclictest11928-21aten2_r7power_v08:05:141
17340991812,4cyclictest7584-21sh11:20:000
1735199170,1cyclictest141rcu_preempt08:50:223
1735199170,1cyclictest141rcu_preempt08:50:213
1734999172,9cyclictest2252-21sshd07:45:142
1734999172,5cyclictest0-21swapper/210:10:132
1734499170,16cyclictest0-21swapper/109:45:251
1734999160,15cyclictest0-21swapper/209:10:272
1734099163,8cyclictest13408-21sendmail12:40:010
1735199152,6cyclictest16563-21apt-get08:15:143
1735199151,11cyclictest7997-21apt-get09:05:123
1734099151,7cyclictest0-21swapper/011:00:010
1735199143,5cyclictest18664-21apt-get10:35:153
17349991411,2cyclictest16753-21sed09:20:252
17349991410,3cyclictest23792-21grep07:20:212
17349991410,2cyclictest14046-21grep10:25:152
1734999140,3cyclictest0-21swapper/212:10:132
1734499149,3cyclictest8976-21latency_hist10:15:001
1734499147,6cyclictest25-21ksoftirqd/108:00:001
1734499141,4cyclictest225-21systemd-journal07:45:361
1735199132,3cyclictest6711-21apt-get10:10:133
1735199130,11cyclictest0-21swapper/311:15:133
1734999138,4cyclictest32-21ksoftirqd/207:50:252
1734999138,3cyclictest5480-21cat09:00:002
17349991310,2cyclictest29378-21wc09:49:592
17349991310,2cyclictest14322-21grep08:10:152
17349991310,2cyclictest13418-21awk12:40:002
1734499139,3cyclictest25342-21grep08:30:261
1734499138,4cyclictest18651-21ntpq09:25:211
1734499135,4cyclictest0-21swapper/108:15:141
17344991310,2cyclictest26864-21sed08:35:181
17344991310,2cyclictest26705-21cron11:59:591
17344991310,2cyclictest17186-21cron11:40:001
17344991310,2cyclictest13505-21cat12:40:001
1734099139,3cyclictest8705-21sed07:55:230
17340991310,2cyclictest3731-21sort11:10:170
17340991310,2cyclictest25866-21wc11:55:220
17340991310,2cyclictest24196-21df08:30:160
1734099130,12cyclictest0-21swapper/009:10:260
1735199129,2cyclictest9901-21sed10:15:183
1735199129,2cyclictest9574-21grep08:00:133
1735199129,2cyclictest928-21sed12:10:253
1735199129,2cyclictest9255-21wc11:20:233
1735199129,2cyclictest7575-21grep12:25:203
1735199129,2cyclictest618-21wc09:55:223
1735199129,2cyclictest5870-21wc07:50:223
1735199129,2cyclictest21659-21sed07:15:233
1735199129,2cyclictest18643-21tr09:25:213
1735199129,2cyclictest17420-21tr10:30:223
1735199129,2cyclictest13810-21ntpq11:30:213
1735199129,2cyclictest13469-21tail09:15:163
1735199128,3cyclictest13811-21chrt08:08:463
1735199127,1cyclictest39-21ksoftirqd/310:55:023
1734999129,2cyclictest8716-21sed11:20:182
1734999129,2cyclictest29176-21cat12:05:012
1734999129,2cyclictest13462-21tail08:05:252
1734999128,3cyclictest7663-21cron09:05:012
1734999128,3cyclictest5699-21grep12:20:252
1734999128,2cyclictest17555-21grep09:25:152
1734999124,3cyclictest225-21systemd-journal11:00:112
1734999121,2cyclictest141rcu_preempt11:50:202
1734999120,7cyclictest0-21swapper/210:35:152
1734999120,6cyclictest0-21swapper/208:15:142
1734499129,2cyclictest5708-21basename10:05:241
1734499129,2cyclictest31477-21sh12:10:001
1734499129,2cyclictest2708-21grep10:00:241
1734499129,2cyclictest23872-21sed11:50:271
1734499129,2cyclictest20624-21sed08:20:241
1734499129,2cyclictest20368-21tr09:30:141
1734499129,2cyclictest1970-21tr12:15:141
1734499128,3cyclictest38150irq/131-snd_hda07:25:251
1734499128,1cyclictest22821-21taskset10:40:491
1734499121,3cyclictest225-21systemd-journal10:35:001
17344991210,2cyclictest38150irq/131-snd_hda08:55:211
17344991210,1cyclictest20596-21head11:45:171
1734499120,1cyclictest0-21swapper/108:10:141
1734499120,11cyclictest14180-21tr10:25:161
1734099129,2cyclictest8246-21grep12:25:270
1734099129,2cyclictest4858-21tr07:50:150
1734099129,2cyclictest4572-21grep10:05:150
1734099129,2cyclictest31093-21sed07:35:220
1734099129,2cyclictest3058-21grep10:00:260
1734099129,2cyclictest29701-21sort07:35:000
1734099129,2cyclictest29341-21cron09:49:590
1734099129,2cyclictest26968-21cron09:45:000
1734099129,2cyclictest22317-21cat09:35:000
1734099129,2cyclictest22090-21sed08:25:190
1734099129,2cyclictest15823-21head11:35:170
1734099129,2cyclictest1441-21grep12:15:000
1734099129,2cyclictest10537-21grep08:00:190
1734099128,3cyclictest15687-21df09:20:150
1734099128,3cyclictest10797-21grep10:15:260
1735199119,1cyclictest26851-21cut08:35:193
1735199119,1cyclictest11137-21cut09:10:313
1735199118,2cyclictest7267-21cut07:55:133
1735199118,2cyclictest6253-21grep09:00:173
1735199118,2cyclictest4723-21sort12:20:163
1735199118,2cyclictest3878-21head11:10:163
1735199118,2cyclictest381rcuc/307:35:273
1735199118,2cyclictest3383-21wc10:00:323
1735199118,2cyclictest3382-21cat08:55:153
1735199118,2cyclictest29666-21cut12:05:153
1735199118,2cyclictest27895-21cron10:55:013
1735199118,2cyclictest27551-21cut07:30:153
1735199118,2cyclictest26999-21apache_processe12:00:133
1735199118,2cyclictest26765-21sed07:25:263
1735199118,2cyclictest25573-21sort11:55:193
1735199118,2cyclictest243132sleep311:54:503
1735199118,2cyclictest23499-21sed07:20:183
1735199118,2cyclictest20427-21taskset11:45:163
1735199118,2cyclictest17586-21cut07:10:093
1735199118,2cyclictest16694-21sed11:35:273
1735199118,2cyclictest15889-21sed09:20:163
1735199118,2cyclictest1450-21expr11:05:173
1735199118,2cyclictest12836-21cut10:20:233
1735199118,2cyclictest11336-21cut12:35:143
1735199118,2cyclictest10520-21grep12:30:263
1735199117,3cyclictest39-21ksoftirqd/309:40:383
1735199117,3cyclictest2184-21sh12:15:153
1735199111,1cyclictest141rcu_preempt10:40:203
1735199110,4cyclictest0-21swapper/308:25:143
1734999119,1cyclictest25859-21grep07:25:182
1734999118,2cyclictest9897-21head10:15:172
1734999118,2cyclictest7582-21cut12:25:212
1734999118,2cyclictest6982-21cat11:15:252
1734999118,2cyclictest6457-21awk10:10:002
1734999118,2cyclictest3334-21grep12:15:262
1734999118,2cyclictest32006-21apt07:39:592
1734999118,2cyclictest31430-21taskset09:51:242
1734999118,2cyclictest30595-21head12:05:232
1734999118,2cyclictest30484-21idleruntime-cro11:00:012
1734999118,2cyclictest27265-21taskset10:50:262
1734999118,2cyclictest22112-21taskset09:33:532
1734999118,2cyclictest21394-21cat11:45:252
1734999118,2cyclictest20314-21sendmail_mailtr08:20:212
1734999118,2cyclictest20191-21tr07:15:122
1734999118,2cyclictest14483-21awk09:15:262
1734999118,2cyclictest11409-21head11:25:202
1734999117,3cyclictest494-21cron08:50:002
1734999117,3cyclictest32554-21grep07:40:142
1734999117,3cyclictest30471-21chrt08:43:472
1734999117,3cyclictest22738-21tr09:35:162
1734999117,2cyclictest9036-21grep12:30:152
1734999113,4cyclictest141rcu_preempt11:10:262
1734999112,2cyclictest225-21systemd-journal11:43:122
1734999110,7cyclictest0-21swapper/208:30:142
1734999110,1cyclictest141rcu_preempt10:40:242
1734499119,2cyclictest20422-21expr07:15:121
1734499119,1cyclictest97482sleep111:21:261
1734499119,1cyclictest512-21cut07:40:161
1734499119,1cyclictest31498-21grep11:00:161
1734499119,1cyclictest26524-21head10:50:181
1734499119,1cyclictest18339-21grep07:10:121
1734499119,1cyclictest13066-21ls11:30:151
1734499119,1cyclictest12314-21head10:20:181
1734499119,1cyclictest12105-21sed09:10:391
1734499118,3cyclictest25-21ksoftirqd/107:30:151
1734499118,2cyclictest8773-21tail09:05:171
1734499118,2cyclictest870-21cut08:50:151
1734499118,2cyclictest870-21cut08:50:141
1734499118,2cyclictest5845-21grep07:50:221
1734499118,2cyclictest28708-21grep12:00:261
1734499118,2cyclictest27178-21cat09:45:001
1734499118,2cyclictest23626-21cat10:45:151
1734499118,2cyclictest23626-21cat10:45:141
1734499118,2cyclictest17678-21egrep11:40:131
1734499118,2cyclictest15949-21cat09:20:171
1734499118,2cyclictest11031-21tail11:25:181
1734499115,2cyclictest225-21systemd-journal12:35:011
1734499115,2cyclictest225-21systemd-journal08:45:011
1734499113,6cyclictest0-21swapper/108:25:151
17344991110,1cyclictest0-21swapper/111:05:131
1734099118,2cyclictest5148-21missed_timers12:20:200
1734099118,2cyclictest382-21taskset11:04:410
1734099118,2cyclictest32758-21grep09:55:200
1734099118,2cyclictest31650-21sshd09:54:140
1734099118,2cyclictest29157-21cut08:40:280
1734099118,2cyclictest24966-21apt-config07:25:000
1734099118,2cyclictest24174-21head10:45:180
1734099118,2cyclictest24174-21head10:45:170
1734099118,2cyclictest23243-21cut09:35:180
1734099118,2cyclictest2313-21wget07:45:150
1734099118,2cyclictest23112-21grep07:20:160
1734099118,2cyclictest23019-21cut11:50:180
1734099118,2cyclictest2281-21dnf12:15:150
1734099118,2cyclictest21623-21taskset11:45:430
1734099118,2cyclictest20203-21tr07:15:110
1734099118,2cyclictest19696-21grep07:15:000
1734099118,2cyclictest19431-21cut10:35:190
1734099118,2cyclictest19274-21taskset11:41:010
1734099118,2cyclictest17746-21tr09:25:160
1734099118,2cyclictest14783-21taskset09:19:160
1734099118,2cyclictest1407-21grep11:05:160
1734099118,2cyclictest13987-21wc08:10:000
1734099117,3cyclictest16015-21turbostat10:30:000
1734099111,5cyclictest225-21systemd-journal11:25:150
1735199109,1cyclictest22695-21wc09:35:173
1735199108,1cyclictest40732sleep307:46:353
1735199108,1cyclictest3902sleep311:04:483
1735199108,1cyclictest18131-21grep11:40:163
1735199107,2cyclictest5417-21sed10:05:213
1735199107,2cyclictest32690-21expr07:40:163
1735199107,2cyclictest31531-21grep08:45:183
1735199107,2cyclictest25183-21taskset10:45:423
1735199107,2cyclictest25183-21taskset10:45:413
1735199107,2cyclictest23775-21wget08:30:143
1735199107,2cyclictest20412-21sed08:20:253
1735199107,2cyclictest20136-21wget09:30:133
1735199107,2cyclictest15214-21grep08:10:203
1735199107,2cyclictest10447-21grep11:25:153
1735199103,3cyclictest27247-21apt-get09:45:143
1734999108,1cyclictest8938-21chrt07:57:252
1734999108,1cyclictest8719-21grep09:05:172
1734999108,1cyclictest710-21basename11:05:122
1734999108,1cyclictest32206-21wc09:55:172
1734999108,1cyclictest267452sleep209:42:102
1734999108,1cyclictest2656-21chrt08:51:582
1734999108,1cyclictest2656-21chrt08:51:572
1734999108,1cyclictest25688-21sed11:55:202
1734999108,1cyclictest254722sleep210:49:262
1734999108,1cyclictest254722sleep210:49:252
1734999108,1cyclictest219122sleep208:25:172
1734999108,1cyclictest19162sleep210:00:002
1734999108,1cyclictest17571-21apache_volume07:10:082
1734999108,1cyclictest16562-21cut10:30:172
1734999108,1cyclictest16130-21sed11:35:212
1734999108,1cyclictest147302sleep211:34:282
1734999107,2cyclictest12024-21df10:20:172
1734999107,2cyclictest10185-21grep08:00:162
1734999103,3cyclictest141rcu_preempt07:30:162
1734999100,9cyclictest26077-21apache_processe08:35:142
1734499109,1cyclictest6414-21cat09:00:181
1734499109,1cyclictest31078-21expr08:45:151
1734499109,1cyclictest23236-21cut09:35:181
1734499109,1cyclictest20712-21seq10:39:301
1734499109,1cyclictest110872sleep110:18:061
1734499108,1cyclictest9455-21cut08:00:141
1734499108,1cyclictest6796-21cat12:25:161
1734499108,1cyclictest59892sleep112:22:241
1734499108,1cyclictest31949-21taskset09:55:081
1734499108,1cyclictest22925-21cat07:20:151
1734499106,4cyclictest1177-21taskset12:11:531
1734499105,2cyclictest225-21systemd-journal09:20:011
1734099108,1cyclictest31155-21cstates08:45:160
1734099108,1cyclictest27610-21chrt10:51:230
1734099108,1cyclictest26468-21ls08:35:160
1734099108,1cyclictest16686-21expr10:30:160
1734099108,1cyclictest147842sleep008:10:170
1734099108,1cyclictest1123-21processes07:40:220
1734099107,2cyclictest9531-21ls11:20:260
1734099107,2cyclictest8584-21forks09:05:160
1734099107,2cyclictest7972-21cut10:10:290
1734099107,2cyclictest6482-21irqstats09:00:190
1734099107,2cyclictest4777-21grep08:55:260
1734099107,2cyclictest29365-21basename12:05:130
1734099107,2cyclictest28454-21cat12:00:250
1734099107,2cyclictest19028-21wget08:20:130
1734099107,2cyclictest1551-21head08:50:180
1734099107,2cyclictest1551-21head08:50:180
1734099107,2cyclictest12714-21tr11:30:140
1734099107,2cyclictest11568-21wget10:20:150
1734099107,2cyclictest10295-21python12:30:250
1734099100,0cyclictest0-21swapper/008:15:130
173519997,1cyclictest28774-21tr08:40:253
173519996,2cyclictest13906-21chrt10:25:143
173519992,3cyclictest225-21systemd-journal09:50:013
173449998,1cyclictest31010-21grep07:35:211
173449998,1cyclictest29826-21ls09:50:141
173449997,1cyclictest3082-21acpi11:10:121
173409998,1cyclictest22184-21ntpq10:40:230
173449987,1cyclictest6537-21ls11:15:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional