You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-28 - 11:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Oct 28, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18985210275,9sleep30-21swapper/319:05:553
1921429582,9sleep10-21swapper/119:08:501
1921328370,9sleep00-21swapper/019:08:490
1900428164,13sleep20-21swapper/219:06:102
44362590,0sleep10-21swapper/100:16:561
236972520,0sleep10-21swapper/122:05:551
1939399291,26cyclictest37850irq/131-snd_hda00:01:252
19393992625,1cyclictest0-21swapper/221:14:272
242212240,0sleep30-21swapper/321:18:343
19384992420,3cyclictest23368-21chrt23:19:030
19393992321,1cyclictest8550-21ntp_states21:30:242
19389992320,2cyclictest14341-21apache_volume00:00:141
19389992319,3cyclictest9211-21vmstat21:30:301
19384992320,2cyclictest8012-21apache_processe21:55:150
19384992319,3cyclictest21240-21grep23:40:260
19395992220,1cyclictest24920-21apache_volume23:20:173
19395992218,3cyclictest16013-21wc22:00:243
19389992219,2cyclictest2834-21df_inode22:15:181
19389992219,2cyclictest27225-21memory21:20:211
1938999220,22cyclictest0-21swapper/123:58:391
19384992220,1cyclictest10698-21diskstats22:45:190
19384992219,2cyclictest2844-21diskstats22:15:180
1938499220,9cyclictest0-21swapper/020:00:160
19395992118,2cyclictest1572-21irqstats23:50:193
19395992118,2cyclictest11835-21df23:10:153
19395992117,2cyclictest22241-21apache_processe22:30:113
19393992119,1cyclictest522-21apache_accesses23:50:122
1939399210,2cyclictest0-21swapper/200:21:572
1939399210,1cyclictest0-21swapper/223:19:092
19389992119,1cyclictest27053-21apache_volume21:45:141
19389992118,2cyclictest28849-21chrt22:35:011
19384992119,1cyclictest27462-21ntp_states23:45:230
19384992118,2cyclictest2307-21/usr/sbin/munin21:50:210
19384992118,1cyclictest0-21swapper/021:24:310
19384992118,1cyclictest0-21swapper/021:04:140
19395992016,3cyclictest32642-21chrt22:37:203
1939399200,19cyclictest0-21swapper/223:49:132
19389992016,3cyclictest6582-21chrt21:29:491
1938999200,19cyclictest0-21swapper/122:40:141
19384992018,1cyclictest4074-21diskstats22:40:180
1938499200,0cyclictest0-21swapper/023:54:070
19395991918,1cyclictest0-21swapper/321:39:343
1939599191,3cyclictest0-21swapper/322:49:503
19395991911,5cyclictest22178-21hddtemp_smartct22:05:193
1939599190,18cyclictest0-21swapper/323:55:143
19393991917,1cyclictest3497-21ntp_states22:15:222
19393991916,2cyclictest59722sleep223:05:202
19389991916,2cyclictest19370-21apache_accesses23:40:121
19395991815,2cyclictest15584-21apache_processe22:25:153
19389991815,2cyclictest160422sleep100:26:291
19384991815,1cyclictest0-21swapper/020:58:130
19384991814,3cyclictest8361-21chrt23:31:020
19395991715,1cyclictest20012-21diskstats23:40:173
1939399173,13cyclictest7464-21latency_hist21:00:012
19393991713,3cyclictest20211-21sed23:40:182
19389991714,2cyclictest19297-21apache_accesses21:15:121
19389991713,3cyclictest4572-21cat23:05:001
1938999170,2cyclictest0-21swapper/122:24:411
19384991715,1cyclictest29197-21ntp_states22:10:240
75112160,0chrt0-21swapper/123:30:271
19395991611,3cyclictest29013-21wc00:10:203
19393991613,2cyclictest13921-21aten2_r7power_p20:05:152
19393991612,3cyclictest31533-21chrt21:48:182
1939399160,14cyclictest0-21swapper/200:19:492
1938999163,9cyclictest21094-21apt-get00:05:111
19389991613,2cyclictest21504-21/usr/sbin/munin21:40:201
1938499164,6cyclictest15952-21munin-run20:10:000
1938499162,13cyclictest0-21swapper/023:05:140
19395991514,1cyclictest0-21swapper/323:48:573
19395991512,2cyclictest299352sleep300:38:533
19395991512,2cyclictest14166-21hddtemp_smartct00:25:213
1939599151,1cyclictest0-21swapper/323:17:093
1939399157,4cyclictest493-21systemd-logind00:06:252
1939399154,7cyclictest224-21systemd-journal00:15:002
19393991510,3cyclictest10610-21wc22:20:292
1938999155,7cyclictest0-21swapper/122:30:381
1938999150,1cyclictest141rcu_preempt21:10:161
1938999150,1cyclictest0-21swapper/122:46:271
1938499151,14cyclictest0-21swapper/021:37:230
1939599147,4cyclictest16792sleep320:45:213
1939599146,7cyclictest9781-21cron21:05:013
1939599145,6cyclictest12165-21diskmemload23:05:173
19395991412,1cyclictest27910-21apt-get22:10:163
19395991411,2cyclictest87782sleep322:44:093
19395991411,2cyclictest180062sleep300:02:193
19395991410,3cyclictest22152-21ssh00:31:193
1939599140,13cyclictest13559-21cron20:04:593
1939399149,4cyclictest0-21swapper/221:20:142
1939399142,10cyclictest0-21swapper/222:40:142
19393991412,1cyclictest10041-21apache_accesses22:45:142
19393991411,2cyclictest1199-21cron19:39:592
1939399140,14cyclictest0-21swapper/222:54:512
1939399140,13cyclictest0-21swapper/223:25:012
1939399140,12cyclictest0-21swapper/223:20:562
19389991411,2cyclictest28804-21head00:10:181
19389991410,3cyclictest29284-21grep20:35:221
19389991410,3cyclictest28095-21grep23:45:291
1938999140,12cyclictest18889-21head23:15:221
1938499149,4cyclictest22817-21sendmail_mailtr22:05:230
19384991411,2cyclictest20478-21apache_volume21:40:130
19384991410,3cyclictest26976-21chrt21:45:110
19384991410,2cyclictest1198-21cron19:40:000
1939599133,7cyclictest28317-21ssh21:45:233
1939599131,1cyclictest141rcu_preempt22:23:303
19395991310,2cyclictest977-21tail23:25:273
19395991310,2cyclictest5852-21ssh23:30:153
19395991310,2cyclictest26534-21grep20:30:173
19395991310,2cyclictest23111-21grep19:15:223
19395991310,2cyclictest14159-21cron22:00:003
19393991311,1cyclictest28122-21/usr/sbin/munin22:10:172
19393991310,2cyclictest5957-21basename23:30:152
19393991310,2cyclictest26451-21chrt22:32:532
19393991310,2cyclictest25722-21cat20:30:002
1939399130,12cyclictest2423-21wc21:50:212
1939399130,12cyclictest21429-21grep22:05:152
1938999139,3cyclictest27529-21tr19:25:191
1938999139,3cyclictest27487-21sed20:30:261
1938999139,3cyclictest25111-21grep19:20:191
1938999139,3cyclictest17139-21tr22:50:171
1938999139,2cyclictest4164-21tr19:45:151
1938999139,2cyclictest21827-21wc22:30:001
1938999138,4cyclictest12722-21aten2_r7power_v23:35:181
1938999137,1cyclictest25-21ksoftirqd/123:21:011
19389991310,2cyclictest6707-21grep23:05:271
19389991310,2cyclictest30775-21grep19:30:281
19389991310,2cyclictest20406-21ssh00:30:201
1938499139,3cyclictest8100-21wc19:50:260
1938499139,3cyclictest2802-21chrt23:27:180
1938499139,2cyclictest3843-21basename19:45:000
1938499139,2cyclictest24060-21grep22:30:260
1938499138,3cyclictest30450-21wc22:35:230
1938499138,3cyclictest28909-21awk19:30:000
1938499136,6cyclictest30220-21munin-run23:00:010
1938499131,7cyclictest18344-21perf20:15:000
1938499131,11cyclictest761-21grep21:25:150
19384991310,2cyclictest24131-21ls19:20:010
19384991310,2cyclictest2181-21grep20:45:250
19384991310,2cyclictest20754-21cron20:19:590
19384991310,2cyclictest11747-21grep21:05:270
1939599129,2cyclictest24060-31su00:06:253
1939599129,2cyclictest2375-21cut22:15:153
1939599129,2cyclictest2359-21tune2fs19:40:173
1939599129,2cyclictest19886-21grep19:10:143
1939599129,2cyclictest17499-21cut22:50:193
1939599129,2cyclictest16485-21tr20:10:153
1939599129,2cyclictest13395-21perf21:35:003
1939599129,2cyclictest13154-21sed23:35:193
1939599129,2cyclictest1292-21tail21:25:183
1939599128,3cyclictest4717-21sort19:45:173
1939599128,3cyclictest12878-21ls00:25:003
1939599120,11cyclictest31337-21cut20:40:203
1939399129,2cyclictest31775-21sed20:40:242
1939399129,2cyclictest30392-21grep19:30:232
1939399129,2cyclictest28488-21grep19:25:302
1939399129,2cyclictest25727-21wc19:20:242
1939399129,2cyclictest20137-21grep21:15:182
1939399129,2cyclictest13854-21basename21:35:152
1939399129,2cyclictest10601-21wc19:55:272
1939399128,3cyclictest7451-21cut19:50:192
1939399128,3cyclictest3484-21taskset19:40:462
1939399124,5cyclictest224-21systemd-journal00:40:012
19393991210,1cyclictest20597-21/usr/sbin/munin21:40:162
19393991210,1cyclictest15899-21ssh22:00:232
1939399120,1cyclictest0-21swapper/220:45:132
1939399120,11cyclictest21751-21taskset22:29:532
1939399120,0cyclictest0-21swapper/200:34:522
1938999129,2cyclictest3222-21sed21:50:291
1938999129,2cyclictest31850-21grep23:25:151
1938999129,2cyclictest2429-21grep19:40:181
1938999129,2cyclictest22994-21cat19:15:221
1938999129,2cyclictest17688-21cat20:10:261
1938999128,4cyclictest3522-21kworker/u8:023:52:581
1938999128,3cyclictest7838-21tr00:20:151
1938999126,2cyclictest5024-21systemd-journal20:55:001
1938999126,2cyclictest5024-21systemd-journal20:55:001
1938999126,2cyclictest224-21systemd-journal20:00:001
19389991210,1cyclictest18935-21grep20:15:161
1938999120,11cyclictest9830-21cat21:55:301
1938999120,11cyclictest0-21swapper/120:00:161
1938499129,2cyclictest9385-21grep23:55:260
1938499129,2cyclictest2659-21/usr/sbin/munin00:15:220
1938499129,2cyclictest15238-21sh22:00:170
1938499129,2cyclictest10713-21grep19:55:280
1938499128,3cyclictest18751-21sed22:50:290
1938499127,3cyclictest23206-21ssh00:05:380
19384991211,1cyclictest0-21swapper/019:12:030
1939599118,2cyclictest6796-21cut19:50:163
1939599118,2cyclictest5004-21cron20:55:003
1939599118,2cyclictest5004-21cron20:54:593
1939599118,2cyclictest3273-21grep21:50:293
1939599118,2cyclictest31612-21cut19:35:143
1939599118,2cyclictest31002-21chrt19:31:473
1939599118,2cyclictest30530-21ssh23:00:053
1939599118,2cyclictest28581-21ssh21:21:013
1939599118,2cyclictest28133-21cut19:25:263
1939599118,2cyclictest25935-21ssh22:55:413
1939599118,2cyclictest25038-21grep20:25:263
1939599118,2cyclictest20514-21taskset20:17:003
1939599118,2cyclictest1704-21tr00:15:163
1939599118,2cyclictest15529-21ssh21:11:253
1939599118,2cyclictest12356-21cat21:10:013
1939599116,2cyclictest21103-21df_inode21:40:173
1939399119,1cyclictest12202-21grep20:00:182
1939399118,2cyclictest7539-21tr23:55:142
1939399118,2cyclictest3659-21grep20:50:182
1939399118,2cyclictest3659-21grep20:50:172
1939399118,2cyclictest32727-21ssh21:25:012
1939399118,2cyclictest30741-21cat22:35:272
1939399118,2cyclictest28901-21expr20:35:202
1939399118,2cyclictest14770-21find00:25:272
1939399118,2cyclictest13387-21grep23:35:212
1939399118,2cyclictest13073-21ssh21:58:542
1939399118,2cyclictest10287-21tr21:05:152
1939399117,3cyclictest5731-21tail19:45:272
1939399112,4cyclictest24119-21cron19:20:012
1939399111,6cyclictest0-21swapper/223:00:232
1939399110,10cyclictest12854-21rm23:10:262
1938999118,2cyclictest7818-21ntpq19:50:211
1938999118,2cyclictest328-21uname19:35:221
1938999118,2cyclictest30874-21wc00:40:001
1938999118,2cyclictest29277-21grep22:10:251
1938999118,2cyclictest23841-21expr20:25:161
1938999118,2cyclictest22928-21chrt20:22:101
1938999118,2cyclictest15318-21fschecks_count22:00:181
1938999118,2cyclictest14061-21tr21:35:161
1938999118,2cyclictest12934-21ssh23:10:281
1938999118,2cyclictest11149-21sed21:05:201
1938999117,3cyclictest2823-21cat20:50:011
1938999117,3cyclictest24398-21uniq22:55:181
1938999113,2cyclictest224-21systemd-journal20:40:351
1938999111,8cyclictest9801-21idleruntime-cro21:05:001
1938999110,10cyclictest5564-21expr20:55:141
1938499118,2cyclictest9931-21ssh22:20:200
1938499118,2cyclictest8119-21tail21:30:200
1938499118,2cyclictest6339-21sed19:50:000
1938499118,2cyclictest31205-21sort20:40:200
1938499118,2cyclictest30521-21chrt00:11:090
1938499118,2cyclictest29154-21apache_accesses19:30:130
1938499118,2cyclictest28792-21ssh23:22:340
1938499118,2cyclictest27121-21rm00:35:280
1938499118,2cyclictest22830-21ssh21:17:070
1938499118,2cyclictest21834-21ssh00:30:580
1938499118,2cyclictest21276-21cut20:20:160
1938499118,2cyclictest18599-21ssh22:26:400
1938499118,2cyclictest16508-21ssh23:37:370
1938499118,2cyclictest14806-21cut00:25:270
1938499118,2cyclictest14584-21cat00:00:160
1938499118,2cyclictest13853-21grep21:10:210
1938499118,2cyclictest13291-21ssh23:10:340
1938499118,2cyclictest11422-21ssh00:22:450
1939599108,1cyclictest9021-21taskset19:55:083
1939599108,1cyclictest7172-21taskset20:57:253
1939599108,1cyclictest14551-21expr20:05:183
1939599107,2cyclictest24442-21tr19:20:163
1939599107,2cyclictest21988-21uname20:20:213
1939399108,1cyclictest8344-21tune2fs21:00:182
1939399108,1cyclictest26007-21cat20:30:142
1939399108,1cyclictest18660-21cut20:15:152
1939399108,1cyclictest16509-21cat20:10:152
1939399107,2cyclictest23650-21ssh22:55:092
1939399107,2cyclictest22952-21taskset20:22:282
1939399102,4cyclictest224-21systemd-journal19:10:182
1938999108,1cyclictest21677-21chrt19:14:261
1938499108,1cyclictest30817-21aten2_r7power_p23:00:160
1938499107,2cyclictest26368-21taskset19:23:160
1938499107,2cyclictest26028-21tr20:30:140
1938499106,2cyclictest3127-21tr20:50:160
1938499106,2cyclictest3127-21tr20:50:150
1938499101,8cyclictest0-21swapper/020:25:150
193959997,1cyclictest28266-21cut20:35:173
193899997,1cyclictest14261-21expr20:05:161
193849985,2cyclictest28858-21grep20:35:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional