You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-13 - 21:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Oct 13, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31695210679,9sleep30-21swapper/307:06:533
31853210490,9sleep10-21swapper/107:08:571
3163528963,8sleep00-21swapper/007:06:070
3179728664,6sleep20-21swapper/207:08:142
32022992423,1cyclictest94-21kswapd009:50:022
32016992420,3cyclictest18053-21needreboot10:50:221
3201699240,1cyclictest0-21swapper/111:50:491
3201699231,2cyclictest0-21swapper/109:14:441
32012992321,2cyclictest13-21ksoftirqd/011:48:230
32026992219,2cyclictest132842chrt09:26:583
32016992217,3cyclictest22404-21taskset10:26:011
32026992117,3cyclictest16806-21taskset11:44:213
32016992118,2cyclictest20945-21memory12:35:251
3202299200,20cyclictest0-21swapper/212:14:442
3202299200,1cyclictest0-21swapper/211:25:172
32016992017,2cyclictest145142chrt09:28:281
3201699200,1cyclictest0-21swapper/111:08:121
32026991919,0cyclictest0-21swapper/310:49:523
32026991916,2cyclictest15133-21fschecks_time09:55:183
3201299185,7cyclictest15337-21awk08:50:010
3201299180,1cyclictest0-21swapper/009:46:530
32026991715,1cyclictest0-21swapper/309:31:593
32022991716,1cyclictest0-21swapper/210:13:292
32022991715,2cyclictest37850irq/131-snd_hda08:40:012
32016991714,2cyclictest82112chrt12:25:591
3202699160,14cyclictest9328-21grep09:50:213
32022991612,3cyclictest0-21swapper/209:15:162
3202299160,1cyclictest0-21swapper/211:02:282
32016991612,3cyclictest1081-21taskset11:04:481
3201299169,2cyclictest0-21swapper/012:26:190
32026991514,1cyclictest0-21swapper/310:31:593
32026991511,3cyclictest19790-21grep08:55:283
32026991511,3cyclictest15365-21cat10:20:253
3202699150,1cyclictest0-21swapper/308:35:163
3202299152,12cyclictest0-21swapper/208:05:132
32022991513,2cyclictest37850irq/131-snd_hda11:30:142
3201699153,5cyclictest224-21systemd-journal08:34:551
3201299150,1cyclictest0-21swapper/011:55:260
3202699147,5cyclictest26259-21apt-get08:05:133
3202699144,8cyclictest20104-21ssh11:20:143
32026991410,3cyclictest31760-21tune2fs08:15:203
32026991410,3cyclictest18637-21sed10:50:303
32026991410,2cyclictest1146-21grep08:20:183
3202699140,13cyclictest0-21swapper/312:14:423
3202299149,4cyclictest28283-21cat10:05:222
32022991414,0cyclictest0-21swapper/211:55:142
3202299140,14cyclictest0-21swapper/210:20:362
3201699147,3cyclictest224-21systemd-journal08:44:551
32016991411,2cyclictest233662chrt10:01:291
32016991410,3cyclictest25724-21apt11:25:001
3201699140,0cyclictest0-21swapper/112:01:061
3201299142,9cyclictest28884-21apache_accesses09:40:130
3202699139,3cyclictest22989-21wc07:55:263
3202699139,2cyclictest0-21swapper/311:35:233
3202699137,5cyclictest5528-21tr10:40:173
32026991310,2cyclictest29443-21grep11:00:263
32026991310,2cyclictest26714-21head11:50:293
32026991310,2cyclictest25568-21grep08:00:273
32026991310,2cyclictest2409-21grep11:05:173
32026991310,2cyclictest20088-21tr12:35:213
32026991310,2cyclictest13630-21cut08:45:213
3202699130,12cyclictest18339-21grep11:45:153
3202299138,5cyclictest30694-21apache210:46:512
3202299138,4cyclictest25056-21diskmemload09:56:252
3202299136,1cyclictest493-21systemd-logind10:54:212
32022991312,1cyclictest6266-21kworker/2:211:16:532
32022991312,1cyclictest30697-21apache212:06:392
32022991312,1cyclictest0-21swapper/209:46:532
3202299131,12cyclictest1378-21ntpd10:25:232
32022991310,2cyclictest15869-21tr08:50:152
3202299130,13cyclictest0-21swapper/211:55:012
3201699139,3cyclictest21238-21taskset09:33:441
3201699139,3cyclictest20687-21taskset12:10:441
3201699139,3cyclictest18348-21grep07:45:251
3201699137,2cyclictest224-21systemd-journal09:10:001
3201699133,5cyclictest17636-21latency_hist11:45:011
32016991310,2cyclictest253382chrt09:36:261
32016991310,2cyclictest23308-21cat10:55:221
32016991310,2cyclictest18399-21grep11:45:151
32016991310,2cyclictest14181-21tr11:15:201
32016991310,2cyclictest13045-21grep12:05:171
3201699130,1cyclictest0-21swapper/112:17:151
3201699130,1cyclictest0-21swapper/112:17:141
3201299136,6cyclictest24665-21munin-run12:15:000
3201299136,6cyclictest22696-21awk09:05:000
32012991312,1cyclictest0-21swapper/012:39:400
32012991311,1cyclictest25056-21diskmemload10:04:530
32012991310,2cyclictest7712-21tail12:00:280
32012991310,2cyclictest25874-21wc11:25:000
32012991310,2cyclictest22910-21grep09:35:170
32012991310,2cyclictest18884-21head08:55:190
32012991310,2cyclictest13-21ksoftirqd/009:20:210
3202699129,2cyclictest7392-21sh12:00:243
3202699129,2cyclictest5579-21cat12:25:013
3202699129,2cyclictest4134-21head08:25:193
3202699129,2cyclictest30788-21cron08:14:593
3202699129,2cyclictest27531-21expr10:05:173
3202699129,2cyclictest2575-21tail10:10:273
3202699129,2cyclictest15575-21sed07:40:233
3202699128,3cyclictest31750-21grep11:55:113
3202699124,4cyclictest224-21systemd-journal11:00:003
3202699122,9cyclictest0-21swapper/310:15:153
32026991210,1cyclictest259202chrt10:04:293
3202699120,12cyclictest0-21swapper/309:36:003
3202299129,2cyclictest9678-21grep11:10:282
3202299129,2cyclictest7663-21grep12:25:262
3202299129,2cyclictest6039-21head10:40:182
3202299129,2cyclictest27741-21cat10:30:302
3202299129,2cyclictest27015-21basename12:40:002
3202299129,2cyclictest26930-21cat09:10:292
3202299129,2cyclictest25476-21tr12:15:232
3202299129,2cyclictest25476-21tr12:15:222
3202299129,2cyclictest24590-21head08:00:192
3202299129,2cyclictest2294-21grep11:05:162
3202299129,2cyclictest22434-21grep10:00:282
3202299129,2cyclictest15767-21basename09:30:012
3202299129,2cyclictest14620-21grep07:40:162
3202299124,5cyclictest20269-21cron09:00:002
3202299123,6cyclictest0-21swapper/212:34:242
3202299120,11cyclictest23435-21grep09:35:212
3201699129,2cyclictest8018-21cron09:50:001
3201699129,2cyclictest7855-21head10:15:181
3201699129,2cyclictest777-21grep10:10:151
3201699129,2cyclictest6367-21grep11:35:221
3201699129,2cyclictest3708-21grep08:25:151
3201699129,2cyclictest30219-21grep09:40:221
3201699129,2cyclictest25995-21munin-run08:05:001
3201699129,2cyclictest22223-21sed09:00:291
3201699129,2cyclictest13917-21grep10:20:141
3201699128,3cyclictest8442-21grep07:25:251
3201699128,3cyclictest31742-21grep08:15:201
3201299129,2cyclictest27357-21tr10:05:180
3201299124,4cyclictest25056-21diskmemload11:16:510
3201299123,8cyclictest23631-21perf08:00:000
3201299121,5cyclictest30694-21apache212:06:380
32012991210,1cyclictest6795-21tr11:35:250
32012991210,1cyclictest26291-21wc12:15:270
32012991210,1cyclictest26291-21wc12:15:270
32012991210,1cyclictest1180-21grep08:20:190
3202699119,1cyclictest5422-21grep09:20:183
3202699118,2cyclictest9760-21sed11:10:293
3202699118,2cyclictest7461-21tr07:25:193
3202699118,2cyclictest3348-21ssh10:38:033
3202699118,2cyclictest3302-21cat09:45:183
3202699118,2cyclictest29784-21sort09:40:193
3202699118,2cyclictest28228-21ssh11:25:403
3202699118,2cyclictest27588-21ssh09:10:533
3202699118,2cyclictest26510-21ssh12:15:303
3202699118,2cyclictest26510-21ssh12:15:293
3202699118,2cyclictest20817-21grep10:25:193
3202699118,2cyclictest1967-21ssh12:21:013
3202699118,2cyclictest18769-21taskset07:49:173
3202699118,2cyclictest16044-21ssh12:32:133
3202699118,2cyclictest14337-21ssh12:05:313
3202699117,3cyclictest15732-21sed08:50:013
3202699117,2cyclictest5833-21cut08:30:123
3202699117,2cyclictest19368-21tr07:50:163
3202699114,4cyclictest0-21swapper/309:00:153
3202699113,7cyclictest12543-21munin-node08:44:213
3202699110,10cyclictest16007-21ssh11:15:473
3202299119,1cyclictest1561-21sort07:10:272
3202299119,1cyclictest1561-21sort07:10:262
3202299119,1cyclictest10710-21sh11:39:592
3202299118,2cyclictest9595-21ssh09:24:322
3202299118,2cyclictest8012-21if_enp2s010:15:192
3202299118,2cyclictest6894-21cat08:30:202
3202299118,2cyclictest31395-21expr08:15:182
3202299118,2cyclictest22532-21grep10:55:162
3202299118,2cyclictest21358-21cat07:55:002
3202299118,2cyclictest13053-21sed11:40:292
3202299117,3cyclictest29393-21cut08:10:192
3202299117,3cyclictest15166-21taskset08:48:112
3202299110,5cyclictest0-21swapper/209:30:142
3201699119,1cyclictest32472-21apt-get07:10:151
3201699119,1cyclictest32472-21apt-get07:10:141
3201699118,2cyclictest997-21chrt08:20:041
3201699118,2cyclictest8841-21ssh10:42:501
3201699118,2cyclictest775-21wc11:55:221
3201699118,2cyclictest6872-21ssh09:20:491
3201699118,2cyclictest429-21grep11:30:191
3201699118,2cyclictest32595-21df12:20:161
3201699118,2cyclictest32533-21ssh09:15:261
3201699118,2cyclictest31207-21ssh11:29:271
3201699118,2cyclictest28523-21ssh10:05:251
3201699118,2cyclictest26153-21ssh10:30:191
3201699118,2cyclictest16575-21ssh12:32:391
3201699118,2cyclictest13727-21expr08:45:221
3201699118,2cyclictest12226-21cut07:35:171
3201699118,2cyclictest11201-21tr10:45:171
3201699118,2cyclictest10423-21sed09:50:301
3201699118,2cyclictest10021-21taskset08:36:471
3201699117,3cyclictest15185-21head09:55:191
3201299119,1cyclictest9871-21tr07:30:170
3201299119,1cyclictest8934-21tr09:50:180
3201299119,1cyclictest7760-21sort10:15:170
3201299119,1cyclictest31604-21grep09:15:180
3201299119,1cyclictest31325-21tr08:15:170
3201299119,1cyclictest2167-21ps10:10:230
3201299119,1cyclictest14502-21cut09:55:160
3201299119,1cyclictest10492-21wc08:40:000
3201299118,2cyclictest6948-21ssh10:40:270
3201299118,2cyclictest6540-21head07:20:280
3201299118,2cyclictest453-21cut11:30:190
3201299118,2cyclictest4410-21chrt07:18:280
3201299118,2cyclictest32719-21grep12:20:160
3201299118,2cyclictest28694-21sh11:26:140
3201299118,2cyclictest275182chrt11:00:030
3201299118,2cyclictest26523-21tail09:10:240
3201299118,2cyclictest26307-21cut08:05:130
3201299118,2cyclictest25573-21grep11:50:200
3201299118,2cyclictest25483-21ssh10:57:160
3201299118,2cyclictest19826-21ssh10:25:040
3201299118,2cyclictest18315-21cat10:50:250
3201299118,2cyclictest16683-21ssh11:44:140
3201299118,2cyclictest14834-21ssh10:48:060
3201299118,2cyclictest14143-21cat07:40:000
3201299118,2cyclictest13778-21grep12:30:200
3201299118,2cyclictest11115-21taskset11:12:080
3201299118,2cyclictest10691-21grep09:25:140
3201299115,5cyclictest30694-21apache211:05:290
3201299111,6cyclictest0-21swapper/007:10:150
3201299111,6cyclictest0-21swapper/007:10:140
3201299110,4cyclictest0-21swapper/008:00:150
3201299110,10cyclictest20697-21sed07:50:270
3202699108,1cyclictest31453-21cat09:15:193
3202699108,1cyclictest24275-21mailstats09:05:263
3202699108,1cyclictest13873-21taskset07:37:383
3202699107,2cyclictest4947-21cut07:20:153
3202699105,2cyclictest224-21systemd-journal07:15:013
3202699105,2cyclictest224-21systemd-journal07:15:003
3202699101,7cyclictest32130-21cut11:30:163
3202699101,6cyclictest3009-21/usr/sbin/munin07:15:173
3202699100,9cyclictest11628-21chrt07:34:223
3202299108,2cyclictest9849-21sh12:02:332
3202299108,2cyclictest3238-21cat07:15:182
3202299108,2cyclictest31678-21ssh09:41:302
3202299108,1cyclictest7160-21chrt07:25:042
3202299108,1cyclictest32384-21grep10:35:162
3202299108,1cyclictest23222-21grep09:05:172
3202299108,1cyclictest22242-21tail07:55:172
3202299108,1cyclictest19719-21rm11:45:262
3202299108,1cyclictest1272-21grep08:20:192
3202299107,2cyclictest22683-21ls09:05:012
3202299107,1cyclictest5639-21wc07:20:182
3202299100,5cyclictest224-21systemd-journal11:23:382
3201699108,1cyclictest7724-21ssh11:10:051
3201699108,1cyclictest3732-21cut07:15:221
3201699108,1cyclictest28280-21chrt08:09:061
3201699107,2cyclictest9898-21ls07:30:161
3201699107,2cyclictest5373-21df07:20:171
3201699107,2cyclictest466-21smartctl10:35:191
3201699107,2cyclictest29860-21grep08:10:241
3201699107,2cyclictest19408-21grep08:55:241
3201699107,2cyclictest19326-21tr07:50:151
3201699107,2cyclictest17475-21grep08:50:341
3201699107,2cyclictest16454-21cron07:45:001
3201699100,9cyclictest23286-21taskset07:56:151
3201299108,1cyclictest9328-21basename07:30:000
3201299108,1cyclictest31945-21ssh10:35:040
3201299108,1cyclictest29173-21sed08:10:180
3201299108,1cyclictest23651-21grep09:05:190
3201299108,1cyclictest17475-21cut07:45:160
3201299108,1cyclictest15848-21ssh10:20:490
3201299107,2cyclictest7672-21chrt08:32:050
3201299107,2cyclictest3734-21cat08:25:150
3201299107,2cyclictest26258-21grep10:30:190
3201299107,2cyclictest16295-21ssh09:30:090
3201299107,2cyclictest15893-21chrt08:50:160
3201299107,2cyclictest11322-21grep08:40:220
3201299103,4cyclictest14360-21apt-get07:40:150
320229998,1cyclictest32101-21aten2_r7power_p12:20:142
320229998,1cyclictest17905-21open_inodes07:45:192
320229998,1cyclictest12541-21dnf07:35:182
320229998,1cyclictest10046-21expr07:30:162
320229991,3cyclictest5194-21sshd08:25:472
320229990,1cyclictest0-21swapper/208:40:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional