You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 02:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Aug 26, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1507129772,20sleep10-21swapper/119:06:501
1526629582,9sleep00-21swapper/019:09:220
1505229164,9sleep20-21swapper/219:06:342
1529028565,16sleep30-21swapper/319:09:413
1541899332,26cyclictest225-21systemd-journal23:45:003
1541899290,20cyclictest0-21swapper/321:50:143
15408992926,2cyclictest8022-21diskmemload21:50:470
1541899272,24cyclictest17651-21munin-run21:35:003
1541899261,24cyclictest8022-21diskmemload23:06:093
15408992624,1cyclictest8135-21apt23:45:000
1540899260,1cyclictest0-21swapper/000:20:520
1541899257,17cyclictest8533-21apt-get21:10:033
1541099250,24cyclictest0-21swapper/100:21:111
1541899249,14cyclictest8022-21diskmemload00:37:503
1541899242,19cyclictest14278-21df22:50:163
15418992413,10cyclictest8022-21diskmemload23:03:273
1541899241,21cyclictest0-21swapper/300:26:583
1541899240,23cyclictest0-21swapper/300:33:553
1541899240,22cyclictest0-21swapper/319:35:143
1541499240,0cyclictest0-21swapper/223:33:192
1541099241,22cyclictest0-21swapper/100:10:441
1541899233,18cyclictest16555-21systemctl20:20:003
1541899230,19cyclictest8022-21diskmemload22:31:123
15408992321,1cyclictest8022-21diskmemload00:26:580
15408992314,8cyclictest0-21swapper/023:09:220
1541899224,17cyclictest8022-21diskmemload23:24:143
15418992216,5cyclictest8022-21diskmemload00:20:373
1541899221,20cyclictest8022-21diskmemload00:04:283
1541899221,19cyclictest0-21swapper/320:55:133
1541899220,21cyclictest28275-21taskset22:20:113
1541899220,21cyclictest28275-21taskset22:20:113
1541899220,21cyclictest0-21swapper/300:08:113
1541099220,2cyclictest0-21swapper/122:15:141
15408992220,1cyclictest0-21swapper/021:10:030
15408992218,3cyclictest17650-21turbostat.cron21:35:000
1541899219,11cyclictest28991-21rm22:59:183
1541899212,14cyclictest8022-21diskmemload22:05:153
15418992112,8cyclictest8429-21grep23:25:173
1541899210,20cyclictest8022-21diskmemload22:46:263
1541899210,20cyclictest0-21swapper/323:19:053
1541899210,20cyclictest0-21swapper/322:18:543
1541899210,19cyclictest8022-21diskmemload21:27:323
1541899210,19cyclictest0-21swapper/320:30:143
1541899210,18cyclictest0-21swapper/323:35:023
1541899210,18cyclictest0-21swapper/322:10:433
1541099211,20cyclictest0-21swapper/100:15:141
15408992118,2cyclictest0-21swapper/000:00:140
1541899209,8cyclictest24362-21sh21:19:353
1541899205,14cyclictest8022-21diskmemload23:46:593
1541899205,13cyclictest19741-21users19:15:273
15418992014,5cyclictest8022-21diskmemload23:50:453
1541899200,19cyclictest0-21swapper/323:30:163
1541899200,19cyclictest0-21swapper/320:45:153
15408992019,1cyclictest0-21swapper/022:50:160
15408992018,1cyclictest8022-21diskmemload23:19:050
15408992018,1cyclictest29726-21sed00:35:270
15408992018,1cyclictest16909-21ssh23:50:020
15408992017,2cyclictest8022-21diskmemload21:58:040
15408992013,6cyclictest8022-21diskmemload22:25:480
1541899190,18cyclictest22075-21sh21:56:163
1541899190,18cyclictest0-21swapper/323:11:583
1541899190,18cyclictest0-21swapper/322:36:283
1541899190,17cyclictest0-21swapper/321:21:473
1541899190,14cyclictest0-21swapper/320:35:143
15414991917,1cyclictest22078-21apt-get22:55:142
1541499190,19cyclictest0-21swapper/200:14:212
1541099191,9cyclictest15939-21ssh00:08:411
1541099191,16cyclictest8022-21diskmemload22:58:501
1540899197,10cyclictest0-21swapper/023:24:070
1540899193,5cyclictest0-21swapper/021:43:320
15408991916,2cyclictest10023-21sh23:25:450
15408991915,1cyclictest8022-21diskmemload00:11:590
15408991910,6cyclictest0-21swapper/019:20:000
1541899188,7cyclictest25450-21ssh23:55:123
1541899182,15cyclictest17561-21grep20:20:173
15418991810,7cyclictest27507-21grep00:15:183
1541899180,17cyclictest0-21swapper/321:49:433
15414991818,0cyclictest0-21swapper/223:27:312
15414991816,1cyclictest23914-21/usr/sbin/munin20:35:262
1541499180,18cyclictest0-21swapper/222:23:262
1541499180,18cyclictest0-21swapper/222:23:262
15410991816,1cyclictest13647-21expr22:30:181
1541099180,1cyclictest0-21swapper/121:29:591
1540899185,5cyclictest225-21systemd-journal00:31:580
15408991816,1cyclictest8022-21diskmemload22:05:170
15408991815,1cyclictest28208-21ps00:15:240
15408991814,3cyclictest161rcuc/023:37:380
15408991814,3cyclictest0-21swapper/022:20:110
15408991814,3cyclictest0-21swapper/022:20:110
1541899177,9cyclictest0-21swapper/320:25:363
1541899172,13cyclictest8022-21diskmemload21:35:243
1541899171,14cyclictest23568-21df_inode19:25:163
1541899170,16cyclictest0-21swapper/322:41:273
1541899170,16cyclictest0-21swapper/321:41:203
1541899170,15cyclictest0-21swapper/322:25:103
1541899170,15cyclictest0-21swapper/322:00:063
15414991717,0cyclictest0-21swapper/223:15:592
15410991715,1cyclictest25470-21apt-get23:55:141
15408991715,2cyclictest0-21swapper/019:30:220
15408991715,1cyclictest16561-21ls20:20:000
15408991714,2cyclictest8022-21diskmemload21:21:560
15408991714,2cyclictest17562-21grep20:20:170
15408991714,2cyclictest0-21swapper/021:49:500
1540899170,3cyclictest0-21swapper/021:16:470
1540899170,1cyclictest0-21swapper/023:33:230
192372160,0sleep20-21swapper/223:12:142
1541899162,13cyclictest26413-21aten2_r7power_e20:40:163
1541899162,12cyclictest0-21swapper/319:55:203
1541499165,6cyclictest225-21systemd-journal22:15:002
15414991612,3cyclictest32-21ksoftirqd/221:45:002
1541499161,15cyclictest0-21swapper/223:23:582
1541499160,16cyclictest0-21swapper/221:55:292
15410991612,3cyclictest22889-21ssh23:53:241
1541099160,1cyclictest0-21swapper/100:01:191
1541099160,15cyclictest0-21swapper/122:36:421
1541099160,14cyclictest8022-21diskmemload22:40:421
15408991614,2cyclictest8483-1kworker/u9:100:09:000
15408991614,1cyclictest8022-21diskmemload21:37:340
15408991614,1cyclictest13-21ksoftirqd/020:59:220
15408991613,2cyclictest8022-21diskmemload21:27:280
15408991613,2cyclictest0-21swapper/023:55:220
15408991613,2cyclictest0-21swapper/022:10:260
1541899159,5cyclictest18659-21expr00:10:153
1541899158,6cyclictest8241-21date21:10:003
1541899151,13cyclictest0-21swapper/319:15:003
1541499153,1cyclictest8022-21diskmemload00:36:372
15414991515,0cyclictest0-21swapper/222:44:422
15414991512,2cyclictest26696-21needreboot23:55:202
1541499150,1cyclictest0-21swapper/223:06:322
1541499150,14cyclictest0-21swapper/223:38:302
1541499150,14cyclictest0-21swapper/221:39:212
15410991513,1cyclictest0-21swapper/121:50:061
1540899159,5cyclictest15312-21ssh23:49:080
1540899158,6cyclictest0-21swapper/022:33:440
15408991514,1cyclictest13-21ksoftirqd/022:47:500
15408991512,2cyclictest26412-21cut20:40:160
15408991512,2cyclictest0-21swapper/019:35:140
15408991510,4cyclictest7041-21idleruntime-cro20:00:000
15408991510,4cyclictest27581-21ssh22:19:520
1541899148,5cyclictest5489-21chrt21:03:073
1541899148,5cyclictest13827-21taskset20:10:553
1541899141,6cyclictest9968-21cstates20:05:153
1541499144,4cyclictest20142-21perf19:20:002
1541499143,10cyclictest18494-21cron21:55:002
15414991413,0cyclictest0-21swapper/200:22:062
1541499141,1cyclictest0-21swapper/221:22:072
15414991411,2cyclictest0-21swapper/221:25:162
1541099149,2cyclictest26460-21apt-get21:40:141
1541099142,1cyclictest0-21swapper/121:45:131
15410991413,1cyclictest0-21swapper/123:09:141
15410991412,2cyclictest0-21swapper/100:32:091
15410991410,3cyclictest27397-21cat20:40:211
15410991410,3cyclictest17902-21sed23:30:281
1540899147,6cyclictest0-21swapper/021:00:140
15408991413,1cyclictest0-21swapper/023:03:270
15408991412,2cyclictest0-21swapper/020:45:150
15408991412,1cyclictest25072-21awk20:35:200
15408991412,1cyclictest0-21swapper/022:36:280
15408991412,1cyclictest0-21swapper/022:00:380
15408991412,1cyclictest0-21swapper/019:15:000
1540899141,1cyclictest0-21swapper/019:29:040
15408991411,2cyclictest0-21swapper/023:11:280
15408991411,2cyclictest0-21swapper/022:55:210
1541899138,4cyclictest21344-21head19:20:163
1541899130,12cyclictest0-21swapper/320:00:133
1541899130,12cyclictest0-21swapper/319:50:173
1541899130,12cyclictest0-21swapper/319:30:223
1541499139,3cyclictest32755-21grep19:45:152
1541499132,1cyclictest8022-21diskmemload21:10:382
15414991311,1cyclictest12649-21uniq20:10:172
15414991310,2cyclictest9312-21idleruntime-cro00:05:002
15414991310,2cyclictest28732-21cat19:35:202
15414991310,2cyclictest28246-21grep22:00:182
15414991310,2cyclictest21877-21grep22:35:162
15414991310,2cyclictest17260-21grep19:10:262
15414991310,2cyclictest12121-21perf22:30:002
1541099139,3cyclictest28395-21sort19:35:171
1541099139,3cyclictest20002-21head20:25:321
1541099138,2cyclictest22502-21taskset19:22:101
1541099137,2cyclictest29067-21ssh21:21:351
1541099133,7cyclictest8435-21grep23:25:171
15410991312,1cyclictest38150irq/131-snd_hda23:37:371
15410991310,2cyclictest7154-21wc23:25:001
15410991310,2cyclictest53182sleep122:25:201
15410991310,2cyclictest18143-21fgrep20:20:221
15410991310,2cyclictest16911-21ssh21:15:061
1540899139,3cyclictest4135-21grep19:50:260
1540899139,3cyclictest11320-21grep20:05:270
15408991312,1cyclictest0-21swapper/022:40:100
1541899126,5cyclictest31230-21basename20:50:233
1541899126,5cyclictest31031-21cat19:40:193
1541899120,11cyclictest694-21fschecks_count19:45:173
1541499129,2cyclictest30925-21cut19:40:182
1541499129,2cyclictest28597-21sed00:15:282
1541499129,2cyclictest25119-21ls19:30:002
1541499129,2cyclictest23035-21ssh22:16:172
1541499129,2cyclictest22307-21sed19:20:252
1541499129,2cyclictest19036-21grep23:50:272
1541499129,2cyclictest1789-21grep20:55:212
1541499129,2cyclictest17079-21tr20:20:142
1541499129,2cyclictest13823-21grep22:50:142
1541499129,2cyclictest12701-21chrt23:47:042
1541499129,2cyclictest11322-21sed20:05:272
1541499129,2cyclictest10102-21grep00:05:162
1541499128,3cyclictest6229-21grep22:45:172
1541499128,2cyclictest6410-21cat19:55:252
15414991211,1cyclictest32-21ksoftirqd/221:17:102
15414991211,1cyclictest0-21swapper/220:50:162
1541499121,10cyclictest0-21swapper/221:30:122
1541499120,11cyclictest0-21swapper/200:25:132
1541499120,0cyclictest0-21swapper/221:45:472
1541099129,2cyclictest7898-21sort20:00:161
1541099129,2cyclictest4295-21grep21:00:171
1541099129,2cyclictest31769-21ls23:00:211
1541099129,2cyclictest29482-21tail20:45:191
1541099129,2cyclictest28576-21grep00:35:181
1541099129,2cyclictest26405-21wc19:30:221
1541099129,2cyclictest25114-21perf19:30:001
1541099128,3cyclictest23180-21sed20:30:271
1541099122,9cyclictest0-21swapper/123:12:241
1541099121,2cyclictest141rcu_preempt22:21:491
1541099121,2cyclictest141rcu_preempt22:21:491
15410991212,0cyclictest0-21swapper/121:55:211
15410991212,0cyclictest0-21swapper/120:50:231
15410991210,1cyclictest5921-21seq22:45:151
1541099120,1cyclictest0-21swapper/122:07:031
1541099120,11cyclictest24158-21head23:15:171
1540899129,2cyclictest21325-21grep19:20:160
1540899128,3cyclictest23673-21cron20:35:000
15408991210,1cyclictest8049-21ssh21:10:000
15408991210,1cyclictest19167-21/usr/sbin/munin20:25:300
1541499118,2cyclictest9555-21cat20:05:002
1541499118,2cyclictest5385-21grep22:05:282
1541499118,2cyclictest3799-21run-parts19:50:222
1541499118,2cyclictest31789-21ssh23:00:212
1541499118,2cyclictest26470-21cat19:30:222
1541499118,2cyclictest20779-21ntp_states00:30:222
1541499118,2cyclictest20723-21cat20:25:402
1541499118,2cyclictest14286-21wc22:30:232
1541499117,3cyclictest28482-21cron20:45:002
15414991110,1cyclictest0-21swapper/220:30:182
1541099118,2cyclictest9928-21cut21:30:141
1541099118,2cyclictest7143-21grep21:05:301
1541099118,2cyclictest663-21expr23:40:151
1541099118,2cyclictest27292-21ssh22:00:101
1541099118,2cyclictest24681-21expr20:35:171
1541099118,2cyclictest2351-21wc19:50:001
1541099118,2cyclictest21851-21ssh21:36:481
1541099118,2cyclictest16852-21ssh22:51:191
1541099118,2cyclictest15649-21basename19:10:141
1541099118,2cyclictest15343-21ssh23:49:101
1541099118,2cyclictest14610-21ssh00:26:441
1541099118,2cyclictest13708-21ssh22:10:301
1541099118,2cyclictest12992-21cut20:10:201
1541099118,2cyclictest10839-21ssh21:10:301
1540899118,2cyclictest309352sleep019:40:180
1540899118,2cyclictest2234-21wc19:50:000
1541499108,1cyclictest34322sleep221:00:012
1541499106,3cyclictest1460-21sort23:40:202
1541099107,2cyclictest31691-21sed19:40:261
1541099107,2cyclictest10212-21sed20:05:161
1541099107,1cyclictest15182-21expr20:15:191
1541099104,2cyclictest225-21systemd-journal19:20:001
1541099100,9cyclictest5128-21expr19:55:141
1540899107,2cyclictest7538-21grep20:00:140
1540899107,2cyclictest31787-21sort20:50:240
1540899107,2cyclictest12333-21tr20:10:150
154149997,1cyclictest28722-21taskset20:45:112
154149996,2cyclictest6732-21cat21:05:262
154149996,2cyclictest14910-21df20:15:162
154109997,1cyclictest2610-21cpu19:50:141
154109997,1cyclictest1884-21hwlatdetect20:55:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional