You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 07:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jun 25, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
815529482,8sleep00-21swapper/019:07:380
621029366,8sleep30-21swapper/319:05:073
800329065,9sleep20-21swapper/219:05:382
822828467,13sleep10-21swapper/119:08:351
101142540,0sleep30-21swapper/321:53:213
843099260,25cyclictest0-21swapper/122:30:071
843399240,24cyclictest0-21swapper/221:13:292
843399210,20cyclictest0-21swapper/223:26:552
843399200,20cyclictest0-21swapper/221:19:252
843099203,9cyclictest0-21swapper/120:00:141
843099200,1cyclictest0-21swapper/122:11:341
842899207,7cyclictest10382-21munin-run23:30:000
8433991917,1cyclictest32-21ksoftirqd/222:16:552
843399190,17cyclictest0-21swapper/222:07:282
8433991816,1cyclictest32-21ksoftirqd/223:35:522
8433991816,1cyclictest0-21swapper/221:45:062
8433991814,1cyclictest32-21ksoftirqd/223:04:512
843099180,2cyclictest0-21swapper/123:29:411
843099180,16cyclictest0-21swapper/122:07:401
8428991817,1cyclictest0-21swapper/022:33:220
842899180,1cyclictest141rcu_preempt21:45:220
843399170,17cyclictest0-21swapper/219:15:122
843099170,3cyclictest3188-21ssh00:22:361
843099170,17cyclictest0-21swapper/123:39:541
843099170,17cyclictest0-21swapper/123:18:281
843099170,16cyclictest1163-21diskmemload21:28:171
843099170,0cyclictest0-21swapper/123:56:341
842899174,9cyclictest24682-21sendmail23:00:000
842899172,1cyclictest141rcu_preempt21:58:190
8428991716,1cyclictest0-21swapper/020:35:130
842899171,1cyclictest141rcu_preempt23:45:100
842899170,16cyclictest0-21swapper/021:10:470
8438991615,1cyclictest0-21swapper/323:20:233
843399162,1cyclictest141rcu_preempt00:37:582
8433991614,1cyclictest0-21swapper/222:29:542
8433991612,3cyclictest32-21ksoftirqd/221:53:092
8433991612,1cyclictest32-21ksoftirqd/223:10:442
8433991612,1cyclictest32-21ksoftirqd/219:10:222
843399160,1cyclictest0-21swapper/222:49:432
843399160,16cyclictest0-21swapper/222:55:282
843099160,2cyclictest0-21swapper/100:01:591
843099160,1cyclictest1163-21diskmemload23:47:131
843099160,16cyclictest0-21swapper/121:11:031
843899150,15cyclictest0-21swapper/300:06:343
843899150,0cyclictest0-21swapper/323:55:393
843399159,1cyclictest141rcu_preempt22:51:542
8433991513,2cyclictest32-21ksoftirqd/223:47:362
8433991513,1cyclictest32-21ksoftirqd/200:08:412
8433991512,2cyclictest32-21ksoftirqd/222:00:512
843099150,9cyclictest0-21swapper/100:08:231
843099150,2cyclictest141rcu_preempt23:32:311
842899153,9cyclictest23745-21/usr/sbin/munin22:20:190
8428991514,1cyclictest0-21swapper/022:42:450
842899151,2cyclictest19956-21ssh22:37:250
842899150,1cyclictest141rcu_preempt00:20:160
842899150,14cyclictest11681-21ssh22:51:280
843899149,5cyclictest4938-21apache222:36:353
8438991414,0cyclictest0-21swapper/321:33:343
8438991412,2cyclictest0-21swapper/323:37:363
8438991411,2cyclictest23671-21sed20:45:273
843899140,2cyclictest0-21swapper/321:23:573
843399142,1cyclictest0-21swapper/221:40:192
843399141,5cyclictest0-21swapper/220:25:112
843399140,13cyclictest0-21swapper/223:53:462
843399140,12cyclictest0-21swapper/200:00:552
843099141,2cyclictest8648-21kworker/1:221:45:141
843099140,14cyclictest0-21swapper/123:06:521
843099140,14cyclictest0-21swapper/121:31:041
842899146,5cyclictest21217-21apt-get22:00:130
842899141,6cyclictest0-21swapper/023:10:150
842899141,1cyclictest0-21swapper/022:10:300
842899141,1cyclictest0-21swapper/021:40:130
8428991410,4cyclictest0-21swapper/021:29:310
843899139,3cyclictest31636-21grep00:40:003
843899139,3cyclictest24669-21grep22:40:153
843899139,3cyclictest13035-21ssh23:11:323
843899139,3cyclictest12741-21sed19:15:263
8438991310,2cyclictest7381-21cron20:15:003
8438991310,2cyclictest3705-21dpkg21:50:003
8438991310,2cyclictest25982-21ssh23:20:003
8438991310,2cyclictest24388-21sed00:15:293
8438991310,2cyclictest24388-21sed00:15:283
8438991310,2cyclictest18674-21sed22:55:273
8438991310,2cyclictest14053-21grep20:25:243
8438991310,2cyclictest12441-21tr23:50:133
8438991310,2cyclictest12315-21cat20:25:013
843899130,1cyclictest0-21swapper/320:15:143
8433991313,0cyclictest0-21swapper/200:10:022
8433991310,2cyclictest20013-21ssh23:15:262
843399130,9cyclictest0-21swapper/223:09:182
843399130,1cyclictest141rcu_preempt20:40:262
843399130,12cyclictest0-21swapper/221:57:112
843099131,3cyclictest28823-21hwlatdetect23:40:171
8430991311,2cyclictest0-21swapper/122:04:361
8430991310,2cyclictest6450-21cron22:30:001
8430991310,2cyclictest290702sleep123:20:551
8430991310,2cyclictest20797-21ssh22:00:011
8430991310,2cyclictest16096-21tune2fs22:35:151
8430991310,2cyclictest16027-21head00:30:171
843099130,9cyclictest38150irq/131-snd_hda19:50:121
843099130,9cyclictest32259-21hddtemp_smartct21:05:151
843099130,11cyclictest0-21swapper/121:38:221
843099130,0cyclictest0-21swapper/120:15:161
842899139,3cyclictest14295-21tr00:10:150
842899133,1cyclictest10924-21grep23:30:120
842899131,9cyclictest30887-21apache_accesses22:25:110
8428991312,1cyclictest0-21swapper/021:52:540
842899130,11cyclictest0-21swapper/000:09:440
843899129,2cyclictest4648-21turbostat.cron00:05:003
843899129,2cyclictest4199-21cut20:05:213
843899129,2cyclictest29942-21tr22:05:143
843899129,2cyclictest28893-21if_enp2s023:40:173
843899129,2cyclictest24636-21rm22:01:313
843899129,2cyclictest20729-21wc22:00:013
843899129,2cyclictest17451-21tr20:35:133
843899129,2cyclictest1259-21head20:00:163
843899129,2cyclictest11717-21grep23:30:173
843899125,3cyclictest225-21systemd-journal19:50:003
843899121,8cyclictest32578-21latency_hist20:00:013
8438991210,1cyclictest17129-21grep19:25:213
843899120,1cyclictest0-21swapper/322:27:043
843399129,2cyclictest310-21grep00:20:252
843399129,2cyclictest29889-21grep23:40:262
843399129,2cyclictest24554-21sort22:40:152
843399129,2cyclictest24417-21grep00:15:292
843399129,2cyclictest24417-21grep00:15:282
843399129,2cyclictest21819-21wc19:35:242
843399129,2cyclictest19758-21grep21:20:182
843399129,2cyclictest18215-21wc20:35:182
843399124,4cyclictest1163-21diskmemload23:20:242
843399120,1cyclictest141rcu_preempt21:25:132
843399120,1cyclictest0-21swapper/221:00:152
843399120,12cyclictest0-21swapper/221:30:572
843399120,11cyclictest7739-21cut22:30:172
843099129,2cyclictest8899-21tr19:10:141
843099129,2cyclictest8005-21head00:25:191
843099129,2cyclictest26960-21cut20:55:131
843099129,2cyclictest23707-21cut00:35:131
843099129,2cyclictest20558-21grep21:40:161
843099127,4cyclictest38150irq/131-snd_hda22:47:121
8430991211,1cyclictest0-21swapper/122:54:391
843099120,12cyclictest0-21swapper/122:22:271
843099120,12cyclictest0-21swapper/121:53:181
843099120,12cyclictest0-21swapper/100:12:301
843099120,11cyclictest0-21swapper/123:54:031
843099120,0cyclictest0-21swapper/123:14:021
842899129,2cyclictest7088-21grep00:25:130
842899129,2cyclictest683-21grep22:45:170
842899129,2cyclictest3101-21sed23:05:240
842899129,2cyclictest30616-21head00:00:220
842899129,2cyclictest27553-21cat19:49:590
842899129,2cyclictest23131-21perf00:35:010
842899129,2cyclictest21280-21df23:55:130
842899122,1cyclictest141rcu_preempt23:35:560
842899120,11cyclictest20679-21grep20:40:200
149122120,1sleep00-21swapper/022:15:170
843899119,2cyclictest0-21swapper/322:33:253
843899118,2cyclictest9198-21sed22:50:173
843899118,2cyclictest8205-21ssh22:10:483
843899118,2cyclictest6784-21sh00:25:123
843899118,2cyclictest5587-21ssh00:24:273
843899118,2cyclictest4737-21head23:45:173
843899118,2cyclictest30824-21who21:00:263
843899118,2cyclictest30712-21ssh21:26:503
843899118,2cyclictest298112sleep319:52:283
843899118,2cyclictest2791-21expr23:25:133
843899118,2cyclictest26442-21grep23:00:203
843899118,2cyclictest25867-21ssh22:21:293
843899118,2cyclictest25430-21grep20:50:203
843899118,2cyclictest24654-21sed19:40:253
843899118,2cyclictest20569-21head21:40:163
843899118,2cyclictest19569-21basename20:40:113
843899118,2cyclictest1879-21sort23:05:163
843899118,2cyclictest15224-21ssh00:30:123
843899118,2cyclictest14401-21grep22:15:153
843899118,2cyclictest14368-21uname19:20:183
843899118,2cyclictest14331-21ssh21:36:143
843899118,2cyclictest14322-21tr00:10:143
843899118,2cyclictest11903-21cat21:15:273
843899118,2cyclictest1042-21cat22:45:183
843899117,3cyclictest2500-21head21:10:173
843899114,2cyclictest225-21systemd-journal21:10:013
843899110,10cyclictest15120-21cat20:30:163
843399119,1cyclictest31019-21tail19:55:172
843399118,2cyclictest8989-21fgrep20:15:232
843399118,2cyclictest8170-21tr00:25:202
843399118,2cyclictest2939-21grep20:05:122
843399118,2cyclictest29138-21head19:50:212
843399118,2cyclictest22960-21sh23:55:352
843399118,2cyclictest22673-21aten2_r7power_v22:20:152
843399118,2cyclictest20404-21cat19:34:592
843399118,2cyclictest16966-21sh21:38:172
843399118,2cyclictest16734-21pluginstate22:35:202
843399118,2cyclictest16229-21wc20:30:262
843399118,2cyclictest14946-21sh00:30:012
843399118,2cyclictest12970-21sh23:30:292
843399118,2cyclictest1199-21expr20:00:162
843399110,2cyclictest0-21swapper/222:12:162
843399110,10cyclictest11054-21sort20:20:202
843099118,2cyclictest32121-21chrt19:56:381
843099118,2cyclictest27619-21ssh22:41:471
843099118,2cyclictest19822-21tr20:40:141
843099118,2cyclictest18812-21wc21:20:131
843099118,2cyclictest17702-21ssh22:16:401
843099118,2cyclictest17581-21tail22:55:181
843099118,2cyclictest12862-21expr20:25:131
843099117,3cyclictest30585-21ssh23:03:051
843099116,4cyclictest19261-21taskset20:39:461
843099116,4cyclictest15358-21sed20:30:161
843099115,2cyclictest225-21systemd-journal20:25:011
843099111,1cyclictest0-21swapper/121:00:121
842899119,1cyclictest32580-21cut21:05:190
842899119,1cyclictest28647-21grep23:40:160
842899119,1cyclictest13760-21pmu-power23:50:210
842899119,1cyclictest0-21swapper/000:15:130
842899119,1cyclictest0-21swapper/000:15:130
842899118,2cyclictest25113-21grep20:50:170
842899118,2cyclictest19315-21head21:20:150
842899118,2cyclictest17565-21chrt19:25:370
842899118,2cyclictest14068-21grep20:25:240
842899118,2cyclictest12682-21grep21:35:220
842899118,2cyclictest12546-21cat19:15:250
842899118,1cyclictest23583-21cut19:40:150
842899114,5cyclictest21958-21apt-get20:45:150
842899114,3cyclictest225-21systemd-journal20:20:000
842899112,4cyclictest16807-21sshd20:33:410
842899111,6cyclictest1163-21diskmemload22:05:100
842899111,2cyclictest141rcu_preempt00:35:160
843899108,1cyclictest9175-21sed19:10:153
843899108,1cyclictest27476-21grep20:55:153
843899107,2cyclictest18556-21expr19:30:163
843399108,2cyclictest24926-21awk20:50:162
843399108,1cyclictest26385-21wc19:45:202
843399108,1cyclictest16057-21tr19:25:132
843399107,2cyclictest5993-21tail20:10:182
843399107,2cyclictest31506-21grep21:05:132
843399107,2cyclictest26940-21tr20:55:132
843399107,2cyclictest24578-21sed19:40:252
843399107,2cyclictest13724-21tr19:20:142
843399100,10cyclictest23004-21missed_timers20:45:212
843099108,1cyclictest6033-21sed20:10:181
843099108,1cyclictest3754-21cat20:05:171
843099108,1cyclictest27297-21taskset19:48:251
843099108,1cyclictest24346-21chrt19:40:231
843099108,1cyclictest21105-21sed19:35:171
843099107,2cyclictest22717-21tail20:45:181
843099107,2cyclictest16578-21grep19:25:161
843099107,2cyclictest14737-21chrt19:20:201
843099107,2cyclictest12430-21chrt19:15:231
843099107,2cyclictest11834-21sh21:15:261
843099100,9cyclictest22186-21ssh00:15:031
843099100,9cyclictest22186-21ssh00:15:021
842899108,2cyclictest3250-21expr20:05:130
842899108,1cyclictest30254-21chrt19:55:090
842899108,1cyclictest27875-21taskset19:50:050
842899108,1cyclictest26438-21ssh23:20:110
842899108,1cyclictest20559-21apache_volume19:35:140
842899108,1cyclictest18093-21ssh23:15:130
842899108,1cyclictest134422sleep019:20:110
842899108,1cyclictest10303-21grep20:20:150
842899107,2cyclictest9281-21fschecks_count19:10:150
842899107,2cyclictest4471-21cat21:30:240
842899107,2cyclictest18936-21head19:30:170
842899107,2cyclictest12047-21grep21:15:270
842899105,3cyclictest24998-21sh23:00:070
842899105,2cyclictest475-21in:imuxsock20:01:420
842899103,6cyclictest28815-21munin-run21:00:000
84389997,1cyclictest20649-21aten2_r7power_p19:35:153
84309997,1cyclictest24795-21expr20:50:141
84309995,3cyclictest19409-21head19:30:221
84289997,1cyclictest310372sleep021:02:570
84289995,4cyclictest5827-21df_abs20:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional