You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 06:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 01, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2713129573,17sleep30-21swapper/319:07:573
2706029164,9sleep00-21swapper/019:07:020
2701928864,13sleep10-21swapper/119:06:311
2698428672,10sleep20-21swapper/219:06:032
73422660,0sleep30-21swapper/300:33:543
2737499321,5cyclictest225-21systemd-journal21:35:000
2737499321,5cyclictest225-21systemd-journal21:35:000
2737899317,23cyclictest19970-21diskmemload22:24:241
2737899310,3cyclictest6186-21/usr/sbin/munin23:15:221
2737899302,26cyclictest30115-21turbostat.cron21:35:001
2737899302,26cyclictest30115-21turbostat.cron21:34:591
2737899301,1cyclictest0-21swapper/121:15:161
2737899293,22cyclictest225-21systemd-journal21:27:301
2737899293,22cyclictest225-21systemd-journal21:27:301
2737899291,2cyclictest16878-21ssh23:40:261
2737899290,2cyclictest0-21swapper/120:35:131
2737499291,5cyclictest0-21swapper/022:05:140
2737899281,1cyclictest7887-21grep21:40:201
2737899281,1cyclictest19970-21diskmemload00:00:221
2737899280,2cyclictest9931-21cron22:40:001
2737899280,27cyclictest0-21swapper/119:45:131
2737499282,2cyclictest225-21systemd-journal21:55:000
2738099270,27cyclictest0-21swapper/200:31:462
2737899271,1cyclictest19970-21diskmemload00:30:311
2737899270,1cyclictest19970-21diskmemload21:10:161
27374992726,1cyclictest0-21swapper/022:49:130
2737899262,2cyclictest19970-21diskmemload22:27:451
2737899262,23cyclictest8550-21ps19:35:211
2737899261,24cyclictest32318-21df_abs23:50:161
2737899261,24cyclictest28805-21apt-get20:20:141
2737899260,25cyclictest0-21swapper/122:05:151
2737899260,1cyclictest19970-21diskmemload22:57:381
2737899260,1cyclictest19970-21diskmemload00:11:421
2737899260,1cyclictest0-21swapper/123:30:011
240872260,0sleep30-21swapper/300:04:573
2737899254,21cyclictest19970-21diskmemload23:48:021
2737899253,1cyclictest19970-21diskmemload22:51:371
2737899251,1cyclictest0-21swapper/123:20:221
2737899250,2cyclictest0-21swapper/100:39:581
2737899250,24cyclictest0-21swapper/119:25:121
2737899250,24cyclictest0-21swapper/100:20:281
2737899250,1cyclictest19970-21diskmemload21:39:431
2737899250,1cyclictest0-21swapper/123:05:121
2737899250,1cyclictest0-21swapper/121:55:141
27383992419,1cyclictest9938-21apt22:40:003
27383992415,7cyclictest0-21swapper/321:30:233
27383992415,7cyclictest0-21swapper/321:30:233
2738399240,24cyclictest0-21swapper/322:50:273
2738099240,24cyclictest0-21swapper/221:45:562
2737899243,20cyclictest19970-21diskmemload21:46:211
2737899243,1cyclictest19970-21diskmemload22:44:331
2737899241,3cyclictest28028-21unixbench-2d20:15:261
2737899240,2cyclictest0-21swapper/121:24:341
2737899240,1cyclictest0-21swapper/123:33:401
2737899240,1cyclictest0-21swapper/100:06:411
2737499240,2cyclictest0-21swapper/021:45:000
27383992315,7cyclictest19970-21diskmemload22:16:443
2738099230,22cyclictest0-21swapper/222:51:592
2737899235,17cyclictest1142-21runrttasks22:16:531
2737899232,1cyclictest19970-21diskmemload00:17:291
2737899231,2cyclictest20590-21grep20:00:221
2737899231,2cyclictest0-21swapper/122:49:271
2737899231,21cyclictest0-21swapper/119:20:141
2737899231,1cyclictest29936-21sed19:15:001
2737899230,1cyclictest0-21swapper/121:54:051
2737499232,1cyclictest19970-21diskmemload21:59:310
2737499231,3cyclictest0-21swapper/023:45:160
2737499230,2cyclictest0-21swapper/019:20:140
27383992221,1cyclictest0-21swapper/323:26:243
27383992213,8cyclictest0-21swapper/321:36:463
27380992222,0cyclictest0-21swapper/221:30:582
27380992222,0cyclictest0-21swapper/221:30:582
2737899223,19cyclictest19970-21diskmemload23:12:391
2737899220,21cyclictest0-21swapper/122:30:161
2737899220,1cyclictest0-21swapper/123:00:011
27374992218,3cyclictest13-21ksoftirqd/000:18:590
2737899213,17cyclictest0-21swapper/123:55:121
2737499210,4cyclictest0-21swapper/023:02:280
2737499210,2cyclictest141rcu_preempt21:24:340
2738399202,18cyclictest0-21swapper/300:26:403
27383992012,7cyclictest19970-21diskmemload21:17:363
2738399200,19cyclictest0-21swapper/323:56:583
2737899200,19cyclictest0-21swapper/100:25:271
2737899200,0cyclictest0-21swapper/122:11:111
2737499202,4cyclictest760-21proc_pri23:50:220
2737499202,3cyclictest0-21swapper/021:45:140
2737499200,4cyclictest29577-21gltestperf20:20:180
2737499200,2cyclictest0-21swapper/022:01:340
27383991917,1cyclictest19970-21diskmemload00:39:583
27383991910,7cyclictest32695-21sh22:34:003
2738099192,17cyclictest0-21swapper/222:06:542
27380991918,1cyclictest0-21swapper/222:26:002
27380991912,4cyclictest9944-21munin-plugin-st22:40:002
2738099190,17cyclictest0-21swapper/221:41:432
2738099190,0cyclictest0-21swapper/222:01:342
2737899191,16cyclictest0-21swapper/123:38:231
2737899190,2cyclictest8329-21cstates22:00:161
2737899190,18cyclictest22194-21grep20:05:151
2737499194,2cyclictest0-21swapper/023:30:000
2737499193,2cyclictest25617-21munin-run22:30:000
97302180,0sleep30-21swapper/322:58:533
27383991816,2cyclictest19970-21diskmemload23:00:473
2738399180,15cyclictest0-21swapper/323:48:453
2738099180,18cyclictest0-21swapper/223:35:362
2737499182,4cyclictest14229-21cron19:50:000
2737499181,5cyclictest0-21swapper/000:10:410
2737499181,4cyclictest0-21swapper/023:55:120
2737499181,2cyclictest13881-21rm23:20:150
2738399177,6cyclictest28229-21fschecks_count19:10:173
2738399174,9cyclictest0-21swapper/323:16:523
2738399174,2cyclictest24673-21sh22:29:163
27383991715,1cyclictest19970-21diskmemload00:20:283
27383991714,2cyclictest0-21swapper/321:55:143
27383991713,1cyclictest16594-21ssh22:05:153
27383991710,2cyclictest18610-21ssh23:41:443
27380991716,1cyclictest0-21swapper/222:43:252
2737899171,2cyclictest32258-21latency_hist19:20:001
2737899170,4cyclictest0-21swapper/120:45:221
2737499177,4cyclictest8828-21latency_hist00:34:590
2737499174,3cyclictest10569-21apache222:43:410
2737499172,1cyclictest29279-21fschecks_time23:10:160
2737499171,4cyclictest9678-21ssh00:35:170
2737499170,2cyclictest0-21swapper/000:26:300
2738399169,6cyclictest19970-21diskmemload23:33:403
27383991613,2cyclictest14538-21apt21:44:593
2738399161,12cyclictest28343-21ssh23:10:043
27383991610,6cyclictest0-21swapper/322:01:003
2738099168,4cyclictest19970-21diskmemload23:23:442
2738099166,1cyclictest291irq_work/221:13:212
27380991615,1cyclictest32-21ksoftirqd/223:30:142
27380991614,2cyclictest32-21ksoftirqd/221:15:222
2737899168,7cyclictest8014-21/usr/sbin/munin20:45:001
2737899167,8cyclictest24260-21tr20:10:151
2737899167,8cyclictest17153-21tr19:55:151
2737899161,13cyclictest32677-21timerandwakeup20:25:251
2737499165,6cyclictest11874-21munin-run19:45:000
2737499164,4cyclictest225-21systemd-journal00:05:580
2737499164,11cyclictest21530-21awk21:30:000
2737499164,11cyclictest21530-21awk21:30:000
2737499163,2cyclictest19970-21diskmemload23:42:020
27374991610,5cyclictest29397-21chrt19:11:020
2737499160,4cyclictest0-21swapper/022:15:050
2737499160,14cyclictest31605-21wc23:30:190
87262150,0sleep00-21swapper/023:16:570
2738399159,5cyclictest0-21swapper/322:13:563
27383991514,1cyclictest19970-21diskmemload00:11:533
27383991513,1cyclictest8552-21wc19:35:213
27383991513,1cyclictest31322-21cron21:55:003
27383991512,3cyclictest0-21swapper/323:50:143
27383991512,1cyclictest10673-21ntp_states22:20:223
27383991511,3cyclictest14226-21cron19:50:003
2738399151,12cyclictest25828-21taskset20:11:153
27383991510,2cyclictest13813-21ssh00:18:323
2738099153,11cyclictest0-21swapper/223:10:132
27380991514,1cyclictest0-21swapper/223:00:352
2738099150,6cyclictest0-21swapper/223:51:002
2737899157,7cyclictest6728-21taskset19:31:141
2737899157,7cyclictest17295-21chrt21:01:301
2737899151,2cyclictest16794-21latency_hist19:55:011
2737499159,5cyclictest10457-21wc22:20:200
2737499158,6cyclictest6521-21ssh22:56:230
2737499158,6cyclictest28430-21taskset20:19:050
2737499158,6cyclictest27500-21ssh22:50:160
2737499157,7cyclictest21771-21ssh23:05:240
2737499157,7cyclictest18661-21ssh00:20:370
2737499155,3cyclictest225-21systemd-journal00:00:560
2737499155,1cyclictest19970-21diskmemload22:30:020
2737499153,6cyclictest21401-21perf20:05:010
2737499153,4cyclictest19970-21diskmemload22:38:230
2737499151,7cyclictest19980-21perf21:10:000
2737499150,14cyclictest16284-21sort21:00:170
2738399146,8cyclictest0-21swapper/321:46:153
2738399144,7cyclictest0-21swapper/319:55:003
2738399141,2cyclictest16721-21sh22:43:413
27383991411,2cyclictest14908-21/usr/sbin/munin23:20:223
27380991414,0cyclictest0-21swapper/222:59:542
27380991410,3cyclictest9777-21grep00:15:272
2738099140,13cyclictest0-21swapper/221:54:272
2737899148,5cyclictest19460-21sed21:05:271
2737899141,2cyclictest225-21systemd-journal20:35:001
2737899140,1cyclictest0-21swapper/120:55:181
2737499149,3cyclictest30003-21wc21:15:210
2737499147,6cyclictest6384-21sed19:30:250
2737499147,6cyclictest30901-21tr21:35:130
2737499147,6cyclictest22969-21wc20:05:220
2737499142,3cyclictest225-21systemd-journal19:20:000
2737499141,3cyclictest225-21systemd-journal20:00:010
27374991411,2cyclictest10554-21apache221:10:250
2738399138,3cyclictest28470-21ssh00:07:593
2738399135,8cyclictest0-21swapper/323:08:053
27383991311,1cyclictest477-21rs:main3
27383991311,1cyclictest477-21rs:main3
27383991311,1cyclictest18851-21rm22:45:103
27383991310,2cyclictest17035-21cat21:00:253
2738399130,8cyclictest0-21swapper/320:35:133
2738099139,3cyclictest17657-21grep19:55:172
2738099138,4cyclictest291irq_work/222:46:142
2738099132,1cyclictest141rcu_preempt00:25:142
27380991312,1cyclictest0-21swapper/200:12:362
27380991311,1cyclictest0-21swapper/223:59:162
27380991310,2cyclictest16237-21sed21:00:162
27380991310,2cyclictest16002-21cat23:40:172
2738099130,1cyclictest141rcu_preempt22:11:522
2738099130,13cyclictest0-21swapper/223:48:572
2737899132,2cyclictest225-21systemd-journal19:41:241
2737899130,2cyclictest0-21swapper/120:50:161
2737499139,2cyclictest3606-21wc19:25:180
2737499138,4cyclictest4521-21memory20:35:190
2737499138,3cyclictest1408-21tr20:30:140
2737499137,5cyclictest15004-21taskset20:57:510
2737499136,2cyclictest31941-21sed20:25:160
2737499136,1cyclictest9615-21tr20:45:220
2737499134,4cyclictest10707-21apt-get20:50:130
2737499133,2cyclictest6783-21sh23:35:160
2737499131,2cyclictest225-21systemd-journal19:37:300
2737499130,13cyclictest0-21swapper/022:10:010
2738399129,2cyclictest5523-21uniq19:30:163
2738399129,2cyclictest28012-21sed20:15:253
2738399129,2cyclictest2470-21cat19:25:003
2738399129,2cyclictest12317-21grep20:50:253
2738399129,2cyclictest11418-21sed19:40:263
2738399129,1cyclictest5900-21apt-get20:40:143
2738399129,1cyclictest25661-21sh21:12:543
2738399126,4cyclictest0-21swapper/321:22:023
2738099129,2cyclictest6519-21sort20:40:162
2738099129,2cyclictest5771-21grep21:20:172
2738099129,2cyclictest31395-21grep21:35:152
2738099129,2cyclictest23372-21wc23:25:222
2738099129,2cyclictest15527-21cat19:50:202
2738099126,2cyclictest225-21systemd-journal22:25:002
2738099126,2cyclictest225-21systemd-journal20:05:012
2738099124,4cyclictest225-21systemd-journal21:00:002
2738099123,4cyclictest25691-21grep20:10:272
27380991212,0cyclictest0-21swapper/200:35:162
27380991210,1cyclictest32-21ksoftirqd/222:31:472
27380991210,1cyclictest32-21ksoftirqd/221:27:592
27380991210,1cyclictest32-21ksoftirqd/221:27:592
2738099120,2cyclictest0-21swapper/223:06:092
2738099120,11cyclictest0-21swapper/222:15:512
2737499127,4cyclictest16585-21chrt19:54:500
2737499126,2cyclictest25567-21sort20:10:260
2737499120,2cyclictest225-21systemd-journal20:41:300
2738399118,2cyclictest31217-21basename20:25:133
2738399118,2cyclictest28817-21hostname20:20:143
2738399118,2cyclictest18737-21taskset21:05:193
2738399116,5cyclictest10569-21apache223:35:053
2738399114,5cyclictest14038-21latency20:55:193
2738399113,7cyclictest0-21swapper/320:00:253
2738099119,1cyclictest4237-21grep20:35:172
2738099118,2cyclictest9986-21ssh23:17:562
2738099118,2cyclictest5826-21cat19:30:182
2738099118,2cyclictest580-21cat19:20:162
2738099118,2cyclictest25275-21sh00:05:412
2738099118,2cyclictest19726-21chrt00:21:272
2738099118,2cyclictest17996-21ssh00:00:262
2738099118,2cyclictest11321-21cat19:40:262
2738099117,3cyclictest30767-21grep19:15:152
2738099111,4cyclictest225-21systemd-journal20:31:372
2738399107,2cyclictest31886-21taskset19:17:273
2738399107,2cyclictest2448-21fgrep20:30:223
2738399107,2cyclictest18670-21taskset19:56:063
2738399102,5cyclictest3936-21sensors_fan19:25:223
2738399101,6cyclictest10068-21taskset20:46:073
2738399101,5cyclictest23390-21runrttasks20:05:293
2738099108,2cyclictest32516-21sh21:55:172
2738099108,1cyclictest9026-21who19:35:262
2738099108,1cyclictest23854-21cat20:10:012
2738099108,1cyclictest139642sleep219:46:382
2738099108,1cyclictest127852sleep220:54:502
2738099107,2cyclictest3265-21expr19:25:162
2738099107,2cyclictest31449-21tr20:25:142
2738099107,2cyclictest27609-21apache_volume19:10:132
2738099100,9cyclictest8293-21cut20:45:142
2738099100,9cyclictest26593-21grep20:15:132
273809998,1cyclictest305542sleep220:20:362
273809994,2cyclictest225-21systemd-journal21:05:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional