You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 02:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Jun 12, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
136292178106,64sleep30-21swapper/307:09:573
136342177103,29sleep10-21swapper/107:09:591
133412172100,65sleep00-21swapper/007:06:120
133282159102,29sleep20-21swapper/207:06:012
49721020,1sleep00-21swapper/011:34:410
114832720,1sleep00-21swapper/009:40:550
304792590,2sleep334-21ksoftirqd/310:20:043
294532570,2sleep10-21swapper/109:55:011
190262560,2sleep00-21swapper/009:23:100
311842550,2sleep10-21swapper/111:32:531
143952540,3chrt0-21swapper/308:25:153
143952540,3chrt0-21swapper/308:25:143
282662520,1sleep00-21swapper/010:18:030
316062500,7sleep122-21ksoftirqd/109:32:061
164462500,1sleep10-21swapper/109:20:351
164402490,1sleep00-21swapper/008:30:130
125902490,1sleep00-21swapper/012:30:480
79452480,3sleep1211rcuc/110:50:511
279152480,7sleep228-21ksoftirqd/211:55:052
296962460,2sleep00-21swapper/009:30:240
72772450,1sleep00-21swapper/010:26:060
62612450,1sleep20-21swapper/209:13:542
267122440,2sleep10-21swapper/112:17:361
267122440,2sleep10-21swapper/112:17:351
13910994218,21cyclictest5274-21ssh11:13:402
232502410,2sleep10-21swapper/111:26:361
13910994122,17cyclictest0-21swapper/210:54:132
1391099390,8cyclictest28195-21rm10:42:062
13910993822,14cyclictest0-21swapper/212:10:582
13910993822,14cyclictest0-21swapper/210:28:492
13910993710,24cyclictest4189-21awk09:35:242
1391099361,32cyclictest26058-21grep07:35:292
1391099361,32cyclictest26058-21grep07:35:282
13910993610,23cyclictest20753-21seq11:49:462
1391099352,30cyclictest10660-21threads12:05:242
1391099351,23cyclictest0-21swapper/211:50:062
13910993512,20cyclictest29278-21sh11:30:572
13910993512,20cyclictest2876-21sh09:59:252
13910993511,22cyclictest30653-21seq09:31:092
1391099350,7cyclictest21762-21ssh10:13:142
1391099349,5cyclictest7760-21cat08:10:152
1391099343,2cyclictest0-21swapper/211:05:092
13910993418,14cyclictest0-21swapper/210:38:302
1391099341,7cyclictest21181-21/usr/sbin/munin11:00:272
13910993411,20cyclictest5662-21sh10:00:262
13910993411,20cyclictest5662-21sh10:00:252
13910993411,20cyclictest24587-21sh09:26:482
13910993411,20cyclictest17179-21sh10:34:272
1391099340,5cyclictest11787-21mailstats12:30:222
1391099339,4cyclictest20769-21expr08:40:152
1391099338,5cyclictest8484-21grep09:15:182
13910993314,16cyclictest30459-21switchtime07:45:262
13910993312,18cyclictest15677-21sh11:20:412
13910993311,19cyclictest9976-21sh11:40:382
13910993311,19cyclictest3277-21sh10:47:542
13910993311,19cyclictest15074-21sh10:56:172
13910993310,20cyclictest29096-21sh12:20:002
13910993310,20cyclictest29096-21sh12:20:002
225662320,3sleep00-21swapper/011:01:510
1391099328,6cyclictest32344-21ntp_states07:50:222
1391099324,8cyclictest23517-21grep07:30:262
1391099321,28cyclictest9213-21ssh12:29:092
1391099321,1cyclictest0-21swapper/212:24:152
13910993211,19cyclictest5869-21sh12:02:032
13910993210,20cyclictest21352-21sshd07:25:252
1390999320,29cyclictest23127-21sh12:39:361
1391099318,4cyclictest19877-21sh12:36:182
1391099316,5cyclictest14001-21/usr/sbin/munin07:10:122
1391099312,8cyclictest3636-21/usr/sbin/munin08:00:132
1391099311,9cyclictest6103-21/usr/sbin/munin08:05:192
1391099311,9cyclictest19455-21/usr/sbin/munin07:20:252
1391099311,8cyclictest9789-21/usr/sbin/munin08:15:132
1391099311,8cyclictest18958-21ssh09:23:072
1391099311,11cyclictest25361-21rm11:28:432
13910993111,17cyclictest634-21diskmemload09:45:342
13910993110,3cyclictest24484-21latency_hist08:50:012
13908993122,6cyclictest634-21diskmemload10:52:270
13908993118,8cyclictest21488-21sh09:25:120
1391099306,5cyclictest18861-21which08:35:172
1391099303,3cyclictest0-21swapper/207:15:112
1391099302,9cyclictest23643-21grep09:50:212
13910993022,7cyclictest0-21swapper/210:24:512
1391099301,8cyclictest5782-21ssh11:38:112
1391099301,8cyclictest13778-21/usr/sbin/munin08:25:002
1391099301,8cyclictest12560-21ssh09:42:012
1391099301,8cyclictest11103-21fschecks_time10:05:152
1391099301,10cyclictest32552-21sshd09:05:262
13910993010,17cyclictest26994-21sh10:16:462
13910993010,17cyclictest1478-21entropy07:55:132
48852290,1chrt0-21swapper/309:12:293
151262290,1sleep10-21swapper/109:44:351
13911992923,4cyclictest10591-21ssh11:17:183
1391199291,26cyclictest26543-21ssh11:53:553
1391099291,8cyclictest17186-21ps08:30:232
1391099291,7cyclictest30162-21mailstats09:00:242
1391099291,7cyclictest27340-21/usr/sbin/munin08:55:152
1391099291,7cyclictest14157-21grep08:25:132
1391099291,7cyclictest14157-21grep08:25:132
13910992910,17cyclictest636-21runrttasks08:51:532
13910992910,16cyclictest9840-21sh11:16:312
1390999291,26cyclictest21858-21sshd10:37:261
1390999290,27cyclictest6530-21ssh12:02:431
1390999290,26cyclictest15471-21ssh12:09:581
1390899293,2cyclictest131rcuc/011:15:320
13908992924,3cyclictest28795-21ssh10:42:420
13908992922,5cyclictest1641-21ssh11:59:460
230402280,2chrt0-21swapper/310:38:383
13911992820,5cyclictest634-21diskmemload10:48:043
1391099281,8cyclictest27584-21/usr/sbin/munin07:40:192
1390999281,3cyclictest4001-21ssh10:24:331
1390999281,25cyclictest634-21diskmemload10:49:031
1390999281,19cyclictest0-21swapper/110:30:531
1390999280,3cyclictest9962-21ssh11:16:401
1390899283,9cyclictest18643-21sh10:59:510
13908992825,2cyclictest131rcuc/011:09:520
1390899281,25cyclictest26275-21ssh12:17:090
1390899281,25cyclictest26275-21ssh12:17:080
1390899281,25cyclictest23461-21ssh12:39:550
1390899281,25cyclictest131rcuc/012:01:390
1390899281,25cyclictest0-21swapper/010:45:050
18832270,8sleep334-21ksoftirqd/309:34:483
13911992722,4cyclictest0-21swapper/311:48:383
1391199271,3cyclictest53050irq/142-enp3s0-12:16:243
1391199271,3cyclictest53050irq/142-enp3s0-12:16:233
1391199271,24cyclictest634-21diskmemload10:52:123
1391199271,24cyclictest21722-21ssh12:14:173
1391199270,25cyclictest1967-21ssh12:23:343
13909992723,2cyclictest0-21swapper/112:25:021
13909992722,3cyclictest7482-21ssh10:26:191
1390999271,2cyclictest0-21swapper/111:10:411
1390999271,25cyclictest0-21swapper/111:07:391
1390999271,24cyclictest0-21swapper/112:20:001
1390999271,24cyclictest0-21swapper/111:51:291
1390999271,24cyclictest0-21swapper/111:23:271
1390999271,1cyclictest0-21swapper/112:13:221
1390999271,1cyclictest0-21swapper/110:40:231
1390999271,19cyclictest0-21swapper/111:55:411
1390899273,6cyclictest12-21ksoftirqd/011:50:190
1390899272,8cyclictest634-21diskmemload11:36:190
13908992725,1cyclictest12-21ksoftirqd/010:36:330
13908992724,2cyclictest0-21swapper/012:13:050
13908992724,2cyclictest0-21swapper/010:34:360
13908992723,2cyclictest3609-21ssh10:24:100
13908992721,4cyclictest17952-21ssh11:22:590
1390899271,24cyclictest0-21swapper/011:10:080
13911992622,2cyclictest0-21swapper/310:29:183
13911992621,3cyclictest27870-21ssh11:55:033
13911992621,3cyclictest22636-21ssh11:25:593
1391199261,2cyclictest0-21swapper/312:01:463
1391199261,23cyclictest0-21swapper/312:28:403
1391199261,23cyclictest0-21swapper/311:08:463
1391199261,23cyclictest0-21swapper/310:40:243
1391199260,2cyclictest0-21swapper/311:10:083
1390999262,16cyclictest15991-21ssh09:45:111
1390999261,24cyclictest0-21swapper/111:35:391
1390999261,1cyclictest0-21swapper/112:30:071
13908992624,1cyclictest0-21swapper/012:08:250
1390899261,24cyclictest0-21swapper/012:28:140
1390899261,24cyclictest0-21swapper/011:49:230
1390899261,23cyclictest0-21swapper/011:25:080
1390899261,1cyclictest0-21swapper/012:23:530
125232260,3sleep112513-21apt-get10:55:141
13911992515,6cyclictest0-21swapper/310:00:133
13911992515,6cyclictest0-21swapper/310:00:133
13911992514,8cyclictest25066-21df_inode08:50:143
1391199251,23cyclictest0-21swapper/312:35:053
1391199251,23cyclictest0-21swapper/312:07:373
1391199251,23cyclictest0-21swapper/311:03:003
1391199251,23cyclictest0-21swapper/310:56:473
1391199251,23cyclictest0-21swapper/310:30:073
1391199250,23cyclictest0-21swapper/311:23:483
13909992523,1cyclictest0-21swapper/111:04:451
1390999251,1cyclictest0-21swapper/111:46:231
1390899254,7cyclictest29011-21/usr/sbin/munin09:00:130
80222240,1sleep00-21swapper/011:40:100
1391199241,1cyclictest0-21swapper/312:32:123
1391199240,23cyclictest0-21swapper/311:34:303
13909992414,7cyclictest560-21dbus-daemon08:05:011
1390899243,5cyclictest5658-21ssh09:36:510
1390899242,6cyclictest20520-21ssh10:11:590
13908992420,3cyclictest131rcuc/009:08:450
1390899241,6cyclictest0-21swapper/009:45:120
13908992414,6cyclictest52950irq/141-enp3s0-10:05:110
313092230,2sleep10-21swapper/109:05:131
1391199234,9cyclictest634-21diskmemload09:53:503
1391199233,18cyclictest17622-21unixbench_multi09:45:253
1390999235,4cyclictest0-21swapper/110:02:331
1390999235,4cyclictest0-21swapper/110:02:321
1390899232,6cyclictest9656-21ssh10:04:290
1390899232,6cyclictest9656-21ssh10:04:280
13908992320,2cyclictest131rcuc/008:49:350
13908992319,3cyclictest131rcuc/007:57:300
1390899231,7cyclictest12-21ksoftirqd/009:13:400
13908992316,4cyclictest6196-21load08:05:190
1390899230,7cyclictest26456-21seq07:38:540
1390899230,7cyclictest26456-21seq07:38:530
76492220,8sleep012-21ksoftirqd/009:15:040
1391199222,14cyclictest634-21diskmemload11:35:123
1391199221,8cyclictest81742tr11:40:123
13911992213,7cyclictest2620-21awk07:55:233
13911992213,6cyclictest278-21systemd-journal09:10:003
13911992210,10cyclictest31918-21ssh09:55:593
1390999221,8cyclictest22874-21cpuspeed_turbos09:50:151
13909992213,6cyclictest7903-21df08:10:161
13909992212,7cyclictest2844-21df09:35:121
13909992211,9cyclictest636-21runrttasks09:11:091
1390899221,8cyclictest29433-21ssh09:55:010
1390899221,8cyclictest22029-21runrttasks08:41:040
13908992218,3cyclictest131rcuc/008:21:420
13908992217,3cyclictest131rcuc/008:02:480
1391199215,3cyclictest165942grep08:30:153
13911992115,3cyclictest634-21diskmemload09:28:353
13911992113,5cyclictest53050irq/142-enp3s0-09:15:133
13911992112,7cyclictest24542-21sed10:15:163
1390999216,3cyclictest141rcu_preempt11:40:131
13909992112,7cyclictest560-21dbus-daemon07:40:001
13909992112,7cyclictest560-21dbus-daemon07:39:591
13909992112,7cyclictest17247-21apt-get10:10:091
1390899213,7cyclictest12-21ksoftirqd/007:24:550
13908992117,3cyclictest131rcuc/008:19:390
1390899211,6cyclictest26876-21apt-get08:55:120
1390899211,6cyclictest12-21ksoftirqd/009:52:020
13908992113,6cyclictest19129-21ls08:35:200
13908992112,6cyclictest32198-21cat07:50:200
1390899210,8cyclictest24178-21taskset07:32:050
1390899210,6cyclictest12-21ksoftirqd/008:25:240
1390899210,6cyclictest12-21ksoftirqd/008:25:230
13911992012,6cyclictest8980-21wc09:40:013
13911992012,6cyclictest560-21dbus-daemon08:40:013
13911992011,6cyclictest30509-21grep09:00:253
1390999208,10cyclictest0-21swapper/110:05:131
1390999204,4cyclictest141rcu_preempt08:00:001
13909992014,4cyclictest636-21runrttasks09:16:451
13909992012,6cyclictest28614-21seq08:58:181
13909992012,6cyclictest17747-21seq07:17:111
13909992012,6cyclictest10885-21awk08:15:241
1390999201,16cyclictest25257-21grep08:50:171
1390899201,6cyclictest12-21ksoftirqd/007:15:280
13908992016,3cyclictest131rcuc/008:10:350
13908992016,3cyclictest131rcuc/007:29:340
1391199191,9cyclictest0-21swapper/310:12:003
1391199191,9cyclictest0-21swapper/309:43:143
13911991912,5cyclictest27165-21expr08:55:133
13911991912,5cyclictest17499-21unixbench_multi07:15:273
13911991911,6cyclictest28215-21wc07:40:273
13911991911,6cyclictest24823-21expr07:35:153
13911991911,6cyclictest24823-21expr07:35:143
13911991911,6cyclictest23030-21tune2fs08:45:173
13911991911,6cyclictest19662-21unixbench_multi07:20:253
13911991911,6cyclictest11634-21ntpq10:05:203
1391199191,13cyclictest14558-21apt-get09:20:123
13911991910,6cyclictest9479-21basename08:15:003
1390999192,14cyclictest30713-21unixbench_multi07:45:271
13909991912,4cyclictest6094-21kernelversion08:05:191
13909991911,6cyclictest24710-21cut10:15:181
13909991911,6cyclictest20133-21cron07:24:591
13909991911,6cyclictest14269-21expr07:10:121
1390999191,15cyclictest21505-21ssh09:25:121
13908991916,2cyclictest131rcuc/007:11:350
13908991910,6cyclictest27827-21ntp_kernel_pll_07:40:220
13911991812,4cyclictest5188-21latency_hist08:05:013
13911991812,4cyclictest20584-21grep08:40:133
13911991811,5cyclictest7441-21date08:09:593
13911991811,5cyclictest32001-21idleruntime07:50:183
1390999188,7cyclictest31422-21apt-get07:50:131
13909991813,3cyclictest15908-21idleruntime-cro08:30:001
13909991813,3cyclictest15908-21idleruntime-cro08:29:591
13909991812,4cyclictest27817-21seq07:40:221
13909991811,5cyclictest23187-21seq08:45:181
13909991811,5cyclictest18971-21sed08:35:181
1390899183,5cyclictest12-21ksoftirqd/007:48:280
13908991814,3cyclictest131rcuc/008:50:050
1391199179,5cyclictest12291-21cut08:20:163
1391199175,9cyclictest13949-21apt-get07:10:113
1391199172,10cyclictest22535-21apt-get07:30:143
13911991711,4cyclictest10308-21basename08:15:163
13911991710,5cyclictest22237-21seq07:29:393
1390999179,6cyclictest22204-21seq08:43:221
1390999178,7cyclictest23216-21cut07:30:201
13909991710,5cyclictest21619-21sed07:25:281
1391199161,11cyclictest29285-21apt-get07:45:123
1390999169,5cyclictest29019-21basename09:00:121
1390999167,7cyclictest12731-21cut08:20:201
13909991612,2cyclictest636-21runrttasks08:34:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional