You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 01:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 29, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4066531998528,14cyclictest275-21dbus-daemon12:26:003
4066517997969,6cyclictest4066525-21grep07:10:020
4066517997533,36cyclictest0-21swapper/011:15:000
406651799654,10cyclictest121ksoftirqd/012:13:520
4066527996416,10cyclictest30-21ksoftirqd/209:20:012
406651799647,12cyclictest12-21ksoftirqd/011:20:010
4066517996329,29cyclictest12-21ksoftirqd/009:18:370
4066522996228,5cyclictest131rcu_preempt07:50:011
4066517996132,10cyclictest12-21ksoftirqd/009:41:200
4066527996012,11cyclictest30-21ksoftirqd/210:50:092
4066527995914,8cyclictest30-21ksoftirqd/212:30:002
4066527995911,11cyclictest30-21ksoftirqd/210:21:502
4066527995911,11cyclictest30-21ksoftirqd/210:21:502
4066527995911,11cyclictest30-21ksoftirqd/207:40:002
406652799589,8cyclictest30-21ksoftirqd/211:25:002
406651799588,5cyclictest131rcu_preempt12:38:450
4066527995715,9cyclictest30-21ksoftirqd/210:40:012
4066527995711,10cyclictest30-21ksoftirqd/211:57:172
406651799573,49cyclictest0-21swapper/009:55:020
4066517995727,14cyclictest4154342-21ssh12:19:020
4066517995726,15cyclictest4157309-21ssh12:25:270
406652799569,9cyclictest30-21ksoftirqd/211:01:132
406652799566,10cyclictest4095445-21ssh10:02:182
4066527995618,9cyclictest30-21ksoftirqd/211:39:552
4066527995618,9cyclictest30-21ksoftirqd/211:39:542
4066527995614,9cyclictest30-21ksoftirqd/211:45:012
406651799569,5cyclictest131rcu_preempt07:15:010
4066517995642,8cyclictest16950irq/224-800000009:59:590
4066527995545,6cyclictest4131022-21kworker/2:211:50:032
4066527995513,9cyclictest30-21ksoftirqd/209:42:012
406652799544,16cyclictest4073024-21ssh09:10:122
4066527995415,9cyclictest30-21ksoftirqd/209:34:292
4066527995412,8cyclictest30-21ksoftirqd/212:37:052
406651799543,45cyclictest4120178-21modprobe10:59:590
4066531995335,13cyclictest4069037-21cron08:00:003
4066527995343,6cyclictest4111623-21kworker/2:011:00:002
4066527995316,12cyclictest4072704-21diskmemload12:21:222
406652299537,7cyclictest131rcu_preempt11:00:021
4066522995324,19cyclictest23-21ksoftirqd/112:05:011
406651799538,6cyclictest131rcu_preempt07:30:010
406651799538,27cyclictest4129713-21rm11:21:380
406651799535,30cyclictest151rcuc/008:35:020
406651799535,11cyclictest12-21ksoftirqd/011:34:590
4066517995344,5cyclictest16950irq/224-800000009:40:010
4066517995327,9cyclictest12-21ksoftirqd/007:39:050
406652799527,10cyclictest30-21ksoftirqd/211:14:042
406652799520,15cyclictest4070919-21cat08:35:012
406652799520,14cyclictest4069635-21latency_hist08:10:012
406651799527,7cyclictest12-21ksoftirqd/009:26:040
4066517995244,6cyclictest12-21ksoftirqd/009:25:010
4066517995235,13cyclictest4069330-21munin-run08:05:000
4066517995226,11cyclictest4144345-21ssh11:55:300
4066517995224,13cyclictest4132649-21ssh11:28:430
4066517995224,13cyclictest4132649-21ssh11:28:420
4066517995219,9cyclictest12-21ksoftirqd/010:08:550
4066517995219,30cyclictest12-21ksoftirqd/010:52:300
4066517995218,11cyclictest12-21ksoftirqd/012:35:010
4066531995136,11cyclictest275-21dbus-daemon11:55:003
406652799516,10cyclictest30-21ksoftirqd/209:56:192
4066527995117,9cyclictest30-21ksoftirqd/209:30:002
4066527995117,12cyclictest30-21ksoftirqd/212:14:102
406652799511,14cyclictest4107735-21ssh10:30:352
4066527995110,7cyclictest30-21ksoftirqd/212:05:572
4066522995141,6cyclictest23-21ksoftirqd/111:14:461
406651799517,10cyclictest12-21ksoftirqd/011:40:020
406651799517,10cyclictest12-21ksoftirqd/011:40:020
4066517995144,4cyclictest16950irq/224-800000008:00:000
4066517995125,12cyclictest4112532-21ssh10:41:540
4066517995125,11cyclictest4123620-21ssh11:07:430
4066517995125,11cyclictest4099840-21ssh10:12:360
4066517995122,13cyclictest4102035-21ssh10:17:450
4066517995116,31cyclictest12-21ksoftirqd/009:31:380
4066527995012,8cyclictest0-21swapper/211:26:572
4066527995012,8cyclictest0-21swapper/211:26:572
4066527995011,9cyclictest0-21swapper/209:40:012
4066527995010,9cyclictest30-21ksoftirqd/209:45:132
406652299503,5cyclictest131rcu_preempt11:46:591
4066522995031,15cyclictest1-21systemd07:40:001
406651799505,13cyclictest12-21ksoftirqd/010:01:230
4066517995022,12cyclictest4143223-21ssh11:53:110
4066517995014,13cyclictest12-21ksoftirqd/011:40:310
4066517995012,31cyclictest4150447-21cat12:10:020
4066517995011,35cyclictest4122605-21cat11:05:030
406652799497,9cyclictest30-21ksoftirqd/212:33:572
406652799496,7cyclictest30-21ksoftirqd/212:17:002
4066527994931,14cyclictest1-21systemd08:50:012
4066527994914,8cyclictest30-21ksoftirqd/211:16:562
4066522994933,12cyclictest275-21dbus-daemon12:20:011
406651799492,12cyclictest12-21ksoftirqd/010:40:010
4066517994911,6cyclictest131rcu_preempt12:21:210
406652799489,8cyclictest0-21swapper/207:55:002
406652799486,8cyclictest30-21ksoftirqd/210:16:162
406652799485,9cyclictest30-21ksoftirqd/211:32:092
406652799485,5cyclictest30-21ksoftirqd/209:53:412
4066527994834,11cyclictest30-21ksoftirqd/209:20:022
4066527994834,10cyclictest1-21systemd09:10:012
4066527994830,13cyclictest275-21dbus-daemon08:55:012
406652799482,16cyclictest364-21snmpd11:54:582
4066527994812,6cyclictest30-21ksoftirqd/210:48:232
4066522994830,13cyclictest275-21dbus-daemon09:25:011
406651799482,41cyclictest3467602-21systemd-journal10:25:010
406651799482,41cyclictest3467602-21systemd-journal10:25:000
4066517994823,10cyclictest4140341-21ssh11:46:080
4066517994822,11cyclictest4073798-21ssh09:12:210
4066517994813,6cyclictest131rcu_preempt07:25:010
4066527994728,13cyclictest275-21dbus-daemon08:25:012
4066527994721,17cyclictest275-21dbus-daemon07:45:012
4066527994711,6cyclictest0-21swapper/210:41:582
4066522994732,11cyclictest275-21dbus-daemon09:05:011
4066517994738,6cyclictest12-21ksoftirqd/012:02:100
4066517994737,8cyclictest12-21ksoftirqd/007:20:000
406653199464,26cyclictest4111552-21kworker/u8:311:05:023
406653199463,4cyclictest131rcu_preempt10:56:053
4066531994633,10cyclictest3467602-21systemd-journal12:10:033
4066531994630,10cyclictest275-21dbus-daemon10:05:023
406653199461,3cyclictest131rcu_preempt09:13:383
406652799468,7cyclictest0-21swapper/210:26:412
4066522994629,12cyclictest120-21kswapd011:10:011
4066522994621,21cyclictest23-21ksoftirqd/112:23:041
406651799466,37cyclictest12-21ksoftirqd/010:27:020
406651799465,11cyclictest12-21ksoftirqd/008:50:010
4066517994629,14cyclictest4108354-21ssh10:32:130
4066517994610,6cyclictest131rcu_preempt07:45:020
4066531994529,13cyclictest37-21ksoftirqd/312:25:013
406653199452,3cyclictest131rcu_preempt07:20:023
406653199451,3cyclictest131rcu_preempt09:44:503
406652299459,5cyclictest131rcu_preempt09:49:591
406652299459,20cyclictest4066513-21cyclictest08:40:011
4066522994533,10cyclictest23-21ksoftirqd/110:05:011
4066522994521,17cyclictest275-21dbus-daemon07:55:001
406651799455,5cyclictest131rcu_preempt09:49:060
4066517994528,12cyclictest3467602-21systemd-journal07:49:590
4066517994526,13cyclictest3467602-21systemd-journal09:00:000
406651799451,10cyclictest12-21ksoftirqd/009:10:020
406653199443,3cyclictest131rcu_preempt11:10:283
4066531994423,16cyclictest4154739-21cron12:19:593
406652799444,8cyclictest30-21ksoftirqd/211:06:542
4066527994437,5cyclictest4072210-21kworker/2:210:07:552
4066527994411,8cyclictest0-21swapper/210:11:582
406652299447,13cyclictest23-21ksoftirqd/112:30:001
4066522994429,11cyclictest23-21ksoftirqd/109:28:251
4066522994426,13cyclictest4111545-21cron10:39:591
4066522994425,13cyclictest275-21dbus-daemon10:55:011
406651799449,7cyclictest131rcu_preempt10:45:570
406653199433,3cyclictest131rcu_preempt10:46:273
4066531994331,9cyclictest4079176-21cron09:25:003
4066531994331,9cyclictest3467602-21systemd-journal08:38:053
4066531994327,11cyclictest309-21in:imuxsock09:00:023
406653199432,3cyclictest131rcu_preempt10:14:163
4066531994320,6cyclictest4135771-21sh11:35:413
4066531994320,6cyclictest4135771-21sh11:35:403
406653199431,3cyclictest131rcu_preempt10:34:593
406652799433,9cyclictest30-21ksoftirqd/207:25:002
4066527994335,5cyclictest4067017-21kworker/2:108:40:022
406652299437,5cyclictest131rcu_preempt12:00:011
4066522994333,8cyclictest23-21ksoftirqd/109:00:021
4066522994333,7cyclictest23-21ksoftirqd/110:17:531
4066522994332,8cyclictest23-21ksoftirqd/109:30:051
4066522994332,7cyclictest4066523-21latency_hist07:10:011
4066522994331,9cyclictest23-21ksoftirqd/109:10:011
4066522994331,8cyclictest309-21in:imuxsock11:25:011
4066522994330,9cyclictest3467602-21systemd-journal10:45:011
4066522994327,11cyclictest4109610-21wc10:35:021
406652299431,5cyclictest131rcu_preempt07:45:021
4066522994310,6cyclictest131rcu_preempt08:55:001
406651799435,21cyclictest19350irq/34-mmc108:25:030
406651799434,8cyclictest12-21ksoftirqd/008:20:030
4066517994312,5cyclictest131rcu_preempt07:35:000
4066531994229,10cyclictest37-21ksoftirqd/310:10:013
4066531994229,10cyclictest1-21systemd07:35:003
4066531994227,12cyclictest1-21systemd08:10:013
4066531994226,11cyclictest275-21dbus-daemon08:15:013
4066531994226,11cyclictest275-21dbus-daemon08:15:013
406653199422,3cyclictest131rcu_preempt09:26:573
406653199422,12cyclictest4081734-21ssh09:30:283
4066527994221,16cyclictest1-21systemd07:35:012
406652799421,19cyclictest0-21swapper/208:20:012
4066527994210,7cyclictest0-21swapper/212:02:042
406652299428,6cyclictest131rcu_preempt11:35:471
406652299428,6cyclictest131rcu_preempt11:35:471
4066522994220,15cyclictest275-21dbus-daemon07:20:021
4066522994218,21cyclictest23-21ksoftirqd/109:45:021
406652299421,4cyclictest131rcu_preempt10:30:011
4066522994213,7cyclictest131rcu_preempt10:11:051
4066517994227,12cyclictest12-21ksoftirqd/008:27:060
4066517994222,13cyclictest3467602-21systemd-journal09:05:010
406651799421,8cyclictest12-21ksoftirqd/008:35:020
4066531994126,12cyclictest275-21dbus-daemon09:00:013
4066531994126,11cyclictest37-21ksoftirqd/309:56:183
4066531994125,13cyclictest37-21ksoftirqd/309:50:293
4066531994125,12cyclictest275-21dbus-daemon09:40:013
406652799415,6cyclictest0-21swapper/207:50:022
406652799414,6cyclictest4066887-21cat07:15:022
406652799411,8cyclictest4070624-21latency_hist08:30:022
406652299417,5cyclictest131rcu_preempt11:41:021
406652299416,5cyclictest131rcu_preempt09:50:361
4066522994131,8cyclictest23-21ksoftirqd/109:17:281
4066522994131,7cyclictest23-21ksoftirqd/111:51:481
4066522994130,8cyclictest4071395-21sshd08:44:051
4066522994129,7cyclictest23-21ksoftirqd/109:13:091
4066522994122,16cyclictest275-21dbus-daemon11:20:001
4066522994117,20cyclictest23-21ksoftirqd/112:33:491
4066522994117,19cyclictest23-21ksoftirqd/112:35:481
4066522994112,6cyclictest131rcu_preempt12:10:011
4066517994124,13cyclictest4069809-21sshd08:12:510
4066517994124,13cyclictest4069809-21sshd08:12:500
4066531994032,5cyclictest4163418-21sh12:40:023
4066531994032,4cyclictest4068804-21sh07:55:013
4066531994027,10cyclictest37-21ksoftirqd/310:37:353
4066531994025,12cyclictest37-21ksoftirqd/310:25:293
4066531994025,11cyclictest37-21ksoftirqd/311:48:553
4066531994016,17cyclictest275-21dbus-daemon08:50:013
406653199401,3cyclictest131rcu_preempt11:15:253
406652799407,7cyclictest30-21ksoftirqd/207:20:012
4066527994028,9cyclictest309-21in:imuxsock08:12:512
4066527994028,9cyclictest309-21in:imuxsock08:12:512
406652299407,6cyclictest131rcu_preempt07:30:021
406652299404,3cyclictest131rcu_preempt09:58:091
4066522994018,4cyclictest131rcu_preempt11:27:291
4066522994018,4cyclictest131rcu_preempt11:27:291
4066522994018,19cyclictest23-21ksoftirqd/110:06:441
4066517994011,6cyclictest131rcu_preempt07:55:030
4066531993929,8cyclictest37-21ksoftirqd/310:54:583
4066531993928,9cyclictest37-21ksoftirqd/312:07:413
4066531993926,10cyclictest4139661-21diskmemload11:45:013
4066531993925,10cyclictest3467602-21systemd-journal09:20:003
406653199392,3cyclictest131rcu_preempt10:41:073
406653199391,4cyclictest131rcu_preempt11:34:233
4066527993928,8cyclictest3467602-21systemd-journal09:03:012
4066527993925,7cyclictest309-21in:imuxsock08:00:012
4066522993930,7cyclictest23-21ksoftirqd/110:49:201
4066522993926,10cyclictest23-21ksoftirqd/112:15:001
4066522993925,11cyclictest275-21dbus-daemon08:00:021
4066522993925,10cyclictest4070260-21wc08:20:041
4066522993924,12cyclictest23-21ksoftirqd/110:22:431
4066522993924,12cyclictest23-21ksoftirqd/110:22:421
4066522993924,11cyclictest23-21ksoftirqd/107:35:001
406652299391,5cyclictest131rcu_preempt11:05:011
4066517993928,9cyclictest12-21ksoftirqd/008:07:480
406651799391,5cyclictest0-21swapper/008:41:030
4066531993831,4cyclictest4145217-21sh11:57:463
4066531993828,7cyclictest4072885-21latency_hist09:10:033
4066531993823,11cyclictest1-21systemd12:35:003
4066531993822,11cyclictest3467602-21systemd-journal08:54:593
4066531993817,15cyclictest3467602-21systemd-journal07:15:013
406652299383,5cyclictest131rcu_preempt08:30:001
4066522993828,8cyclictest23-21ksoftirqd/108:15:021
4066522993828,8cyclictest23-21ksoftirqd/108:15:021
4066522993828,5cyclictest23-21ksoftirqd/108:35:011
406652299381,9cyclictest4069390-21latency_hist08:05:011
4066522993811,10cyclictest23-21ksoftirqd/107:15:011
4066531993729,5cyclictest4132438-21sh11:28:133
4066531993729,5cyclictest4132438-21sh11:28:123
4066531993726,9cyclictest4147095-21sshd12:02:043
4066531993726,5cyclictest309-21in:imuxsock07:40:013
4066531993724,10cyclictest37-21ksoftirqd/310:19:043
406653199372,3cyclictest131rcu_preempt10:22:523
406653199372,3cyclictest131rcu_preempt10:22:513
406652799377,6cyclictest131rcu_preempt08:45:002
4066527993725,7cyclictest4069328-21munin-plugin-st08:05:002
4066522993728,7cyclictest23-21ksoftirqd/109:39:231
4066522993725,10cyclictest23-21ksoftirqd/107:25:011
406652299371,9cyclictest3467602-21systemd-journal08:24:591
4066517993714,8cyclictest12-21ksoftirqd/008:54:040
406653199366,24cyclictest4067398-21wc07:25:023
406653199364,3cyclictest131rcu_preempt11:07:353
4066527993619,12cyclictest30-21ksoftirqd/209:00:002
406652299367,5cyclictest131rcu_preempt11:32:071
4066522993626,8cyclictest23-21ksoftirqd/108:10:011
406653199354,6cyclictest131rcu_preempt08:30:013
406653199354,4cyclictest131rcu_preempt08:20:003
406653199354,3cyclictest131rcu_preempt11:20:513
406653199346,3cyclictest131rcu_preempt08:35:023
4066531993421,9cyclictest3467602-21systemd-journal08:20:033
4066531993420,10cyclictest37-21ksoftirqd/308:41:013
4066527993425,5cyclictest4066604-21latency_hist07:10:022
406653199334,3cyclictest131rcu_preempt09:45:053
4066531993325,5cyclictest37-21ksoftirqd/307:10:013
4066531993323,7cyclictest4069316-21cron08:04:593
4066527993319,9cyclictest364-21snmpd07:28:132
406653199323,4cyclictest131rcu_preempt07:45:003
406653199322,3cyclictest131rcu_preempt07:49:593
4066522993121,6cyclictest23-21ksoftirqd/108:50:031
4066531992921,6cyclictest37-21ksoftirqd/307:30:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional