You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 08:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jan 10, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
352755899893,83cyclictest0-21swapper/019:15:030
352755899891,86cyclictest0-21swapper/022:35:020
352755899891,86cyclictest0-21swapper/022:35:020
3527558997773,2cyclictest0-21swapper/021:15:020
3527558997773,2cyclictest0-21swapper/021:15:020
3527558997725,8cyclictest3579619-21ssh22:45:590
3527558997725,8cyclictest3579619-21ssh22:45:590
352755899771,3cyclictest0-21swapper/021:25:020
352757099733,4cyclictest131rcu_preempt00:20:013
3527570997225,15cyclictest3550449-21latency_hist21:45:023
3527562996722,18cyclictest3558434-21sh22:01:361
3527570996513,6cyclictest131rcu_preempt20:00:003
3527558996335,21cyclictest19350irq/34-mmc123:35:010
3527558996238,19cyclictest16850irq/134-800000023:59:590
3527558996238,19cyclictest16850irq/134-800000000:00:000
3527566996146,10cyclictest3614581-21sh23:59:592
3527566996146,10cyclictest3614581-21sh23:59:592
3527562996125,6cyclictest3619290-21/usr/sbin/munin00:10:001
352757099605,11cyclictest3614636-21sendmail-msp00:00:013
352757099605,11cyclictest3614636-21sendmail-msp00:00:003
352755899601,48cyclictest3552781-21ssh21:50:020
352755899601,48cyclictest3552781-21ssh21:50:010
3527562995919,12cyclictest3632735-21rm00:38:221
3527558995932,12cyclictest3605716-21ssh23:40:550
3527570995841,13cyclictest1-21systemd20:25:003
3527562995819,12cyclictest0-21swapper/122:23:381
3527558995827,15cyclictest3577406-21ssh22:41:160
3527562995730,21cyclictest3530590-21cron20:09:591
3527558995736,14cyclictest275-21dbus-daemon21:00:000
3527558995720,33cyclictest16850irq/134-800000000:24:070
3527558995720,33cyclictest16850irq/134-800000000:24:060
3527558995719,34cyclictest16850irq/134-800000023:12:200
3527570995616,7cyclictest131rcu_preempt23:15:023
3527566995636,15cyclictest3467602-21systemd-journal00:30:002
3527558995624,30cyclictest12-21ksoftirqd/022:24:440
3527558995618,34cyclictest16850irq/134-800000022:59:510
3527562995533,19cyclictest3592592-21ssh23:13:241
3527562995517,12cyclictest0-21swapper/122:36:021
3527558995544,5cyclictest16950irq/224-800000021:40:000
352755899553,19cyclictest3556846-21sh21:58:310
352755899553,19cyclictest3556846-21sh21:58:310
3527558995514,32cyclictest16850irq/134-800000021:54:120
3527558995512,8cyclictest12-21ksoftirqd/000:02:440
3527558995512,8cyclictest12-21ksoftirqd/000:02:440
3527562995440,11cyclictest3609682-21ssh23:49:431
3527562995436,14cyclictest3611137-21ssh23:52:321
352756299543,4cyclictest131rcu_preempt22:20:001
3527562995430,20cyclictest3538757-21ssh21:20:111
3527562995426,23cyclictest3557532-21sendmail-msp22:00:001
3527562995426,23cyclictest3557532-21sendmail-msp22:00:001
3527562995426,22cyclictest3588627-21sort23:05:031
3527562995426,22cyclictest3588627-21sort23:05:021
3527558995415,9cyclictest12-21ksoftirqd/023:19:130
3527570995344,4cyclictest37-21ksoftirqd/320:15:013
3527570995312,5cyclictest131rcu_preempt23:50:013
3527566995318,18cyclictest3533673-21kworker/2:221:20:022
3527562995338,12cyclictest3599425-21ssh23:27:441
3527562995333,15cyclictest3581702-21sh22:50:261
352755899534,18cyclictest3586718-21sh23:00:500
352755899534,18cyclictest3586718-21sh23:00:490
3527558995314,6cyclictest12-21ksoftirqd/021:19:010
3527562995234,11cyclictest3605109-21cron23:40:001
3527562995234,11cyclictest3605109-21cron23:40:001
3527562995233,15cyclictest364-21snmpd22:46:441
3527562995233,15cyclictest364-21snmpd22:46:441
3527562995224,24cyclictest3630488-21ssh00:33:331
3527558995243,6cyclictest151rcuc/000:20:000
3527558995224,14cyclictest3609796-21ssh23:49:590
3527558995216,31cyclictest16850irq/134-800000022:50:070
3527558995216,15cyclictest12-21ksoftirqd/023:22:590
3527558995213,36cyclictest16850irq/134-800000022:16:420
3527570995110,6cyclictest131rcu_preempt19:29:563
352756299517,33cyclictest0-21swapper/120:30:011
352756299515,7cyclictest131rcu_preempt00:00:001
352756299515,7cyclictest131rcu_preempt00:00:001
3527562995139,9cyclictest3551070-21ssh21:46:111
3527562995139,9cyclictest3551070-21ssh21:46:101
3527562995137,10cyclictest3530055-21sshd19:58:551
3527562995133,14cyclictest3542837-21ssh21:29:031
3527562995132,14cyclictest3532665-21cron20:50:001
3527562995129,18cyclictest3615187-21ssh00:00:591
3527562995129,18cyclictest3615187-21ssh00:00:581
3527562995125,21cyclictest3528273-21cron19:25:001
3527562995125,21cyclictest3528273-21cron19:25:001
3527562995123,23cyclictest3594819-21sh23:18:051
352755899517,8cyclictest12-21ksoftirqd/000:10:240
3527558995122,14cyclictest3562925-21ssh22:11:010
3527558995120,3cyclictest358571250kworker/u8:300:31:190
3527562995031,16cyclictest3598073-21ssh23:25:011
3527562995029,14cyclictest3572791-21ssh22:31:391
3527562995029,14cyclictest3572791-21ssh22:31:381
3527562995027,18cyclictest3619842-21ssh00:10:471
3527562995026,20cyclictest3623929-21ssh00:19:491
3527562995025,21cyclictest3585814-21sh22:59:201
3527558995041,5cyclictest12-21ksoftirqd/023:40:030
3527558995041,5cyclictest12-21ksoftirqd/023:40:020
352755899504,13cyclictest3533432-21sshd21:01:370
352755899504,13cyclictest3533432-21sshd21:01:360
3527566994933,11cyclictest1-21systemd00:05:022
3527566994933,11cyclictest1-21systemd00:05:012
3527562994937,10cyclictest3578373-21ssh22:43:381
3527562994936,10cyclictest3534944-21ssh21:12:311
3527562994936,10cyclictest3534944-21ssh21:12:301
3527562994932,14cyclictest3588936-21ssh23:05:291
3527562994932,14cyclictest3545333-21ssh21:34:241
3527562994931,15cyclictest3625927-21ssh00:24:001
3527562994931,15cyclictest3625927-21ssh00:23:591
3527562994931,14cyclictest3601275-21ssh23:31:301
3527562994930,13cyclictest3531118-21latency_hist20:20:001
3527562994930,13cyclictest3531118-21latency_hist20:20:001
3527562994928,17cyclictest3537947-21ssh21:18:561
3527562994927,18cyclictest3554176-21ssh21:52:471
3527562994924,21cyclictest3560833-21ssh22:06:411
3527562994924,21cyclictest3549028-21ssh21:41:531
3527562994924,19cyclictest3570013-21ssh22:25:401
352755899499,5cyclictest131rcu_preempt23:29:150
352755899497,10cyclictest3561291-21ssh22:07:420
352757099486,3cyclictest131rcu_preempt00:15:023
3527562994837,9cyclictest3627218-21ssh00:26:261
3527562994832,13cyclictest3605451-21ssh23:40:121
3527562994832,12cyclictest3529557-21sshd19:49:561
3527562994831,14cyclictest3547376-21ssh21:38:421
3527562994831,13cyclictest3529812-21cron19:55:001
3527562994830,16cyclictest3563282-21ssh22:11:501
352755899485,6cyclictest12-21ksoftirqd/019:36:560
3527558994832,10cyclictest0-21swapper/019:10:020
3527558994823,11cyclictest3575555-21ssh22:37:370
3527558994811,6cyclictest16850irq/134-800000023:09:590
3527558994810,32cyclictest3628703-21latency_hist00:30:010
352757099471,9cyclictest131rcu_preempt19:45:023
3527566994712,6cyclictest131rcu_preempt21:00:002
3527562994731,12cyclictest3533697-21cron21:09:591
3527562994727,16cyclictest275-21dbus-daemon20:55:011
3527562994724,18cyclictest3532176-21sendmail-msp20:40:011
352755899474,39cyclictest0-21swapper/000:40:020
3527558994733,11cyclictest12-21ksoftirqd/022:29:190
352755899472,40cyclictest3612199-21munin-run23:55:010
3527558994714,5cyclictest0-21swapper/021:27:550
3527570994623,19cyclictest3528939-21sed19:35:023
352756699469,11cyclictest30-21ksoftirqd/222:20:002
3527566994634,10cyclictest30-21ksoftirqd/200:15:092
3527566994629,13cyclictest1-21systemd21:25:012
3527566994627,14cyclictest275-21dbus-daemon22:50:022
3527566994627,14cyclictest275-21dbus-daemon22:50:012
3527566994625,15cyclictest3467602-21systemd-journal19:40:002
3527562994627,16cyclictest3530868-21munin-run20:15:001
3527562994625,17cyclictest3529299-21sshd19:42:561
3527562994624,18cyclictest3528695-21latency_hist19:30:011
352755899465,10cyclictest1-21systemd19:41:560
3527558994638,5cyclictest151rcuc/020:40:000
3527558994620,9cyclictest16850irq/134-800000000:09:590
352757099459,3cyclictest131rcu_preempt21:15:023
352757099459,3cyclictest131rcu_preempt21:15:023
3527570994531,10cyclictest37-21ksoftirqd/321:10:003
3527570994527,14cyclictest3633436-21cron00:39:593
3527566994535,7cyclictest30-21ksoftirqd/223:51:122
3527566994530,11cyclictest3559891-21systemd-run22:05:002
3527566994520,4cyclictest30-21ksoftirqd/221:27:362
3527562994533,9cyclictest3531383-21cron20:25:001
3527562994531,10cyclictest364-21snmpd21:02:561
3527562994531,10cyclictest364-21snmpd21:02:561
3527562994531,10cyclictest3527971-21sshd19:16:561
3527562994530,9cyclictest3467602-21systemd-journal20:05:001
3527562994530,9cyclictest3467602-21systemd-journal20:05:001
3527562994523,18cyclictest3529168-21latency_hist19:40:031
352755899457,6cyclictest151rcuc/021:30:450
3527558994534,8cyclictest3521033-21kworker/0:019:51:570
3527558994534,7cyclictest3558841-21ssh22:02:420
3527558994531,11cyclictest12-21ksoftirqd/021:10:010
3527558994529,12cyclictest3548839-21ssh21:41:280
352755899451,40cyclictest0-21swapper/020:40:020
3527558994512,29cyclictest0-21swapper/019:50:000
352757099444,8cyclictest37-21ksoftirqd/323:05:563
3527570994432,8cyclictest1-21systemd23:30:013
3527570994429,11cyclictest275-21dbus-daemon22:30:003
3527570994419,4cyclictest37-21ksoftirqd/320:53:563
3527566994435,6cyclictest30-21ksoftirqd/221:48:532
3527566994435,6cyclictest30-21ksoftirqd/221:48:532
3527566994433,8cyclictest30-21ksoftirqd/200:23:392
3527566994433,8cyclictest30-21ksoftirqd/200:23:392
3527566994431,8cyclictest30-21ksoftirqd/220:30:002
3527566994431,10cyclictest30-21ksoftirqd/223:11:522
3527566994421,19cyclictest30-21ksoftirqd/220:37:562
3527566994417,7cyclictest291rcuc/221:10:022
3527562994428,13cyclictest3532442-21munin-run20:45:011
3527562994428,13cyclictest3531869-21sshd20:31:311
3527570994331,6cyclictest3578969-21sh22:45:003
3527570994329,10cyclictest3556665-21sh21:58:073
3527570994329,10cyclictest3556665-21sh21:58:063
3527570994321,18cyclictest3467602-21systemd-journal22:30:023
3527570994321,18cyclictest3467602-21systemd-journal22:30:023
352756699437,4cyclictest3527549-21cyclictest23:49:552
352756699436,23cyclictest3607512-21sh23:45:012
3527566994333,8cyclictest30-21ksoftirqd/223:24:592
3527566994331,9cyclictest3576706-21ssh22:40:032
3527566994322,14cyclictest3605140-21systemctl23:40:002
3527566994322,14cyclictest3605140-21systemctl23:40:002
3527562994331,10cyclictest3527714-21sshd19:10:561
3527562994328,12cyclictest3528792-21munin-run19:35:001
3527562994324,15cyclictest364-21snmpd20:55:251
3527570994228,11cyclictest275-21dbus-daemon00:10:003
3527570994228,10cyclictest3629921-21sh00:32:173
3527570994227,11cyclictest37-21ksoftirqd/320:20:013
3527570994227,11cyclictest37-21ksoftirqd/320:20:003
3527570994227,11cyclictest3546089-21sh21:35:373
3527570994227,11cyclictest3467602-21systemd-journal21:05:013
3527570994227,11cyclictest3467602-21systemd-journal21:05:003
352757099421,36cyclictest275-21dbus-daemon19:20:003
3527566994234,6cyclictest30-21ksoftirqd/222:13:072
3527566994230,9cyclictest1-21systemd22:10:002
3527566994229,11cyclictest30-21ksoftirqd/221:40:232
3527566994228,11cyclictest30-21ksoftirqd/219:55:022
352755899426,6cyclictest12-21ksoftirqd/020:25:000
352755899424,11cyclictest12-21ksoftirqd/020:50:030
3527570994128,9cyclictest1-21systemd21:35:013
3527570994126,11cyclictest3597934-21sh23:24:413
3527570994125,13cyclictest1-21systemd22:40:013
3527570994125,12cyclictest37-21ksoftirqd/321:26:313
352757099411,14cyclictest3581581-21ssh22:50:083
352757099411,13cyclictest3594034-21ssh23:16:193
3527566994129,8cyclictest309-21in:imuxsock21:34:572
3527566994129,10cyclictest30-21ksoftirqd/223:02:502
3527566994129,10cyclictest30-21ksoftirqd/223:02:502
3527566994126,12cyclictest3467602-21systemd-journal20:20:562
3527566994126,12cyclictest275-21dbus-daemon23:25:492
352755899414,9cyclictest12-21ksoftirqd/019:35:000
3527558994124,12cyclictest3467602-21systemd-journal20:10:000
352757099407,6cyclictest131rcu_preempt21:50:193
352757099404,4cyclictest131rcu_preempt22:20:013
3527570994030,5cyclictest309-21in:imuxsock20:45:013
3527570994027,11cyclictest37-21ksoftirqd/300:05:003
3527570994027,11cyclictest37-21ksoftirqd/300:04:593
3527566994032,6cyclictest30-21ksoftirqd/222:55:222
3527558994025,12cyclictest3467602-21systemd-journal19:25:020
352757099396,5cyclictest131rcu_preempt22:50:013
352757099396,5cyclictest131rcu_preempt22:50:003
352757099396,25cyclictest3562264-21grep22:10:003
3527570993928,9cyclictest37-21ksoftirqd/322:58:073
3527570993927,10cyclictest37-21ksoftirqd/320:05:023
3527570993927,10cyclictest37-21ksoftirqd/320:05:013
3527570993927,10cyclictest3587579-21sshd23:02:563
3527570993927,10cyclictest3587579-21sshd23:02:553
3527570993926,10cyclictest3603979-21sh23:37:243
3527570993926,10cyclictest3603979-21sh23:37:233
3527570993926,10cyclictest3538396-21sh21:19:583
3527570993925,10cyclictest3568092-21sh22:21:443
352756699393,6cyclictest131rcu_preempt00:35:282
3527566993929,7cyclictest3536064-21latency_hist21:15:022
3527566993929,7cyclictest3536064-21latency_hist21:15:012
3527566993927,8cyclictest3530572-21sshd20:06:272
3527566993924,9cyclictest3530854-21munin-plugin-st20:15:002
3527566993924,11cyclictest3467602-21systemd-journal19:35:012
352756699392,32cyclictest3578979-21ssh22:45:002
3527566993922,12cyclictest3467602-21systemd-journal23:35:002
3527566993922,12cyclictest1-21systemd19:15:012
3527558993931,5cyclictest151rcuc/020:15:010
3527558993926,10cyclictest12-21ksoftirqd/020:02:560
3527558993926,10cyclictest12-21ksoftirqd/020:02:560
3527558993924,12cyclictest3528256-21sshd19:21:190
3527558993924,12cyclictest3528256-21sshd19:21:180
3527558993917,15cyclictest12-21ksoftirqd/020:35:010
3527570993826,9cyclictest0-21swapper/322:00:263
3527570993826,8cyclictest3531710-21wc20:30:013
352757099382,3cyclictest131rcu_preempt23:44:303
3527566993829,7cyclictest30-21ksoftirqd/221:52:152
3527566993827,8cyclictest30-21ksoftirqd/200:10:122
3527566993826,9cyclictest30-21ksoftirqd/221:59:322
3527566993826,9cyclictest30-21ksoftirqd/221:59:312
3527566993825,9cyclictest3532935-21systemd-run20:55:012
352756699382,4cyclictest131rcu_preempt00:06:582
3527566993824,12cyclictest30-21ksoftirqd/220:45:022
352756699381,4cyclictest131rcu_preempt22:23:562
3527566993814,7cyclictest131rcu_preempt23:09:002
3527566993810,5cyclictest131rcu_preempt00:35:012
352755899382,4cyclictest3530079-21modprobe20:00:000
3527570993725,9cyclictest3564871-21sed22:15:023
3527570993723,11cyclictest37-21ksoftirqd/321:46:023
3527570993723,11cyclictest37-21ksoftirqd/321:46:013
352757099371,5cyclictest3602749-21ssh23:35:013
352756699375,6cyclictest131rcu_preempt21:04:572
352756699375,6cyclictest131rcu_preempt21:04:562
3527566993728,5cyclictest3530337-21cron20:05:012
3527566993728,5cyclictest3530337-21cron20:05:002
352756699372,7cyclictest0-21swapper/222:52:292
3527566993725,9cyclictest30-21ksoftirqd/220:35:022
3527566993724,10cyclictest30-21ksoftirqd/221:38:292
3527562993722,10cyclictest3527597-21cat19:10:031
352755899373,5cyclictest3527761-21munin-run19:15:010
3527558993725,8cyclictest3531143-21sendmail-msp20:20:010
3527558993725,8cyclictest3531143-21sendmail-msp20:20:000
3527558993724,10cyclictest12-21ksoftirqd/020:30:000
352755899371,7cyclictest12-21ksoftirqd/020:54:570
352757099365,11cyclictest0-21swapper/321:21:113
3527570993621,11cyclictest275-21dbus-daemon20:10:003
3527570993619,13cyclictest275-21dbus-daemon19:40:013
3527570993618,15cyclictest3627928-21rm00:28:153
352757099361,4cyclictest3624554-21modprobe00:20:533
352757099361,4cyclictest3624554-21modprobe00:20:523
3527566993624,9cyclictest3532725-21date20:50:022
3527566993623,10cyclictest30-21ksoftirqd/222:31:562
3527566993623,10cyclictest30-21ksoftirqd/222:31:552
352756699361,5cyclictest131rcu_preempt19:25:012
352756699361,5cyclictest131rcu_preempt19:25:012
3527566993611,5cyclictest131rcu_preempt19:16:562
3527570993527,5cyclictest274-21cron23:55:003
3527570993523,9cyclictest37-21ksoftirqd/321:00:003
3527570993521,9cyclictest3467602-21systemd-journal19:25:013
3527570993521,9cyclictest3467602-21systemd-journal19:25:013
3527570993521,10cyclictest3467602-21systemd-journal19:14:563
352756699353,11cyclictest131rcu_preempt23:16:402
3527566993525,7cyclictest30-21ksoftirqd/220:00:042
3527566993515,4cyclictest30-21ksoftirqd/222:29:352
352756699344,7cyclictest131rcu_preempt19:10:022
3527570993323,6cyclictest3527638-21wc19:10:023
352756699334,5cyclictest131rcu_preempt20:18:562
352756699334,5cyclictest131rcu_preempt20:18:562
3527566993319,11cyclictest3467602-21systemd-journal19:45:012
3527566993313,16cyclictest275-21dbus-daemon19:50:002
3527570993219,9cyclictest3467602-21systemd-journal19:55:013
3527570993218,9cyclictest3467602-21systemd-journal20:38:563
3527570993117,8cyclictest37-21ksoftirqd/320:50:003
3527566993121,5cyclictest30-21ksoftirqd/219:30:002
352757099304,21cyclictest3529575-21grep19:50:003
352757099293,20cyclictest3532085-21latency_hist20:35:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional