You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 22:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 15, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39717339911670,41cyclictest12-21ksoftirqd/011:45:020
397173399972,81cyclictest0-21swapper/007:20:020
397173399887,78cyclictest16950irq/224-800000011:25:030
397173899842,48cyclictest4061582-21ssh12:23:422
397173899821,46cyclictest4023781-21ssh10:55:482
397173899821,43cyclictest3976854-21systemd-run08:50:002
397173899801,46cyclictest30-21ksoftirqd/211:08:012
397173899791,40cyclictest4032944-21rm11:17:142
3971738997826,20cyclictest0-21swapper/211:30:002
3971733997769,3cyclictest0-21swapper/011:35:030
3971733997718,41cyclictest12-21ksoftirqd/010:38:220
397173899761,68cyclictest398923950ssh09:35:572
397173899752,44cyclictest3467602-21systemd-journal12:37:002
397173899752,44cyclictest3467602-21systemd-journal12:36:592
397173899682,62cyclictest3992363-21ssh09:43:262
3971738996824,39cyclictest4009168-21sh10:22:042
397173899671,62cyclictest3985663-21ssh09:27:442
397173899671,62cyclictest3985663-21ssh09:27:432
397173899671,43cyclictest3977852-21sshd09:08:122
397173899661,61cyclictest4058324-21ssh12:15:562
397173899661,61cyclictest4052154-21ssh12:01:392
397173899661,61cyclictest4043423-21ssh11:41:222
397173899661,61cyclictest4035745-21ssh11:23:552
397173899661,61cyclictest3997951-21ssh09:56:032
397173899661,61cyclictest3997056-21ssh09:54:162
397173899661,61cyclictest3994706-21ssh09:48:552
397173899661,42cyclictest4005257-21ssh10:13:172
3971733996618,5cyclictest12-21ksoftirqd/010:08:560
397173899651,60cyclictest4065544-21ssh12:32:512
397173899651,60cyclictest4063392-21ssh12:27:462
397173899651,60cyclictest4054036-21ssh12:05:542
397173899651,60cyclictest4040900-21ssh11:35:312
397173899651,60cyclictest4025787-21ssh11:00:272
397173899651,60cyclictest4022242-21ssh10:52:302
397173899651,60cyclictest4007049-21ssh10:17:162
397173899651,60cyclictest4001880-21ssh10:05:032
397173899651,60cyclictest3982995-21ssh09:21:252
397173899651,59cyclictest4010690-21ssh10:25:252
397173899651,41cyclictest3973705-21sshd07:48:292
397173899651,37cyclictest4001022-21ssh10:03:222
397173899642,58cyclictest3972943-21munin-run07:35:012
397173899641,59cyclictest4056312-21ssh12:11:132
397173899641,59cyclictest4049675-21ssh11:55:522
397173899641,59cyclictest4045545-21ssh11:46:212
397173899641,59cyclictest4039976-21ssh11:33:432
397173899641,59cyclictest4030040-21ssh11:10:152
397173899641,59cyclictest4020184-21ssh10:47:382
397173899641,59cyclictest4015696-21ssh10:37:112
397173899641,59cyclictest4013452-21ssh10:32:032
397173899641,59cyclictest3987170-21ssh09:31:072
397173899641,59cyclictest3979012-21ssh09:12:182
3971733996419,4cyclictest12-21ksoftirqd/009:59:030
397173899632,57cyclictest4047763-21ssh11:51:252
397173899631,58cyclictest4017449-21ssh10:41:112
397173899631,58cyclictest3980648-21ssh09:15:572
397173899631,58cyclictest3980648-21ssh09:15:562
397173899631,58cyclictest3972128-21sshd07:17:112
397173899631,57cyclictest3974209-21sshd07:58:302
397173399635,36cyclictest16950irq/224-800000010:25:010
3971742996212,6cyclictest131rcu_preempt09:56:543
397173899622,55cyclictest3977626-21munin-run09:05:002
397173899622,53cyclictest3974778-21munin-run08:10:002
397173899621,57cyclictest3975234-21sshd08:16:102
3971733996212,41cyclictest12-21ksoftirqd/012:30:390
397173899612,54cyclictest3973170-21sshd07:38:112
3971738996116,39cyclictest1-21systemd08:05:012
397173899611,55cyclictest48-21kcompactd008:13:022
397173899611,55cyclictest48-21kcompactd008:13:012
3971742996027,9cyclictest37-21ksoftirqd/311:35:003
397173899601,55cyclictest3976561-21sshd08:41:112
3971733996019,37cyclictest3988886-21latency_hist09:35:020
3971738995938,15cyclictest3467602-21systemd-journal08:35:002
397173899591,54cyclictest3975532-21sshd08:23:122
397173899591,54cyclictest3972402-21sshd07:20:102
3971733995924,3cyclictest131rcu_preempt12:10:460
397173899582,52cyclictest364-21snmpd08:53:162
397173899581,54cyclictest3973949-21sshd07:52:102
397173899581,52cyclictest3971898-21munin-run07:15:002
3971733995827,15cyclictest4006643-21ssh10:16:100
397173899571,51cyclictest3972653-21sshd07:25:112
397173399573,38cyclictest3995285-21ssh09:50:020
3971733995726,15cyclictest4026195-21ssh11:01:240
3971742995612,12cyclictest37-21ksoftirqd/308:05:003
397173899562,49cyclictest3976294-21modprobe08:38:342
397173899561,50cyclictest3975811-21munin-run08:30:002
397173899561,50cyclictest3975811-21munin-run08:30:002
3971733995630,9cyclictest4023458-21ssh10:55:030
3971733995628,12cyclictest4061013-21ssh12:22:120
3971733995626,15cyclictest4063650-21ssh12:28:240
397174299559,5cyclictest131rcu_preempt12:07:373
3971742995539,12cyclictest3467602-21systemd-journal11:45:003
3971742995515,6cyclictest131rcu_preempt12:29:493
397173899551,50cyclictest364-21snmpd08:56:372
397173899551,49cyclictest3973442-21sshd07:41:112
3971733995514,36cyclictest3467602-21systemd-journal10:55:010
397174299549,5cyclictest131rcu_preempt11:02:393
3971742995439,12cyclictest37-21ksoftirqd/310:15:193
3971733995426,13cyclictest3982826-21ssh09:21:010
3971733995420,16cyclictest3982264-21ssh09:20:010
3971733995420,16cyclictest3982264-21ssh09:20:000
397173399541,9cyclictest4029939-21latency_hist11:10:030
397173399541,48cyclictest4021194-21cat10:50:010
3971742995312,9cyclictest131rcu_preempt10:25:013
3971736995335,14cyclictest3467602-21systemd-journal08:15:001
3971736995335,14cyclictest3467602-21systemd-journal08:15:001
397173399533,15cyclictest4038407-21latency_hist11:30:010
3971733995327,22cyclictest3977932-21latency_hist09:10:010
3971733995327,10cyclictest4013580-21ssh10:32:210
3971733995324,14cyclictest3979407-21ssh09:13:230
3971733995322,27cyclictest4018255-21sh10:43:180
3971736995232,15cyclictest275-21dbus-daemon10:10:011
3971736995213,27cyclictest3971608-21kworker/u8:307:30:021
3971733995241,8cyclictest12-21ksoftirqd/010:30:020
3971733995228,5cyclictest131rcu_preempt12:10:020
3971733995227,11cyclictest3984762-21ssh09:25:240
3971733995227,11cyclictest3984762-21ssh09:25:230
3971733995224,13cyclictest3990191-21ssh09:38:250
3971733995220,16cyclictest3995795-21ssh09:51:090
397174299519,6cyclictest131rcu_preempt09:25:083
397174299519,6cyclictest131rcu_preempt09:25:073
3971742995140,8cyclictest37-21ksoftirqd/308:30:023
3971742995140,8cyclictest37-21ksoftirqd/308:30:013
3971742995118,11cyclictest37-21ksoftirqd/309:21:123
3971742995113,6cyclictest131rcu_preempt12:00:023
397173399515,40cyclictest0-21swapper/007:50:020
397174299504,9cyclictest361rcuc/309:10:003
3971742995033,12cyclictest1-21systemd09:50:013
3971742995018,12cyclictest37-21ksoftirqd/312:11:263
3971742995011,7cyclictest131rcu_preempt11:55:013
3971736995038,7cyclictest23-21ksoftirqd/110:40:011
3971736995033,12cyclictest1-21systemd11:15:011
3971733995034,12cyclictest275-21dbus-daemon07:45:000
3971733995025,9cyclictest3999937-21ssh10:00:380
3971733995024,11cyclictest4067592-21ssh12:37:290
3971733995024,11cyclictest4067592-21ssh12:37:290
3971742994936,10cyclictest37-21ksoftirqd/310:49:263
397174299492,5cyclictest131rcu_preempt09:11:393
3971742994912,6cyclictest131rcu_preempt10:15:013
3971742994911,5cyclictest131rcu_preempt09:39:023
3971742994910,6cyclictest131rcu_preempt09:50:193
3971733994931,12cyclictest3976583-21modprobe08:45:000
397174299489,5cyclictest131rcu_preempt07:40:003
397174299488,16cyclictest131rcu_preempt11:38:223
3971742994813,7cyclictest131rcu_preempt08:15:013
3971742994813,7cyclictest131rcu_preempt08:15:003
3971742994812,6cyclictest131rcu_preempt11:00:003
3971742994811,6cyclictest131rcu_preempt10:06:193
3971742994811,5cyclictest131rcu_preempt10:50:393
3971733994833,11cyclictest275-21dbus-daemon08:50:010
3971733994830,14cyclictest4051054-21ssh11:59:300
3971733994814,26cyclictest131rcu_preempt11:49:120
3971736994734,10cyclictest23-21ksoftirqd/112:10:011
397173399477,21cyclictest3467602-21systemd-journal11:15:010
3971733994732,11cyclictest12-21ksoftirqd/009:45:000
397173399472,15cyclictest4059911-21cron12:19:590
3971742994637,6cyclictest37-21ksoftirqd/307:20:013
3971742994634,9cyclictest37-21ksoftirqd/310:27:103
3971742994627,15cyclictest275-21dbus-daemon11:10:023
397174299462,10cyclictest37-21ksoftirqd/311:23:003
3971742994614,6cyclictest131rcu_preempt09:16:043
3971742994614,6cyclictest131rcu_preempt09:16:033
397173899461,40cyclictest3971822-21latency_hist07:10:012
397173699465,25cyclictest221rcuc/107:30:031
3971736994634,8cyclictest1-21systemd12:00:011
3971736994633,10cyclictest23-21ksoftirqd/108:35:021
397174299457,5cyclictest131rcu_preempt10:36:403
397174299454,6cyclictest37-21ksoftirqd/310:04:523
3971742994530,12cyclictest37-21ksoftirqd/307:45:023
397174299452,5cyclictest131rcu_preempt12:22:203
397174299452,10cyclictest37-21ksoftirqd/311:49:373
3971742994514,6cyclictest131rcu_preempt12:05:013
3971742994514,5cyclictest131rcu_preempt09:43:413
3971742994511,6cyclictest131rcu_preempt08:40:013
3971736994530,13cyclictest23-21ksoftirqd/111:15:271
397173399453,9cyclictest12-21ksoftirqd/008:30:010
397173399453,9cyclictest12-21ksoftirqd/008:30:000
3971733994538,4cyclictest16950irq/224-800000008:00:000
3971733994538,4cyclictest16850irq/134-800000011:55:020
3971733994538,3cyclictest0-21swapper/008:35:020
3971733994536,7cyclictest12-21ksoftirqd/012:03:080
397173399453,37cyclictest3974752-21cron08:09:590
3971733994527,15cyclictest3975525-21sshd08:21:120
3971733994527,14cyclictest4033678-21ssh11:19:080
397174299448,9cyclictest37-21ksoftirqd/311:18:453
397174299442,10cyclictest37-21ksoftirqd/311:25:043
3971742994412,6cyclictest131rcu_preempt09:32:463
3971742994412,6cyclictest131rcu_preempt07:35:003
3971742994410,6cyclictest131rcu_preempt10:42:563
3971736994436,6cyclictest23-21ksoftirqd/112:20:021
3971736994434,7cyclictest23-21ksoftirqd/110:05:021
397173699443,3cyclictest131rcu_preempt09:30:011
397173699443,3cyclictest131rcu_preempt09:30:001
397173699441,3cyclictest131rcu_preempt09:30:301
3971733994438,4cyclictest16950irq/224-800000007:20:100
3971733994428,13cyclictest4004184-21ssh10:10:290
397174299437,5cyclictest131rcu_preempt12:16:353
3971742994330,9cyclictest37-21ksoftirqd/307:50:013
397174299432,9cyclictest37-21ksoftirqd/312:33:123
3971742994312,6cyclictest131rcu_preempt12:35:333
3971742994312,6cyclictest131rcu_preempt12:35:333
3971742994311,9cyclictest131rcu_preempt10:33:083
3971736994328,11cyclictest3467602-21systemd-journal08:51:111
397173699431,4cyclictest131rcu_preempt11:04:521
3971733994329,9cyclictest3467602-21systemd-journal07:40:000
397173399432,21cyclictest131rcu_preempt11:30:570
3971742994210,6cyclictest131rcu_preempt11:10:413
397173699423,6cyclictest0-21swapper/111:05:241
3971736994233,6cyclictest4068590-21latency_hist12:40:031
3971736994233,6cyclictest4068590-21latency_hist12:40:021
3971736994228,11cyclictest3467602-21systemd-journal09:40:011
397173699421,3cyclictest131rcu_preempt09:45:151
3971733994230,7cyclictest3975281-21latency_hist08:19:590
3971733994227,10cyclictest3977389-21unin-run09:00:020
397174299416,9cyclictest37-21ksoftirqd/307:10:023
3971742994126,11cyclictest309-21in:imuxsock08:45:013
3971742994118,16cyclictest275-21dbus-daemon07:15:003
397173699414,3cyclictest131rcu_preempt09:57:281
3971736994130,6cyclictest309-21in:imuxsock07:50:011
3971736994126,10cyclictest309-21in:imuxsock11:29:101
3971736994122,14cyclictest275-21dbus-daemon07:25:001
3971733994134,4cyclictest16950irq/224-800000007:55:000
3971733994126,12cyclictest3975001-21sshd08:14:120
3971733994126,12cyclictest3975001-21sshd08:14:120
397173399411,8cyclictest3971999-21latency_hist07:15:010
3971742994025,11cyclictest37-21ksoftirqd/309:05:023
3971742994022,14cyclictest37-21ksoftirqd/307:25:023
397174299401,9cyclictest37-21ksoftirqd/307:55:013
397173699404,10cyclictest0-21swapper/112:27:221
397173699403,3cyclictest131rcu_preempt09:12:241
3971736994031,5cyclictest3997325-21sh09:55:011
3971736994029,8cyclictest4062238-21sed12:25:021
3971736994028,10cyclictest23-21ksoftirqd/108:25:011
3971736994027,10cyclictest3467602-21systemd-journal07:16:111
3971736994027,10cyclictest23-21ksoftirqd/109:19:181
3971736994027,10cyclictest23-21ksoftirqd/109:19:181
3971736994026,11cyclictest23-21ksoftirqd/111:31:271
3971736994025,11cyclictest1-21systemd11:55:001
3971736994023,13cyclictest275-21dbus-daemon09:25:001
397173399401,35cyclictest12-21ksoftirqd/008:35:010
3971742993929,7cyclictest37-21ksoftirqd/308:35:013
397173699393,3cyclictest131rcu_preempt10:19:241
3971736993929,6cyclictest4052162-21sh12:01:401
3971736993929,6cyclictest3975282-21sendmail08:20:001
3971736993927,9cyclictest23-21ksoftirqd/110:57:461
3971736993927,8cyclictest309-21in:imuxsock08:05:021
3971736993924,12cyclictest23-21ksoftirqd/109:42:401
3971733993922,12cyclictest275-21dbus-daemon09:05:020
397174299387,6cyclictest131rcu_preempt08:55:003
397174299387,5cyclictest131rcu_preempt08:20:023
3971742993811,5cyclictest131rcu_preempt08:50:003
3971742993810,10cyclictest37-21ksoftirqd/308:10:023
3971736993829,6cyclictest23-21ksoftirqd/109:00:011
3971736993827,8cyclictest4018942-21systemctl10:45:011
3971736993825,10cyclictest23-21ksoftirqd/110:31:421
3971736993824,11cyclictest0-21swapper/111:25:021
3971736993823,9cyclictest3467602-21systemd-journal08:30:011
3971736993823,9cyclictest3467602-21systemd-journal08:30:011
397173699381,3cyclictest131rcu_preempt12:10:381
397174299373,4cyclictest131rcu_preempt07:28:003
3971742993725,10cyclictest37-21ksoftirqd/308:25:023
397173699373,3cyclictest131rcu_preempt10:51:201
3971736993728,5cyclictest3973195-21sh07:40:001
3971736993726,8cyclictest3977856-21sshd09:08:381
3971736993725,9cyclictest23-21ksoftirqd/111:35:261
3971736993724,11cyclictest23-21ksoftirqd/112:34:201
3971736993724,10cyclictest23-21ksoftirqd/111:41:281
3971736993718,15cyclictest275-21dbus-daemon08:00:011
3971742993620,12cyclictest0-21swapper/308:00:003
3971736993629,4cyclictest3976574-21sshd08:44:111
3971736993626,8cyclictest23-21ksoftirqd/109:05:021
397173699362,5cyclictest131rcu_preempt10:10:511
3971736993624,10cyclictest23-21ksoftirqd/110:49:061
3971736993624,10cyclictest23-21ksoftirqd/110:24:451
397173699362,3cyclictest131rcu_preempt11:47:351
3971736993623,9cyclictest275-21dbus-daemon07:45:021
3971733993627,7cyclictest12-21ksoftirqd/008:05:010
3971733993627,6cyclictest309-21in:imuxsock07:35:010
397173399361,6cyclictest131rcu_preempt07:30:000
397173399361,31cyclictest0-21swapper/008:50:020
397173699354,7cyclictest131rcu_preempt08:04:591
397174299341,5cyclictest131rcu_preempt09:00:003
3971736993426,5cyclictest3971874-21sshd07:13:271
3971736993424,7cyclictest3467602-21systemd-journal07:55:001
3971736993423,8cyclictest0-21swapper/110:25:021
397173699332,4cyclictest131rcu_preempt08:40:011
3971736993323,7cyclictest23-21ksoftirqd/108:46:111
3971736992817,8cyclictest3971849-21cat07:10:011
397173399281,4cyclictest131rcu_preempt07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional