You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 15:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 14, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34612509910118,5cyclictest131rcu_preempt07:50:022
34612509910118,5cyclictest131rcu_preempt07:50:022
346125099772,44cyclictest3500544-21rm10:19:352
346125099772,44cyclictest3500544-21rm10:19:342
346125099772,40cyclictest3467422-21diskmemload12:09:452
346125099753,35cyclictest3472952-21ssh09:21:142
346125099752,41cyclictest3498343-21munin-run10:15:002
346125099751,41cyclictest3549084-21ssh12:00:362
346125099751,41cyclictest3549084-21ssh12:00:352
346125099748,31cyclictest3536333-21ssh11:34:172
346125099739,35cyclictest3503394-21ssh10:25:042
346125099739,35cyclictest3503394-21ssh10:25:032
346125099731,41cyclictest3509007-21ssh10:37:062
346125099731,41cyclictest3509007-21ssh10:37:062
346125099722,35cyclictest3529212-21ssh11:19:352
346125099711,33cyclictest3563712-21ssh12:31:172
346125099692,39cyclictest3479122-21ssh09:34:282
3461250996828,36cyclictest3555651-21sh12:14:462
3461250996828,36cyclictest3555651-21sh12:14:452
346125099672,32cyclictest3482674-21ssh09:41:442
346125099671,62cyclictest3525964-21ssh11:12:342
346125099671,61cyclictest48-21kcompactd010:30:132
346125099671,38cyclictest3467602-21systemd-journal08:52:152
346125099662,42cyclictest3465077-21sshd08:23:382
346125099661,61cyclictest3565874-21ssh12:35:522
346125099661,61cyclictest3556836-21ssh12:16:572
346125099661,61cyclictest3516663-21ssh10:53:152
346125099661,61cyclictest3476346-21ssh09:28:402
346125099661,60cyclictest3545641-21ssh11:53:332
346125099661,33cyclictest3543482-21sshd11:49:172
3461241996648,15cyclictest3497404-21kworker/u8:110:15:020
3461241996648,15cyclictest3497404-21kworker/u8:110:15:020
3461241996619,43cyclictest12-21ksoftirqd/010:10:000
3461241996619,43cyclictest12-21ksoftirqd/010:09:590
346125099651,60cyclictest3502564-21ssh10:23:372
346125099651,60cyclictest3495093-21ssh10:07:542
346125099651,60cyclictest3495093-21ssh10:07:542
346125099651,60cyclictest3486220-21ssh09:49:242
346125099651,59cyclictest3518795-21ssh10:57:372
346125099642,58cyclictest3510764-21ssh10:40:342
346125099641,59cyclictest3562184-21ssh12:28:282
346125099641,59cyclictest3546848-21ssh11:55:592
346125099641,59cyclictest3540956-21ssh11:43:492
346125099641,59cyclictest3540956-21ssh11:43:482
346125099641,59cyclictest3530862-21ssh11:22:472
346125099641,59cyclictest3520504-21ssh11:00:582
346125099641,59cyclictest3513940-21ssh10:47:192
346125099641,59cyclictest3513940-21ssh10:47:192
346125099641,59cyclictest3492434-21ssh10:02:132
346125099641,59cyclictest3489437-21ssh09:55:532
346125099641,59cyclictest3471799-21ssh09:19:142
346125099641,58cyclictest364-21snmpd12:21:202
3461246996444,16cyclictest1-21systemd11:40:001
3461246996442,16cyclictest3467602-21systemd-journal12:25:001
346125099632,56cyclictest3462206-21sshd07:28:162
346125099632,56cyclictest3461979-21munin-run07:25:012
346125099631,59cyclictest3486910-21ssh09:50:352
346125099631,58cyclictest3537218-21ssh11:35:492
346125099631,58cyclictest3522662-21ssh11:05:322
346125099631,58cyclictest3480189-21ssh09:36:312
346125099631,57cyclictest3532570-21ssh11:26:092
346125099631,57cyclictest3467724-21ssh09:10:092
346124199631,41cyclictest35641101ssh12:32:180
346125099621,56cyclictest3466851-21sshd08:55:162
346125099621,56cyclictest3466851-21sshd08:55:162
3461241996223,5cyclictest131rcu_preempt11:25:010
3461250996126,30cyclictest3467602-21systemd-journal09:05:172
3461241996123,34cyclictest12-21ksoftirqd/010:55:010
346125099602,54cyclictest3466366-21sshd08:49:162
346125099592,53cyclictest3466130-21munin-run08:45:002
346125099592,53cyclictest3466130-21munin-run08:45:002
346125099592,53cyclictest3464019-21sshd08:01:162
346125099591,54cyclictest3467153-21sshd09:04:162
346125099591,54cyclictest3465330-21sshd08:29:162
3461241995931,23cyclictest12-21ksoftirqd/012:40:000
346125099582,52cyclictest3465601-21munin-run08:35:002
346125099581,52cyclictest3461421-21sshd07:14:162
3461241995829,14cyclictest3532372-21ssh11:25:380
3461241995829,14cyclictest3523722-21ssh11:07:580
3461241995828,14cyclictest3520857-21ssh11:01:420
3461241995828,14cyclictest3497888-21ssh10:13:570
3461241995823,31cyclictest12-21ksoftirqd/011:40:320
3461241995823,31cyclictest12-21ksoftirqd/011:40:310
346124199582,11cyclictest3515307-21cut10:50:020
346124199581,50cyclictest3553530-21grep12:10:010
3461241995810,9cyclictest12-21ksoftirqd/010:40:000
3461241995810,9cyclictest12-21ksoftirqd/010:39:590
346125099573,49cyclictest364-21snmpd07:55:412
346125099571,52cyclictest3461678-21sshd07:18:312
3461246995739,13cyclictest3467602-21systemd-journal08:35:001
3461241995729,13cyclictest3487567-21ssh09:52:090
3461241995728,13cyclictest3486345-21ssh09:49:390
346124199572,50cyclictest309-21in:imuxsock09:10:000
3461241995718,34cyclictest311-21rs:main0
3461241995646,6cyclictest16850irq/134-800000009:15:000
3461241995628,13cyclictest3543201-21ssh11:48:310
346125099551,50cyclictest364-21snmpd07:33:372
346125099551,50cyclictest364-21snmpd07:33:372
3461246995535,15cyclictest3467602-21systemd-journal10:17:161
3461246995535,15cyclictest3467602-21systemd-journal10:17:161
3461241995531,17cyclictest3493644-21munin-run10:05:000
3461241995528,23cyclictest3514165-21ssh10:47:480
3461241995527,13cyclictest3481616-21ssh09:39:540
346125099541,49cyclictest364-21snmpd08:13:172
346125099541,49cyclictest364-21snmpd07:38:402
346125099541,48cyclictest3464304-21munin-run08:10:002
3461241995446,5cyclictest12-21ksoftirqd/010:43:220
3461241995428,11cyclictest3478342-21ssh09:32:390
3461241995426,12cyclictest3501359-21ssh10:20:450
3461241995424,13cyclictest3561007-21ssh12:25:350
3461241995418,9cyclictest12-21ksoftirqd/007:40:000
346125099533,46cyclictest3462992-21sshd07:43:322
346125099532,46cyclictest364-21snmpd08:15:092
346125099531,48cyclictest3465837-21cron08:40:002
3461246995336,12cyclictest275-21dbus-daemon11:35:001
3461241995344,5cyclictest16850irq/134-800000012:00:000
3461241995327,11cyclictest3505175-21ssh10:29:160
3461241995327,11cyclictest3505175-21ssh10:29:150
3461241995320,30cyclictest3548283-21kworker/u8:212:05:030
3461241995320,30cyclictest3548283-21kworker/u8:212:05:020
346124199531,36cyclictest3545478-21ssh11:53:130
3461241995232,15cyclictest275-21dbus-daemon10:00:010
3461241995226,10cyclictest3472711-21ssh09:20:430
3461241995222,14cyclictest3526443-21ssh11:13:400
346124199521,4cyclictest131rcu_preempt08:45:000
346124199521,4cyclictest131rcu_preempt08:45:000
346124199521,45cyclictest3467602-21systemd-journal07:20:020
3461255995137,9cyclictest1-21systemd11:40:013
3461241995142,6cyclictest16850irq/134-800000011:40:000
3461241995133,12cyclictest1-21systemd11:00:010
3461241995116,29cyclictest3484094-21kworker/u8:210:35:010
346125599503,4cyclictest131rcu_preempt11:18:033
3461241995020,14cyclictest3557983-21ssh12:19:400
3461241994920,13cyclictest3483615-21ssh09:43:510
3461255994838,4cyclictest37-21ksoftirqd/311:00:023
3461255994835,6cyclictest3505592-21ssh10:30:033
3461255994835,6cyclictest3505592-21ssh10:30:023
3461255994832,12cyclictest3467602-21systemd-journal11:10:153
3461255994831,12cyclictest37-21ksoftirqd/309:40:003
3461255994829,13cyclictest275-21dbus-daemon09:30:013
3461241994841,4cyclictest16850irq/134-800000011:19:160
3461241994832,12cyclictest3475381-21ssh09:26:280
346124199482,5cyclictest0-21swapper/007:30:010
3461255994732,11cyclictest3467602-21systemd-journal08:10:003
3461255994730,13cyclictest3467602-21systemd-journal07:15:033
346124199473,39cyclictest0-21swapper/008:00:010
3461241994722,10cyclictest3535448-21ssh11:32:170
3461246994633,9cyclictest23-21ksoftirqd/111:25:021
3461246994629,13cyclictest3474560-21latency_hist09:25:021
3461241994638,4cyclictest16950irq/224-800000008:35:010
3461241994628,14cyclictest3560353-21ssh12:24:350
346125599457,5cyclictest131rcu_preempt09:55:433
3461255994535,6cyclictest3507949-21munin-plugin-st10:35:003
3461255994528,13cyclictest37-21ksoftirqd/309:35:013
3461255994526,13cyclictest37-21ksoftirqd/308:40:003
346125599452,12cyclictest3517036-21ssh10:54:013
346125599451,3cyclictest131rcu_preempt12:23:133
346125599451,3cyclictest131rcu_preempt09:23:363
346125099451,41cyclictest3467562-21sshd07:53:162
3461246994532,10cyclictest3482921-21sshd09:42:161
3461246994532,10cyclictest23-21ksoftirqd/111:53:041
3461246994532,10cyclictest23-21ksoftirqd/109:35:031
3461246994529,11cyclictest275-21dbus-daemon09:50:011
3461246994525,16cyclictest3467718-21lldpd10:27:341
3461246994525,16cyclictest3467718-21lldpd10:27:341
346124199452,40cyclictest3554868-21ssh12:12:580
346124199452,40cyclictest3554868-21ssh12:12:570
346124199452,38cyclictest3461695-21modprobe07:20:000
3461241994517,25cyclictest12-21ksoftirqd/007:55:010
3461255994416,22cyclictest37-21ksoftirqd/309:10:243
3461246994432,8cyclictest23-21ksoftirqd/111:10:001
346124699442,7cyclictest131rcu_preempt12:00:021
3461246994427,12cyclictest1-21systemd08:30:011
346124199444,27cyclictest3461244-21latency_hist07:10:010
3461255994329,11cyclictest37-21ksoftirqd/310:36:173
3461255994329,11cyclictest37-21ksoftirqd/310:36:173
3461255994327,11cyclictest275-21dbus-daemon12:10:003
3461255994323,14cyclictest275-21dbus-daemon12:40:003
346125599431,3cyclictest131rcu_preempt12:03:493
346125599431,3cyclictest131rcu_preempt12:03:483
3461246994328,10cyclictest1-21systemd09:40:011
3461241994311,6cyclictest131rcu_preempt09:00:010
3461241994311,6cyclictest131rcu_preempt09:00:000
3461255994228,11cyclictest37-21ksoftirqd/311:57:283
3461255994225,12cyclictest275-21dbus-daemon08:00:013
3461255994219,20cyclictest37-21ksoftirqd/312:25:343
346125599421,8cyclictest131rcu_preempt09:10:023
346125599421,3cyclictest131rcu_preempt10:47:203
346125599421,3cyclictest131rcu_preempt10:47:193
346125099421,35cyclictest3461298-21latency_hist07:10:022
346124699424,3cyclictest131rcu_preempt09:57:111
3461246994230,9cyclictest3467402-21sshd09:09:161
3461246994228,10cyclictest3503162-21systemd-run10:25:001
3461246994228,10cyclictest3466606-21sshd08:51:161
3461241994235,5cyclictest12-21ksoftirqd/008:55:010
346124199422,35cyclictest3466387-21modprobe08:50:010
346125599413,3cyclictest131rcu_preempt10:13:553
3461255994129,10cyclictest37-21ksoftirqd/311:07:443
3461255994128,10cyclictest37-21ksoftirqd/312:17:033
3461255994125,13cyclictest3465391-21sed08:30:023
3461255994120,6cyclictest37-21ksoftirqd/309:41:463
346125599411,3cyclictest131rcu_preempt07:25:033
3461246994129,9cyclictest3554591-21sshd12:12:161
3461246994129,9cyclictest3554591-21sshd12:12:161
3461246994129,8cyclictest1-21systemd08:05:021
3461246994126,10cyclictest3567650-21systemctl12:39:591
3461241994132,6cyclictest12-21ksoftirqd/008:25:000
3461241994132,5cyclictest16950irq/224-800000008:40:010
3461241994126,11cyclictest3464804-21cron08:20:000
3461241994114,8cyclictest12-21ksoftirqd/007:49:380
3461241994114,8cyclictest12-21ksoftirqd/007:49:380
346125599404,3cyclictest131rcu_preempt11:29:593
3461255994028,8cyclictest3465096-21munin-plugin-st08:25:003
3461255994025,11cyclictest37-21ksoftirqd/310:40:173
346125599401,4cyclictest131rcu_preempt09:52:323
346124699409,24cyclictest0-21swapper/110:49:051
346124699409,24cyclictest0-21swapper/110:49:041
3461246994028,6cyclictest3467562-21sshd07:33:171
3461246994028,6cyclictest3467562-21sshd07:33:161
3461246994027,10cyclictest3466993-21sed09:00:021
3461246994027,10cyclictest3466993-21sed09:00:021
3461246994025,12cyclictest23-21ksoftirqd/111:15:211
3461246994025,11cyclictest23-21ksoftirqd/111:02:251
3461246994025,10cyclictest275-21dbus-daemon09:05:011
3461246994024,11cyclictest309-21in:imuxsock09:25:031
346124699402,3cyclictest131rcu_preempt09:10:301
3461246994023,12cyclictest275-21dbus-daemon10:40:021
3461246994023,12cyclictest275-21dbus-daemon10:40:011
346124699401,5cyclictest131rcu_preempt11:14:591
346124199406,20cyclictest19350irq/34-mmc108:15:020
346124199405,9cyclictest12-21ksoftirqd/008:10:000
3461241994012,5cyclictest131rcu_preempt08:05:020
3461255993931,5cyclictest3519863-21latency_hist11:00:033
3461255993930,6cyclictest37-21ksoftirqd/308:20:013
3461255993928,8cyclictest37-21ksoftirqd/311:45:013
3461255993928,8cyclictest37-21ksoftirqd/311:45:003
3461255993923,4cyclictest37-21ksoftirqd/310:25:013
3461255993912,6cyclictest131rcu_preempt08:45:023
346125599391,11cyclictest3467602-21systemd-journal10:10:003
346125599391,11cyclictest3467602-21systemd-journal10:10:003
3461246993927,9cyclictest23-21ksoftirqd/110:09:241
3461246993927,9cyclictest23-21ksoftirqd/110:09:241
3461246993927,8cyclictest309-21in:imuxsock12:28:151
3461246993925,11cyclictest23-21ksoftirqd/112:34:171
3461246993925,11cyclictest23-21ksoftirqd/110:33:351
3461246993924,11cyclictest3467602-21systemd-journal11:40:041
3461246993924,11cyclictest3467602-21systemd-journal11:40:031
3461246993924,11cyclictest275-21dbus-daemon08:25:001
346124699391,5cyclictest131rcu_preempt12:09:171
346124699391,3cyclictest131rcu_preempt10:56:041
346124199398,6cyclictest131rcu_preempt09:04:160
3461241993930,6cyclictest19350irq/34-mmc107:13:180
346124199391,9cyclictest0-21swapper/007:32:520
346124199391,9cyclictest0-21swapper/007:32:520
3461255993826,9cyclictest3467602-21systemd-journal08:45:013
3461255993826,9cyclictest3467602-21systemd-journal08:45:003
3461255993825,11cyclictest37-21ksoftirqd/311:51:523
3461255993825,10cyclictest37-21ksoftirqd/312:32:093
3461255993824,10cyclictest275-21dbus-daemon09:05:003
3461255993819,13cyclictest275-21dbus-daemon08:15:013
346125599381,4cyclictest131rcu_preempt11:30:103
346125599381,3cyclictest131rcu_preempt10:04:343
3461246993828,8cyclictest23-21ksoftirqd/110:02:271
3461246993826,9cyclictest23-21ksoftirqd/112:01:251
3461246993826,9cyclictest23-21ksoftirqd/112:01:251
3461246993826,8cyclictest3467602-21systemd-journal08:39:161
346124699382,3cyclictest131rcu_preempt12:16:301
346124199381,10cyclictest3467602-21systemd-journal08:27:410
346125599377,5cyclictest150-21pr/ttyS207:45:003
346125599373,3cyclictest131rcu_preempt08:35:013
3461255993727,8cyclictest37-21ksoftirqd/312:10:473
3461255993727,8cyclictest37-21ksoftirqd/312:10:463
3461255993727,7cyclictest3467602-21systemd-journal09:45:163
3461255993726,9cyclictest37-21ksoftirqd/309:17:203
3461255993725,9cyclictest37-21ksoftirqd/311:24:213
3461246993725,9cyclictest23-21ksoftirqd/107:15:021
3461246993724,10cyclictest23-21ksoftirqd/110:15:021
3461246993724,10cyclictest23-21ksoftirqd/107:21:361
3461241993726,7cyclictest3462994-21sshd07:43:320
3461255993628,5cyclictest3461500-21latency_hist07:15:023
3461255993627,5cyclictest3498771-21sh10:15:283
3461255993627,5cyclictest3498771-21sh10:15:273
3461255993626,8cyclictest37-21ksoftirqd/311:47:253
3461246993626,8cyclictest23-21ksoftirqd/109:53:401
3461246993626,8cyclictest23-21ksoftirqd/107:40:001
3461246993625,9cyclictest23-21ksoftirqd/111:47:291
3461246993625,9cyclictest23-21ksoftirqd/111:26:031
3461246993625,9cyclictest23-21ksoftirqd/110:54:001
3461246993624,8cyclictest3512859-21latency_hist10:45:011
346124699361,4cyclictest131rcu_preempt09:18:071
3461255993524,5cyclictest37-21ksoftirqd/307:30:003
346125599351,3cyclictest131rcu_preempt08:05:023
3461246993525,8cyclictest23-21ksoftirqd/107:45:141
3461246993525,8cyclictest23-21ksoftirqd/107:45:141
3461246993523,9cyclictest3463515-21cron07:55:001
346124699351,4cyclictest150-21pr/ttyS208:45:001
346124699351,4cyclictest150-21pr/ttyS208:45:001
346125599342,6cyclictest131rcu_preempt07:40:003
3461255993425,6cyclictest3466869-21sshd08:58:413
3461255993425,6cyclictest3466869-21sshd08:58:413
3461255993416,14cyclictest3467602-21systemd-journal07:35:003
3461255993416,14cyclictest3467602-21systemd-journal07:35:003
3461255993325,5cyclictest3463469-21latency_hist07:50:023
346124699333,24cyclictest3464827-21munin-run08:20:011
3461246993315,13cyclictest275-21dbus-daemon07:20:011
346124699331,3cyclictest131rcu_preempt08:10:001
346124699331,3cyclictest131rcu_preempt07:10:011
3461255993220,10cyclictest37-21ksoftirqd/307:45:023
3461255993219,9cyclictest3467602-21systemd-journal07:53:343
3461246993221,7cyclictest3463000-21cron07:45:001
3461246993215,5cyclictest0-21swapper/107:30:021
346124699321,20cyclictest364-21snmpd08:50:011
3461246993125,4cyclictest3463775-21latency_hist08:00:021
3461246993124,4cyclictest3464552-21latency_hist08:15:011
346125599304,3cyclictest131rcu_preempt08:55:003
346125599232,3cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional