You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 15:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 28, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3851360991008,9cyclictest131rcu_preempt10:50:041
3851369998215,34cyclictest3903227-21ssh10:45:023
3851369998215,34cyclictest3903227-21ssh10:45:023
3851360997836,38cyclictest1-21systemd09:10:011
3851357997571,2cyclictest0-21swapper/011:10:020
385136099713,18cyclictest3921068-21ssh11:22:291
3851360996918,45cyclictest3948616-21cron12:20:001
3851360996843,19cyclictest1-21systemd09:25:011
385136099682,18cyclictest3941476-21munin-run12:05:011
385136499672,12cyclictest3891109-21latency_hist10:20:002
385136099673,56cyclictest3895970-21ssh10:30:021
3851360996630,32cyclictest3926793-21sh11:34:331
385135799667,43cyclictest12-21ksoftirqd/011:45:560
3851357996625,7cyclictest3958230-21cron12:40:000
385135799651,60cyclictest0-21swapper/010:45:020
3851360996431,29cyclictest3853361-21sshd07:48:481
3851357996424,6cyclictest3895901-21latency_hist10:30:020
3851360996328,29cyclictest3932569-21sh11:46:211
3851360996323,36cyclictest3945907-21sh12:14:111
3851360996221,36cyclictest3870633-21sh09:36:541
3851360996213,6cyclictest3851445-21cat07:10:021
3851357996228,15cyclictest3912680-21ssh11:05:010
3851364996151,8cyclictest291rcuc/208:35:022
3851360996123,34cyclictest3869672-21latency_hist09:35:011
3851360996120,35cyclictest3922694-21sh11:25:411
3851357996129,15cyclictest3869598-21ssh09:35:010
385136099602,17cyclictest3943812-21modprobe12:10:001
385136099602,16cyclictest1-21systemd08:40:001
385136099601,15cyclictest3873333-21ssh09:42:381
385136099601,14cyclictest3891013-21sshd10:19:501
385135799601,55cyclictest0-21swapper/007:35:020
385136999594,36cyclictest3939043-21kworker/u8:412:05:023
385136499598,7cyclictest131rcu_preempt12:40:012
385136099592,15cyclictest3892983-21ssh10:23:491
3851360995919,34cyclictest3937993-21sh11:57:401
385136499586,7cyclictest30-21ksoftirqd/207:40:012
3851364995813,5cyclictest131rcu_preempt12:10:012
3851364995810,6cyclictest131rcu_preempt10:37:412
3851364995810,6cyclictest131rcu_preempt10:37:412
385136099584,16cyclictest3888659-21modprobe10:15:001
385136099583,15cyclictest3931396-21ssh11:44:031
385136099582,20cyclictest3853640-21munin-run07:55:001
385136099582,19cyclictest3881475-21cron10:00:001
385136099582,16cyclictest3910248-21munin-plugin-st11:00:001
385136099581,16cyclictest3874603-21ssh09:45:061
385136099581,13cyclictest3883885-21munin-run10:05:011
385136099581,13cyclictest3883885-21munin-run10:05:011
385136099573,16cyclictest3956740-21ssh12:36:271
385136099572,15cyclictest3903056-21munin-run10:45:001
385136099572,15cyclictest3897909-21ssh10:34:121
3851360995718,10cyclictest23-21ksoftirqd/109:29:261
3851360995716,4cyclictest3898948-21sh10:36:031
3851360995716,4cyclictest3898948-21sh10:36:031
3851360995715,4cyclictest0-21swapper/111:18:441
3851357995727,14cyclictest3873572-21ssh09:43:120
3851357995726,15cyclictest3943334-21ssh12:08:560
385136499567,10cyclictest30-21ksoftirqd/211:55:042
385136099564,47cyclictest150-21pr/ttyS212:30:391
385136099563,14cyclictest3854922-21cron08:20:001
385136099562,15cyclictest3951277-21latency_hist12:25:021
385136099562,15cyclictest3951277-21latency_hist12:25:021
385136099562,15cyclictest3856511-21systemd-run08:50:011
3851360995615,10cyclictest275-21dbus-daemon08:00:011
3851357995637,4cyclictest16850irq/134-800000010:20:000
3851357995629,12cyclictest3871253-21ssh09:38:210
3851357995628,13cyclictest3898163-21ssh10:34:440
3851357995627,13cyclictest3919520-21ssh11:19:260
385135799562,3cyclictest131rcu_preempt11:00:000
385136499558,11cyclictest30-21ksoftirqd/210:30:232
385136499558,10cyclictest30-21ksoftirqd/209:54:592
385136499557,9cyclictest30-21ksoftirqd/211:43:262
385136499556,11cyclictest30-21ksoftirqd/211:50:002
3851364995544,8cyclictest291rcuc/209:28:342
3851364995525,8cyclictest30-21ksoftirqd/212:26:432
385136099554,12cyclictest3927943-21sshd11:36:511
385136099552,15cyclictest3936793-21grep11:55:021
385136099552,15cyclictest3854666-21modprobe08:15:001
385136099552,15cyclictest284-21systemd-logind07:20:011
385136099552,15cyclictest284-21systemd-logind07:20:011
3851360995519,31cyclictest23-21ksoftirqd/108:30:021
3851360995519,31cyclictest23-21ksoftirqd/108:30:021
3851360995511,8cyclictest3897315-1kworker/u9:110:48:341
3851360995511,8cyclictest3897315-1kworker/u9:110:48:341
385136099551,17cyclictest3952433-21ssh12:27:411
385136099551,17cyclictest3857863-21ssh09:10:121
385136099551,13cyclictest3852707-21latency_hist07:35:011
3851357995528,12cyclictest3945432-21ssh12:13:110
3851357995525,14cyclictest3930800-21ssh11:42:490
3851357995521,4cyclictest0-21swapper/009:16:530
385135799551,38cyclictest3857903-21ssh09:10:170
385136499545,9cyclictest30-21ksoftirqd/210:48:262
385136499545,9cyclictest30-21ksoftirqd/210:48:262
385136499545,8cyclictest30-21ksoftirqd/210:02:372
385136499545,8cyclictest30-21ksoftirqd/210:02:372
385136499541,9cyclictest3919816-21modprobe11:20:002
385136099543,9cyclictest3917408-21cut11:15:001
385136099543,15cyclictest3911294-21ssh11:01:521
3851360995421,30cyclictest23-21ksoftirqd/108:10:021
3851360995414,4cyclictest3878802-21sh09:54:201
385136099541,15cyclictest3914338-21ssh11:08:251
3851357995426,13cyclictest3953742-21ssh12:30:040
3851357995421,4cyclictest131rcu_preempt10:46:080
3851357995421,4cyclictest131rcu_preempt10:46:080
3851357995418,5cyclictest3865824-21sh09:26:500
385136999536,6cyclictest131rcu_preempt11:20:013
3851369995326,9cyclictest37-21ksoftirqd/307:40:023
385136499539,6cyclictest131rcu_preempt10:52:282
385136499534,10cyclictest30-21ksoftirqd/209:40:142
3851364995311,7cyclictest131rcu_preempt10:10:232
385136099531,16cyclictest3862118-21ssh09:19:251
385136099531,15cyclictest3885426-21ssh10:08:071
385135799533,15cyclictest3467602-21systemd-journal09:00:000
3851357995330,7cyclictest3948575-21ssh12:19:550
3851357995328,10cyclictest3891384-21ssh10:20:040
3851357995325,13cyclictest3941288-21ssh12:04:390
3851357995325,13cyclictest3923965-21ssh11:28:390
3851357995325,13cyclictest3915303-21ssh11:10:130
3851357995316,33cyclictest12-21ksoftirqd/009:22:180
3851369995244,6cyclictest37-21ksoftirqd/309:00:013
385136499527,8cyclictest30-21ksoftirqd/211:29:142
385136499527,12cyclictest30-21ksoftirqd/212:31:032
3851364995211,5cyclictest131rcu_preempt12:17:532
385136099523,13cyclictest1-21systemd07:40:011
3851360995218,30cyclictest3856261-21sh08:45:011
385135799524,26cyclictest131rcu_preempt11:56:380
385135799523,3cyclictest131rcu_preempt10:50:150
3851357995226,11cyclictest3951517-21ssh12:25:350
3851357995226,11cyclictest3898772-21ssh10:35:410
3851357995226,11cyclictest3898772-21ssh10:35:410
3851357995226,11cyclictest3876151-21ssh09:48:440
3851364995142,6cyclictest30-21ksoftirqd/211:52:132
3851364995133,13cyclictest3951133-21latency_hist12:25:022
3851364995133,13cyclictest3951133-21latency_hist12:25:022
3851364995120,10cyclictest30-21ksoftirqd/209:59:572
3851364995115,10cyclictest30-21ksoftirqd/210:21:052
385136499511,16cyclictest30-21ksoftirqd/212:10:032
385136099515,9cyclictest3852345-21modprobe07:30:001
3851360995113,9cyclictest275-21dbus-daemon07:25:011
385135799512,35cyclictest0-21swapper/009:52:340
385135799512,16cyclictest3855984-21munin-plugin-st08:40:000
385136999503,6cyclictest131rcu_preempt10:15:023
3851364995042,5cyclictest3869603-21kworker/2:010:43:562
3851364995042,5cyclictest3856512-21kworker/2:311:11:202
3851364995042,5cyclictest3856512-21kworker/2:309:33:072
3851364995042,5cyclictest3842845-21kworker/2:008:15:212
3851364995023,19cyclictest275-21dbus-daemon07:45:012
385136499502,13cyclictest30-21ksoftirqd/210:58:322
3851364995020,7cyclictest30-21ksoftirqd/209:12:132
385136099502,10cyclictest3857280-21modprobe09:05:001
3851360995011,8cyclictest23-21ksoftirqd/108:03:341
3851360995011,10cyclictest275-21dbus-daemon08:35:011
3851357995043,4cyclictest16850irq/134-800000011:55:010
385135799503,3cyclictest131rcu_preempt11:20:210
3851357995031,14cyclictest12-21ksoftirqd/007:20:000
3851357995031,14cyclictest12-21ksoftirqd/007:20:000
385135799502,14cyclictest3881497-21systemctl10:00:010
385136499497,6cyclictest131rcu_preempt09:35:372
385136499496,6cyclictest131rcu_preempt11:24:492
3851364994942,4cyclictest3851263-21kworker/2:107:15:372
3851364994942,4cyclictest3851263-21kworker/2:107:15:372
3851364994941,5cyclictest3869603-21kworker/2:011:04:032
3851364994941,5cyclictest3869603-21kworker/2:010:06:412
3851364994941,5cyclictest3856512-21kworker/2:312:02:432
3851364994941,5cyclictest3856512-21kworker/2:311:32:112
3851364994941,5cyclictest3856512-21kworker/2:309:46:432
3851364994941,5cyclictest3856512-21kworker/2:309:24:092
3851364994941,5cyclictest3851263-21kworker/2:107:14:142
3851364994940,5cyclictest3856512-21kworker/2:311:39:002
385136499492,9cyclictest30-21ksoftirqd/210:26:172
385136499492,10cyclictest30-21ksoftirqd/211:06:422
385136499491,17cyclictest0-21swapper/208:14:242
385136099494,6cyclictest3856998-21modprobe09:00:001
385135799493,3cyclictest131rcu_preempt10:15:000
3851357994924,10cyclictest3925788-21ssh11:32:180
3851369994827,10cyclictest37-21ksoftirqd/307:35:023
3851364994840,5cyclictest3857300-21kworker/2:209:15:102
3851364994840,5cyclictest3853397-21kworker/2:308:09:372
3851364994840,5cyclictest3851263-21kworker/2:107:20:432
3851364994840,5cyclictest3842845-21kworker/2:008:47:502
3851364994840,5cyclictest3842845-21kworker/2:008:44:462
3851364994840,5cyclictest3842845-21kworker/2:008:20:022
3851364994840,5cyclictest3842845-21kworker/2:008:04:402
3851364994840,5cyclictest3842845-21kworker/2:007:56:462
3851364994840,5cyclictest3842845-21kworker/2:007:51:052
3851364994840,5cyclictest3842845-21kworker/2:007:46:572
3851364994839,6cyclictest3842845-21kworker/2:008:33:392
3851364994839,5cyclictest3855223-21kworker/2:108:57:142
3851357994824,21cyclictest3950662-21ssh12:24:100
3851357994824,21cyclictest3950662-21ssh12:24:100
3851369994733,11cyclictest3854909-21sshd08:18:473
3851369994721,18cyclictest37-21ksoftirqd/311:52:023
3851364994740,4cyclictest3842845-21kworker/2:008:29:192
3851364994740,4cyclictest3842845-21kworker/2:008:29:182
3851364994739,5cyclictest3857300-21kworker/2:209:08:112
3851364994739,5cyclictest3855223-21kworker/2:109:02:272
3851364994739,5cyclictest3855223-21kworker/2:108:52:142
3851364994739,5cyclictest3852362-21kworker/2:207:33:232
3851364994739,5cyclictest3842845-21kworker/2:007:27:192
3851357994724,8cyclictest12-21ksoftirqd/011:35:330
385136499466,10cyclictest30-21ksoftirqd/207:10:022
385136099463,16cyclictest3851556-21modprobe07:15:001
3851360994614,10cyclictest23-21ksoftirqd/107:45:011
3851357994639,4cyclictest16950irq/224-800000007:25:000
3851357994629,13cyclictest3467602-21systemd-journal08:34:140
3851357994627,15cyclictest3882448-21ssh10:01:410
3851357994627,15cyclictest3882448-21ssh10:01:410
3851357994613,6cyclictest12-21ksoftirqd/010:06:390
3851369994534,8cyclictest37-21ksoftirqd/311:46:353
3851369994529,7cyclictest37-21ksoftirqd/312:23:413
3851369994529,7cyclictest37-21ksoftirqd/312:23:413
3851369994527,13cyclictest37-21ksoftirqd/310:00:033
3851369994527,13cyclictest37-21ksoftirqd/310:00:033
3851369994521,20cyclictest37-21ksoftirqd/309:15:003
3851369994519,6cyclictest131rcu_preempt09:21:563
385136999451,8cyclictest0-21swapper/312:26:413
385136999451,10cyclictest0-21swapper/310:52:423
385135799452,30cyclictest0-21swapper/007:55:020
385136999447,6cyclictest131rcu_preempt08:05:003
3851369994433,8cyclictest37-21ksoftirqd/311:14:023
3851369994433,7cyclictest37-21ksoftirqd/309:55:003
3851369994431,9cyclictest37-21ksoftirqd/307:50:013
3851369994430,9cyclictest309-21in:imuxsock12:40:003
3851369994430,10cyclictest3868630-21sshd09:32:503
3851369994421,8cyclictest37-21ksoftirqd/308:15:023
3851369994420,19cyclictest37-21ksoftirqd/311:05:393
3851369994418,16cyclictest37-21ksoftirqd/309:29:373
3851369994417,6cyclictest131rcu_preempt10:28:273
3851369994417,16cyclictest0-21swapper/309:42:403
385136999434,6cyclictest131rcu_preempt09:20:003
3851369994327,11cyclictest37-21ksoftirqd/311:20:323
3851369994326,13cyclictest37-21ksoftirqd/311:42:153
3851369994317,21cyclictest37-21ksoftirqd/310:35:403
3851369994317,21cyclictest37-21ksoftirqd/310:35:403
385136999431,6cyclictest0-21swapper/311:25:363
3851369994313,17cyclictest37-21ksoftirqd/311:04:103
3851369994310,7cyclictest131rcu_preempt10:10:013
385135799432,6cyclictest3857551-21ssh09:10:000
385136999427,6cyclictest131rcu_preempt11:37:463
385136999425,7cyclictest131rcu_preempt09:40:003
385136999424,27cyclictest0-21swapper/310:45:013
3851369994231,8cyclictest37-21ksoftirqd/310:20:003
3851369994220,16cyclictest275-21dbus-daemon08:25:013
385136999421,5cyclictest0-21swapper/312:05:143
385135799423,36cyclictest3851366-21cat07:10:020
3851357994230,9cyclictest12-21ksoftirqd/007:30:010
3851369994124,9cyclictest37-21ksoftirqd/309:48:443
3851369994123,13cyclictest3467602-21systemd-journal07:15:003
3851369994119,6cyclictest131rcu_preempt11:32:243
3851369994117,20cyclictest37-21ksoftirqd/312:13:153
3851369994116,6cyclictest131rcu_preempt12:31:193
3851369994115,9cyclictest131rcu_preempt10:34:293
3851360994114,6cyclictest150-21pr/ttyS208:54:361
385136999406,6cyclictest131rcu_preempt08:40:003
3851369994028,9cyclictest37-21ksoftirqd/310:57:353
3851369994023,11cyclictest310-21in:imklog09:55:333
385136999401,4cyclictest3891967-21ssh10:21:243
3851357994031,5cyclictest3467602-21systemd-journal07:57:020
3851357994024,12cyclictest3855441-21sshd08:28:470
3851357994024,12cyclictest3855441-21sshd08:28:470
3851369993927,8cyclictest37-21ksoftirqd/309:10:013
385136999392,4cyclictest131rcu_preempt12:20:003
3851369993916,19cyclictest37-21ksoftirqd/307:10:023
3851357993929,7cyclictest3856510-21sh08:50:010
3851357993929,7cyclictest12-21ksoftirqd/008:15:030
385135799392,6cyclictest3854181-21systemd-run08:05:000
3851357993926,10cyclictest12-21ksoftirqd/008:52:160
3851357993924,11cyclictest12-21ksoftirqd/007:45:020
385135799391,35cyclictest3852818-21sshd07:35:470
385136999383,7cyclictest131rcu_preempt08:35:003
3851369993829,6cyclictest37-21ksoftirqd/308:45:023
385136999382,5cyclictest131rcu_preempt08:50:003
3851369993825,9cyclictest3851807-21cron07:20:013
3851369993825,9cyclictest3851807-21cron07:20:003
3851369993824,10cyclictest37-21ksoftirqd/307:55:023
3851369993814,5cyclictest131rcu_preempt11:56:003
385135799381,6cyclictest3855122-21latency_hist08:20:020
3851369993720,13cyclictest3856924-21sed08:55:023
3851369993626,7cyclictest37-21ksoftirqd/307:55:463
3851360993620,11cyclictest275-21dbus-daemon08:25:011
385135799362,5cyclictest3855214-21munin-run08:25:000
385135799361,31cyclictest0-21swapper/008:41:170
385135799361,31cyclictest0-21swapper/008:08:090
385135799351,31cyclictest0-21swapper/009:05:020
3851369993426,6cyclictest37-21ksoftirqd/307:20:413
3851369993423,8cyclictest37-21ksoftirqd/308:05:473
3851369993421,9cyclictest37-21ksoftirqd/308:25:063
3851369993421,9cyclictest37-21ksoftirqd/308:25:063
385135799342,5cyclictest3851649-21sed07:15:020
3851369993321,9cyclictest37-21ksoftirqd/307:44:003
385135799331,29cyclictest0-21swapper/007:46:200
3851369993219,10cyclictest3857390-21sort09:05:023
385136999321,4cyclictest131rcu_preempt07:25:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional