You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 00:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 01, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3906541997972,3cyclictest0-21swapper/007:40:030
3906541997972,3cyclictest0-21swapper/007:40:020
3906541997869,3cyclictest0-21swapper/008:00:020
390654199781,73cyclictest0-21swapper/008:30:020
3906541997453,5cyclictest3936503-21latency_hist10:05:030
390655299681,21cyclictest3924864-21ssh09:38:183
390655299681,21cyclictest3924864-21ssh09:38:173
3906546996826,5cyclictest3912712-21diskmemload09:14:591
3906546996728,35cyclictest3982486-21sh11:51:441
3906541996748,15cyclictest275-21dbus-daemon12:40:000
3906546996619,10cyclictest3938713-21cat10:10:021
390654199667,40cyclictest19350irq/34-mmc112:05:020
390654199661,10cyclictest12-21ksoftirqd/011:40:010
3906552996531,31cyclictest37-21ksoftirqd/311:45:023
3906552996529,32cyclictest3976237-21sshd11:37:153
390655299652,58cyclictest3996021-21ssh12:23:193
390655299642,11cyclictest3467602-21systemd-journal11:58:123
390655299642,11cyclictest3467602-21systemd-journal11:58:123
390655299641,9cyclictest3994348-21ssh12:19:383
390655299641,4cyclictest3972753-21ssh11:29:253
390655299641,19cyclictest3968474-21ssh11:19:223
390655299632,55cyclictest3975157-21munin-run11:35:003
3906552996324,35cyclictest3962174-21sh11:04:513
3906552996324,35cyclictest3962174-21sh11:04:513
390655299632,17cyclictest3916003-21ssh09:17:313
390655299632,12cyclictest3938293-21ssh10:09:233
390655299631,58cyclictest3922842-21ssh09:33:323
390655299631,56cyclictest3969491-21ssh11:21:223
390655299631,16cyclictest3958593-21sshd10:56:093
3906546996326,9cyclictest23-21ksoftirqd/109:31:431
3906546996325,5cyclictest3992634-21sh12:15:031
390654199635,38cyclictest12-21ksoftirqd/009:50:270
3906552996233,25cyclictest3946285-21sh10:27:493
3906552996222,35cyclictest275-21dbus-daemon07:25:003
390655299622,10cyclictest3467602-21systemd-journal10:15:283
390655299621,9cyclictest4001530-21ssh12:35:463
3906552996218,39cyclictest3930029-21sed09:50:013
390655299621,4cyclictest3998257-21ssh12:28:243
390655299621,4cyclictest3983598-21ssh11:54:313
3906552996211,47cyclictest3990282-21systemd-run12:10:023
390655299612,13cyclictest3910614-21sshd08:28:193
390655299611,9cyclictest3962757-21ssh11:05:503
390655299611,9cyclictest3933776-21ssh09:59:003
3906552996119,37cyclictest3949814-21ssh10:35:463
3906552996119,37cyclictest3949814-21ssh10:35:453
390655299611,8cyclictest3930791-21ssh09:51:513
390655299611,8cyclictest3921023-21ssh09:29:243
390655299611,7cyclictest3953536-21ssh10:44:493
390655299611,15cyclictest3913445-21sshd09:11:313
390655299611,13cyclictest3907499-21sshd07:26:063
390655299611,10cyclictest3947941-21ssh10:31:303
3906546996127,30cyclictest3957412-21sh10:53:441
3906546996126,10cyclictest3467602-21systemd-journal08:15:331
3906546996124,5cyclictest3973004-21cron11:30:011
3906541996157,2cyclictest0-21swapper/011:25:020
3906541996136,19cyclictest16850irq/134-800000010:55:000
3906541996121,36cyclictest151rcuc/011:20:010
3906541996117,41cyclictest12-21ksoftirqd/010:42:430
390655299602,6cyclictest3944638-21ssh10:24:083
390655299601,9cyclictest3934723-21ssh10:00:463
390655299601,7cyclictest3956469-21ssh10:51:213
390655299601,6cyclictest3965134-21ssh11:11:223
390655299601,6cyclictest3926758-21ssh09:42:313
3906546996025,4cyclictest3916394-21sh09:18:351
390654199604,16cyclictest12-21ksoftirqd/009:50:000
390655299592,5cyclictest3909845-21munin-run08:15:003
390655299591,7cyclictest3986285-21ssh12:00:153
390655299591,7cyclictest3939093-21ssh10:10:573
390655299591,7cyclictest3918001-21ssh09:22:083
390655299591,6cyclictest3979946-21ssh11:45:343
390655299591,6cyclictest3979946-21ssh11:45:333
390655299591,5cyclictest3991301-21ssh12:12:153
3906541995926,17cyclictest3964069-21ssh11:09:140
3906552995820,33cyclictest37-21ksoftirqd/307:45:023
3906552995820,33cyclictest37-21ksoftirqd/307:45:023
390655299581,6cyclictest4000067-21ssh12:32:383
390655299581,6cyclictest3910352-21sshd08:20:173
3906552995813,40cyclictest0-21swapper/308:19:413
390654699584,32cyclictest0-21swapper/111:43:161
3906546995824,4cyclictest3962553-21sh11:05:111
390654199587,6cyclictest131rcu_preempt12:35:010
390655299572,4cyclictest3908534-21sshd07:47:063
390655299572,4cyclictest364-21snmpd08:33:323
3906552995722,31cyclictest3907777-21latency_hist07:35:023
390655299571,5cyclictest364-21snmpd08:38:343
390655299571,4cyclictest3911399-21sshd08:43:293
390654699573,20cyclictest3467602-21systemd-journal10:14:541
3906546995723,10cyclictest23-21ksoftirqd/111:35:321
3906546995722,4cyclictest3944711-21sh10:24:181
3906541995728,12cyclictest3995331-21ssh12:21:300
3906541995722,7cyclictest12-21ksoftirqd/010:13:240
3906541995711,15cyclictest12-21ksoftirqd/012:20:000
390655299561,4cyclictest3955394-21ssh10:49:083
3906546995625,11cyclictest23-21ksoftirqd/109:10:021
3906546995623,10cyclictest23-21ksoftirqd/110:30:011
3906546995621,7cyclictest23-21ksoftirqd/112:20:251
3906546995620,8cyclictest23-21ksoftirqd/111:58:471
3906546995620,8cyclictest23-21ksoftirqd/111:58:471
390654199568,6cyclictest131rcu_preempt12:30:010
3906541995628,13cyclictest3917501-21ssh09:20:460
3906541995627,13cyclictest3961792-21ssh11:03:560
3906541995627,13cyclictest3961792-21ssh11:03:550
3906541995626,14cyclictest3959247-21ssh10:57:520
3906541995618,32cyclictest16850irq/134-800000010:29:190
390654199561,39cyclictest3920389-21ssh09:27:430
390655299552,17cyclictest3912187-21grep08:59:593
3906546995527,5cyclictest3955331-21sh10:48:511
3906541995528,12cyclictest3916524-21ssh09:18:530
3906541995527,13cyclictest3985026-21ssh11:57:420
3906541995527,13cyclictest3985026-21ssh11:57:410
3906541995518,31cyclictest0-21swapper/011:33:420
390655299542,13cyclictest3909120-21ls08:00:013
390655299541,48cyclictest3911887-21sshd08:50:293
3906550995444,6cyclictest3900240-21kworker/2:210:05:012
3906550995434,16cyclictest3467602-21systemd-journal08:25:012
3906546995421,4cyclictest3921316-21latency_hist09:30:011
3906541995432,17cyclictest275-21dbus-daemon08:20:000
3906541995425,13cyclictest3971145-21ssh11:25:050
3906550995345,5cyclictest3900240-21kworker/2:209:52:342
3906550995343,6cyclictest3981650-21kworker/2:011:54:262
3906550995342,8cyclictest3900240-21kworker/2:211:15:002
3906546995322,8cyclictest3968384-21sshd11:19:111
3906546995322,8cyclictest23-21ksoftirqd/112:05:041
3906546995311,10cyclictest131rcu_preempt09:45:001
390654199534,17cyclictest3907781-21munin-run07:35:000
3906541995334,15cyclictest1-21systemd08:45:010
3906541995327,11cyclictest3992052-21ssh12:14:130
3906541995327,10cyclictest3913762-21ssh09:12:270
3906541995316,32cyclictest16850irq/134-800000010:36:210
3906541995316,32cyclictest16850irq/134-800000010:36:200
3906541995315,34cyclictest151rcuc/009:55:230
3906541995310,38cyclictest3979608-21grep11:45:020
3906550995244,5cyclictest3981650-21kworker/2:011:56:512
3906550995244,5cyclictest3981650-21kworker/2:011:56:502
3906550995242,7cyclictest3900240-21kworker/2:208:35:032
3906550995233,12cyclictest3467602-21systemd-journal10:10:002
3906550995231,16cyclictest0-21swapper/211:50:022
3906550995231,16cyclictest0-21swapper/211:50:012
3906550995226,19cyclictest3467602-21systemd-journal10:40:002
3906550995226,19cyclictest3467602-21systemd-journal10:39:592
3906546995220,8cyclictest23-21ksoftirqd/110:36:011
3906546995220,8cyclictest23-21ksoftirqd/110:36:011
3906546995217,11cyclictest23-21ksoftirqd/109:58:061
390654199527,11cyclictest12-21ksoftirqd/011:55:010
390654199526,5cyclictest131rcu_preempt09:43:100
390654199522,45cyclictest3467602-21systemd-journal09:35:000
3906541995218,30cyclictest12-21ksoftirqd/010:22:420
390655299511,6cyclictest3912420-21sshd09:00:323
3906550995144,4cyclictest3981650-21kworker/2:012:38:302
3906550995143,5cyclictest3900240-21kworker/2:211:22:532
3906550995142,6cyclictest3907289-21kworker/2:108:20:012
3906546995120,8cyclictest23-21ksoftirqd/111:02:181
3906546995120,8cyclictest23-21ksoftirqd/111:02:181
3906541995119,15cyclictest3988042-21ssh12:04:540
390655299503,40cyclictest3911810-21cat08:50:033
390655299503,40cyclictest3911810-21cat08:50:023
390655299502,6cyclictest275-21dbus-daemon08:05:013
3906550995042,5cyclictest3912217-21kworker/2:009:35:012
3906550995042,5cyclictest3900240-21kworker/2:210:30:012
3906550995041,5cyclictest3900240-21kworker/2:211:05:422
3906550995041,5cyclictest3900240-21kworker/2:209:10:002
3906550995041,5cyclictest3900240-21kworker/2:208:55:002
3906550995039,7cyclictest3981650-21kworker/2:012:21:352
3906550995039,7cyclictest3900240-21kworker/2:207:15:012
3906546995025,8cyclictest23-21ksoftirqd/110:32:271
390654199503,6cyclictest131rcu_preempt11:47:140
390654199503,6cyclictest131rcu_preempt11:47:130
3906541995029,14cyclictest3906542-21latency_hist07:10:010
3906541995028,7cyclictest3942232-21ssh10:18:310
390654199502,41cyclictest3925691-21latency_hist09:40:020
390654199502,41cyclictest3925691-21latency_hist09:40:010
3906541995022,13cyclictest3965005-21ssh11:11:010
3906541995013,10cyclictest12-21ksoftirqd/010:31:300
390655299491,4cyclictest3912679-21sshd09:07:313
3906550994941,5cyclictest3981650-21kworker/2:012:26:142
3906550994941,5cyclictest3900240-21kworker/2:209:59:302
3906550994940,6cyclictest3900240-21kworker/2:211:26:412
3906550994940,6cyclictest3900240-21kworker/2:211:20:012
3906550994940,6cyclictest3900240-21kworker/2:210:15:082
3906550994940,5cyclictest3981650-21kworker/2:012:11:322
3906550994940,5cyclictest3900240-21kworker/2:211:40:032
3906546994921,7cyclictest23-21ksoftirqd/111:12:201
390655299482,6cyclictest3906751-21munin-run07:15:013
390655299482,15cyclictest3907009-21munin-run07:20:013
3906550994840,5cyclictest3900240-21kworker/2:210:42:002
3906550994839,5cyclictest3907289-21kworker/2:107:58:022
3906550994833,9cyclictest30-21ksoftirqd/208:10:022
3906546994837,8cyclictest3980170-21sshd11:46:161
3906546994837,8cyclictest3980170-21sshd11:46:161
3906541994839,5cyclictest16950irq/224-800000010:10:000
3906541994823,5cyclictest131rcu_preempt08:35:000
3906541994818,14cyclictest3955574-21ssh10:49:350
3906552994725,17cyclictest3909593-21munin-plugin-st08:10:003
3906550994740,5cyclictest3900240-21kworker/2:209:37:232
3906550994740,5cyclictest3900240-21kworker/2:209:37:232
3906550994740,4cyclictest3981650-21kworker/2:012:18:582
3906550994739,5cyclictest3912217-21kworker/2:009:22:542
3906550994739,5cyclictest3900240-21kworker/2:211:45:022
3906550994738,6cyclictest3900240-21kworker/2:210:53:062
3906550994738,6cyclictest3900240-21kworker/2:209:42:352
3906550994738,5cyclictest3907289-21kworker/2:108:14:592
3906550994737,6cyclictest3900240-21kworker/2:211:02:402
3906550994737,6cyclictest3900240-21kworker/2:211:02:402
3906550994737,6cyclictest3900240-21kworker/2:210:23:552
3906550994737,6cyclictest3900240-21kworker/2:209:45:182
3906550994733,10cyclictest30-21ksoftirqd/207:30:022
3906550994728,13cyclictest1-21systemd10:35:022
3906546994732,12cyclictest3960032-21cron10:59:591
3906541994734,9cyclictest16950irq/224-800000009:09:590
390655299462,5cyclictest364-21snmpd07:35:183
390655299461,3cyclictest3908783-21sshd07:51:083
390655299461,3cyclictest3908783-21sshd07:51:073
3906550994638,5cyclictest3900240-21kworker/2:207:45:242
3906550994634,7cyclictest3900240-21kworker/2:210:10:252
390654699468,8cyclictest23-21ksoftirqd/109:49:591
390654199468,34cyclictest151rcuc/008:10:010
3906541994631,12cyclictest12-21ksoftirqd/009:05:010
3906550994537,5cyclictest3981650-21kworker/2:012:32:272
3906550994537,5cyclictest3900240-21kworker/2:210:55:042
3906550994534,7cyclictest3900240-21kworker/2:210:50:002
3906550994529,11cyclictest1-21systemd09:30:022
390654699453,12cyclictest3909077-21munin-run07:59:591
390654199452,38cyclictest0-21swapper/007:55:020
390654199452,38cyclictest0-21swapper/007:55:020
3906550994436,5cyclictest3981650-21kworker/2:012:05:452
3906550994436,5cyclictest3912217-21kworker/2:009:18:212
3906550994436,4cyclictest3981650-21kworker/2:012:00:562
3906550994435,6cyclictest3900240-21kworker/2:211:30:512
3906550994435,5cyclictest3900240-21kworker/2:209:00:012
3906550994426,14cyclictest3467602-21systemd-journal07:16:592
390654199445,8cyclictest12-21ksoftirqd/007:50:020
3906541994422,15cyclictest3467602-21systemd-journal07:15:000
3906550994317,18cyclictest275-21dbus-daemon09:05:012
3906546994333,7cyclictest3953699-21latency_hist10:45:001
390654699431,3cyclictest131rcu_preempt09:25:031
3906541994333,6cyclictest19350irq/34-mmc107:30:010
390654699421,37cyclictest23-21ksoftirqd/112:37:401
3906541994226,11cyclictest3467602-21systemd-journal07:40:010
390655299411,11cyclictest3906598-21latency_hist07:10:013
3906550994132,6cyclictest3912217-21kworker/2:009:15:012
3906550994132,6cyclictest3900240-21kworker/2:208:45:062
3906550994132,6cyclictest3900240-21kworker/2:208:45:062
3906546994129,9cyclictest3970901-21systemd-run11:25:011
3906546994128,8cyclictest3912166-21cron09:00:001
390654199412,35cyclictest3906974-21sshd07:18:010
390654699408,6cyclictest131rcu_preempt08:55:021
3906546994032,5cyclictest3910385-21munin-plugin-st08:24:591
390654199403,4cyclictest131rcu_preempt09:00:010
3906541994032,6cyclictest12-21ksoftirqd/008:05:020
3906541994031,6cyclictest12-21ksoftirqd/008:15:020
390654199401,10cyclictest3911977-21cut08:55:010
3906550993931,5cyclictest3900240-21kworker/2:207:25:032
3906546993930,7cyclictest23-21ksoftirqd/111:32:411
3906546993928,9cyclictest23-21ksoftirqd/112:01:121
3906546993925,8cyclictest23-21ksoftirqd/107:30:001
390654699392,13cyclictest3908067-21true07:40:011
390654199394,12cyclictest12-21ksoftirqd/008:48:290
390654199394,12cyclictest12-21ksoftirqd/008:48:280
3906541993921,13cyclictest3467602-21systemd-journal07:25:000
3906550993830,5cyclictest3900240-21kworker/2:207:32:092
3906550993829,7cyclictest30-21ksoftirqd/208:05:002
3906550993828,6cyclictest309-21in:imuxsock08:38:292
3906550993822,13cyclictest3908044-21systemctl07:40:012
3906546993828,5cyclictest3911144-21sendmail08:39:591
3906546993827,9cyclictest23-21ksoftirqd/108:10:011
3906546993826,9cyclictest23-21ksoftirqd/107:20:011
3906546993824,10cyclictest23-21ksoftirqd/110:16:311
390654199381,10cyclictest12-21ksoftirqd/008:25:010
3906550993727,8cyclictest30-21ksoftirqd/207:45:022
3906550993727,8cyclictest30-21ksoftirqd/207:45:012
3906546993723,6cyclictest3925605-21grep09:40:011
3906546993723,6cyclictest3925605-21grep09:40:001
390654699371,3cyclictest131rcu_preempt07:55:011
390654699371,3cyclictest131rcu_preempt07:55:011
3906541993724,10cyclictest12-21ksoftirqd/008:40:010
3906550993627,5cyclictest3907289-21kworker/2:107:55:012
3906550993627,5cyclictest3907289-21kworker/2:107:55:002
3906550993626,8cyclictest30-21ksoftirqd/208:25:012
3906546993624,8cyclictest275-21dbus-daemon08:35:011
3906546993623,10cyclictest4001143-21sed12:35:021
390654699358,5cyclictest131rcu_preempt07:34:221
3906546993527,4cyclictest3930379-21sh09:50:401
3906546993525,7cyclictest3907449-21sed07:25:021
3906546993523,9cyclictest23-21ksoftirqd/112:10:081
3906546993523,9cyclictest23-21ksoftirqd/108:10:301
3906546993522,10cyclictest23-21ksoftirqd/108:43:351
3906550993427,4cyclictest3900240-21kworker/2:208:43:092
3906546993425,7cyclictest3997366-21ssh12:26:071
3906546993425,6cyclictest3906478-21latency_hist07:10:031
3906546993422,9cyclictest3912585-21sed09:05:011
3906546993420,9cyclictest309-21in:imuxsock07:45:011
3906546993420,9cyclictest309-21in:imuxsock07:45:001
3906546993324,7cyclictest23-21ksoftirqd/108:26:191
390654699325,8cyclictest23-21ksoftirqd/110:03:341
3906546993120,8cyclictest3909331-21cron08:05:011
3906546993118,10cyclictest3906802-21sort07:15:001
3906546993023,5cyclictest23-21ksoftirqd/107:48:241
3906546993016,11cyclictest3911699-21cat08:50:021
3906546993016,11cyclictest3911699-21cat08:50:011
3906550992918,8cyclictest3906672-21cat07:10:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional