You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 13:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 31, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
344522599944,73cyclictest3492438-21ssh22:45:023
344522199681,15cyclictest3471500-21ssh21:56:222
3445211996826,10cyclictest3529054-21munin-run00:10:010
3445211996644,7cyclictest12-21ksoftirqd/021:40:470
3445211996644,7cyclictest12-21ksoftirqd/021:40:470
3445221996531,30cyclictest30-21ksoftirqd/222:16:572
344522199652,15cyclictest3456214-21ssh21:20:302
344522199652,15cyclictest3456214-21ssh21:20:302
3445211996528,5cyclictest3539763-21munin-run00:35:000
3445211996510,38cyclictest3512828-21ssh23:32:140
344522199643,53cyclictest3453818-21wc21:15:022
3445221996430,30cyclictest3470453-21sh21:54:132
3445221996430,30cyclictest3470453-21sh21:54:132
344522199641,9cyclictest3462495-21ssh21:35:132
344522199641,10cyclictest3496161-21ssh22:53:452
344522199632,15cyclictest3448834-21munin-run20:20:012
344522199631,9cyclictest3513494-21ssh23:33:572
344522199631,8cyclictest3474176-21ssh22:02:332
344522199631,8cyclictest3474176-21ssh22:02:332
344522199631,7cyclictest3504547-21ssh23:13:142
344522199631,7cyclictest3504547-21ssh23:13:142
344522199631,7cyclictest3493599-21ssh22:47:412
344522199631,13cyclictest3450838-21sshd20:57:272
3445211996311,36cyclictest3486806-21ssh22:31:590
3445225996254,4cyclictest37-21ksoftirqd/321:55:023
3445225996254,4cyclictest37-21ksoftirqd/321:55:013
3445225996214,13cyclictest37-21ksoftirqd/322:34:283
344522199625,52cyclictest3492503-21latency_hist22:45:022
344522199623,49cyclictest3450359-21modprobe20:50:012
344522199623,49cyclictest3450359-21modprobe20:50:012
344522199622,15cyclictest3448293-21munin-run20:10:022
344522199621,9cyclictest3516771-21ssh23:41:162
344522199621,8cyclictest3518873-21ssh23:46:092
344522199621,7cyclictest3506577-21ssh23:17:542
344522199621,19cyclictest3533203-21ssh00:19:462
344521199622,43cyclictest3462187-21ssh21:35:000
3445211996223,35cyclictest16850irq/134-800000023:22:440
3445211996223,35cyclictest16850irq/134-800000023:22:440
3445211996219,39cyclictest3451420-21systemd-run21:10:000
3445211996219,37cyclictest16850irq/134-800000021:21:300
3445211996219,37cyclictest16850irq/134-800000021:21:300
3445225996115,7cyclictest37-21ksoftirqd/300:25:013
3445225996111,9cyclictest37-21ksoftirqd/323:55:003
344522199612,13cyclictest48-21kcompactd022:25:252
344522199612,13cyclictest48-21kcompactd022:25:252
344522199612,13cyclictest364-21snmpd00:35:092
344522199612,11cyclictest3450578-21sshd20:51:282
344522199611,8cyclictest3525403-21ssh00:01:112
344522199611,8cyclictest3522924-21ssh23:55:272
344522199611,8cyclictest3498225-21ssh22:58:342
344522199611,7cyclictest3535312-21ssh00:24:352
344522199611,7cyclictest3515733-21ssh23:39:142
344522199611,7cyclictest3500426-21ssh23:03:432
344522199611,7cyclictest3467552-21ssh21:47:152
344522199611,7cyclictest3455211-21ssh21:18:342
344522199611,5cyclictest3503139-21munin-run23:10:012
344522199611,14cyclictest3466008-21ssh21:43:462
344522199611,14cyclictest3466008-21ssh21:43:462
344522199611,11cyclictest48-21kcompactd021:25:152
344522199611,10cyclictest3486802-21ssh22:31:582
3445211996124,33cyclictest3477264-21munin-run22:10:010
344522199602,5cyclictest3447494-21munin-run19:55:002
344522199602,12cyclictest3446426-21sshd19:31:252
344522199601,9cyclictest3538312-21ssh00:31:162
344522199601,8cyclictest3527280-21ssh00:05:242
344522199601,8cyclictest3477654-21ssh22:10:282
344522199601,7cyclictest3521530-21ssh23:52:292
344522199601,7cyclictest3510672-21ssh23:27:142
344522199601,7cyclictest3488601-21ssh22:36:012
344522199601,7cyclictest3482691-21ssh22:22:302
344522199601,7cyclictest3482691-21ssh22:22:302
344522199601,6cyclictest3507936-21ssh23:20:442
344522199601,6cyclictest3507936-21ssh23:20:442
3445216996038,18cyclictest1-21systemd19:40:011
3445211996024,4cyclictest3485832-21munin-run22:30:010
3445211996024,4cyclictest3485832-21munin-run22:30:010
3445225995932,8cyclictest37-21ksoftirqd/321:16:393
3445225995917,8cyclictest37-21ksoftirqd/322:18:473
344522199594,49cyclictest275-21dbus-daemon19:25:002
344522199592,7cyclictest3530712-21ssh00:13:532
344522199592,5cyclictest3446208-21munin-run19:30:002
344522199591,14cyclictest3446721-21munin-run19:40:002
344522199591,11cyclictest3448529-21munin-run20:15:002
3445211995931,12cyclictest3498491-21ssh22:59:110
3445211995922,32cyclictest16850irq/134-800000023:25:080
3445211995922,14cyclictest3454923-21sh21:17:520
3445225995816,9cyclictest37-21ksoftirqd/300:00:363
344522199581,7cyclictest3462010-21ssh21:34:372
344522199581,13cyclictest3476026-21ssh22:06:562
3445211995819,34cyclictest3536308-21sshd00:26:340
344522599577,11cyclictest37-21ksoftirqd/300:35:023
3445225995715,8cyclictest37-21ksoftirqd/322:50:403
3445225995715,7cyclictest37-21ksoftirqd/323:37:523
344522599571,16cyclictest3482451-21ssh22:21:453
344522599571,16cyclictest3482451-21ssh22:21:453
344522199573,46cyclictest1-21systemd21:05:012
344522199573,46cyclictest1-21systemd21:05:002
344522199572,12cyclictest3535690-21ssh00:25:032
3445211995728,13cyclictest3514647-21ssh23:36:220
3445211995712,40cyclictest3452325-21rm21:11:490
3445211995711,28cyclictest16850irq/134-800000000:23:110
3445221995648,5cyclictest3447268-21kworker/2:220:40:012
344522199562,7cyclictest3451407-21munin-run21:10:002
3445211995620,5cyclictest3492323-21ssh22:45:000
3445211995610,41cyclictest3480952-21ssh22:18:380
3445225995537,12cyclictest275-21dbus-daemon20:40:003
3445225995516,8cyclictest37-21ksoftirqd/322:04:543
3445225995516,8cyclictest37-21ksoftirqd/322:04:543
3445221995538,13cyclictest1-21systemd19:20:012
344521199558,43cyclictest3495465-21ssh22:51:550
3445211995525,14cyclictest3501300-21ssh23:05:260
3445211995520,32cyclictest16850irq/134-800000022:46:000
3445211995518,7cyclictest16850irq/134-800000023:53:400
3445211995518,33cyclictest0-21swapper/023:45:020
344522599545,11cyclictest3452205-21ssh21:11:233
3445225995419,7cyclictest37-21ksoftirqd/300:10:523
3445225995411,9cyclictest37-21ksoftirqd/323:33:113
344522199543,5cyclictest0-21swapper/220:29:462
344522199543,45cyclictest3467602-21systemd-journal20:45:012
344522199542,6cyclictest3447243-21munin-run19:50:012
3445221995412,36cyclictest275-21dbus-daemon20:00:012
3445211995421,28cyclictest16850irq/134-800000022:10:110
3445211995419,18cyclictest16850irq/134-800000021:45:480
3445211995419,18cyclictest16850irq/134-800000021:28:020
3445211995418,18cyclictest16850irq/134-800000022:04:490
3445211995418,18cyclictest16850irq/134-800000022:04:490
3445211995415,34cyclictest3540488-21ssh00:36:230
3445211995414,34cyclictest309-21in:imuxsock21:56:260
3445211995414,17cyclictest16850irq/134-800000021:51:160
3445211995414,17cyclictest16850irq/134-800000021:51:160
344522599537,6cyclictest37-21ksoftirqd/300:10:023
344522599537,10cyclictest37-21ksoftirqd/323:10:483
344522599537,10cyclictest37-21ksoftirqd/323:10:483
3445225995315,8cyclictest37-21ksoftirqd/321:42:173
3445225995315,8cyclictest37-21ksoftirqd/321:42:173
3445225995312,9cyclictest0-21swapper/321:10:003
344522199532,19cyclictest3446995-21modprobe19:45:002
3445211995340,10cyclictest3445218-21latency_hist19:10:010
3445211995340,10cyclictest3445218-21latency_hist19:10:000
3445211995318,6cyclictest16850irq/134-800000000:16:350
3445211995318,18cyclictest16850irq/134-800000023:57:000
3445211995317,18cyclictest16850irq/134-800000023:10:050
3445211995317,18cyclictest16850irq/134-800000023:10:050
344522199521,6cyclictest3449538-21sshd20:30:252
344522199521,5cyclictest3445411-21munin-run19:14:592
3445216995242,7cyclictest23-21ksoftirqd/123:53:351
3445211995221,19cyclictest16850irq/134-800000022:23:470
3445211995221,19cyclictest16850irq/134-800000022:23:470
3445211995217,6cyclictest16850irq/134-800000022:35:410
3445211995217,17cyclictest16850irq/134-800000023:00:300
3445211995217,17cyclictest16850irq/134-800000000:04:270
3445211995216,30cyclictest3447343-21wc19:50:020
3445211995216,18cyclictest16850irq/134-800000000:11:540
344521199521,36cyclictest3505613-21ssh23:15:240
344522599518,6cyclictest3449586-21unin-run20:35:013
3445225995113,8cyclictest37-21ksoftirqd/322:47:143
3445225995112,12cyclictest37-21ksoftirqd/323:17:413
3445225995111,11cyclictest0-21swapper/322:10:583
344522199512,5cyclictest0-21swapper/220:03:302
344521699512,30cyclictest3479553-21ssh22:15:031
3445211995117,17cyclictest16850irq/134-800000021:37:590
3445211995116,30cyclictest16850irq/134-800000023:49:500
3445225995011,8cyclictest37-21ksoftirqd/320:50:023
3445225995011,8cyclictest37-21ksoftirqd/320:50:023
3445225995010,7cyclictest0-21swapper/323:25:003
3445225995010,7cyclictest0-21swapper/323:25:003
3445211995012,32cyclictest3446451-21modprobe19:35:000
344522599497,13cyclictest37-21ksoftirqd/323:59:393
344522599495,8cyclictest37-21ksoftirqd/322:36:003
344522599495,7cyclictest37-21ksoftirqd/322:29:163
344522599495,7cyclictest37-21ksoftirqd/322:29:163
3445216994940,6cyclictest23-21ksoftirqd/120:40:011
3445216994938,8cyclictest23-21ksoftirqd/100:15:011
344521199491,42cyclictest3467602-21systemd-journal20:55:000
3445211994913,4cyclictest3446741-21lockfile-remove19:40:010
344522599489,8cyclictest37-21ksoftirqd/323:41:593
344522599487,8cyclictest37-21ksoftirqd/321:25:433
344522599486,7cyclictest37-21ksoftirqd/323:01:073
344522599485,6cyclictest37-21ksoftirqd/321:30:373
344522599483,10cyclictest37-21ksoftirqd/300:29:063
3445225994829,14cyclictest275-21dbus-daemon21:05:013
3445225994829,14cyclictest275-21dbus-daemon21:05:013
3445216994835,10cyclictest3467602-21systemd-journal21:34:591
3445216994831,12cyclictest1-21systemd23:05:011
344521199482,16cyclictest3449837-21munin-run20:40:010
344522599479,10cyclictest0-21swapper/323:06:033
344522599475,8cyclictest37-21ksoftirqd/321:39:013
3445225994730,13cyclictest275-21dbus-daemon19:50:013
3445225994728,16cyclictest275-21dbus-daemon23:30:003
3445225994712,8cyclictest0-21swapper/321:20:263
3445225994712,8cyclictest0-21swapper/321:20:263
344522199471,10cyclictest3449039-21sshd20:21:272
344521699477,6cyclictest131rcu_preempt21:45:151
3445216994715,6cyclictest131rcu_preempt21:20:001
344522599468,5cyclictest131rcu_preempt23:48:483
344522599465,6cyclictest37-21ksoftirqd/321:57:223
344522599464,7cyclictest37-21ksoftirqd/322:59:003
344522599461,41cyclictest37-21ksoftirqd/320:20:013
344521699464,22cyclictest131rcu_preempt22:41:531
3445216994635,8cyclictest23-21ksoftirqd/123:30:011
3445216994630,11cyclictest23-21ksoftirqd/100:35:031
3445216994621,6cyclictest131rcu_preempt00:23:371
3445216994612,6cyclictest131rcu_preempt00:27:121
344521199469,13cyclictest3447457-21sshd19:50:260
344521199467,35cyclictest0-21swapper/019:25:070
344521199465,37cyclictest0-21swapper/020:40:300
344522599459,9cyclictest0-21swapper/321:48:183
344522599455,8cyclictest37-21ksoftirqd/300:30:363
344522599454,8cyclictest37-21ksoftirqd/322:09:293
3445225994526,15cyclictest3446575-21sed19:35:023
3445216994534,7cyclictest23-21ksoftirqd/100:00:011
344521699451,9cyclictest0-21swapper/122:21:091
344521699451,9cyclictest0-21swapper/122:21:091
3445216994517,6cyclictest131rcu_preempt21:37:131
3445216994511,6cyclictest131rcu_preempt23:35:331
3445211994529,12cyclictest3447758-21systemctl20:00:000
3445211994517,23cyclictest1-21systemd19:25:010
344522599445,10cyclictest37-21ksoftirqd/300:18:103
344522599443,7cyclictest37-21ksoftirqd/319:26:073
3445225994427,11cyclictest275-21dbus-daemon20:45:013
344522199442,9cyclictest3445154-21latency_hist19:10:022
344522199442,9cyclictest3445154-21latency_hist19:10:022
344521699445,6cyclictest131rcu_preempt00:20:031
3445216994435,6cyclictest3494509-21munin-plugin-st22:50:001
3445216994434,8cyclictest23-21ksoftirqd/121:56:281
344521699441,7cyclictest0-21swapper/123:19:501
344521699441,7cyclictest0-21swapper/121:13:031
3445216994415,18cyclictest23-21ksoftirqd/123:10:151
3445216994415,18cyclictest23-21ksoftirqd/123:10:151
3445216994414,6cyclictest131rcu_preempt23:40:501
3445216994413,21cyclictest23-21ksoftirqd/121:41:251
3445216994413,21cyclictest23-21ksoftirqd/121:41:251
3445211994418,22cyclictest3451139-21latency_hist21:05:030
3445211994418,22cyclictest3451139-21latency_hist21:05:030
3445211994418,20cyclictest3451075-21latency_hist21:00:020
3445211994411,29cyclictest1895574-1kworker/0:1H20:10:020
3445225994326,12cyclictest37-21ksoftirqd/319:40:013
344522599431,8cyclictest3450607-21latency_hist20:55:013
344521699437,6cyclictest131rcu_preempt23:50:021
3445216994333,7cyclictest23-21ksoftirqd/122:50:291
3445216994318,21cyclictest23-21ksoftirqd/123:20:251
3445216994318,21cyclictest23-21ksoftirqd/123:20:251
344521699431,7cyclictest0-21swapper/100:34:371
3445216994311,5cyclictest131rcu_preempt00:02:261
344521699431,12cyclictest23-21ksoftirqd/123:32:271
3445211994332,8cyclictest12-21ksoftirqd/020:05:020
3445211994327,11cyclictest3445398-21cron19:14:590
3445211994314,25cyclictest3445857-21latency_hist19:20:020
3445216994232,7cyclictest23-21ksoftirqd/122:39:531
3445216994228,10cyclictest309-21in:imuxsock19:10:021
344521699421,7cyclictest0-21swapper/122:33:481
344521699421,6cyclictest0-21swapper/122:11:341
3445216994213,5cyclictest131rcu_preempt22:26:491
3445216994213,5cyclictest131rcu_preempt22:26:481
3445211994229,10cyclictest3448501-21sshd20:12:110
3445211994227,11cyclictest3448804-21systemctl20:20:000
3445211994226,12cyclictest12-21ksoftirqd/020:29:260
344522599415,8cyclictest37-21ksoftirqd/319:50:263
3445225994125,11cyclictest275-21dbus-daemon20:15:013
3445225994111,7cyclictest0-21swapper/320:25:313
3445216994130,5cyclictest3451432-21latency_hist21:10:011
344521699411,9cyclictest0-21swapper/121:53:191
344521699411,9cyclictest0-21swapper/121:53:191
344521699411,7cyclictest0-21swapper/121:28:411
344521699411,33cyclictest309-21in:imuxsock22:09:281
3445216994112,9cyclictest131rcu_preempt22:02:271
3445216994112,9cyclictest131rcu_preempt22:02:261
3445216994111,6cyclictest131rcu_preempt20:00:001
344521199412,35cyclictest3447003-21systemctl19:45:000
3445211994121,15cyclictest275-21dbus-daemon20:25:010
3445216994031,7cyclictest23-21ksoftirqd/121:23:271
3445216994031,7cyclictest23-21ksoftirqd/121:23:271
3445216994029,6cyclictest3450864-21sendmail21:00:001
3445216994025,11cyclictest1-21systemd20:10:011
3445216994024,11cyclictest3467602-21systemd-journal20:45:001
344521699401,6cyclictest0-21swapper/122:55:401
344521699401,5cyclictest0-21swapper/100:05:391
3445216994013,17cyclictest23-21ksoftirqd/123:06:521
3445216994011,9cyclictest23-21ksoftirqd/120:20:011
3445211994022,13cyclictest364-21snmpd20:32:310
344522599396,8cyclictest37-21ksoftirqd/321:00:023
3445216993924,11cyclictest23-21ksoftirqd/119:55:021
3445216993914,19cyclictest0-21swapper/120:30:021
3445211993920,15cyclictest3450324-21sshd20:45:270
3445211993920,15cyclictest3450324-21sshd20:45:260
3445225993820,13cyclictest275-21dbus-daemon19:45:013
344522599381,5cyclictest131rcu_preempt20:10:023
3445216993828,6cyclictest309-21in:imuxsock20:30:001
3445216993825,10cyclictest3467602-21systemd-journal20:49:271
3445216993825,10cyclictest3467602-21systemd-journal20:49:261
3445225993721,10cyclictest1-21systemd20:25:013
3445225993718,14cyclictest275-21dbus-daemon19:25:013
344521699379,5cyclictest131rcu_preempt21:01:271
344521699379,5cyclictest131rcu_preempt21:01:261
3445216993728,6cyclictest3445899-21latency_hist19:20:031
3445216993727,7cyclictest23-21ksoftirqd/119:35:011
3445216993725,9cyclictest23-21ksoftirqd/120:05:001
3445216993721,12cyclictest23-21ksoftirqd/120:15:011
3445225993628,4cyclictest309-21in:imuxsock20:00:003
344522599362,6cyclictest37-21ksoftirqd/319:11:013
3445225993516,14cyclictest275-21dbus-daemon19:20:003
3445216993524,9cyclictest23-21ksoftirqd/119:50:001
3445216993524,9cyclictest23-21ksoftirqd/119:24:551
3445216993522,10cyclictest23-21ksoftirqd/119:43:441
344521699349,5cyclictest131rcu_preempt19:30:011
3445216993423,8cyclictest23-21ksoftirqd/120:55:001
3445216993325,5cyclictest3449151-21latency_hist20:25:021
3445216993323,6cyclictest3445340-21latency_hist19:10:021
3445216993323,6cyclictest3445340-21latency_hist19:10:011
3445225993122,6cyclictest37-21ksoftirqd/320:01:313
3445225992715,8cyclictest3445347-21cut19:10:023
3445225992715,8cyclictest3445347-21cut19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional