You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 17:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jun 19, 2026 12:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3702986991043,3cyclictest131rcu_preempt12:20:021
370297899851,2cyclictest0-21swapper/007:55:020
3702986996917,13cyclictest3715029-21sh09:22:391
3702993996823,6cyclictest131rcu_preempt11:35:003
3702986996618,16cyclictest3776229-21ssh11:38:161
3702986996618,16cyclictest3776229-21ssh11:38:151
370299199657,4cyclictest131rcu_preempt07:20:012
3702991996536,23cyclictest30-21ksoftirqd/208:28:342
3702991996518,17cyclictest3709007-21wc09:05:022
3702991996442,18cyclictest3755631-21ssh10:52:432
3702991996442,18cyclictest3755631-21ssh10:52:432
3702991996343,16cyclictest364-21snmpd11:43:492
3702991996321,37cyclictest3705490-21cron08:00:002
3702991996313,10cyclictest131rcu_preempt09:12:362
3702978996328,11cyclictest3729228-21ssh09:54:190
3702991996241,16cyclictest3782947-21ssh11:53:252
3702991996241,16cyclictest3716634-21ssh09:26:022
3702991996241,16cyclictest3716634-21ssh09:26:012
3702991996240,17cyclictest3727569-21ssh09:50:162
3702978996230,29cyclictest3796115-21sshd12:22:320
370297899621,4cyclictest131rcu_preempt12:00:010
370297899621,4cyclictest131rcu_preempt12:00:000
3702991996152,6cyclictest3705009-21kworker/2:210:45:192
3702991996141,13cyclictest3758794-21cron10:59:592
3702978996128,30cyclictest3741050-21ssh10:20:070
3702978996128,30cyclictest3741050-21ssh10:20:070
3702991996043,14cyclictest3713984-21ssh09:20:092
3702991996040,16cyclictest3720735-21ssh09:35:122
3702991996040,16cyclictest3720735-21ssh09:35:112
3702991996039,17cyclictest3771753-21ssh11:28:242
3702991996039,17cyclictest3750798-21ssh10:41:502
3702991996039,17cyclictest3733686-21ssh10:04:102
3702991996039,17cyclictest364-21snmpd11:34:342
3702991996038,18cyclictest3787199-21ssh12:02:482
3702991995943,13cyclictest3767932-21munin-run11:20:012
3702991995943,13cyclictest3767932-21munin-run11:20:012
3702991995943,13cyclictest3731417-21ssh09:59:122
3702991995943,12cyclictest3793255-21ssh12:16:082
3702991995942,13cyclictest3739136-21ssh10:15:562
3702991995940,16cyclictest3765458-21ssh11:14:442
3702991995940,15cyclictest3711931-21ssh09:15:382
3702991995939,16cyclictest3789484-21ssh12:07:562
3702991995939,16cyclictest3789484-21ssh12:07:552
3702991995939,16cyclictest3768344-21ssh11:20:292
3702991995938,17cyclictest3719228-21ssh09:31:582
3702991995936,19cyclictest3737126-21ssh10:11:352
370297899599,45cyclictest3747809-21ssh10:35:080
370297899599,45cyclictest3747809-21ssh10:35:070
370297899597,48cyclictest3716568-21ssh09:25:530
370297899597,48cyclictest3716568-21ssh09:25:520
3702978995927,28cyclictest3779609-21sshd11:45:320
3702978995927,28cyclictest3779609-21sshd11:45:310
3702978995918,32cyclictest0-21swapper/011:08:390
3702991995842,14cyclictest3798728-21ssh12:28:242
3702991995842,13cyclictest3723289-21ssh09:40:422
3702991995842,12cyclictest3800053-21ssh12:31:132
3702991995842,12cyclictest3759346-21ssh11:00:492
3702991995841,14cyclictest3743514-21ssh10:25:372
3702991995841,14cyclictest3743514-21ssh10:25:362
3702991995840,15cyclictest3792098-21ssh12:13:432
3702991995839,16cyclictest3783884-21ssh11:55:092
3702991995839,16cyclictest3783884-21ssh11:55:092
3702991995839,16cyclictest3741500-21ssh10:21:172
3702991995839,16cyclictest3741500-21ssh10:21:162
3702991995838,16cyclictest3745638-21ssh10:30:152
3702991995838,16cyclictest3734904-21ssh10:06:372
3702986995833,21cyclictest3723253-21ssh09:40:381
3702978995836,18cyclictest3726398-21ssh09:48:000
3702978995833,8cyclictest3735328-21ssh10:07:430
3702978995830,13cyclictest3714702-21ssh09:21:550
3702978995825,29cyclictest12-21ksoftirqd/010:18:440
3702993995712,27cyclictest3783750-21ssh11:55:013
3702991995744,10cyclictest3705229-21sshd07:53:322
3702991995743,12cyclictest3748102-21ssh10:35:452
3702991995743,12cyclictest3748102-21ssh10:35:452
3702991995742,12cyclictest3795513-21ssh12:21:022
3702991995741,13cyclictest3762826-21ssh11:08:432
3702991995741,12cyclictest3779487-21ssh11:45:152
3702991995741,12cyclictest3779487-21ssh11:45:142
3702991995739,15cyclictest3801804-21ssh12:35:042
3702991995739,14cyclictest3707602-21sshd08:36:322
3702991995738,16cyclictest3775483-21ssh11:36:252
3702991995738,16cyclictest3775483-21ssh11:36:242
3702991995738,14cyclictest3725460-21ssh09:45:362
3702978995728,14cyclictest3730565-21ssh09:57:100
3702991995641,12cyclictest3704727-21sshd07:43:322
3702991995634,15cyclictest3706543-21modprobe08:20:002
3702986995634,18cyclictest3780141-21ssh11:46:541
3702986995634,18cyclictest3780141-21ssh11:46:531
3702978995620,32cyclictest3752258-21cat10:45:030
3702978995618,34cyclictest16850irq/134-800000011:29:230
3702978995613,39cyclictest0-21swapper/012:12:080
3702991995541,10cyclictest3707883-21sshd08:41:332
3702991995541,10cyclictest3707883-21sshd08:41:322
3702991995540,11cyclictest3708152-21cron08:49:592
3702991995540,11cyclictest3703187-21munin-run07:15:012
3702991995537,14cyclictest364-21snmpd08:31:522
3702991995536,15cyclictest364-21snmpd07:22:362
3702986995535,16cyclictest3770941-21ssh11:26:261
3702986995532,19cyclictest3726594-21ssh09:48:251
3702978995540,12cyclictest16950irq/224-800000012:20:010
3702991995440,10cyclictest3708398-21sshd08:53:452
3702991995439,11cyclictest3704509-21munin-run07:40:022
3702986995438,13cyclictest3730420-21ssh09:56:491
3702986995436,15cyclictest3784699-21ssh11:57:171
3702986995436,15cyclictest3784699-21ssh11:57:161
3702986995433,17cyclictest3792354-21ssh12:14:271
3702986995433,17cyclictest3712336-21ssh09:16:411
3702986995430,19cyclictest3467602-21systemd-journal07:40:011
3702986995430,18cyclictest3749047-21ssh10:38:101
3702986995430,18cyclictest3749047-21ssh10:38:091
3702986995428,15cyclictest3758796-21cron11:00:001
3702978995426,13cyclictest3759391-21ssh11:00:550
3702978995425,25cyclictest3733592-21ssh10:03:570
3702978995418,8cyclictest3467602-21systemd-journal10:45:270
3702978995417,32cyclictest16850irq/134-800000009:18:500
370297899541,38cyclictest3766062-21ssh11:15:340
370297899541,38cyclictest3766062-21ssh11:15:340
3702991995341,9cyclictest3708645-21sshd08:58:312
3702991995339,10cyclictest3706792-21sshd08:20:322
3702991995339,10cyclictest3705757-21sshd08:02:332
3702991995327,21cyclictest275-21dbus-daemon07:35:002
3702986995337,13cyclictest3797806-21ssh12:26:111
3702986995337,12cyclictest1-21systemd12:05:001
3702986995334,15cyclictest3732053-21ssh10:00:131
3702986995333,16cyclictest3795924-21ssh12:22:031
3702986995333,16cyclictest3778313-21ssh11:42:591
3702986995333,16cyclictest3738981-21ssh10:15:341
3702986995333,15cyclictest3736943-21ssh10:11:021
3702986995332,17cyclictest3800332-21ssh12:31:521
3702986995332,17cyclictest3766072-21ssh11:15:371
3702986995332,17cyclictest3766072-21ssh11:15:371
3702978995333,15cyclictest3702972-21cyclictest07:10:010
3702978995328,10cyclictest3747375-21ssh10:34:390
3702978995325,13cyclictest3743484-21ssh10:25:330
3702978995325,13cyclictest3743484-21ssh10:25:320
3702978995324,14cyclictest3803390-21ssh12:39:070
370297899532,13cyclictest3776929-21modprobe11:40:010
370297899532,13cyclictest3776929-21modprobe11:40:000
3702978995317,32cyclictest16850irq/134-800000012:34:070
3702978995317,32cyclictest16850irq/134-800000011:42:000
3702978995317,31cyclictest16850irq/134-800000009:42:260
3702978995316,32cyclictest16850irq/134-800000010:56:140
3702978995311,38cyclictest3788182-21latency_hist12:05:020
3702978995311,11cyclictest3798140-21ssh12:26:570
3702993995236,13cyclictest1-21systemd10:55:013
3702993995236,13cyclictest1-21systemd10:55:003
3702993995233,15cyclictest364-21snmpd12:04:593
3702991995244,5cyclictest3709173-21sshd09:07:322
3702991995244,5cyclictest3709173-21sshd09:07:312
3702991995238,9cyclictest48-21kcompactd007:47:582
3702991995238,9cyclictest48-21kcompactd007:47:572
3702986995239,10cyclictest3728056-21ssh09:51:351
3702986995237,11cyclictest48-21kcompactd012:35:061
3702986995236,13cyclictest3755504-21ssh10:52:221
3702986995236,13cyclictest3755504-21ssh10:52:211
3702986995236,13cyclictest3744467-21ssh10:28:011
3702986995236,13cyclictest3744467-21ssh10:28:001
3702986995236,13cyclictest3734386-21ssh10:05:211
3702986995236,13cyclictest3706528-21sshd08:18:371
3702986995236,12cyclictest3761459-21ssh11:05:171
3702986995236,12cyclictest3718883-21ssh09:31:131
3702986995235,13cyclictest3764184-21ssh11:11:361
3702986995235,13cyclictest3745658-21ssh10:30:181
3702986995234,15cyclictest3789873-21ssh12:08:481
3702986995234,15cyclictest3789873-21ssh12:08:481
3702986995232,17cyclictest3710695-21ssh09:13:111
3702986995231,17cyclictest3721029-21ssh09:35:501
3702986995231,17cyclictest3721029-21ssh09:35:501
3702978995224,20cyclictest0-21swapper/011:24:320
3702978995218,31cyclictest16850irq/134-800000012:07:580
3702978995218,31cyclictest16850irq/134-800000012:07:570
3702978995218,19cyclictest16850irq/134-800000009:11:340
3702978995217,31cyclictest16850irq/134-800000009:38:460
3702978995217,31cyclictest16850irq/134-800000009:38:460
3702978995216,21cyclictest16850irq/134-800000010:11:040
3702991995138,9cyclictest364-21snmpd08:14:102
3702991995137,11cyclictest364-21snmpd07:29:292
3702986995136,12cyclictest3768413-21ssh11:20:431
3702986995136,12cyclictest3760063-21sshd11:02:311
3702986995136,12cyclictest3753234-21ssh10:47:151
3702986995136,12cyclictest3750189-21ssh10:40:151
3702986995136,10cyclictest3703702-21cron07:25:001
3702986995135,13cyclictest3781658-21ssh11:50:071
3702986995135,13cyclictest3742155-21ssh10:22:541
3702986995135,13cyclictest3742155-21ssh10:22:541
3702986995134,13cyclictest3772758-21ssh11:30:161
3702986995132,14cyclictest3468525-21sendmail-mta07:40:051
3702986995132,14cyclictest275-21dbus-daemon08:55:001
3702986995130,17cyclictest3716453-21ssh09:25:371
3702986995130,17cyclictest3716453-21ssh09:25:371
3702978995118,29cyclictest16850irq/134-800000009:30:200
3702978995117,19cyclictest16850irq/134-800000011:51:510
3702978995113,34cyclictest16850irq/134-800000011:35:010
3702993995010,9cyclictest131rcu_preempt12:20:573
3702986995036,10cyclictest3706036-21munin-run08:10:011
370297899505,40cyclictest3709417-21latency_hist09:10:030
370297899505,40cyclictest3709417-21latency_hist09:10:030
3702978995026,9cyclictest3764713-21ssh11:12:590
3702993994920,9cyclictest131rcu_preempt09:51:563
3702993994920,26cyclictest37-21ksoftirqd/309:21:543
3702986994935,10cyclictest3707383-21munin-run08:35:001
3702978994914,32cyclictest16850irq/134-800000010:52:120
3702978994914,32cyclictest16850irq/134-800000010:52:110
3702993994837,8cyclictest37-21ksoftirqd/311:10:013
3702993994829,14cyclictest0-21swapper/309:50:013
3702991994840,5cyclictest3696568-21kworker/2:307:10:022
3702986994835,10cyclictest3703476-21munin-run07:20:021
3702986994833,10cyclictest3705547-21munin-run08:00:021
3702986994832,12cyclictest3707048-21sshd08:26:321
3702986994824,16cyclictest3705773-21cron08:05:001
3702991994733,10cyclictest364-21snmpd08:07:322
3702986994734,10cyclictest3704190-21sshd07:32:311
3702986994730,13cyclictest3704964-21sshd07:45:331
3702986994730,13cyclictest3704964-21sshd07:45:321
3702986994728,15cyclictest364-21snmpd07:27:571
3702986994723,19cyclictest3708116-21cat08:45:021
370297899476,4cyclictest3704997-21modprobe07:50:010
370297899476,4cyclictest3704997-21modprobe07:50:000
3702978994732,12cyclictest3704442-21sshd07:37:320
3702978994725,17cyclictest3707100-21apt-config08:28:340
370299399465,11cyclictest37-21ksoftirqd/307:40:003
3702993994636,7cyclictest37-21ksoftirqd/311:05:013
3702993994634,7cyclictest309-21in:imuxsock09:05:003
3702986994634,9cyclictest3709177-21sshd09:08:321
3702986994634,9cyclictest3709177-21sshd09:08:311
3702986994632,10cyclictest3706251-21sendmail-mta08:10:041
3702986994627,15cyclictest364-21snmpd08:40:221
3702986994622,18cyclictest3703233-21cat07:15:021
370297899466,36cyclictest0-21swapper/007:20:000
3702978994630,11cyclictest3703946-21sshd07:28:300
3702978994628,13cyclictest275-21dbus-daemon07:15:020
370299399454,7cyclictest131rcu_preempt10:35:013
3702993994533,9cyclictest37-21ksoftirqd/310:43:053
3702993994531,12cyclictest37-21ksoftirqd/310:20:293
3702993994531,12cyclictest37-21ksoftirqd/310:20:283
3702993994530,10cyclictest309-21in:imuxsock09:25:333
3702993994530,10cyclictest309-21in:imuxsock09:25:323
370299399451,8cyclictest0-21swapper/309:39:213
370299399451,8cyclictest0-21swapper/309:39:203
370299399451,4cyclictest0-21swapper/309:58:063
3702986994533,9cyclictest48-21kcompactd008:55:051
3702986994532,9cyclictest48-21kcompactd008:22:551
3702986994527,14cyclictest364-21snmpd08:38:081
3702978994530,11cyclictest3704182-21sshd07:30:300
3702978994529,13cyclictest3705480-21sshd07:58:310
3702978994527,12cyclictest1-21systemd08:55:010
370299399449,5cyclictest131rcu_preempt10:25:383
370299399449,5cyclictest131rcu_preempt10:25:383
370299399448,7cyclictest37-21ksoftirqd/309:15:003
3702993994428,12cyclictest37-21ksoftirqd/308:05:003
370299399441,8cyclictest0-21swapper/311:56:563
370299399441,8cyclictest0-21swapper/311:56:563
370299399441,7cyclictest0-21swapper/309:19:153
370299399441,6cyclictest0-21swapper/311:27:173
3702993994413,9cyclictest131rcu_preempt10:10:143
3702986994437,4cyclictest3708963-21munin-run09:05:011
3702986994426,14cyclictest364-21snmpd07:52:121
3702986994425,14cyclictest3703022-21cat07:10:021
3702993994334,7cyclictest37-21ksoftirqd/312:25:163
3702993994333,7cyclictest37-21ksoftirqd/309:33:103
3702993994331,9cyclictest37-21ksoftirqd/312:11:173
3702993994329,10cyclictest37-21ksoftirqd/312:35:013
3702993994326,12cyclictest275-21dbus-daemon12:20:013
370299399431,8cyclictest37-21ksoftirqd/311:44:403
370299399431,5cyclictest0-21swapper/311:45:083
370299399431,5cyclictest0-21swapper/311:45:083
3702986994328,12cyclictest364-21snmpd08:47:101
3702978994329,10cyclictest3707364-21sshd08:34:310
370299399424,5cyclictest131rcu_preempt08:13:323
370299399423,6cyclictest131rcu_preempt10:09:063
3702993994232,7cyclictest37-21ksoftirqd/311:15:013
3702993994226,12cyclictest37-21ksoftirqd/310:50:023
370299399421,5cyclictest0-21swapper/312:09:013
370299399421,5cyclictest0-21swapper/312:09:003
370299399421,5cyclictest0-21swapper/307:40:333
3702993994211,5cyclictest131rcu_preempt11:36:213
3702993994211,5cyclictest131rcu_preempt11:36:203
3702978994230,9cyclictest3703698-21sshd07:24:290
3702978994224,12cyclictest3706545-21modprobe08:20:010
3702993994132,7cyclictest37-21ksoftirqd/310:02:523
3702993994130,8cyclictest3769919-21sshd11:24:313
3702993994129,9cyclictest37-21ksoftirqd/310:59:543
370299399411,5cyclictest0-21swapper/310:20:003
3702993994110,5cyclictest131rcu_preempt11:16:313
3702993994110,5cyclictest131rcu_preempt11:16:303
3702978994134,4cyclictest16950irq/224-800000008:50:000
3702978994128,10cyclictest3704762-21munin-run07:45:000
3702978994127,9cyclictest48-21kcompactd008:04:360
370299399409,6cyclictest131rcu_preempt08:32:523
370299399404,5cyclictest131rcu_preempt08:08:243
370299399401,7cyclictest0-21swapper/310:36:173
370299399401,7cyclictest0-21swapper/310:36:173
370299399401,7cyclictest0-21swapper/309:43:193
370299399401,14cyclictest0-21swapper/307:57:323
3702978994027,10cyclictest3707674-21munin-run08:40:010
370297899401,8cyclictest0-21swapper/008:25:000
3702978994013,22cyclictest3706295-21true08:15:000
370299399397,5cyclictest131rcu_preempt12:38:443
3702993993929,8cyclictest37-21ksoftirqd/308:55:013
3702993993926,10cyclictest3708151-21cron08:49:593
370299399391,5cyclictest0-21swapper/307:45:553
370299399391,5cyclictest0-21swapper/307:45:553
3702993993911,8cyclictest131rcu_preempt07:13:423
3702993993911,4cyclictest131rcu_preempt08:18:323
3702978993927,8cyclictest48-21kcompactd009:02:170
3702978993926,9cyclictest48-21kcompactd008:07:550
3702978993925,9cyclictest3707907-21modprobe08:45:000
3702978993925,9cyclictest3707907-21modprobe08:45:000
3702978993916,19cyclictest3708655-21cron09:00:000
3702993993810,5cyclictest131rcu_preempt07:27:113
370299399374,5cyclictest131rcu_preempt07:54:543
3702993993726,7cyclictest37-21ksoftirqd/307:20:003
370299399361,7cyclictest0-21swapper/308:37:523
3702993993612,4cyclictest131rcu_preempt08:27:323
3702993993611,5cyclictest131rcu_preempt08:43:453
3702993993611,5cyclictest131rcu_preempt08:43:443
370299399356,5cyclictest131rcu_preempt07:23:273
3702993993525,8cyclictest37-21ksoftirqd/307:35:023
370299399349,5cyclictest131rcu_preempt08:20:483
370299399341,6cyclictest0-21swapper/308:55:323
3702993993310,6cyclictest131rcu_preempt09:05:093
3702993993310,6cyclictest131rcu_preempt09:05:083
3702993993120,8cyclictest3702999-21sed07:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional