You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-18 - 13:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Mar 18, 2026 00:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3650690991237,3cyclictest131rcu_preempt00:30:011
365069099912,49cyclictest3713589-21sh23:16:221
3650687998581,2cyclictest0-21swapper/021:25:020
3650697998115,39cyclictest3721792-21cron23:35:022
365068799795,69cyclictest3650695-21cat19:10:010
365068799795,69cyclictest3650695-21cat19:10:010
365069799721,36cyclictest3720280-21ssh23:31:172
3650697997044,5cyclictest3644161-21kworker/2:320:35:022
3650697997044,5cyclictest3644161-21kworker/2:320:35:012
365069799691,34cyclictest3704946-21ssh22:56:572
3650690996923,41cyclictest3674127-21sh21:48:321
3650697996723,39cyclictest30-21ksoftirqd/222:10:012
3650697996527,34cyclictest3662959-21sh21:23:262
365069099654,56cyclictest3651668-21cron19:29:591
365069099652,60cyclictest3690467-21ssh22:25:011
3650697996425,34cyclictest3738291-21sh00:11:362
3650697996425,34cyclictest3738291-21sh00:11:352
3650697996422,37cyclictest1-21systemd00:05:022
3650690996331,26cyclictest275-21dbus-daemon21:40:011
365069099632,55cyclictest3680211-21sshd22:01:491
365069099631,16cyclictest3687111-21ssh22:17:181
3650687996318,40cyclictest16850irq/134-800000000:05:010
3650697996226,32cyclictest30-21ksoftirqd/223:07:092
3650697996224,35cyclictest30-21ksoftirqd/222:25:132
3650697996123,34cyclictest3653752-21latency_hist20:10:002
3650690996122,35cyclictest3729105-21sh23:50:541
3650690996122,34cyclictest3694857-21sh22:34:541
365069099611,16cyclictest3663522-21rm21:24:551
365069099611,15cyclictest3735763-21sh00:05:531
365069099611,15cyclictest3697404-21sh22:40:081
365069799602,52cyclictest3673244-21ssh21:46:142
365069799601,55cyclictest0-21swapper/220:30:022
365069799601,54cyclictest3746881-21ssh00:30:452
365069799601,53cyclictest3743090-21ssh00:22:322
365069799601,53cyclictest3701787-21latency_hist22:50:022
3650690996029,27cyclictest3667360-21sh21:33:241
365069099602,53cyclictest3676986-21cron21:55:011
365069099602,53cyclictest3664577-21sh21:26:501
365069099602,19cyclictest3657257-21ssh21:10:231
3650690996019,36cyclictest1-21systemd00:00:011
365069099601,17cyclictest3740047-21ssh00:15:241
3650697995929,25cyclictest3693883-21sh22:32:272
3650697995925,30cyclictest3713762-21sh23:16:482
365069799591,54cyclictest3732888-21ssh23:59:512
365069099592,53cyclictest3733714-21ssh00:01:191
365069099592,52cyclictest3651138-21cron19:20:001
365069099591,54cyclictest3692152-21ssh22:28:441
365069099591,53cyclictest3746606-21ssh00:30:061
365069099591,18cyclictest3742148-21ssh00:20:071
365069799582,51cyclictest3679447-21cut22:00:022
3650697995819,35cyclictest3740379-21sh00:16:262
365069799581,53cyclictest3724960-21ssh23:41:422
365069799581,52cyclictest3698622-21ssh22:43:082
365069799581,52cyclictest3668381-21ssh21:35:162
3650690995828,26cyclictest3725397-21sh23:42:451
365069099582,52cyclictest3720639-21sh23:32:071
365069099582,52cyclictest3701454-21sh22:49:371
365069099582,52cyclictest3682997-21ssh22:08:131
365069099582,52cyclictest3671834-21ssh21:43:171
365069099581,52cyclictest3677688-21ssh21:56:071
365069099581,17cyclictest3656876-21munin-run21:10:011
365069099581,16cyclictest3727869-21ssh23:48:201
3650687995826,12cyclictest3739859-21ssh00:15:030
3650687995826,12cyclictest3739859-21ssh00:15:020
3650697995744,7cyclictest3642836-21kworker/2:019:20:002
3650697995728,25cyclictest3718202-21sh23:26:352
3650697995725,28cyclictest3735561-21sh00:05:242
3650697995725,28cyclictest3711331-21sh23:11:102
365069799572,51cyclictest3702963-21ssh22:52:402
365069799572,51cyclictest3702963-21ssh22:52:392
3650697995723,30cyclictest30-21ksoftirqd/222:37:002
365069799571,52cyclictest3717166-21ssh23:24:322
365069799571,52cyclictest3717166-21ssh23:24:312
365069799571,52cyclictest3664099-21ssh21:25:442
365069799571,52cyclictest3659858-21ssh21:16:262
365069799571,51cyclictest3751095-21latency_hist00:40:032
365069799571,51cyclictest3684050-21ssh22:10:212
365069799571,51cyclictest3684050-21ssh22:10:202
365069099572,50cyclictest3684931-21ssh22:12:341
365069099572,50cyclictest3684931-21ssh22:12:331
365069099572,16cyclictest3738414-21sh00:11:521
365069099572,16cyclictest3738414-21sh00:11:521
365069099571,51cyclictest3722364-21ssh23:35:501
365069099571,51cyclictest3715385-21ssh23:20:071
365069099571,51cyclictest3715385-21ssh23:20:071
365068799572,39cyclictest3727703-21ssh23:47:570
365069799562,50cyclictest3744682-21sh00:25:562
365069799562,49cyclictest3657271-21sh21:10:242
365069799561,51cyclictest3729901-21ssh23:52:572
365069799561,51cyclictest3727669-21ssh23:47:532
365069799561,51cyclictest3686510-21ssh22:15:552
365069799561,51cyclictest3680788-21ssh22:03:212
365069799561,51cyclictest3676238-21ssh21:53:052
365069099562,50cyclictest3702086-21ssh22:50:341
365069099562,50cyclictest3702086-21ssh22:50:341
365069099562,49cyclictest3659967-21sshd21:16:401
365069099561,51cyclictest3695799-21ssh22:36:481
365069099561,50cyclictest3711654-21rm23:11:571
365069099561,18cyclictest3706070-21rm22:59:471
3650687995652,2cyclictest0-21swapper/020:05:020
365068799562,38cyclictest3659827-21ssh21:16:220
365069799552,49cyclictest3666555-21ssh21:31:182
365069799551,50cyclictest3688686-21ssh22:20:352
365069799551,50cyclictest3670808-21ssh21:40:432
365069799551,49cyclictest3656837-21sshd21:06:312
365069099552,49cyclictest3707754-21ssh23:03:301
365069099552,49cyclictest3707754-21ssh23:03:291
3650690995518,32cyclictest23-21ksoftirqd/123:07:291
365069099551,50cyclictest3718231-21ssh23:26:401
3650687995551,2cyclictest0-21swapper/021:30:020
3650687995545,5cyclictest16850irq/134-800000023:20:000
3650687995527,12cyclictest3717797-21ssh23:25:370
365070299547,27cyclictest0-21swapper/321:35:033
365070299542,33cyclictest3651901-21kworker/u8:219:50:023
3650697995443,7cyclictest0-21swapper/219:10:012
3650697995443,7cyclictest0-21swapper/219:10:012
365069799541,50cyclictest3707455-21ssh23:02:382
365069799541,50cyclictest3707455-21ssh23:02:372
365069799541,49cyclictest3654556-21systemd-run20:25:012
365069099542,47cyclictest3654310-21munin-run20:20:021
365069099542,47cyclictest3654310-21munin-run20:20:011
365069099542,45cyclictest3750118-21ssh00:38:191
3650687995427,12cyclictest3687091-21ssh22:17:160
3650687995426,13cyclictest3692364-21ssh22:29:120
3650687995425,13cyclictest3666661-21ssh21:31:340
365070299536,34cyclictest3687325-21kworker/u8:122:25:033
365069799539,38cyclictest275-21dbus-daemon20:15:012
365069799531,48cyclictest0-21swapper/220:20:002
365069799531,48cyclictest0-21swapper/220:20:002
365069099532,47cyclictest3656380-21munin-run21:00:001
365069099532,46cyclictest3655807-21sshd20:47:331
365069099532,46cyclictest3655807-21sshd20:47:321
365069099532,46cyclictest364-21snmpd19:35:441
365069099532,46cyclictest364-21snmpd19:35:441
365069099532,44cyclictest3654550-21cut20:24:591
365069099531,6cyclictest3653265-21munin-run20:00:021
365069099531,47cyclictest3655593-21munin-run20:45:011
3650690995310,39cyclictest3654823-21ls20:30:011
3650687995327,11cyclictest3746065-21ssh00:29:210
3650687995327,11cyclictest3707169-21ssh23:01:550
3650687995327,11cyclictest3707169-21ssh23:01:550
365068799532,4cyclictest131rcu_preempt21:45:330
3650687995324,14cyclictest3693371-21ssh22:31:150
3650697995226,23cyclictest3650882-21latency_hist19:15:022
365069799521,47cyclictest3656814-21date21:05:032
365069799521,47cyclictest3652938-21sshd19:51:212
365069099522,44cyclictest3655287-21cron20:40:001
365069099521,47cyclictest364-21snmpd20:05:511
365069099521,47cyclictest364-21snmpd19:45:461
3650687995226,23cyclictest3669366-21ssh21:37:430
3650687995225,12cyclictest3684081-21ssh22:10:250
3650687995225,12cyclictest3684081-21ssh22:10:240
3650687995217,17cyclictest3708657-21ssh23:05:060
3650687995211,7cyclictest16850irq/134-800000022:55:000
3650687995211,7cyclictest16850irq/134-800000022:55:000
365069799512,45cyclictest3655803-21sshd20:46:332
365069799512,45cyclictest3655803-21sshd20:46:332
365069799512,45cyclictest3655611-21cat20:45:022
365069799512,45cyclictest3651401-21sshd19:21:332
365069799512,44cyclictest3652718-21munin-run19:50:012
365069099512,45cyclictest364-21snmpd19:40:461
365069099512,45cyclictest364-21snmpd19:20:421
365069099512,44cyclictest3652990-21cat19:55:021
3650690995124,23cyclictest3653983-21sshd20:12:321
365069099511,10cyclictest3653678-21latency_hist20:05:021
3650687995129,8cyclictest3723439-21sshd23:38:340
3650702995034,13cyclictest275-21dbus-daemon22:45:013
365069799502,44cyclictest3656084-21munin-run20:55:012
365069799502,43cyclictest3655536-21latency_hist20:40:022
365069799501,45cyclictest3651648-21sshd19:26:172
3650690995022,24cyclictest3656615-21latency_hist21:05:021
3650690995022,24cyclictest3656615-21latency_hist21:05:011
365069099501,5cyclictest3655019-21sshd20:31:271
365069099501,5cyclictest3655019-21sshd20:31:261
3650687995034,13cyclictest3750827-21cron00:40:010
3650687995024,11cyclictest3696949-21ssh22:39:400
3650687995013,32cyclictest309-21in:imuxsock20:34:340
3650687995013,32cyclictest309-21in:imuxsock20:34:330
3650687995013,31cyclictest16850irq/134-800000000:20:010
3650702994935,10cyclictest3467602-21systemd-journal20:40:013
3650702994917,8cyclictest0-21swapper/322:49:073
3650702994914,6cyclictest131rcu_preempt23:07:283
365069799492,41cyclictest3656425-21latency_hist21:00:012
365069099491,43cyclictest364-21snmpd20:54:061
3650690994910,35cyclictest23-21ksoftirqd/119:11:121
3650687994934,11cyclictest12-21ksoftirqd/023:59:590
3650687994915,30cyclictest12-21ksoftirqd/021:13:390
3650687994914,31cyclictest12-21ksoftirqd/021:55:000
365070299484,3cyclictest131rcu_preempt00:06:073
3650702994834,10cyclictest3706181-21systemctl22:59:593
3650702994832,12cyclictest37-21ksoftirqd/321:00:013
3650702994812,6cyclictest131rcu_preempt22:16:543
3650702994812,6cyclictest131rcu_preempt21:38:493
365069799481,42cyclictest3652220-21/usr/sbin/munin19:40:022
365069799481,42cyclictest3652220-21/usr/sbin/munin19:40:012
365069099482,41cyclictest3652113-21latency_hist19:35:031
3650690994820,24cyclictest3650729-21cat19:10:031
3650690994820,24cyclictest3650729-21cat19:10:031
3650687994820,13cyclictest3677268-21ssh21:55:090
3650687994816,28cyclictest0-21swapper/022:07:590
3650702994730,13cyclictest275-21dbus-daemon20:00:003
365070299472,13cyclictest3661283-21ssh21:19:533
365069799472,40cyclictest364-21snmpd19:32:552
365069799472,38cyclictest3653516-21unin-run20:05:022
365069799471,41cyclictest3656779-21sed21:05:022
3650687994740,4cyclictest16950irq/224-800000022:50:000
3650687994740,4cyclictest16950irq/224-800000019:15:000
3650687994732,9cyclictest3699416-21cut22:45:000
365068799472,40cyclictest3656369-21modprobe21:00:010
3650687994714,29cyclictest16850irq/134-800000022:58:530
365068799471,41cyclictest3712253-21sshd23:13:330
365068799471,3cyclictest131rcu_preempt21:05:030
365068799471,15cyclictest3717351-21ssh23:25:020
365068799471,15cyclictest3717351-21ssh23:25:020
3650702994624,5cyclictest37-21ksoftirqd/323:45:283
3650702994619,18cyclictest275-21dbus-daemon23:20:003
365069799461,41cyclictest364-21snmpd19:55:482
3650687994639,4cyclictest16850irq/134-800000021:45:010
3650687994632,10cyclictest3467602-21systemd-journal23:50:330
3650687994629,13cyclictest3656588-21sshd21:01:320
3650687994619,12cyclictest3679608-21ssh22:00:210
3650687994613,29cyclictest3726002-21sh23:44:210
3650702994535,5cyclictest3735167-21cron00:05:013
3650702994523,6cyclictest37-21ksoftirqd/323:10:193
3650702994514,18cyclictest131rcu_preempt19:40:013
3650702994514,18cyclictest131rcu_preempt19:40:003
3650687994530,9cyclictest3748537-21cron00:35:000
3650687994523,15cyclictest3721813-21ssh23:35:010
365068799451,39cyclictest3467602-21systemd-journal22:23:470
3650702994425,14cyclictest3703909-21cron22:55:013
3650702994425,14cyclictest3703909-21cron22:55:003
3650687994429,10cyclictest1-21systemd20:05:020
3650687994427,13cyclictest3742334-21ssh00:20:320
3650702994327,13cyclictest1-21systemd23:59:593
365070299430,7cyclictest150-21pr/ttyS222:38:103
365069799432,37cyclictest364-21snmpd19:41:402
3650687994330,7cyclictest16950irq/224-800000020:20:010
3650687994330,7cyclictest16950irq/224-800000020:20:000
3650687994329,11cyclictest3735580-21ssh00:05:260
3650687994325,14cyclictest3655264-21sshd20:35:320
3650687994323,14cyclictest3467602-21systemd-journal20:00:000
3650702994230,8cyclictest37-21ksoftirqd/322:05:013
3650702994229,9cyclictest275-21dbus-daemon00:30:013
3650702994225,13cyclictest3467602-21systemd-journal23:40:033
3650702994224,13cyclictest275-21dbus-daemon20:20:023
3650702994224,13cyclictest275-21dbus-daemon20:20:013
3650687994235,4cyclictest16950irq/224-800000019:16:150
3650687994225,13cyclictest3656070-21sshd20:54:340
3650702994125,12cyclictest1-21systemd19:20:013
365070299411,11cyclictest3743465-21ssh00:23:233
3650687994129,7cyclictest19350irq/34-mmc119:35:000
3650687994128,7cyclictest3656874-21modprobe21:10:010
365068799411,5cyclictest131rcu_preempt20:30:020
365070299404,5cyclictest131rcu_preempt23:01:523
365070299404,5cyclictest131rcu_preempt23:01:513
3650702994030,5cyclictest3656872-21diskmemload22:00:013
3650702994028,9cyclictest3655807-21sshd20:47:333
3650702994028,9cyclictest3655807-21sshd20:47:333
3650702994027,10cyclictest3677014-21systemctl21:55:003
3650702994026,9cyclictest3467602-21systemd-journal19:28:333
3650702994026,10cyclictest309-21in:imuxsock19:51:343
3650702994025,11cyclictest3657335-21sshd21:10:323
3650702994018,16cyclictest275-21dbus-daemon00:20:013
365068799402,5cyclictest3651103-21kworker/u8:119:25:330
365068799401,35cyclictest1-21systemd19:22:330
3650702993930,7cyclictest37-21ksoftirqd/300:40:013
3650702993930,5cyclictest3656973-21latency_hist21:10:023
3650702993929,7cyclictest3730754-21cron23:55:013
365070299383,27cyclictest3739686-21true00:15:013
365070299383,27cyclictest3739686-21true00:15:003
3650702993828,7cyclictest3664807-21sshd21:27:333
3650702993827,9cyclictest37-21ksoftirqd/323:25:023
3650702993827,9cyclictest37-21ksoftirqd/323:25:013
3650702993823,12cyclictest3674447-21sh21:49:173
3650702993820,13cyclictest3467602-21systemd-journal20:45:003
365070299381,3cyclictest131rcu_preempt22:35:023
3650702993812,5cyclictest37-21ksoftirqd/321:23:383
3650687993827,7cyclictest12-21ksoftirqd/020:25:010
3650702993730,4cyclictest3690703-21sh22:25:133
3650702993729,5cyclictest3650874-21cron19:14:593
3650702993728,7cyclictest37-21ksoftirqd/323:26:363
3650702993728,5cyclictest3656071-21cron20:54:593
3650687993730,4cyclictest0-21swapper/019:55:010
365068799372,6cyclictest3655832-21munin-run20:50:010
365068799372,6cyclictest3655832-21munin-run20:50:000
365068799372,5cyclictest3652478-21munin-run19:45:000
365070299362,7cyclictest131rcu_preempt23:35:003
3650702993624,9cyclictest37-21ksoftirqd/322:10:063
3650702993624,9cyclictest37-21ksoftirqd/322:10:053
3650702993624,9cyclictest3653731-21sshd20:07:343
3650687993624,9cyclictest12-21ksoftirqd/020:44:330
365068799361,33cyclictest0-21swapper/019:36:030
365068799361,33cyclictest0-21swapper/019:36:020
365068799361,31cyclictest3652865-21cut19:50:030
365070299353,7cyclictest3650821-21latency_hist19:10:033
365070299353,7cyclictest3650821-21latency_hist19:10:023
3650702993518,11cyclictest309-21in:imuxsock23:35:033
3650702993510,6cyclictest131rcu_preempt00:34:463
3650702993425,5cyclictest37-21ksoftirqd/322:06:483
3650702993422,9cyclictest3671266-21sshd21:41:453
3650702993422,8cyclictest309-21in:imuxsock19:40:333
365070299342,24cyclictest3655050-21munin-plugin-st20:35:003
365070299342,24cyclictest3655050-21munin-plugin-st20:35:003
365068799343,26cyclictest0-21swapper/020:12:520
365070299335,22cyclictest3467602-21systemd-journal20:30:013
365070299332,28cyclictest48-21kcompactd019:22:373
365070299323,23cyclictest3656620-21grep21:05:003
365070299323,23cyclictest3656620-21grep21:05:003
3650702993221,8cyclictest37-21ksoftirqd/320:15:013
3650702993219,9cyclictest0-21swapper/320:25:003
365070299321,5cyclictest131rcu_preempt19:35:013
3650702993118,10cyclictest1-21systemd20:05:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional