You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 20:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Aug 10, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3607428999084,3cyclictest0-21swapper/010:10:020
3607443998438,15cyclictest3629162-21ssh09:42:083
3607443998438,15cyclictest3629162-21ssh09:42:083
3607428998337,6cyclictest12-21ksoftirqd/011:21:480
3607443998220,43cyclictest0-21swapper/312:30:023
3607428998129,18cyclictest0-21swapper/012:05:000
3607428997928,8cyclictest3615327-21ssh09:13:400
3607428997928,15cyclictest3704374-21ssh12:19:060
3607428997769,3cyclictest0-21swapper/011:00:020
3607443997631,11cyclictest3643325-21ssh10:11:463
3607443997426,15cyclictest3695198-21cat12:00:023
3607443997426,15cyclictest3695198-21cat12:00:023
3607443997424,15cyclictest3638073-21ssh10:00:383
3607443997325,14cyclictest3704432-21sh12:19:143
3607428997324,12cyclictest3684676-21ssh11:38:090
3607428997322,16cyclictest3692754-21ssh11:55:010
3607443997226,14cyclictest3608031-21sed07:20:013
3607443997225,16cyclictest3610617-21cat08:10:023
3607443997113,49cyclictest3647200-21ssh10:20:023
3607443997023,13cyclictest3611889-21latency_hist08:35:023
3607443997019,21cyclictest3613603-21diskmemload09:19:093
3607443996924,15cyclictest3608428-21munin-run07:30:013
3607443996923,15cyclictest3658005-21rm10:42:243
3607443996911,4cyclictest131rcu_preempt12:10:013
360743899691,9cyclictest131rcu_preempt11:52:202
360742899691,14cyclictest3467602-21systemd-journal10:38:500
3607443996822,14cyclictest3651761-21ssh10:29:413
3607428996824,29cyclictest12-21ksoftirqd/011:05:220
3607443996726,11cyclictest3608923-21latency_hist07:40:013
3607443996723,14cyclictest3637490-21ssh09:59:573
3607443996653,9cyclictest48-21kcompactd011:50:443
3607428996633,7cyclictest16950irq/224-800000010:00:000
3607443996549,13cyclictest3649439-21ssh10:24:533
3607443996452,9cyclictest3702103-21ssh12:14:233
3607443996448,13cyclictest3690199-21ssh11:49:423
3607443996447,14cyclictest3695643-21ssh12:00:373
3607443996428,11cyclictest3705716-21ssh12:21:403
3607443996424,12cyclictest3684196-21ssh11:37:093
3607428996429,16cyclictest3467602-21systemd-journal07:40:030
3607443996348,12cyclictest3710515-21ssh12:31:433
3607443996348,12cyclictest3678535-21ssh11:25:053
3607443996347,13cyclictest3655706-21ssh10:37:503
3607443996347,13cyclictest3619457-21ssh09:22:013
3607443996346,14cyclictest3712691-21ssh12:36:043
3607443996346,13cyclictest3672955-21ssh11:13:543
3607443996346,13cyclictest3672955-21ssh11:13:543
3607443996346,13cyclictest3652917-21ssh10:31:453
3607443996346,13cyclictest3634070-21ssh09:52:283
3607443996346,13cyclictest3634070-21ssh09:52:283
3607443996321,12cyclictest3613603-21diskmemload11:06:053
3607443996247,12cyclictest3630869-21ssh09:45:353
3607443996246,13cyclictest3664623-21ssh10:56:073
3607443996246,13cyclictest3663818-21ssh10:54:533
3607443996246,13cyclictest3640653-21ssh10:06:113
3607443996245,13cyclictest3627261-21ssh09:38:213
3607443996245,13cyclictest3614147-21ssh09:10:553
3607443996221,11cyclictest3677010-21ssh11:22:113
3607428996228,12cyclictest3661906-21rm10:50:240
3607443996146,13cyclictest3667303-21ssh11:01:423
3607443996146,12cyclictest3624101-21ssh09:31:423
3607443996146,12cyclictest3624101-21ssh09:31:423
3607443996146,11cyclictest3681177-21ssh11:30:433
3607443996144,13cyclictest3685946-21ssh11:40:343
3607443996144,13cyclictest3621565-21ssh09:26:273
3607443996120,12cyclictest1-21systemd07:50:013
3607443996120,12cyclictest1-21systemd07:50:003
3607428996123,8cyclictest12-21ksoftirqd/010:04:470
3607443996047,10cyclictest3659646-21ssh10:45:373
3607443996044,13cyclictest3674304-21ssh11:16:263
3607438996013,10cyclictest131rcu_preempt10:23:362
360742899603,44cyclictest19350irq/34-mmc109:00:020
3607428996033,11cyclictest3708243-21ssh12:26:560
3607428996031,13cyclictest3711483-21ssh12:33:530
3607443995944,12cyclictest3612780-21sshd08:50:353
3607428995912,42cyclictest0-21swapper/009:25:000
3607443995846,9cyclictest3611511-21sshd08:29:303
3607443995843,11cyclictest3613581-21sshd09:09:353
3607443995843,11cyclictest3613581-21sshd09:09:353
3607428995827,27cyclictest12-21ksoftirqd/012:38:020
3607443995744,9cyclictest48-21kcompactd007:23:153
3607443995744,9cyclictest48-21kcompactd007:23:153
3607443995744,10cyclictest3609917-21sshd07:55:093
3607443995742,11cyclictest3611006-21munin-run08:20:003
3607428995728,14cyclictest3651103-21ssh10:28:100
3607428995728,13cyclictest3624066-21ssh09:31:380
3607428995728,13cyclictest3624066-21ssh09:31:380
3607428995726,14cyclictest3675931-21ssh11:20:010
3607443995643,10cyclictest48-21kcompactd007:32:083
3607443995641,12cyclictest3467562-21sshd09:00:263
3607428995627,13cyclictest3629546-21ssh09:43:090
3607428995627,13cyclictest3629546-21ssh09:43:080
3607428995624,16cyclictest3699566-21ssh12:09:050
3607443995530,20cyclictest3613094-21lockfile-remove09:00:013
3607428995527,13cyclictest3679046-21ssh11:26:170
3607443995442,9cyclictest364-21snmpd08:42:533
3607443995440,11cyclictest364-21snmpd08:48:163
3607443995437,13cyclictest364-21snmpd07:10:243
3607438995444,6cyclictest3671430-21kworker/2:011:27:352
3607438995419,9cyclictest30-21ksoftirqd/212:35:262
360742899542,15cyclictest3618338-21modprobe09:20:010
3607443995346,4cyclictest3612072-21munin-run08:40:023
3607443995346,4cyclictest3612072-21munin-run08:40:023
3607438995345,5cyclictest3599758-21kworker/2:411:58:352
3607438995345,5cyclictest3599758-21kworker/2:411:58:352
3607438995345,5cyclictest3599758-21kworker/2:410:56:112
360742899535,33cyclictest16950irq/224-800000008:15:010
3607428995335,13cyclictest1-21systemd10:45:010
360742899532,3cyclictest131rcu_preempt09:40:010
360742899532,17cyclictest1-21systemd12:25:010
360742899531,3cyclictest131rcu_preempt09:49:520
3607443995242,8cyclictest0-21swapper/307:45:013
3607443995239,10cyclictest3609674-21sshd07:51:113
3607438995244,5cyclictest3599758-21kworker/2:412:15:192
3607438995244,5cyclictest3599758-21kworker/2:411:07:592
3607438995243,6cyclictest3599758-21kworker/2:410:08:162
3607438995243,6cyclictest3599758-21kworker/2:409:28:022
3607438995242,6cyclictest3599758-21kworker/2:412:20:252
3607438995216,5cyclictest131rcu_preempt10:14:092
360742899528,38cyclictest311-21rs:main0
3607443995137,10cyclictest364-21snmpd08:02:263
3607438995143,5cyclictest3697576-21kworker/2:112:30:432
3607438995143,5cyclictest3599758-21kworker/2:411:42:272
3607438995143,5cyclictest3599758-21kworker/2:411:31:122
3607438995143,5cyclictest3599758-21kworker/2:410:50:132
3607438995143,5cyclictest3599758-21kworker/2:410:35:082
3607438995143,5cyclictest3599758-21kworker/2:409:12:292
3607438995142,6cyclictest3671430-21kworker/2:011:20:492
3607438995142,5cyclictest3625574-21kworker/2:109:52:312
3607438995142,5cyclictest3625574-21kworker/2:109:52:312
3607438995120,6cyclictest30-21ksoftirqd/209:44:512
3607438995120,6cyclictest30-21ksoftirqd/209:44:512
3607435995113,10cyclictest23-21ksoftirqd/110:35:001
3607428995134,12cyclictest3647075-21cron10:20:000
360742899511,3cyclictest131rcu_preempt08:37:360
360742899511,3cyclictest131rcu_preempt08:37:360
3607438995042,5cyclictest3697576-21kworker/2:112:29:212
3607438995042,5cyclictest3625574-21kworker/2:110:19:082
3607438995042,5cyclictest3625574-21kworker/2:109:55:142
3607438995042,5cyclictest3625574-21kworker/2:109:37:302
3607438995042,5cyclictest3607921-21kworker/2:207:40:022
3607438995042,5cyclictest3599758-21kworker/2:412:00:192
3607438995042,5cyclictest3599758-21kworker/2:411:37:142
3607438995042,5cyclictest3599758-21kworker/2:411:00:542
3607438995042,5cyclictest3599758-21kworker/2:410:02:372
3607438995042,5cyclictest3599758-21kworker/2:409:31:152
3607438995042,5cyclictest3599758-21kworker/2:409:31:142
3607438995042,5cyclictest3599758-21kworker/2:409:22:102
3607438995041,6cyclictest3625574-21kworker/2:109:48:362
3607438995041,6cyclictest3599758-21kworker/2:409:16:592
360743899501,7cyclictest131rcu_preempt07:20:012
3607435995033,14cyclictest3613070-21cat09:00:011
360743599502,6cyclictest131rcu_preempt11:20:001
360743599502,6cyclictest131rcu_preempt09:20:001
360743599501,5cyclictest131rcu_preempt08:00:001
3607428995035,12cyclictest1-21systemd08:55:000
3607443994943,3cyclictest3610704-21sshd08:10:313
3607443994937,8cyclictest364-21snmpd08:23:003
3607443994930,16cyclictest3607442-21latency_hist07:10:023
3607438994941,6cyclictest3599758-21kworker/2:410:44:282
3607438994941,5cyclictest3671430-21kworker/2:011:16:422
3607438994941,5cyclictest3599758-21kworker/2:412:12:232
3607438994941,5cyclictest3599758-21kworker/2:412:09:052
3607438994941,5cyclictest3599758-21kworker/2:411:46:522
3607438994941,5cyclictest3599758-21kworker/2:411:10:252
3607438994941,5cyclictest3599758-21kworker/2:411:10:252
3607438994941,5cyclictest3599758-21kworker/2:410:30:512
3607438994941,5cyclictest3599758-21kworker/2:410:27:102
3607438994941,5cyclictest3599758-21kworker/2:409:07:012
3607438994941,5cyclictest3599758-21kworker/2:409:07:012
3607438994941,5cyclictest3599758-21kworker/2:409:00:202
3607438994941,5cyclictest3599758-21kworker/2:408:20:312
360743599497,5cyclictest131rcu_preempt10:11:481
3607428994942,4cyclictest16950irq/224-800000007:30:000
3607428994925,4cyclictest12-21ksoftirqd/010:35:020
3607428994922,12cyclictest3659424-21ssh10:45:090
3607428994922,12cyclictest3648976-21ssh10:23:560
3607438994841,4cyclictest3599758-21kworker/2:408:46:542
3607438994841,4cyclictest3599758-21kworker/2:408:05:252
3607438994840,5cyclictest3625574-21kworker/2:110:49:212
3607438994840,5cyclictest3607921-21kworker/2:207:41:102
3607438994840,5cyclictest3599758-21kworker/2:407:27:062
3607438994839,6cyclictest3599758-21kworker/2:408:42:292
3607428994835,9cyclictest3613606-21modprobe09:10:000
3607428994835,9cyclictest3613606-21modprobe09:10:000
3607428994831,13cyclictest12-21ksoftirqd/011:15:000
3607428994831,13cyclictest12-21ksoftirqd/011:15:000
3607428994822,10cyclictest3702491-21ssh12:15:020
360742899481,43cyclictest0-21swapper/009:55:020
360742899481,43cyclictest0-21swapper/009:55:020
3607438994739,5cyclictest3599758-21kworker/2:408:34:472
3607438994739,5cyclictest3599758-21kworker/2:407:22:172
3607438994739,5cyclictest3599758-21kworker/2:407:22:162
3607438994734,9cyclictest1-21systemd08:15:012
3607428994734,8cyclictest12-21ksoftirqd/011:05:000
3607428994719,13cyclictest3687230-21ssh11:43:250
3607428994714,28cyclictest0-21swapper/008:31:250
3607438994639,4cyclictest3607921-21kworker/2:207:55:482
3607438994639,4cyclictest3607921-21kworker/2:207:50:502
3607438994638,5cyclictest3607921-21kworker/2:208:00:082
3607438994638,5cyclictest3599758-21kworker/2:408:56:432
3607438994638,5cyclictest3599758-21kworker/2:408:50:542
3607438994638,5cyclictest3599758-21kworker/2:408:39:542
3607438994638,5cyclictest3599758-21kworker/2:408:39:532
3607438994638,5cyclictest3599758-21kworker/2:408:16:112
3607438994638,5cyclictest3599758-21kworker/2:407:11:322
3607435994631,11cyclictest275-21dbus-daemon10:20:011
3607435994631,11cyclictest1-21systemd11:15:011
3607435994631,11cyclictest1-21systemd11:15:011
3607435994621,21cyclictest23-21ksoftirqd/112:11:571
3607435994613,11cyclictest0-21swapper/107:40:021
360742899469,24cyclictest151rcuc/008:05:030
3607428994636,7cyclictest12-21ksoftirqd/011:49:210
3607428994632,11cyclictest16950irq/224-800000011:35:010
3607428994618,14cyclictest3643755-21ssh10:12:500
360742899461,3cyclictest131rcu_preempt08:00:000
3607438994537,5cyclictest3607921-21kworker/2:207:48:522
3607438994537,5cyclictest3607921-21kworker/2:207:48:512
3607438994537,5cyclictest3607921-21kworker/2:207:30:592
3607438994537,5cyclictest3599758-21kworker/2:408:25:392
3607435994530,11cyclictest23-21ksoftirqd/112:35:021
360743599451,28cyclictest3635153-21kworker/u8:410:30:021
360742899458,32cyclictest0-21swapper/007:15:010
360742899451,40cyclictest1-21systemd08:20:010
3607435994430,9cyclictest0-21swapper/112:30:021
3607435994427,11cyclictest3666289-21cron11:00:001
360743599442,13cyclictest3467602-21systemd-journal11:45:021
3607435994420,20cyclictest23-21ksoftirqd/112:20:411
3607428994440,2cyclictest0-21swapper/009:05:020
360742899442,8cyclictest3607893-21modprobe07:20:000
3607428994427,14cyclictest3695080-21ssh11:59:570
3607428994427,14cyclictest3695080-21ssh11:59:570
360743599438,22cyclictest23-21ksoftirqd/111:55:021
360743599433,5cyclictest131rcu_preempt12:40:001
3607435994332,7cyclictest3608015-21cat07:20:011
3607435994330,9cyclictest3467602-21systemd-journal07:40:031
3607435994330,9cyclictest23-21ksoftirqd/109:32:151
3607435994330,9cyclictest23-21ksoftirqd/109:32:151
3607435994328,12cyclictest23-21ksoftirqd/109:59:181
3607435994328,12cyclictest23-21ksoftirqd/109:36:361
3607435994326,13cyclictest23-21ksoftirqd/109:42:491
3607435994326,13cyclictest23-21ksoftirqd/109:42:491
3607435994325,12cyclictest1-21systemd09:15:011
360743599431,10cyclictest23-21ksoftirqd/110:06:111
360742899432,14cyclictest3623176-21munin-run09:30:020
360743599426,5cyclictest131rcu_preempt10:35:271
3607435994232,6cyclictest3613603-21diskmemload09:46:321
3607435994229,10cyclictest3467602-21systemd-journal12:00:321
3607435994229,10cyclictest275-21dbus-daemon12:00:011
3607435994229,10cyclictest275-21dbus-daemon12:00:011
3607435994228,10cyclictest1-21systemd08:20:001
360742899424,34cyclictest3612413-21latency_hist08:45:010
3607428994235,4cyclictest16950irq/224-800000007:50:010
3607428994235,4cyclictest16950irq/224-800000007:50:010
360742899421,37cyclictest12-21ksoftirqd/008:23:430
3607435994119,19cyclictest23-21ksoftirqd/110:53:481
3607435994116,14cyclictest23-21ksoftirqd/111:31:451
360743599411,5cyclictest131rcu_preempt12:20:001
3607435994030,7cyclictest3699793-21sshd12:09:411
3607435994029,8cyclictest23-21ksoftirqd/108:15:031
3607435994027,10cyclictest23-21ksoftirqd/111:49:141
3607435994027,10cyclictest23-21ksoftirqd/109:50:441
3607435994027,10cyclictest23-21ksoftirqd/109:50:441
3607428994030,8cyclictest12-21ksoftirqd/007:40:020
3607435993932,4cyclictest3612560-21munin-plugin-st08:50:001
3607435993928,8cyclictest23-21ksoftirqd/109:10:011
3607435993928,8cyclictest23-21ksoftirqd/109:10:011
3607435993928,7cyclictest3676305-21sshd11:20:351
3607435993923,13cyclictest23-21ksoftirqd/109:26:541
3607435993923,12cyclictest23-21ksoftirqd/110:05:001
3607435993917,4cyclictest23-21ksoftirqd/110:47:301
3607428993924,11cyclictest12-21ksoftirqd/007:25:020
3607428993924,11cyclictest12-21ksoftirqd/007:25:010
3607428993923,12cyclictest1-21systemd08:30:010
360743599386,5cyclictest131rcu_preempt11:36:321
3607435993826,9cyclictest23-21ksoftirqd/111:29:381
3607435993825,10cyclictest23-21ksoftirqd/110:42:521
3607435993821,12cyclictest275-21dbus-daemon07:55:001
360743599381,5cyclictest131rcu_preempt11:08:331
3607428993833,3cyclictest16950irq/224-800000008:46:350
3607428993823,10cyclictest3607422-21cyclictest07:10:020
3607438993725,9cyclictest3607566-21sed07:10:032
360743599378,7cyclictest131rcu_preempt07:15:021
360743599375,5cyclictest131rcu_preempt08:30:011
3607435993728,6cyclictest309-21in:imuxsock07:45:051
3607435993728,6cyclictest309-21in:imuxsock07:45:051
3607435993727,8cyclictest23-21ksoftirqd/110:25:011
3607435993724,9cyclictest23-21ksoftirqd/107:25:011
3607435993724,9cyclictest23-21ksoftirqd/107:25:011
3607435993724,8cyclictest3608673-21systemd-run07:35:011
3607435993724,10cyclictest23-21ksoftirqd/111:01:391
3607428993730,4cyclictest16950irq/224-800000008:10:000
360743599364,6cyclictest131rcu_preempt07:30:011
3607435993624,9cyclictest23-21ksoftirqd/108:05:011
360743599361,5cyclictest131rcu_preempt09:20:121
3607435993524,8cyclictest3612010-21sshd08:38:361
3607435993524,8cyclictest3612010-21sshd08:38:361
3607435993522,9cyclictest23-21ksoftirqd/107:10:021
3607435993520,9cyclictest309-21in:imuxsock08:06:321
360742899353,4cyclictest131rcu_preempt07:55:010
3607435993423,8cyclictest23-21ksoftirqd/108:55:011
3607435993421,10cyclictest23-21ksoftirqd/108:31:331
3607435993411,17cyclictest3467602-21systemd-journal09:05:001
3607435993317,8cyclictest23-21ksoftirqd/108:41:171
360743599311,5cyclictest131rcu_preempt08:20:471
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional