You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-20 - 13:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jun 20, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
386228099935,12cyclictest3467602-21systemd-journal00:10:000
386228699921,54cyclictest3864266-21sshd19:46:362
3862280998914,63cyclictest16850irq/134-800000022:10:010
3862280998025,37cyclictest3915375-21ssh22:55:290
3862280998014,6cyclictest16850irq/134-800000000:39:360
3862280997916,6cyclictest16850irq/134-800000022:25:450
386228099772,70cyclictest3886027-21munin-run21:50:020
3862280997726,47cyclictest3908779-21ssh22:40:440
3862280997719,15cyclictest3890063-21ssh21:58:570
3862280997718,6cyclictest16850irq/134-800000023:58:450
3862280997714,5cyclictest16850irq/134-800000021:20:120
3862280997529,28cyclictest3925512-21ssh23:18:460
3862280997423,32cyclictest3920840-21ssh23:08:030
3862280997418,6cyclictest16850irq/134-800000021:19:230
3862291997228,39cyclictest3923986-21sed23:15:023
386228699721,41cyclictest3934711-21ssh23:39:352
3862280997235,33cyclictest16850irq/134-800000023:31:280
386228099722,7cyclictest3863770-21systemctl19:40:000
3862280997220,45cyclictest275-21dbus-daemon20:50:010
386228099721,53cyclictest0-21swapper/000:30:030
3862280997125,30cyclictest0-21swapper/023:26:340
3862280997119,13cyclictest3879669-21ssh21:35:210
3862280997025,41cyclictest3904439-21ssh22:30:530
3862280997022,32cyclictest3886677-21ssh21:51:020
3862280997022,18cyclictest3868476-21ssh21:10:010
3862280997021,9cyclictest3913303-21ssh22:50:500
3862280997014,46cyclictest16850irq/134-800000000:11:060
3862286996825,39cyclictest275-21dbus-daemon19:20:012
386228099681,63cyclictest3879377-21latency_hist21:35:010
386228099681,5cyclictest3876476-21ssh21:28:300
386228099681,5cyclictest3876476-21ssh21:28:290
3862280996815,7cyclictest16850irq/134-800000023:45:470
386228099673,24cyclictest3956944-21cut00:30:000
386228099672,7cyclictest3864806-21munin-run20:00:010
3862280996725,38cyclictest3928161-21sh23:24:510
386228099669,13cyclictest3891471-21ssh22:01:510
386228099669,13cyclictest3891471-21ssh22:01:510
386228099662,27cyclictest3941427-21sh23:55:000
3862280996620,2cyclictest0-21swapper/022:10:450
3862280996620,2cyclictest0-21swapper/022:10:450
3862280996617,12cyclictest3899233-21ssh22:19:370
3862280996616,6cyclictest16850irq/134-800000022:46:550
3862280996616,6cyclictest16850irq/134-800000022:46:550
3862291996525,36cyclictest3928928-21sh23:26:143
3862291996524,36cyclictest3867651-21cron20:54:593
3862286996524,37cyclictest3909713-21sh22:43:002
386228099654,20cyclictest3934860-21latency_hist23:39:590
386228099654,14cyclictest3866126-21grep20:25:000
3862280996513,47cyclictest16850irq/134-800000021:11:110
386228099651,2cyclictest0-21swapper/019:15:360
386228699642,57cyclictest3939848-21ssh23:50:502
386228699642,56cyclictest3929460-21ssh23:27:362
3862286996425,34cyclictest3942778-21sshd23:57:472
3862286996423,37cyclictest275-21dbus-daemon19:30:012
386228699641,58cyclictest3914069-21ssh22:52:432
386228699641,10cyclictest3902563-21ssh22:26:372
386228099649,20cyclictest3883740-21modprobe21:45:000
386228099644,24cyclictest3867925-21cut21:00:000
386228099643,19cyclictest3865582-21munin-plugin-st20:15:000
3862280996427,33cyclictest12-21ksoftirqd/023:10:330
3862280996422,3cyclictest0-21swapper/023:04:210
3862291996324,35cyclictest3924683-21sh23:16:443
3862286996327,32cyclictest3922880-21sh23:12:382
386228699632,57cyclictest3892617-21ssh22:04:482
386228699632,57cyclictest3892617-21ssh22:04:472
386228699631,21cyclictest3927814-21ssh23:24:042
386228699631,16cyclictest48-21kcompactd000:00:172
386228699631,15cyclictest3959598-21sh00:35:332
386228699631,14cyclictest3872051-21ssh21:18:292
386228699631,13cyclictest3918520-21ssh23:02:442
386228699631,13cyclictest3894451-21ssh22:08:472
3862280996326,33cyclictest309-21in:imuxsock19:30:000
3862280996323,34cyclictest3865351-21grep20:10:020
3862280996323,34cyclictest3865351-21grep20:10:020
3862280996319,5cyclictest3863542-21latency_hist19:35:020
386228099631,57cyclictest0-21swapper/022:22:410
3862291996223,35cyclictest3869494-21sh21:12:333
386228699621,8cyclictest3948652-21ssh00:10:442
386228699621,57cyclictest3911967-21ssh22:48:002
386228699621,57cyclictest3911967-21ssh22:48:002
386228699621,57cyclictest3881252-21ssh21:39:252
386228699621,56cyclictest3899024-21ssh22:18:582
386228699621,13cyclictest3896371-21ssh22:13:072
386228699621,13cyclictest3896371-21ssh22:13:062
386228099624,21cyclictest309-21in:imuxsock20:05:010
386229199612,19cyclictest3895585-21ssh22:11:023
386229199612,19cyclictest3895585-21ssh22:11:013
386229199612,17cyclictest3866011-21cat20:20:023
3862291996121,36cyclictest3874949-21latency_hist21:25:023
386229199611,19cyclictest3871910-21ssh21:18:113
386228699612,55cyclictest3938594-21ssh23:48:092
386228699612,55cyclictest3924155-21ssh23:15:242
386228699612,55cyclictest3888716-21ssh21:55:342
386228699612,55cyclictest3868949-21ssh21:11:072
386228699612,54cyclictest3952906-21ssh00:20:182
386228699612,53cyclictest3866377-21munin-run20:30:002
386228699611,8cyclictest3955100-21ssh00:25:192
386228699611,8cyclictest3951823-21ssh00:18:062
386228699611,8cyclictest3919858-21ssh23:05:372
386228699611,7cyclictest3884114-21ssh21:45:202
386228699611,56cyclictest3879224-21ssh21:34:522
386228699611,16cyclictest3882858-21rm21:42:512
386228099616,12cyclictest3867172-21unin-run20:45:010
386228099612,9cyclictest3944440-21ssh00:01:130
386228099612,55cyclictest3914922-21kworker/u8:023:40:120
386228099612,18cyclictest3866585-21latency_hist20:30:020
3862291996027,29cyclictest3467602-21systemd-journal22:50:003
3862291996027,29cyclictest3467602-21systemd-journal22:50:003
386229199602,19cyclictest3878846-21ssh21:33:513
386229199602,17cyclictest3952610-21latency_hist00:20:023
386228699602,53cyclictest3874143-21ssh21:23:082
386228699601,9cyclictest3899841-21ssh22:20:232
386228699601,8cyclictest3935384-21ssh23:40:442
386228699601,7cyclictest3930769-21ssh23:30:222
386228699601,6cyclictest3907945-21ssh22:39:142
386228699601,55cyclictest3957513-21ssh00:30:502
386228699601,55cyclictest3875321-21ssh21:25:312
386228699601,55cyclictest3875321-21ssh21:25:312
386228699601,54cyclictest3887635-21ssh21:53:302
386228099602,5cyclictest3908355-21latency_hist22:40:020
3862280996024,32cyclictest0-21swapper/000:16:400
3862280996022,35cyclictest12-21ksoftirqd/019:45:020
3862280996022,35cyclictest12-21ksoftirqd/019:45:020
386228099601,55cyclictest0-21swapper/000:24:360
386229199593,50cyclictest3467602-21systemd-journal22:45:003
3862291995922,33cyclictest3921659-21latency_hist23:10:013
386228699591,6cyclictest3905421-21ssh22:33:182
386228699591,53cyclictest3947473-21ssh00:08:252
386228699591,12cyclictest3915537-21ssh22:55:522
386228099592,16cyclictest3862672-21latency_hist19:15:020
3862280995912,7cyclictest0-21swapper/019:55:000
386229199583,18cyclictest3953015-21ssh00:20:313
386229199582,17cyclictest3890648-21wc22:00:033
386229199582,14cyclictest3883863-21cat21:45:023
386228699581,52cyclictest3868157-21sendmail-mta21:00:052
386228699581,51cyclictest3868434-21cron21:10:002
386228699581,51cyclictest3865056-21sshd20:02:362
3862280995820,5cyclictest16950irq/224-800000020:32:360
3862280995820,5cyclictest16950irq/224-800000020:32:360
386229199572,9cyclictest3917178-21latency_hist23:00:003
386229199572,16cyclictest3936127-21ssh23:42:343
386229199572,10cyclictest3956956-21ssh00:30:023
386229199571,14cyclictest3942666-21ssh23:57:303
3862291995710,11cyclictest3892751-21sh22:05:023
3862291995710,11cyclictest3892751-21sh22:05:013
386228099575,8cyclictest3865854-21systemctl20:20:000
386228099573,15cyclictest3866889-21munin-plugin-st20:40:000
3862291995626,27cyclictest37-21ksoftirqd/300:03:283
386229199562,5cyclictest3888247-21munin-run21:55:003
386229199562,5cyclictest275-21dbus-daemon21:25:483
386229199562,5cyclictest275-21dbus-daemon21:25:473
386229199562,15cyclictest3904296-21ssh22:30:323
386229199562,14cyclictest3919389-21munin-run23:05:013
386229199561,7cyclictest364-21snmpd22:27:163
386229199561,14cyclictest3946316-21ssh00:05:263
386228699562,49cyclictest3866630-21munin-run20:35:002
386228699562,49cyclictest3866630-21munin-run20:35:002
386228699561,50cyclictest3864548-21munin-run19:55:012
386229199557,6cyclictest131rcu_preempt00:40:003
386229199552,17cyclictest3950122-21ssh00:14:233
386229199551,16cyclictest3928195-21rm23:24:593
386229199551,10cyclictest3939859-21ssh23:50:523
386229199551,10cyclictest311-21rs:main3
386228699551,50cyclictest3862513-21latency_hist19:15:022
386228699551,49cyclictest3863472-21sendmail-mta19:30:052
386228499557,10cyclictest23-21ksoftirqd/100:29:331
386228499556,12cyclictest3889490-21sh21:57:301
3862284995514,11cyclictest23-21ksoftirqd/122:01:451
3862284995514,11cyclictest23-21ksoftirqd/122:01:451
386228099553,16cyclictest3863009-21modprobe19:25:000
386229199543,16cyclictest3934927-21latency_hist23:40:013
3862291995430,20cyclictest3897624-21sh22:15:353
386229199542,9cyclictest3883983-21modprobe21:45:023
386229199542,8cyclictest3864044-21grep19:45:003
386229199542,8cyclictest3864044-21grep19:45:003
386229199542,7cyclictest3866089-21sshd20:20:363
386229199542,16cyclictest3932074-21ssh23:33:383
386229199542,13cyclictest3866907-21lockfile-create20:40:013
386229199541,15cyclictest3880909-21ssh21:38:263
386229199541,13cyclictest3467602-21systemd-journal19:50:003
386228699543,46cyclictest0-21swapper/220:54:592
3862286995422,28cyclictest3865405-21sed20:10:022
3862286995422,28cyclictest3865405-21sed20:10:022
386228699541,48cyclictest3863740-21modprobe19:37:362
386228499547,9cyclictest23-21ksoftirqd/100:15:011
386228499547,11cyclictest23-21ksoftirqd/121:15:021
3862284995415,8cyclictest23-21ksoftirqd/123:56:531
3862284995414,11cyclictest23-21ksoftirqd/100:30:221
3862284995414,10cyclictest23-21ksoftirqd/123:00:011
386228099543,11cyclictest3864316-21cat19:50:010
386229199538,11cyclictest3865301-21sshd20:05:423
386229199538,11cyclictest3865301-21sshd20:05:423
386229199533,6cyclictest3863748-21cron19:40:003
386229199532,8cyclictest3907471-21ssh22:37:593
386229199532,17cyclictest3900004-21ssh22:20:463
386229199532,12cyclictest3865106-21sort20:05:013
386229199531,8cyclictest3863517-21systemd19:35:013
3862291995318,30cyclictest275-21dbus-daemon21:05:013
3862291995312,36cyclictest274-21cron20:35:003
3862291995312,36cyclictest274-21cron20:35:003
386229199531,14cyclictest3958862-21ssh00:34:153
386228699532,47cyclictest3864178-21latency_hist19:45:022
386228699532,47cyclictest3864178-21latency_hist19:45:022
386228699532,46cyclictest3863209-21latency_hist19:25:022
3862284995337,12cyclictest1-21systemd21:20:011
3862291995218,29cyclictest309-21in:imuxsock19:57:363
386229199521,13cyclictest3937675-21ssh23:45:563
3862291995211,37cyclictest3868095-21latency_hist21:00:023
386228699523,44cyclictest3467602-21systemd-journal20:45:002
386228699522,43cyclictest3866127-21cut20:25:002
386228699521,46cyclictest3867412-21munin-run20:50:002
386228499525,8cyclictest23-21ksoftirqd/120:20:021
3862284995232,14cyclictest3467602-21systemd-journal20:30:001
3862280995221,11cyclictest12-21ksoftirqd/021:05:030
386229199513,7cyclictest275-21dbus-daemon19:20:013
386229199512,6cyclictest3865722-21wc20:15:013
3862291995126,21cyclictest3893110-21sh22:05:233
386229199512,5cyclictest3864514-21sshd19:50:403
386229199511,16cyclictest3914677-21ssh22:54:163
386228699512,42cyclictest3864778-21sshd19:57:362
3862286995122,25cyclictest3867895-21sshd20:58:362
3862286995121,27cyclictest3866965-21latency_hist20:40:012
386228499519,7cyclictest23-21ksoftirqd/121:21:091
3862284995143,5cyclictest221rcuc/122:13:571
3862284995143,5cyclictest221rcuc/122:13:571
3862284995136,10cyclictest1-21systemd22:35:011
3862284995133,14cyclictest3467602-21systemd-journal23:50:011
3862284995119,8cyclictest23-21ksoftirqd/123:20:181
386228499511,7cyclictest23-21ksoftirqd/122:45:111
386228499511,7cyclictest23-21ksoftirqd/122:45:111
386228099517,13cyclictest3867668-21modprobe20:55:000
3862280995118,6cyclictest12-21ksoftirqd/019:10:020
386229199508,35cyclictest3868439-21modprobe21:10:003
386229199502,9cyclictest3863257-21munin-run19:30:003
386228499506,8cyclictest23-21ksoftirqd/123:31:121
386228499505,7cyclictest23-21ksoftirqd/123:07:291
3862284995042,5cyclictest23-21ksoftirqd/121:52:491
386229199494,6cyclictest3867418-21true20:50:013
386229199492,6cyclictest3866370-21modprobe20:30:003
386229199492,6cyclictest364-21snmpd19:15:003
386228699491,12cyclictest3865805-21sshd20:16:362
386228499498,24cyclictest0-21swapper/122:40:021
386228499494,10cyclictest23-21ksoftirqd/122:15:551
386228499493,13cyclictest23-21ksoftirqd/100:38:501
386228499491,14cyclictest23-21ksoftirqd/123:19:321
386228499488,8cyclictest23-21ksoftirqd/121:40:011
386228499483,9cyclictest23-21ksoftirqd/122:45:031
3862284994833,11cyclictest275-21dbus-daemon20:40:001
3862284994812,6cyclictest23-21ksoftirqd/122:09:111
386229199471,11cyclictest3467602-21systemd-journal19:23:573
386228699471,42cyclictest1-21systemd20:12:322
386228499479,7cyclictest23-21ksoftirqd/123:12:251
386228499478,9cyclictest0-21swapper/119:59:361
386228499476,9cyclictest0-21swapper/121:49:511
386228499474,8cyclictest23-21ksoftirqd/100:05:511
3862284994738,6cyclictest221rcuc/123:27:181
386228499473,5cyclictest23-21ksoftirqd/123:02:161
3862284994711,9cyclictest23-21ksoftirqd/123:52:501
386228499465,8cyclictest23-21ksoftirqd/122:24:351
386228499464,6cyclictest23-21ksoftirqd/121:28:181
386228499464,6cyclictest23-21ksoftirqd/121:28:181
3862284994611,9cyclictest23-21ksoftirqd/120:50:011
386229199453,5cyclictest0-21swapper/319:10:023
386228499459,9cyclictest23-21ksoftirqd/122:26:231
3862284994535,6cyclictest3934842-21cron23:40:001
386228499451,7cyclictest3951517-21ssh00:17:271
386228499445,11cyclictest23-21ksoftirqd/123:44:081
386228499444,8cyclictest23-21ksoftirqd/120:15:011
3862284994428,13cyclictest23-21ksoftirqd/121:05:011
3862284994427,13cyclictest3467602-21systemd-journal19:41:361
3862284994427,13cyclictest3467602-21systemd-journal19:41:361
386228499441,6cyclictest0-21swapper/119:20:011
386228699431,38cyclictest3862208-21latency_hist19:10:012
386228499436,7cyclictest23-21ksoftirqd/100:02:121
386228499435,7cyclictest23-21ksoftirqd/121:31:371
386228499435,7cyclictest23-21ksoftirqd/120:10:011
386228499435,7cyclictest23-21ksoftirqd/120:10:011
386228499434,9cyclictest23-21ksoftirqd/122:54:301
386228499434,8cyclictest23-21ksoftirqd/121:40:561
386228499427,6cyclictest23-21ksoftirqd/100:21:501
386228499426,9cyclictest23-21ksoftirqd/120:05:001
3862284994030,7cyclictest23-21ksoftirqd/119:40:011
3862284994027,9cyclictest309-21in:imuxsock20:30:021
3862284994027,9cyclictest309-21in:imuxsock20:30:021
386228499401,7cyclictest0-21swapper/121:08:551
386228499400,5cyclictest150-21pr/ttyS219:26:001
386228499391,5cyclictest0-21swapper/119:50:311
386228499385,8cyclictest0-21swapper/119:49:481
386228499381,6cyclictest150-21pr/ttyS220:58:211
386228499381,5cyclictest150-21pr/ttyS219:31:551
386228499380,6cyclictest150-21pr/ttyS219:11:031
386228499374,7cyclictest23-21ksoftirqd/120:25:001
386228499360,5cyclictest150-21pr/ttyS220:51:351
3862284993526,7cyclictest23-21ksoftirqd/119:25:021
3862284993323,7cyclictest3867125-21sshd20:40:361
3862284993122,6cyclictest3862287-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional