You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 11:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Jun 19, 2026 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3545522998129,15cyclictest3546519-21munin-run19:30:011
3545522998029,16cyclictest3589705-21ssh22:34:091
3545531997326,42cyclictest275-21dbus-daemon00:15:023
3545526997326,14cyclictest3551537-21cat21:05:012
3545526997230,13cyclictest3600564-21sh22:58:082
3545522997228,13cyclictest3640394-21ssh00:26:291
3545526997120,14cyclictest3601913-21ssh23:00:472
3545531997020,45cyclictest275-21dbus-daemon00:10:013
3545526997028,14cyclictest3560263-21ssh21:28:532
3545526997024,15cyclictest3605265-21sh23:08:332
3545526997024,15cyclictest3605265-21sh23:08:332
3545526997022,13cyclictest3606323-21ssh23:10:332
3545522996926,37cyclictest3549907-21cron20:35:011
3545522996828,13cyclictest3613174-21ssh23:25:341
3545531996711,53cyclictest3589042-21ssh22:32:273
3545522996725,13cyclictest150-21pr/ttyS223:24:551
3545531996613,49cyclictest3555715-21ssh21:18:423
3545526996630,7cyclictest3633649-21ssh00:11:422
354553199657,46cyclictest3592351-21sh22:40:013
3545531996520,40cyclictest37-21ksoftirqd/319:55:003
354553199645,50cyclictest0-21swapper/320:14:593
354553199643,17cyclictest3566104-21ssh21:41:313
354553199642,6cyclictest3623023-21ssh23:48:033
3545522996425,11cyclictest150-21pr/ttyS221:57:431
354553199635,54cyclictest3610418-21modprobe23:19:593
354553199633,19cyclictest3563009-21latency_hist21:35:023
354553199633,19cyclictest3563009-21latency_hist21:35:013
354553199633,17cyclictest3553983-21modprobe21:15:013
354553199632,7cyclictest3568637-21sshd21:47:233
354553199632,16cyclictest3578745-21cron22:10:003
354553199631,10cyclictest3636442-21ssh00:18:003
3545522996341,18cyclictest3575103-21ssh22:01:281
3545522996324,14cyclictest150-21pr/ttyS223:13:291
3545517996324,35cyclictest12-21ksoftirqd/020:25:000
354553199622,19cyclictest3574229-21munin-run22:00:013
354553199622,16cyclictest3557150-21ssh21:21:413
3545531996221,36cyclictest275-21dbus-daemon23:45:013
3545526996246,13cyclictest3591460-21ssh22:38:042
3545522996224,13cyclictest150-21pr/ttyS200:30:471
354553199614,49cyclictest3545715-21munin-run19:15:013
354553199613,48cyclictest3617191-21modprobe23:34:593
354553199613,15cyclictest3619366-21true23:40:013
3545531996126,32cyclictest3586209-21sh22:26:103
3545531996123,34cyclictest37-21ksoftirqd/323:11:183
354553199612,14cyclictest3597647-21ssh22:51:353
3545531996121,35cyclictest3581434-21sh22:15:263
3545531996113,45cyclictest3576001-21ssh22:03:463
3545531996113,44cyclictest3604564-21ssh23:06:513
3545531996113,44cyclictest3604564-21ssh23:06:513
354553199611,13cyclictest3596522-21ssh22:49:123
3545526996142,16cyclictest3611703-21ssh23:22:312
3545526996124,9cyclictest0-21swapper/221:30:402
3545526996124,9cyclictest0-21swapper/221:30:402
3545526996122,13cyclictest3596502-21ssh22:49:092
3545522996140,16cyclictest3622937-21ssh23:47:521
3545522996124,10cyclictest23-21ksoftirqd/123:09:091
3545522996124,10cyclictest23-21ksoftirqd/123:09:091
3545522996123,13cyclictest150-21pr/ttyS221:15:241
3545522996123,12cyclictest150-21pr/ttyS222:08:151
3545517996129,15cyclictest3628089-21ssh23:59:240
354553199609,46cyclictest309-21in:imuxsock22:45:013
354553199609,46cyclictest309-21in:imuxsock22:45:003
354553199605,48cyclictest3548336-21systemd-run20:05:013
354553199605,48cyclictest3548336-21systemd-run20:05:003
354553199604,17cyclictest3639552-21grep00:25:003
354553199604,12cyclictest3547561-21cat19:50:013
354553199602,8cyclictest3563455-21ssh21:35:443
354553199602,8cyclictest120-21kswapd000:00:023
354553199602,14cyclictest3629376-21ssh00:02:083
354553199601,9cyclictest3579648-21ssh22:11:413
354553199601,18cyclictest3611648-21ssh23:22:223
354553199601,14cyclictest3614581-21ssh23:29:133
354553199601,10cyclictest3559807-21ssh21:27:423
3545526996044,13cyclictest3614561-21ssh23:29:102
3545526996044,13cyclictest3586233-21ssh22:26:142
3545526996044,12cyclictest3646008-21ssh00:39:142
3545526996044,12cyclictest3640553-21ssh00:26:502
3545526996043,14cyclictest3632515-21ssh00:09:162
3545526996043,14cyclictest3584271-21ssh22:21:512
3545526996043,13cyclictest3626977-21ssh23:56:462
3545526996042,14cyclictest3552008-21ssh21:10:052
3545526996041,16cyclictest3618398-21ssh23:37:492
3545526996040,16cyclictest3593110-21ssh22:41:252
3545526996040,16cyclictest3593110-21ssh22:41:242
3545526996040,16cyclictest3580981-21ssh22:14:572
3545526996039,17cyclictest3572636-21ssh21:56:012
3545526996036,19cyclictest284-21systemd-logind22:10:012
3545522996040,16cyclictest3599913-21ssh22:56:321
3545522996024,11cyclictest150-21pr/ttyS219:11:451
3545522996023,11cyclictest150-21pr/ttyS221:04:301
354553199593,17cyclictest3643358-21ssh00:33:163
3545531995913,39cyclictest3467602-21systemd-journal23:02:173
354553199591,11cyclictest3585469-21ssh22:24:483
3545526995944,12cyclictest3574650-21ssh22:00:252
3545526995943,13cyclictest3637737-21ssh00:20:302
3545526995943,13cyclictest3581376-21ssh22:15:192
3545526995942,14cyclictest3621377-21ssh23:44:322
3545526995942,14cyclictest3565650-21ssh21:40:312
3545526995941,14cyclictest3597307-21ssh22:50:372
3545526995941,14cyclictest3569495-21ssh21:49:242
3545526995939,16cyclictest3556771-21ssh21:20:402
3545522995941,15cyclictest3627057-21ssh23:56:571
3545522995941,14cyclictest3630558-21ssh00:04:521
3545522995940,15cyclictest3625978-21ssh23:54:541
3545522995939,16cyclictest3587413-21ssh22:29:031
3545522995938,17cyclictest3609540-21ssh23:17:481
3545522995938,17cyclictest3559422-21ssh21:26:491
354553199585,47cyclictest3551751-21grep21:10:003
354553199584,9cyclictest3549124-21cron20:20:003
354553199584,49cyclictest3646384-21cut00:40:003
354553199582,9cyclictest3549895-21sshd20:32:163
354553199582,6cyclictest3546255-21munin-run19:25:013
354553199582,6cyclictest3546255-21munin-run19:25:003
354553199582,13cyclictest3641855-21latency_hist00:30:003
354553199582,13cyclictest3600526-21ssh22:57:553
354553199581,16cyclictest3550214-21munin-run20:40:003
3545526995845,10cyclictest3563225-21ssh21:35:052
3545526995843,12cyclictest3624420-21ssh23:50:572
3545526995843,12cyclictest3622163-21ssh23:45:572
3545526995842,13cyclictest3570723-21ssh21:51:572
3545526995842,12cyclictest3615672-21ssh23:31:182
3545526995841,14cyclictest3588052-21ssh22:30:052
3545526995839,15cyclictest3642259-21ssh00:30:342
3545522995844,11cyclictest3618305-21ssh23:37:371
3545522995843,12cyclictest3646357-21ssh00:39:591
3545522995843,12cyclictest3635492-21ssh00:15:361
3545522995843,12cyclictest3619698-21ssh23:40:251
3545522995842,13cyclictest3561422-21ssh21:31:131
3545522995842,13cyclictest3561422-21ssh21:31:131
3545522995842,12cyclictest3571016-21ssh21:52:361
3545522995840,14cyclictest3590905-21ssh22:36:381
3545522995838,16cyclictest3631873-21ssh00:07:411
3545522995838,16cyclictest3582999-21ssh22:19:291
3545522995838,16cyclictest3564828-21ssh21:39:081
3545522995837,17cyclictest3583831-21ssh22:20:441
3545522995830,23cyclictest3550749-21latency_hist20:50:011
3545517995834,19cyclictest16850irq/134-800000000:10:010
3545517995818,36cyclictest12-21ksoftirqd/000:02:460
354553199574,43cyclictest3571968-21munin-run21:55:003
354553199572,14cyclictest3549393-21sshd20:22:153
354553199571,12cyclictest3551452-21sendmail-mta21:00:033
354553199571,10cyclictest3624435-21ssh23:50:583
3545526995744,10cyclictest3546457-21cat19:25:042
3545526995744,10cyclictest3546457-21cat19:25:042
3545526995743,12cyclictest3554592-21ssh21:15:522
3545526995742,13cyclictest3636123-21ssh00:17:102
3545526995722,12cyclictest3628652-21ssh00:00:142
3545522995744,10cyclictest3601591-21ssh23:00:061
3545522995742,12cyclictest3598871-21ssh22:54:241
3545522995741,14cyclictest3639439-21ssh00:24:451
3545522995740,14cyclictest3616211-21ssh23:32:411
3545522995740,13cyclictest3580074-21ssh22:12:361
3545522995739,15cyclictest3595271-21ssh22:46:181
3545522995737,16cyclictest3633519-21ssh00:11:251
3545522995733,20cyclictest3557369-21ssh21:22:101
3545517995728,25cyclictest12-21ksoftirqd/022:00:020
3545517995726,14cyclictest3601739-21ssh23:00:240
3545517995720,33cyclictest12-21ksoftirqd/000:35:320
354553199565,22cyclictest3548576-21munin-run20:10:003
354553199561,6cyclictest3546758-21munin-run19:35:003
3545526995644,9cyclictest3549637-21sshd20:26:172
3545526995642,11cyclictest3608383-21ssh23:15:062
3545522995643,10cyclictest3568908-21ssh21:48:011
3545522995641,13cyclictest3567168-21ssh21:44:151
3545522995641,13cyclictest3553966-21ssh21:14:591
3545517995643,10cyclictest12-21ksoftirqd/000:16:570
3545517995642,11cyclictest12-21ksoftirqd/020:45:020
3545517995623,30cyclictest12-21ksoftirqd/021:29:500
3545517995620,32cyclictest12-21ksoftirqd/019:10:020
354553199555,44cyclictest3551194-21cron21:00:013
354553199553,6cyclictest3550718-21munin-run20:50:013
354553199553,5cyclictest364-21snmpd19:15:343
354553199552,6cyclictest3547289-21/usr/sbin/munin19:45:013
354553199551,7cyclictest3550467-21munin-run20:45:013
3545526995544,8cyclictest3550970-21munin-run20:55:012
3545526995541,10cyclictest364-21snmpd20:55:172
3545526995528,22cyclictest3547895-21latency_hist19:55:052
3545522995542,10cyclictest3550956-21munin-run20:55:001
3545522995528,23cyclictest3594561-21munin-plugin-st22:45:001
3545522995528,23cyclictest3594561-21munin-plugin-st22:45:001
3545522995526,24cyclictest3548079-21true20:00:011
3545517995524,15cyclictest3598284-21ssh22:53:040
3545517995522,15cyclictest3637686-21ssh00:20:220
354551799551,39cyclictest3609333-21ssh23:17:200
3545517995513,37cyclictest275-21dbus-daemon00:30:010
354553199543,5cyclictest364-21snmpd20:54:243
3545526995432,18cyclictest3546003-21sendmail-msp19:19:592
3545522995424,24cyclictest3551728-21cron21:09:591
3545517995428,21cyclictest3554009-21ssh21:15:010
3545517995427,12cyclictest3625873-21ssh23:54:290
354553199533,8cyclictest1-21systemd19:30:013
354553199532,18cyclictest3547178-21latency_hist19:40:033
3545526995337,11cyclictest3548054-21munin-run20:00:002
3545526995335,14cyclictest3548692-21latency_hist20:10:022
3545526995333,15cyclictest3548859-21apt-get20:15:012
3545522995335,13cyclictest1-21systemd19:45:011
3545522995331,17cyclictest3547779-21cron19:55:001
3545522995328,21cyclictest3550241-21latency_hist20:40:021
354551799532,14cyclictest3467602-21systemd-journal19:40:010
3545526995239,10cyclictest3547541-21munin-run19:50:012
3545526995232,15cyclictest3549150-21munin-plugin-st20:19:592
3545522995232,15cyclictest275-21dbus-daemon20:15:011
3545522995230,17cyclictest3549521-21ls20:25:031
3545517995216,8cyclictest151rcuc/023:00:010
3545526995130,17cyclictest3546903-21latency_hist19:35:022
3545522995137,10cyclictest275-21dbus-daemon20:45:011
3545522995127,19cyclictest3546393-21latency_hist19:25:031
3545522995127,19cyclictest3546393-21latency_hist19:25:031
354551799516,6cyclictest131rcu_preempt19:20:000
3545517995134,12cyclictest12-21ksoftirqd/021:18:480
3545517995127,9cyclictest12-21ksoftirqd/023:25:030
3545517995124,23cyclictest12-21ksoftirqd/019:35:020
3545517995122,26cyclictest3583014-21ssh22:19:300
3545517995119,28cyclictest12-21ksoftirqd/023:13:570
354553199502,7cyclictest364-21snmpd19:58:113
354553199502,7cyclictest3549644-21sshd20:27:173
3545526995036,11cyclictest3547017-21munin-run19:40:012
3545526995033,12cyclictest3467602-21systemd-journal20:37:112
3545526995031,16cyclictest364-21snmpd20:34:502
3545522995029,17cyclictest3549326-21sed20:20:021
3545522995028,17cyclictest3545645-21latency_hist19:10:031
3545522995026,19cyclictest3547020-21sendmail19:40:011
3545517995041,5cyclictest16850irq/134-800000023:30:000
3545517995033,14cyclictest16950irq/224-800000021:50:010
354551799502,43cyclictest3592306-21sendmail22:40:010
3545517995023,12cyclictest3594875-21ssh22:45:140
3545517995021,14cyclictest12-21ksoftirqd/022:34:010
354551799501,25cyclictest12-21ksoftirqd/021:37:400
3545526994942,4cyclictest3549417-21munin-run20:25:012
3545526994936,10cyclictest3548296-21sshd20:02:062
3545526994936,10cyclictest3548296-21sshd20:02:062
3545526994931,14cyclictest3551719-21sshd21:08:162
3545526994923,22cyclictest3545593-21cat19:10:022
3545522994930,14cyclictest3548577-21grep20:10:001
354551799496,4cyclictest131rcu_preempt22:00:360
3545517994934,10cyclictest1-21systemd19:25:010
3545517994934,10cyclictest1-21systemd19:25:010
3545517994932,12cyclictest16950irq/224-800000019:59:590
354551799492,10cyclictest12-21ksoftirqd/023:35:010
3545517994920,25cyclictest12-21ksoftirqd/022:27:570
3545526994837,8cyclictest3545681-21sshd19:11:162
3545526994833,11cyclictest1-21systemd20:45:012
3545522994836,9cyclictest364-21snmpd20:00:071
3545522994836,9cyclictest364-21snmpd20:00:071
3545522994835,10cyclictest3549636-21sshd20:25:161
3545517994833,11cyclictest16950irq/224-800000021:45:010
3545517994829,15cyclictest3622164-21ssh23:45:570
3545517994819,14cyclictest3562698-21ssh21:34:260
3545517994819,14cyclictest3562698-21ssh21:34:260
3545517994813,6cyclictest131rcu_preempt20:35:010
354551799481,36cyclictest12-21ksoftirqd/022:41:160
354551799481,36cyclictest12-21ksoftirqd/022:41:160
3545526994727,17cyclictest364-21snmpd20:49:572
3545517994738,6cyclictest12-21ksoftirqd/022:20:340
3545517994728,15cyclictest3619984-21ssh23:41:050
3545517994728,15cyclictest3550181-21munin-run20:40:000
3545517994719,10cyclictest12-21ksoftirqd/023:10:010
3545517994719,10cyclictest12-21ksoftirqd/023:10:000
3545526994633,10cyclictest3547255-21sshd19:40:172
3545522994632,11cyclictest364-21snmpd20:55:191
3545522994618,25cyclictest3546814-21latency_hist19:35:021
3545517994637,6cyclictest12-21ksoftirqd/020:55:010
3545517994633,8cyclictest12-21ksoftirqd/000:14:350
3545517994628,14cyclictest3617777-21ssh23:35:580
3545517994611,9cyclictest12-21ksoftirqd/000:32:240
3545517994610,32cyclictest12-21ksoftirqd/019:45:000
3545522994528,12cyclictest3545964-21cron19:20:001
3545517994529,12cyclictest12-21ksoftirqd/022:09:150
3545517994528,13cyclictest3579907-21ssh22:12:140
354551799452,39cyclictest12-21ksoftirqd/020:15:020
354553199441,8cyclictest3545536-21latency_hist19:10:013
3545526994426,14cyclictest3546470-21sshd19:25:152
3545522994432,9cyclictest3547512-21sshd19:47:021
354551799442,6cyclictest3549146-21systemctl20:20:000
3545517994422,7cyclictest3571898-21ssh21:54:530
354551799425,11cyclictest0-21swapper/019:30:010
3545517994235,4cyclictest16950irq/224-800000021:00:010
3545517994231,8cyclictest12-21ksoftirqd/019:50:010
3545517994229,10cyclictest3551711-21sshd21:07:160
354551799421,14cyclictest3557321-21sh21:22:040
3545517994134,4cyclictest16950irq/224-800000020:05:020
3545517994134,4cyclictest16950irq/224-800000020:05:020
354551799412,6cyclictest151rcuc/021:01:170
3545517994126,11cyclictest12-21ksoftirqd/019:55:010
354551799411,36cyclictest3545697-21sshd19:14:160
3545517993825,11cyclictest12-21ksoftirqd/020:09:590
354551799378,11cyclictest12-21ksoftirqd/020:49:590
3545517993629,5cyclictest12-21ksoftirqd/020:28:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional