You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 16:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon Nov 24, 2025 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
35131332970,4sleep10-21swapper/107:20:161
35425682910,6sleep20-21swapper/208:45:022
3508588998221,43cyclictest438-20systemd-journal10:24:211
350815028062,11sleep30-21swapper/307:06:073
162399790,65rtkit-daemon0-21swapper/107:06:331
35970622740,6sleep30-21swapper/311:20:153
350822127439,9sleep20-21swapper/207:07:022
350821427451,17sleep00-21swapper/007:06:560
350858899641,37cyclictest3582830-21sed10:40:011
35591632620,7chrt2197014-21PK-Backend09:30:270
3508590996230,27cyclictest3548066-21cpuspeed_turbos09:00:122
350859499611,14cyclictest3539194-21taskset08:35:113
350859499611,14cyclictest3539194-21taskset08:35:103
350858899612,34cyclictest1753-21gnome-shell07:35:021
350858899602,35cyclictest1753-21gnome-shell11:40:221
350858899602,34cyclictest1753-21gnome-shell12:10:351
3508588996015,40cyclictest1795-21gmain08:20:021
350858899601,34cyclictest3611384-21awk12:00:221
3508588995925,30cyclictest3526453-21sh08:00:001
3508584995925,30cyclictest3529903-21sshd08:09:200
3508594995823,30cyclictest438-20systemd-journal10:36:203
3508588995819,34cyclictest1-21systemd10:05:011
350858899572,50cyclictest1753-21gnome-shell09:30:191
350858899572,32cyclictest3555307-21irqstats09:20:211
350858899572,32cyclictest3555307-21irqstats09:20:201
3508588995720,32cyclictest3551944-21memory09:10:241
3508588995718,34cyclictest3623374-21cat12:35:231
3508588995718,34cyclictest3560305-21expr09:35:161
350858899571,34cyclictest3544472-21cpuspeed08:50:111
350858899567,29cyclictest26-21ksoftirqd/112:19:221
3508588995618,33cyclictest3522544-21cat07:45:301
350858899561,50cyclictest3557042-21irqstats09:25:211
350859099551,13cyclictest860997-21snmpd08:07:372
350858899552,49cyclictest3591271-21apt11:04:591
3508588995518,32cyclictest3509817-21wc07:10:311
350858899551,48cyclictest3530238-21apt08:10:111
3508588995513,37cyclictest3548895-21cat09:00:301
3508584995521,29cyclictest3522303-21awk07:45:260
3508584995518,11cyclictest3523482-21ls07:50:140
3508584995516,15cyclictest855-21dbus-daemon08:50:000
36048332540,6chrt0-21swapper/311:40:443
350859099542,8cyclictest1753-21gnome-shell07:11:512
3508590995422,27cyclictest3567683-21ntpq09:55:252
350858899542,47cyclictest3577469-21grep10:25:141
3508588995423,27cyclictest1069-21runrttasks08:05:371
3508588995417,32cyclictest3585923-21sshd10:48:201
350858899541,32cyclictest3606214-21ntpq11:45:281
162399540,44rtkit-daemon3619019-21chrt12:24:400
350859499536,42cyclictest3611451-21ps12:00:263
3508594995317,31cyclictest3514203-21cron07:25:003
350859099535,43cyclictest3511872-21uname07:15:272
3508590995320,28cyclictest3579386-21expr10:30:152
3508590995319,29cyclictest3565533-21expr09:50:152
3508588995320,29cyclictest3533803-21cpuspeed_turbos08:20:141
3508588995316,33cyclictest3242255-21gdbus08:50:011
3508588995315,33cyclictest3523718-21if_enp2s007:50:191
350858899531,47cyclictest3570734-21expr10:05:171
350858899531,47cyclictest3570734-21expr10:05:171
350858899531,35cyclictest3512594-21sed07:20:021
3508584995310,17cyclictest1753-21gnome-shell08:56:440
350859499528,5cyclictest141rcu_preempt08:25:283
350859499522,14cyclictest1753-21gnome-shell09:50:023
3508594995212,36cyclictest855-21dbus-daemon11:20:013
3508590995217,30cyclictest3598917-21smartctl11:25:202
3508588995220,28cyclictest1069-21runrttasks10:13:151
3508588995219,29cyclictest3596835-21cpuspeed_turbos11:20:131
3508588995218,30cyclictest1-21systemd07:45:011
3508588995215,33cyclictest3519034-21grep07:35:311
350858899521,47cyclictest3620017-21netstat12:25:241
350858899521,46cyclictest3562016-21expr09:40:151
3508588995213,34cyclictest3598819-21expr11:25:161
350858499524,20cyclictest3568878-21cpuspeed_turbos10:00:150
3508584995221,27cyclictest13-21ksoftirqd/008:39:490
3508584995221,27cyclictest13-21ksoftirqd/008:39:490
3508584995219,29cyclictest3577184-21sed10:25:010
3508584995211,13cyclictest3516035-21pool-org.gnome.07:30:010
3508584995210,17cyclictest1753-21gnome-shell08:13:280
35552282510,13sleep2331rcuc/209:20:192
35552282510,13sleep2331rcuc/209:20:192
350859499512,45cyclictest1753-21gnome-shell10:43:213
350859499512,45cyclictest1753-21gnome-shell10:43:203
350859499512,44cyclictest854-21avahi-daemon09:57:493
350859499512,43cyclictest1753-21gnome-shell10:20:093
3508594995123,24cyclictest3510260-21latency_hist07:15:013
3508594995120,27cyclictest3527485-21ntp_states08:00:263
3508594995119,27cyclictest1753-21gnome-shell10:47:583
3508594995118,28cyclictest3537517-21cat08:30:133
3508594995115,31cyclictest3547482-21seq08:58:273
3508594995113,34cyclictest3555543-21ntp_kernel_pll_09:20:243
3508594995113,34cyclictest3555543-21ntp_kernel_pll_09:20:243
3508590995121,25cyclictest3528204-21/usr/sbin/munin08:05:012
3508590995120,26cyclictest3578965-21sed10:30:012
3508590995118,29cyclictest3605936-21interrupts11:45:222
350859099511,7cyclictest1-21systemd08:35:022
3508590995116,30cyclictest3588648-21cat10:55:202
350858899512,27cyclictest0-21swapper/109:05:281
3508588995117,30cyclictest1954-21gsd-print-notif09:52:111
3508588995115,32cyclictest3580522-21seq10:34:391
350858899511,28cyclictest0-21swapper/111:17:501
3508588995111,35cyclictest3621056-21cat12:30:101
350858499512,9cyclictest1753-21gnome-shell10:13:180
3508584995117,9cyclictest3593089-21ls11:10:010
3508584995117,9cyclictest3593089-21ls11:10:010
3508584995117,8cyclictest3551947-21missed_timers09:10:230
3508584995114,11cyclictest3557056-21kernelversion09:25:230
350859499505,40cyclictest3605547-21cpuspeed_turbos11:45:143
350859499504,41cyclictest3511893-21runrttasks07:15:273
350859499503,42cyclictest3531876-21date08:15:023
350859499502,43cyclictest1753-21gnome-shell07:40:203
3508594995016,29cyclictest3548318-21cat09:00:203
3508594995014,32cyclictest1632-21dbus-daemon10:55:023
3508590995021,24cyclictest1753-21gnome-shell07:21:482
3508590995014,30cyclictest3552809-21cron09:15:032
3508590995013,32cyclictest438-20systemd-journal12:36:252
350858899502,28cyclictest1753-21gnome-shell09:57:351
3508588995018,28cyclictest3601960-21/usr/sbin/munin11:35:261
3508588995017,28cyclictest2962346-21in:imuxsock08:28:191
3508588995016,29cyclictest3537701-21expr08:30:151
350858899501,45cyclictest0-21swapper/108:40:111
350858899501,44cyclictest3608895-21/usr/sbin/munin11:55:281
350858899501,44cyclictest3600485-21ls11:30:141
350858899501,44cyclictest3600485-21ls11:30:141
3508588995012,33cyclictest438-20systemd-journal08:05:001
350858499509,14cyclictest855-21dbus-daemon09:40:010
350858499509,13cyclictest3242255-21gdbus10:35:040
350858499508,17cyclictest1753-21gnome-shell11:12:130
3508584995017,10cyclictest1753-21gnome-shell07:43:050
3508584995014,13cyclictest13-21ksoftirqd/012:25:240
3508584995010,18cyclictest1753-21gnome-shell11:28:060
350859499493,42cyclictest3559096-21wc09:30:263
350859499492,43cyclictest1753-21gnome-shell12:15:593
350859499492,42cyclictest1753-21gnome-shell12:26:423
350859499492,42cyclictest1753-21gnome-shell09:36:163
350859499492,42cyclictest1753-21gnome-shell07:59:243
350859499492,42cyclictest1753-21gnome-shell07:52:333
3508594994922,22cyclictest1753-21gnome-shell07:25:153
350859499492,11cyclictest1753-21gnome-shell11:34:103
350859499492,11cyclictest1753-21gnome-shell11:34:093
3508594994919,27cyclictest1753-21gnome-shell12:32:363
3508594994918,27cyclictest3531949-21/usr/sbin/munin08:15:323
3508594994915,30cyclictest3607735-21wc11:50:223
3508594994915,30cyclictest3607735-21wc11:50:213
3508594994915,30cyclictest3544138-21cron08:50:003
350859499491,43cyclictest3589692-21wc11:00:023
3508594994914,30cyclictest3569252-21sort10:00:223
350859099498,36cyclictest1753-21gnome-shell07:51:512
3508590994922,23cyclictest3612396-21dpkg12:05:122
3508590994916,29cyclictest3586745-21smartctl10:50:192
3508590994915,28cyclictest3517364-21sshd07:30:322
3508588994915,30cyclictest3546967-21cut08:55:291
350858499499,18cyclictest1753-21gnome-shell11:32:070
350858499499,18cyclictest1753-21gnome-shell11:32:070
350858499499,18cyclictest1753-21gnome-shell10:54:190
350858499499,18cyclictest1753-21gnome-shell10:46:190
350858499499,17cyclictest1753-21gnome-shell10:16:300
350858499499,16cyclictest1753-21gnome-shell10:08:240
350858499499,16cyclictest1753-21gnome-shell10:08:240
350858499494,18cyclictest3608854-21sort11:55:030
350858499494,18cyclictest3608854-21sort11:55:020
350858499492,18cyclictest1753-21gnome-shell10:35:010
3508584994917,7cyclictest3532185-21cstates08:15:150
3508584994917,12cyclictest13-21ksoftirqd/007:35:250
3508584994914,10cyclictest3549839-21wc09:05:160
350858499491,22cyclictest1753-21gnome-shell07:31:560
3508584994912,11cyclictest3562170-21idleruntime09:40:210
3508584994910,19cyclictest1753-21gnome-shell08:44:460
3508584994910,18cyclictest1753-21gnome-shell10:56:120
3508584994910,16cyclictest1753-21gnome-shell08:03:420
350859499482,42cyclictest1753-21gnome-shell09:16:143
350859499482,40cyclictest1753-21gnome-shell08:20:073
350859499481,43cyclictest1753-21gnome-shell10:11:503
3508594994814,29cyclictest3622867-21cat12:35:143
3508594994814,29cyclictest3517883-21cut07:35:033
350859499481,10cyclictest3544882-21cat08:50:173
350859099489,35cyclictest1753-21gnome-shell10:48:112
350859099488,35cyclictest1753-21gnome-shell10:02:222
3508590994819,24cyclictest1069-21runrttasks10:23:572
3508590994817,27cyclictest3594768-21pool-org.gnome.11:15:012
3508590994817,26cyclictest3602332-21expr11:35:152
3508590994816,27cyclictest3581215-21wc10:35:152
3508590994816,27cyclictest3543635-21cat08:45:332
3508590994815,29cyclictest3540255-21cat08:35:322
3508590994815,29cyclictest3540255-21cat08:35:312
3508590994815,28cyclictest3526659-21latency_hist08:00:032
3508590994814,29cyclictest3571384-21timerandwakeup10:05:322
3508590994814,29cyclictest3571384-21timerandwakeup10:05:322
3508590994814,28cyclictest3619440-21cat12:25:142
350858899486,39cyclictest231irq_work/107:28:061
350858899482,31cyclictest1753-21gnome-shell12:05:451
350858899482,29cyclictest3574254-21df10:15:121
3508588994820,24cyclictest3540693-21latency_hist08:40:031
3508588994820,24cyclictest3540693-21latency_hist08:40:031
350858899481,42cyclictest3563858-21seq09:45:161
350858899481,33cyclictest1753-21gnome-shell11:53:391
350858899481,33cyclictest1753-21gnome-shell11:53:391
3508588994811,32cyclictest1753-21gnome-shell11:12:591
350858499489,18cyclictest1753-21gnome-shell08:23:220
350858499489,17cyclictest1753-21gnome-shell11:20:060
3508584994819,7cyclictest3609667-21ntp_states11:55:270
350858499481,8cyclictest3549360-21apt09:05:010
3508584994816,7cyclictest3509523-21ntp_kernel_pll_07:10:240
350858499481,22cyclictest438-20systemd-journal12:04:550
3508584994810,17cyclictest1753-21gnome-shell09:58:310
3508584994810,16cyclictest1753-21gnome-shell12:11:550
3508584994810,16cyclictest1753-21gnome-shell12:06:010
3508584994810,16cyclictest1753-21gnome-shell09:54:320
3508584994810,16cyclictest1753-21gnome-shell09:22:290
3508584994810,16cyclictest1753-21gnome-shell09:22:290
350859499479,16cyclictest1753-21gnome-shell11:35:583
3508594994718,25cyclictest1753-21gnome-shell10:05:263
3508594994718,25cyclictest1753-21gnome-shell10:05:263
3508594994717,26cyclictest3528442-21/usr/sbin/munin08:05:313
3508594994715,28cyclictest3579295-21cstates10:30:153
350859099478,34cyclictest1753-21gnome-shell08:25:302
350858899479,34cyclictest3591737-21cat11:05:161
350858899479,34cyclictest3591737-21cat11:05:151
350858899471,42cyclictest3584399-21cat10:45:021
350858899471,42cyclictest3584399-21cat10:45:011
350858899471,41cyclictest1753-21gnome-shell10:50:041
350858899471,30cyclictest3554258-21seq09:16:231
350858499479,17cyclictest1753-21gnome-shell08:26:580
350858499479,16cyclictest1753-21gnome-shell07:17:210
350858499474,16cyclictest1753-21gnome-shell11:45:000
350858499472,16cyclictest1753-21gnome-shell11:00:270
3508584994714,8cyclictest3623341-21latency12:35:220
3508584994713,6cyclictest0-21swapper/012:30:130
350859499469,18cyclictest1753-21gnome-shell07:49:523
350859499469,16cyclictest1753-21gnome-shell12:20:313
350859499462,7cyclictest1753-21gnome-shell12:10:003
350859499462,39cyclictest3614882-21ntp_states12:10:283
350859499462,19cyclictest1753-21gnome-shell11:25:193
3508594994611,11cyclictest3574619-21ntp_kernel_pll_10:15:223
350859099469,32cyclictest1753-21gnome-shell09:40:242
350859099466,18cyclictest1753-21gnome-shell11:30:042
350859099466,18cyclictest1753-21gnome-shell11:30:042
3508590994620,22cyclictest1753-21gnome-shell09:47:382
3508590994618,24cyclictest1753-21gnome-shell09:26:312
3508590994617,25cyclictest3520225-21basename07:40:202
350859099461,40cyclictest1753-21gnome-shell12:15:452
350859099461,22cyclictest3561014-21timerandwakeup09:35:322
3508588994615,27cyclictest3588961-21switchtime10:55:291
350858899461,41cyclictest1069-21runrttasks12:23:421
350858499465,16cyclictest3525209-21cstates07:55:150
3508584994613,8cyclictest3577574-21cstates10:25:150
350859499452,7cyclictest1753-21gnome-shell11:03:293
3508594994521,20cyclictest42-21ksoftirqd/311:13:173
3508594994519,22cyclictest3542043-21users08:40:353
350859499451,39cyclictest3557683-21sshd09:25:523
350859499451,23cyclictest1753-21gnome-shell10:26:113
3508594994511,11cyclictest3552395-21cat09:10:313
3508594994510,17cyclictest1753-21gnome-shell07:36:253
350859099458,33cyclictest3546905-21seq08:55:262
350859099458,32cyclictest3611777-21taskset12:00:522
350859099457,33cyclictest1753-21gnome-shell08:22:592
3508590994518,23cyclictest3584388-21sed10:45:022
3508590994518,23cyclictest3584388-21sed10:45:022
3508590994518,22cyclictest3618045-21if_enp2s012:20:192
350859099451,40cyclictest1753-21gnome-shell08:54:512
350859099451,39cyclictest1753-21gnome-shell07:49:312
3508590994511,30cyclictest1753-21gnome-shell11:51:462
3508590994511,30cyclictest1753-21gnome-shell11:51:462
350858499453,12cyclictest1753-21gnome-shell08:30:250
350858499452,16cyclictest1753-21gnome-shell11:16:390
3508584994516,7cyclictest1069-21runrttasks09:47:010
3508594994415,25cyclictest3565342-21cpuspeed_turbos09:50:153
350859099448,32cyclictest1753-21gnome-shell11:41:502
350859099448,32cyclictest1753-21gnome-shell09:06:232
350859099442,37cyclictest1753-21gnome-shell10:14:172
3508590994418,22cyclictest34-21ksoftirqd/207:35:292
3508590994414,25cyclictest3608994-21cat11:55:142
3508590994413,26cyclictest3559223-21sed09:30:292
3508590994410,30cyclictest1753-21gnome-shell12:32:542
3508590994410,29cyclictest1753-21gnome-shell08:17:092
350858499449,9cyclictest3605455-21sh11:45:120
350858499448,10cyclictest3602401-21basename11:35:200
350859499436,16cyclictest1753-21gnome-shell09:44:243
350859499431,39cyclictest0-21swapper/311:08:083
350859499431,39cyclictest0-21swapper/311:08:073
350859499431,13cyclictest860997-21snmpd09:09:053
350859099438,31cyclictest1753-21gnome-shell10:18:192
3508590994314,25cyclictest3595099-21cpuspeed_turbos11:15:122
3508590994310,29cyclictest1753-21gnome-shell11:21:592
350858499431,16cyclictest3544422-21taskset08:50:100
350859099427,16cyclictest1753-21gnome-shell07:29:302
350858499421,15cyclictest1753-21gnome-shell07:20:390
350859099412,16cyclictest1753-21gnome-shell09:13:592
350859099411,13cyclictest3592191-21/usr/sbin/munin11:05:252
350859099411,13cyclictest3592191-21/usr/sbin/munin11:05:252
3508584994121,4cyclictest2197014-21PK-Backend09:17:470
3508590994022,14cyclictest3615103-21timerandwakeup12:10:332
3508590994011,24cyclictest0-21swapper/208:10:252
350858499401,5cyclictest2197014-21PK-Backend10:41:310
350858499401,5cyclictest2197014-21PK-Backend10:41:310
350859499391,13cyclictest3610006-21chrt11:55:333
350858499392,15cyclictest1753-21gnome-shell12:15:070
3508590993817,18cyclictest35911292sleep211:03:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional