You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 06:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System (updated Tue Dec 02, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991460,8rtkit-daemon1622-21rtkit-daemon00:00:352
357593821140,5sleep00-21swapper/019:05:190
35767292920,4sleep10-21swapper/119:06:241
357692528145,14sleep20-21swapper/219:08:472
35967932800,7sleep31753-21gnome-shell20:05:023
357690528060,14sleep30-21swapper/319:08:323
36168142760,8sleep33616589-40apt-check21:00:153
162399750,6rtkit-daemon0-21swapper/000:27:440
36927422700,5sleep30-21swapper/300:35:213
357715699680,64cyclictest1622-21rtkit-daemon00:00:363
162399590,8rtkit-daemon1622-21rtkit-daemon00:18:323
357715099575,27cyclictest854-21avahi-daemon22:55:461
36508962560,8sleep22197014-21PK-Backend22:35:262
162399560,9rtkit-daemon1622-21rtkit-daemon20:22:381
162399560,9rtkit-daemon1622-21rtkit-daemon20:21:150
3577152995516,21cyclictest3589766-21sh19:45:012
357715099559,19cyclictest3669949-21expr23:30:171
357715099559,19cyclictest3669949-21expr23:30:171
357715099557,15cyclictest3689003-21cstates00:25:141
3577150995511,14cyclictest3615931-21runrttasks20:58:141
3577156995430,20cyclictest1753-21gnome-shell19:23:123
357715699530,49cyclictest1622-21rtkit-daemon00:31:473
357715099539,19cyclictest1753-21gnome-shell22:20:211
3577150995313,15cyclictest1753-21gnome-shell00:21:331
3577150995310,18cyclictest1753-21gnome-shell23:45:251
162399530,5rtkit-daemon0-21swapper/222:01:172
357715099528,20cyclictest3625848-21cstates21:25:181
357715099528,20cyclictest1617-10pipewire-media-20:35:151
3577150995215,32cyclictest3651177-21cat22:35:321
162399520,5rtkit-daemon2197014-21PK-Backend22:33:042
3577156995117,19cyclictest42-21ksoftirqd/300:24:593
357715099518,21cyclictest3663650-21sed23:10:311
357715099517,21cyclictest3613136-21expr20:50:151
3577150995114,12cyclictest1753-21gnome-shell22:34:461
357715699501,26cyclictest994-21lldpd19:55:013
357715099509,17cyclictest1753-21gnome-shell21:10:281
357715099508,19cyclictest3678954-21missed_timers23:55:261
357715099508,15cyclictest3665065-21memory23:15:231
3577150995012,15cyclictest3605280-21seq20:29:011
3577150995012,10cyclictest869-21irqbalance22:07:341
357715099499,16cyclictest3623898-21apt21:20:121
357715099499,16cyclictest3606320-21hddtemp_smartct20:30:211
357715099498,15cyclictest3595622-21chrt20:00:201
357715099497,19cyclictest3676797-21expr23:50:141
357715099496,17cyclictest3635693-21users21:50:321
357715099496,15cyclictest3668372-21basename23:25:211
3577150994915,29cyclictest1825-21JS14:31:441
3577150994912,22cyclictest1753-21gnome-shell19:39:251
3577150994911,16cyclictest3681820-21taskset00:05:051
3577150994910,15cyclictest3674388-21python323:42:181
162399490,35rtkit-daemon2197014-21PK-Backend19:56:192
357715099489,16cyclictest904-21systemd-logind19:15:001
357715099489,16cyclictest3579140-21missed_timers19:15:021
357715099488,16cyclictest3638438-21df22:00:141
357715099488,16cyclictest1753-21gnome-shell22:25:281
357715099487,16cyclictest3680427-21cstates00:00:141
3577150994812,10cyclictest3583487-21/usr/sbin/munin19:25:211
3577150994810,15cyclictest2197014-21PK-Backend20:15:291
3577150994810,14cyclictest3633752-21python321:45:321
36769312470,6sleep0161rcuc/023:50:140
35944432470,6sleep3411rcuc/319:55:313
35867142470,5sleep22197014-21PK-Backend19:35:142
357715299477,16cyclictest1753-21gnome-shell22:50:142
3577152994724,20cyclictest3634359-21awk21:50:012
357715099477,18cyclictest3588692-21entropy19:40:191
357715099477,15cyclictest3655453-21apt22:50:111
3577150994712,20cyclictest1753-21gnome-shell23:38:061
3577146994726,16cyclictest3242253-21nautilus23:05:000
36186832460,8sleep31753-21gnome-shell21:05:163
36186832460,8sleep31753-21gnome-shell21:05:153
3577156994636,6cyclictest42-21ksoftirqd/319:34:513
3577156994618,12cyclictest141rcu_preempt22:30:493
357715099468,14cyclictest3684343-21ntp_states00:10:291
357715099466,16cyclictest1-21systemd20:15:011
3577150994612,29cyclictest1738-21gdbus23:21:561
3577150994612,14cyclictest3659928-21/usr/sbin/munin23:00:331
162399460,31rtkit-daemon1753-21gnome-shell23:30:583
162399460,31rtkit-daemon1753-21gnome-shell23:30:583
357715699459,20cyclictest42-21ksoftirqd/322:45:123
357715699459,19cyclictest42-21ksoftirqd/321:52:513
357715699454,14cyclictest0-21swapper/319:50:003
357715699453,17cyclictest42-21ksoftirqd/320:34:413
3577156994527,13cyclictest438-20systemd-journal22:10:013
3577156994523,9cyclictest42-21ksoftirqd/322:50:463
3577156994512,15cyclictest0-21swapper/322:10:313
357715699451,18cyclictest1753-21gnome-shell20:53:193
357715299458,17cyclictest141rcu_preempt20:35:002
3577152994511,12cyclictest141rcu_preempt20:50:392
357715099458,14cyclictest3617687-21sort21:00:341
357715099457,15cyclictest908-21gdbus22:45:111
357715099455,15cyclictest3693660-21sshd00:38:541
3577150994510,14cyclictest3592357-21memory19:50:231
162399450,30rtkit-daemon1753-21gnome-shell23:10:172
357715699446,20cyclictest42-21ksoftirqd/319:25:293
357715699443,19cyclictest42-21ksoftirqd/323:20:563
3577156994412,18cyclictest42-21ksoftirqd/321:21:363
3577156994411,12cyclictest42-21ksoftirqd/321:45:153
357715099448,15cyclictest1753-21gnome-shell20:10:021
357715099448,15cyclictest1753-21gnome-shell20:10:021
357715099447,15cyclictest3686808-21basename00:20:011
357715099447,14cyclictest3662330-21taskset23:08:241
357715099446,15cyclictest3636651-21expr21:55:161
3577150994415,24cyclictest3623039-21python321:15:281
3577146994422,14cyclictest13-21ksoftirqd/022:45:130
357715699439,19cyclictest42-21ksoftirqd/320:22:563
357715699437,14cyclictest42-21ksoftirqd/321:40:323
3577156994322,8cyclictest42-21ksoftirqd/300:10:083
357715699432,19cyclictest42-21ksoftirqd/322:40:253
3577156994321,10cyclictest42-21ksoftirqd/323:10:233
3577156994316,9cyclictest42-21ksoftirqd/322:35:163
3577156994316,14cyclictest0-21swapper/323:06:243
3577156994310,19cyclictest42-21ksoftirqd/321:34:043
357715299437,5cyclictest141rcu_preempt19:25:372
357715299432,13cyclictest3675641-21ntp_states23:45:262
3577152994315,12cyclictest141rcu_preempt22:22:082
357715099436,15cyclictest3631524-21diskstats21:40:191
357715099436,14cyclictest3644035-21awk22:15:281
357715099436,14cyclictest0-21swapper/121:38:111
3577150994317,21cyclictest3590013-21/usr/sbin/munin19:45:251
357714699431,14cyclictest3588039-21turbostat19:40:010
162399430,36rtkit-daemon2197014-21PK-Backend00:19:212
36779332420,7sleep22197014-21PK-Backend23:53:182
357715699428,19cyclictest42-21ksoftirqd/322:55:553
357715699427,19cyclictest42-21ksoftirqd/300:25:323
357715699425,10cyclictest141rcu_preempt19:19:553
3577156994227,11cyclictest1-21systemd23:05:003
3577156994222,7cyclictest42-21ksoftirqd/320:25:253
357715699422,20cyclictest42-21ksoftirqd/320:19:593
357715699422,10cyclictest42-21ksoftirqd/323:40:253
3577156994219,5cyclictest141rcu_preempt20:11:493
3577156994216,7cyclictest42-21ksoftirqd/300:06:333
3577156994214,10cyclictest42-21ksoftirqd/322:22:373
357715699421,26cyclictest2197014-21PK-Backend21:18:483
3577156994211,8cyclictest42-21ksoftirqd/321:25:243
357715699421,17cyclictest0-21swapper/323:55:483
3577152994217,12cyclictest141rcu_preempt20:18:482
3577152994213,11cyclictest141rcu_preempt21:15:022
3577152994211,12cyclictest141rcu_preempt20:14:272
3577150994216,10cyclictest3652656-21/usr/sbin/munin22:40:261
3577150994215,11cyclictest1753-21gnome-shell19:55:211
3577150994214,23cyclictest3691570-21cat00:30:291
3577146994223,14cyclictest13-21ksoftirqd/020:52:370
357715699417,20cyclictest42-21ksoftirqd/323:27:183
357715699417,19cyclictest42-21ksoftirqd/319:41:433
357715699415,19cyclictest42-21ksoftirqd/321:11:503
357715699414,18cyclictest42-21ksoftirqd/322:01:243
357715699412,19cyclictest42-21ksoftirqd/321:35:403
3577156994119,8cyclictest42-21ksoftirqd/323:45:253
3577156994115,6cyclictest141rcu_preempt19:10:343
3577156994112,8cyclictest42-21ksoftirqd/320:44:583
3577152994129,9cyclictest1753-21gnome-shell21:25:242
357715299412,5cyclictest141rcu_preempt19:25:012
3577152994118,6cyclictest141rcu_preempt20:45:002
3577152994114,10cyclictest141rcu_preempt21:25:002
3577152994112,11cyclictest141rcu_preempt22:25:072
357715099412,15cyclictest0-21swapper/119:33:321
3577150994114,23cyclictest3641863-21expr22:10:141
3577146994123,14cyclictest13-21ksoftirqd/022:10:200
3577146994123,13cyclictest438-20systemd-journal19:51:110
3577146994122,6cyclictest13-21ksoftirqd/022:00:010
162399410,30rtkit-daemon2197014-21PK-Backend23:17:182
162399410,29rtkit-daemon2197014-21PK-Backend22:46:062
162399410,29rtkit-daemon2197014-21PK-Backend21:42:462
357715699408,8cyclictest42-21ksoftirqd/322:17:503
357715699406,19cyclictest42-21ksoftirqd/323:35:403
357715699403,18cyclictest42-21ksoftirqd/320:35:053
3577156994023,14cyclictest194-1kworker/3:1H19:35:273
3577156994018,8cyclictest42-21ksoftirqd/322:25:293
3577156994013,8cyclictest42-21ksoftirqd/321:56:523
357715299408,16cyclictest0-21swapper/221:31:182
357715299406,6cyclictest141rcu_preempt00:09:252
3577152994022,14cyclictest3616589-40apt-check21:00:142
3577152994022,14cyclictest34-21ksoftirqd/221:10:022
3577152994022,14cyclictest34-21ksoftirqd/221:10:022
357715299401,9cyclictest2197014-21PK-Backend22:07:542
3577152994014,12cyclictest141rcu_preempt00:35:322
3577152994012,12cyclictest141rcu_preempt23:41:172
3577152994011,12cyclictest141rcu_preempt00:25:232
3577150994017,6cyclictest3619263-21python321:05:301
3577150994017,6cyclictest3619263-21python321:05:291
3577150994015,20cyclictest3609793-21df_inode20:40:171
3577150994014,22cyclictest1825-21JS14:31:441
3577146994024,12cyclictest855-21dbus-daemon19:15:360
357714699402,34cyclictest1753-21gnome-shell23:43:100
3577146994023,12cyclictest3605379-21apt-get20:30:010
3577146994022,13cyclictest855-21dbus-daemon22:55:020
3577146994019,15cyclictest1-21systemd22:45:000
3577146994017,12cyclictest3595801-21sed20:00:250
162399400,28rtkit-daemon2197014-21PK-Backend23:04:402
162399400,28rtkit-daemon2197014-21PK-Backend20:09:092
162399400,28rtkit-daemon2197014-21PK-Backend20:09:092
35776952390,5sleep20-21swapper/219:10:142
357715699395,8cyclictest42-21ksoftirqd/320:10:033
357715699395,8cyclictest42-21ksoftirqd/320:10:023
357715699395,17cyclictest42-21ksoftirqd/323:50:033
3577156993923,12cyclictest855-21dbus-daemon20:50:023
357715299397,10cyclictest0-21swapper/220:25:162
3577152993924,11cyclictest3579748-21tr19:15:152
3577152993922,13cyclictest34-21ksoftirqd/223:35:582
3577152993921,13cyclictest34-21ksoftirqd/223:58:062
3577152993920,14cyclictest34-21ksoftirqd/222:10:292
3577152993914,11cyclictest141rcu_preempt19:45:102
3577152993911,6cyclictest141rcu_preempt23:25:242
3577152993910,4cyclictest141rcu_preempt20:20:232
357714699392,8cyclictest1753-21gnome-shell21:17:450
3577146993923,12cyclictest3628092-21awk21:30:210
3577146993919,7cyclictest13-21ksoftirqd/000:10:420
3577146993912,6cyclictest36186852cstates21:05:160
3577146993912,6cyclictest36186852cstates21:05:160
357715699381,9cyclictest2197014-21PK-Backend23:15:253
357715699381,13cyclictest42-21ksoftirqd/320:57:373
357715299386,10cyclictest34-21ksoftirqd/221:15:132
357715299382,32cyclictest1753-21gnome-shell21:56:352
3577152993822,12cyclictest34-21ksoftirqd/222:40:522
3577152993822,12cyclictest34-21ksoftirqd/219:33:142
3577152993821,13cyclictest34-21ksoftirqd/223:24:092
3577152993821,12cyclictest34-21ksoftirqd/221:50:332
3577152993821,12cyclictest34-21ksoftirqd/220:00:492
3577152993810,8cyclictest141rcu_preempt00:10:202
3577150993824,10cyclictest3629138-21grep21:35:011
357714699385,30cyclictest1358-1kworker/u9:123:57:510
357714699385,16cyclictest1358-1kworker/u9:120:46:040
357714699383,31cyclictest1753-21gnome-shell21:49:570
3577146993826,8cyclictest3670677-21wc23:30:320
3577146993826,8cyclictest3670677-21wc23:30:320
357714699382,32cyclictest1753-21gnome-shell21:29:190
357714699382,32cyclictest1753-21gnome-shell21:02:350
3577146993821,13cyclictest3680555-21if_enp2s000:00:210
3577146993821,12cyclictest438-20systemd-journal22:34:550
3577146993820,5cyclictest13-21ksoftirqd/000:10:000
357714699381,10cyclictest3639542-21turbostat22:05:010
162399380,31rtkit-daemon0-21swapper/020:32:520
162399380,27rtkit-daemon2197014-21PK-Backend22:59:472
162399380,27rtkit-daemon2197014-21PK-Backend22:17:002
162399380,27rtkit-daemon2197014-21PK-Backend21:38:142
162399380,26rtkit-daemon2197014-21PK-Backend23:33:232
162399380,26rtkit-daemon2197014-21PK-Backend23:33:222
3577152993720,13cyclictest3607795-21cat20:35:152
357714699372,31cyclictest0-21swapper/000:21:560
3577146993722,11cyclictest3668604-21grep23:25:240
3577146993721,12cyclictest3650463-21expr22:35:160
3577146993721,12cyclictest3635679-21cat21:50:310
3577146993721,12cyclictest3601829-21sshd20:19:140
3577146993721,11cyclictest438-20systemd-journal23:46:500
3577146993720,13cyclictest13-21ksoftirqd/020:38:150
3577146993720,12cyclictest438-20systemd-journal19:28:120
3577146993720,12cyclictest3624001-21cat21:20:130
3577146993719,13cyclictest3597173-21cat20:05:190
3577146993719,13cyclictest3597173-21cat20:05:180
3577146993712,5cyclictest141rcu_preempt00:30:250
162399370,26rtkit-daemon2197014-21PK-Backend20:58:522
357715299362,4cyclictest2197014-21PK-Backend00:31:132
3577146993620,11cyclictest3593479-21sshd19:55:130
3577146993620,11cyclictest3578588-21sshd19:12:130
3577146993619,12cyclictest438-20systemd-journal22:19:590
3577146993619,12cyclictest438-20systemd-journal19:44:550
162399360,29rtkit-daemon0-21swapper/020:41:090
357715299355,9cyclictest34-21ksoftirqd/223:06:412
357714699357,6cyclictest141rcu_preempt23:35:260
357714699353,13cyclictest13-21ksoftirqd/022:22:530
357714699352,7cyclictest1753-21gnome-shell23:06:190
357714699352,29cyclictest1753-21gnome-shell21:43:450
3577146993519,12cyclictest855-21dbus-daemon23:00:000
3577146993519,12cyclictest3664859-21diskstats23:15:190
3577146993518,6cyclictest13-21ksoftirqd/019:25:000
3577146993518,13cyclictest3662742-21apt23:10:120
3577146993517,13cyclictest2181-21gdbus22:10:000
3577146993512,5cyclictest141rcu_preempt21:12:270
357715299346,8cyclictest141rcu_preempt00:21:572
3577152993416,14cyclictest34-21ksoftirqd/220:50:002
357714699345,12cyclictest141rcu_preempt20:14:150
357714699342,7cyclictest1753-21gnome-shell20:59:590
3577146993426,5cyclictest0-21swapper/023:24:240
3577146993423,7cyclictest1753-21gnome-shell00:15:040
3577146993421,9cyclictest1753-21gnome-shell19:47:280
3577146993416,13cyclictest3692799-21kernelversion00:35:220
357714699341,29cyclictest3647147-21hddtemp_smartct22:25:210
3577146993321,8cyclictest3630371-21grep21:35:310
357714699331,6cyclictest141rcu_preempt19:34:430
357715299329,5cyclictest141rcu_preempt19:50:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional