You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 00:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 09, 2025 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3644139210075,16sleep30-21swapper/307:09:143
36741822970,5sleep20-21swapper/208:30:222
364407928565,14sleep20-21swapper/207:08:282
37269472790,4chrt3726204-21/usr/sbin/munin11:00:261
162399790,6rtkit-daemon2197014-21PK-Backend12:25:013
162399790,6rtkit-daemon2197014-21PK-Backend12:25:013
364387327759,10sleep00-21swapper/007:05:460
162399740,8rtkit-daemon856-21gmain07:43:220
162399740,5rtkit-daemon2197014-21PK-Backend12:35:570
364397627343,23sleep10-21swapper/107:07:061
162399730,6rtkit-daemon2197014-21PK-Backend12:15:363
162399730,6rtkit-daemon2197014-21PK-Backend07:46:250
37124642720,7sleep22197014-21PK-Backend10:20:152
162399710,5rtkit-daemon0-21swapper/012:01:040
36943892600,7sleep01753-21gnome-shell09:30:130
162399590,9rtkit-daemon1622-21rtkit-daemon08:09:080
162399580,9rtkit-daemon1622-21rtkit-daemon08:33:591
162399580,9rtkit-daemon1622-21rtkit-daemon08:26:230
36980542570,7chrt1753-21gnome-shell09:40:142
3644343995714,19cyclictest3714680-21wc10:25:242
162399570,8rtkit-daemon1622-21rtkit-daemon12:29:561
162399570,8rtkit-daemon1622-21rtkit-daemon07:21:101
162399570,8rtkit-daemon1622-21rtkit-daemon07:14:331
3644343995626,8cyclictest28750irq/122-ahci[0009:30:302
3644341995613,29cyclictest26-21ksoftirqd/107:40:001
36760772550,12chrt341ksoftirqd/208:35:252
3644343995519,12cyclictest3660164-21sh07:50:232
162399540,8rtkit-daemon1622-21rtkit-daemon07:41:411
162399540,8rtkit-daemon1622-21rtkit-daemon07:24:170
3644343995319,11cyclictest3729421-21/usr/sbin/munin11:10:012
3644343995218,11cyclictest3733536-21df_inode11:20:182
3644343995216,14cyclictest1753-21gnome-shell07:25:162
162399520,8rtkit-daemon1622-21rtkit-daemon08:21:540
162399520,8rtkit-daemon1622-21rtkit-daemon08:01:051
162399520,8rtkit-daemon1622-21rtkit-daemon08:01:051
3644341995119,10cyclictest26-21ksoftirqd/109:50:001
36568492500,8sleep30-21swapper/307:40:333
364433699509,15cyclictest855-21dbus-daemon07:30:030
162399500,6rtkit-daemon1622-21rtkit-daemon08:13:261
162399500,6rtkit-daemon1622-21rtkit-daemon08:13:261
3644341994913,18cyclictest26-21ksoftirqd/108:00:031
162399490,8rtkit-daemon856-21gmain10:34:060
364434699481,15cyclictest37424442expr11:45:133
3644336994826,7cyclictest13-21ksoftirqd/009:15:200
162399480,35rtkit-daemon2287-21gmain08:51:481
37495962470,17sleep234-21ksoftirqd/212:05:192
36831392470,8sleep02197014-21PK-Backend08:55:310
3644346994724,19cyclictest3662661-21munin-run08:00:013
3644341994724,6cyclictest26-21ksoftirqd/109:30:011
3644341994721,12cyclictest1753-21gnome-shell09:21:141
3644336994723,20cyclictest3663537-40lsb_release08:00:210
3644336994723,20cyclictest3663537-40lsb_release08:00:200
36860342460,17sleep126-21ksoftirqd/109:05:141
3644346994622,20cyclictest42-21ksoftirqd/310:45:143
364434199469,18cyclictest26-21ksoftirqd/112:15:011
3644341994637,6cyclictest251rcuc/110:02:201
3644341994621,9cyclictest26-21ksoftirqd/111:27:281
3644341994614,21cyclictest251rcuc/111:23:151
364434199461,10cyclictest26-21ksoftirqd/111:45:031
364434199461,10cyclictest26-21ksoftirqd/111:45:021
37611632450,7sleep21753-21gnome-shell12:35:302
364434199459,19cyclictest26-21ksoftirqd/109:10:301
364434199458,20cyclictest26-21ksoftirqd/111:32:291
364434199452,18cyclictest26-21ksoftirqd/110:00:011
364434199452,18cyclictest26-21ksoftirqd/110:00:011
3644341994513,18cyclictest26-21ksoftirqd/107:30:041
3644341994513,18cyclictest26-21ksoftirqd/107:20:001
364434199451,15cyclictest26-21ksoftirqd/110:55:251
3644341994510,19cyclictest26-21ksoftirqd/108:35:131
364433699453,9cyclictest1753-21gnome-shell09:28:190
364434699442,7cyclictest0-21swapper/311:55:013
3644343994429,11cyclictest1753-21gnome-shell10:17:362
3644343994429,11cyclictest1753-21gnome-shell10:17:362
3644343994414,10cyclictest1753-21gnome-shell11:25:042
364434199444,23cyclictest26-21ksoftirqd/111:15:271
364434199441,39cyclictest3761077-21sshd12:35:291
364434199441,14cyclictest26-21ksoftirqd/108:55:191
3644336994426,13cyclictest438-20systemd-journal11:30:000
162399440,31rtkit-daemon0-21swapper/108:48:281
37585492430,12sleep2331rcuc/212:30:142
37420802430,7chrt0-21swapper/211:45:022
37420802430,7chrt0-21swapper/211:45:022
36766822430,7chrt2197014-21PK-Backend08:38:483
364434199437,20cyclictest26-21ksoftirqd/109:45:021
364434199436,20cyclictest26-21ksoftirqd/109:51:261
3644341994312,12cyclictest26-21ksoftirqd/112:20:001
364434199431,13cyclictest141rcu_preempt11:55:001
364434199431,12cyclictest26-21ksoftirqd/111:35:151
36916612420,3sleep20-21swapper/209:20:252
364434699429,15cyclictest1753-21gnome-shell07:46:303
3644346994225,6cyclictest42-21ksoftirqd/310:33:353
364434199429,18cyclictest26-21ksoftirqd/108:45:021
364434199426,19cyclictest26-21ksoftirqd/108:25:251
364434199425,17cyclictest26-21ksoftirqd/109:35:271
364434199425,17cyclictest26-21ksoftirqd/109:35:271
3644341994224,13cyclictest438-20systemd-journal11:15:001
3644341994221,8cyclictest26-21ksoftirqd/108:21:541
3644341994212,9cyclictest26-21ksoftirqd/108:05:301
364434199421,19cyclictest26-21ksoftirqd/110:11:381
364434199421,12cyclictest26-21ksoftirqd/112:05:011
364434199421,12cyclictest26-21ksoftirqd/110:38:551
364434199421,12cyclictest26-21ksoftirqd/110:38:551
3644336994223,5cyclictest13-21ksoftirqd/009:45:170
364433699422,13cyclictest141rcu_preempt09:24:590
162399420,35rtkit-daemon3646620-21munin-node12:30:151
364434699419,28cyclictest1753-21gnome-shell10:41:553
3644346994129,7cyclictest438-20systemd-journal09:50:023
3644346994120,10cyclictest42-21ksoftirqd/311:25:223
3644341994124,13cyclictest3750539-21pkexec12:06:271
364434199412,22cyclictest26-21ksoftirqd/109:15:261
3644341994116,9cyclictest26-21ksoftirqd/107:50:291
364434199411,36cyclictest1622-21rtkit-daemon07:48:071
364434199411,22cyclictest3711980-21seq10:18:551
364434199411,22cyclictest3711980-21seq10:18:551
364434199411,13cyclictest3720166-21/usr/sbin/munin10:40:281
364434199411,11cyclictest26-21ksoftirqd/109:00:131
3644336994130,7cyclictest3705124-21taskset09:59:550
3644336994130,7cyclictest3705124-21taskset09:59:550
3644336994129,8cyclictest13-21ksoftirqd/008:40:000
37313162400,6chrt0-21swapper/011:15:020
37002592400,6sleep20-21swapper/209:45:202
36880522400,6chrt0-21swapper/209:10:222
364434699406,16cyclictest1753-21gnome-shell07:24:103
364434699402,34cyclictest1753-21gnome-shell09:56:383
364434699402,34cyclictest1753-21gnome-shell09:56:373
3644346994022,6cyclictest42-21ksoftirqd/311:35:053
3644346994022,14cyclictest1753-21gnome-shell08:00:323
3644346994022,14cyclictest1753-21gnome-shell08:00:323
364434399408,17cyclictest34-21ksoftirqd/207:15:032
364434399402,19cyclictest1753-21gnome-shell09:50:382
364434199406,10cyclictest26-21ksoftirqd/108:15:251
364434199404,19cyclictest26-21ksoftirqd/111:57:441
3644341994022,13cyclictest438-20systemd-journal09:35:001
364434199402,11cyclictest26-21ksoftirqd/111:45:181
364434199401,22cyclictest26-21ksoftirqd/110:25:141
364434199401,21cyclictest26-21ksoftirqd/111:10:011
364434199401,21cyclictest26-21ksoftirqd/110:30:301
364434199401,21cyclictest26-21ksoftirqd/107:30:111
364433699406,5cyclictest141rcu_preempt10:11:380
364433699401,34cyclictest1753-21gnome-shell11:48:540
162399400,28rtkit-daemon2197014-21PK-Backend10:49:330
364434699397,13cyclictest1753-21gnome-shell11:34:053
3644346993925,10cyclictest3669645-21sshd08:17:303
364434699392,33cyclictest1753-21gnome-shell11:56:043
3644346993922,5cyclictest42-21ksoftirqd/309:24:243
364434399392,7cyclictest1753-21gnome-shell10:39:282
364434399392,7cyclictest1753-21gnome-shell10:39:282
364434399392,7cyclictest1753-21gnome-shell08:22:292
364434199398,15cyclictest26-21ksoftirqd/110:07:361
364434199393,21cyclictest26-21ksoftirqd/110:55:011
3644341993925,10cyclictest3713680-21sshd10:23:301
364434199391,21cyclictest26-21ksoftirqd/110:50:001
364433699394,11cyclictest13-21ksoftirqd/008:10:090
364433699394,11cyclictest13-21ksoftirqd/008:10:090
3644336993927,8cyclictest13-21ksoftirqd/007:30:020
3644336993926,9cyclictest13-21ksoftirqd/012:10:000
3644336993922,13cyclictest13-21ksoftirqd/008:40:300
364433699390,35cyclictest1622-21rtkit-daemon07:52:270
36719242380,5chrt0-21swapper/308:25:113
364434699389,7cyclictest37230702expr10:50:153
364434699382,32cyclictest3678532-21munin-run08:45:013
3644346993822,12cyclictest42-21ksoftirqd/309:48:523
3644346993822,12cyclictest1632-21dbus-daemon08:35:023
3644346993815,20cyclictest391irq_work/308:08:013
364434399387,11cyclictest34-21ksoftirqd/207:35:562
3644343993827,7cyclictest3669867-21turbostat08:20:002
3644343993826,8cyclictest1753-21gnome-shell10:54:282
3644343993824,9cyclictest1753-21gnome-shell08:00:192
3644343993824,9cyclictest1753-21gnome-shell08:00:192
364434399382,32cyclictest1753-21gnome-shell09:36:552
364434399382,32cyclictest1753-21gnome-shell09:36:552
3644343993822,12cyclictest3752646-40lsb_release12:12:252
3644343993822,12cyclictest3680203-21sshd08:49:302
3644343993822,12cyclictest3668123-21/usr/sbin/munin08:15:012
3644343993822,12cyclictest3668123-21/usr/sbin/munin08:15:012
364433699388,27cyclictest1069-21runrttasks12:18:290
3644336993827,7cyclictest1753-21gnome-shell11:44:350
3644336993827,7cyclictest1753-21gnome-shell11:44:350
3644336993823,11cyclictest3699324-21cron09:44:590
3644336993821,13cyclictest3668825-21cat08:15:210
3644336993820,6cyclictest13-21ksoftirqd/009:38:450
3644336993820,6cyclictest13-21ksoftirqd/009:38:440
3644336993813,8cyclictest13-21ksoftirqd/007:35:300
3644336993812,22cyclictest3733649-21irqstats11:20:240
162399380,24rtkit-daemon2197014-21PK-Backend11:06:490
162399380,12rtkit-daemon906-21gmain11:31:552
3644346993720,13cyclictest3707332-21grep10:05:153
3644346993720,12cyclictest3748790-21apt-get12:05:003
3644346993720,12cyclictest3711486-21python310:15:303
3644346993720,12cyclictest3711486-21python310:15:293
3644346993713,5cyclictest0-21swapper/307:36:063
3644346993713,20cyclictest3687440-21acpi09:10:123
364434399373,9cyclictest34-21ksoftirqd/209:07:092
364434399372,12cyclictest1753-21gnome-shell12:02:202
3644343993721,12cyclictest3706273-21rt-features10:00:292
3644343993720,12cyclictest438-20systemd-journal11:40:022
364434399371,10cyclictest2197014-21PK-Backend12:20:132
364434399371,10cyclictest2197014-21PK-Backend12:20:132
364434199376,17cyclictest26-21ksoftirqd/112:25:021
364434199376,17cyclictest26-21ksoftirqd/112:25:021
3644336993726,7cyclictest1753-21gnome-shell12:23:540
3644336993726,7cyclictest1753-21gnome-shell12:23:540
3644336993725,8cyclictest1753-21gnome-shell10:51:400
3644336993725,7cyclictest13-21ksoftirqd/010:40:280
3644336993725,7cyclictest13-21ksoftirqd/009:01:240
3644336993724,10cyclictest3701896-21awk09:50:100
3644336993722,11cyclictest1-21systemd08:55:010
3644336993720,12cyclictest3646724-21load07:15:030
162399370,25rtkit-daemon2197014-21PK-Backend09:08:170
162399370,23rtkit-daemon0-21swapper/010:25:050
3644346993624,9cyclictest1753-21gnome-shell12:35:293
364434699362,30cyclictest1753-21gnome-shell12:07:473
3644346993621,11cyclictest3644832-21expr07:10:163
3644346993618,14cyclictest3740994-21memory11:40:243
3644346993618,14cyclictest3740994-21memory11:40:243
3644346993618,13cyclictest3697631-21systemd-run09:40:023
3644346993618,13cyclictest3697631-21systemd-run09:40:013
364434699361,15cyclictest1753-21gnome-shell09:32:373
3644346993611,21cyclictest3242253-21nautilus11:00:023
364434399365,5cyclictest141rcu_preempt08:55:272
3644343993623,9cyclictest1753-21gnome-shell08:51:532
3644343993622,10cyclictest3645800-21gpgv07:11:522
3644343993621,11cyclictest3661871-21ntp_kernel_pll_07:55:242
3644343993620,12cyclictest3731848-21basename11:15:232
3644343993620,12cyclictest2962348-21rs:main2
3644343993619,12cyclictest438-20systemd-journal09:18:312
364434399361,11cyclictest2197014-21PK-Backend11:45:212
3644336993625,7cyclictest1753-21gnome-shell11:30:250
3644336993625,7cyclictest1753-21gnome-shell08:46:530
3644336993624,7cyclictest13-21ksoftirqd/010:55:070
3644336993622,10cyclictest3740119-21cron11:40:000
364433699362,18cyclictest1753-21gnome-shell08:00:000
3644336993621,11cyclictest855-21dbus-daemon10:05:000
3644336993620,11cyclictest438-20systemd-journal10:20:020
3644336993620,11cyclictest438-20systemd-journal10:20:010
3644336993619,5cyclictest13-21ksoftirqd/011:00:160
3644336993619,13cyclictest3717697-21expr10:35:160
3644336993619,13cyclictest3717697-21expr10:35:150
3644336993616,16cyclictest3646073-21unattended-upgr07:12:030
36527322350,5sleep30-21swapper/307:30:133
3644346993524,7cyclictest3682329-21cpuspeed08:55:163
3644346993523,9cyclictest3713453-21awk10:20:333
3644346993521,10cyclictest42-21ksoftirqd/311:10:113
3644346993521,10cyclictest42-21ksoftirqd/309:07:083
364434699351,9cyclictest2197014-21PK-Backend09:04:063
364434699351,9cyclictest2197014-21PK-Backend07:19:383
3644346993517,13cyclictest855-21dbus-daemon08:15:013
3644346993517,13cyclictest855-21dbus-daemon08:15:013
3644346993517,11cyclictest42-21ksoftirqd/311:09:553
364434699351,14cyclictest3731829-21hddtemp_smartct11:15:223
3644346993511,21cyclictest1753-21gnome-shell09:43:203
364434399357,5cyclictest141rcu_preempt08:06:542
3644343993519,12cyclictest3754324-21sshd12:16:302
364434399351,7cyclictest2197014-21PK-Backend07:20:272
3644343993510,8cyclictest0-21swapper/211:57:422
3644336993524,7cyclictest1753-21gnome-shell08:34:200
364433699352,30cyclictest0-21swapper/011:55:460
3644336993521,10cyclictest3745197-21sshd11:53:300
3644336993516,14cyclictest1632-21dbus-daemon12:15:000
364434699347,7cyclictest141rcu_preempt10:35:313
364434699347,7cyclictest141rcu_preempt10:35:303
3644346993422,8cyclictest1753-21gnome-shell07:50:243
364434699342,27cyclictest1753-21gnome-shell08:45:193
364434699342,10cyclictest3752616-40lsb_release12:12:193
3644346993420,10cyclictest3733471-21expr11:20:163
3644346993419,11cyclictest3670551-21basename08:20:203
3644346993417,13cyclictest1753-21gnome-shell09:18:483
3644346993417,12cyclictest904-21systemd-logind12:26:503
364434699341,4cyclictest2197014-21PK-Backend08:51:533
364434399345,5cyclictest141rcu_preempt07:41:322
364434399343,5cyclictest141rcu_preempt07:30:452
3644343993423,7cyclictest1069-21runrttasks10:49:262
364434399341,5cyclictest141rcu_preempt10:55:162
364434399341,12cyclictest1753-21gnome-shell10:10:232
3644336993423,6cyclictest13-21ksoftirqd/010:20:190
3644336993418,12cyclictest3759097-21ntpq12:30:260
3644336993416,14cyclictest3756811-21cstates12:25:150
364434699332,9cyclictest1753-21gnome-shell10:00:233
3644346993320,10cyclictest3652098-21awk07:30:003
3644346993318,11cyclictest42-21ksoftirqd/312:32:213
364434699331,13cyclictest2197014-21PK-Backend11:00:343
3644346993310,20cyclictest0-21swapper/309:25:443
364434399337,5cyclictest141rcu_preempt09:25:292
364434399337,5cyclictest141rcu_preempt08:40:312
364434399335,8cyclictest1753-21gnome-shell09:00:282
3644343993322,7cyclictest1753-21gnome-shell11:11:162
3644343993320,9cyclictest34-21ksoftirqd/210:40:372
3644343993312,5cyclictest141rcu_preempt08:25:272
3644336993318,11cyclictest3731761-21expr11:15:170
364433699331,10cyclictest1358-1kworker/u9:110:07:200
3644346993224,5cyclictest2197014-21PK-Backend10:25:423
364434399322,12cyclictest1753-21gnome-shell11:51:372
3644343993217,11cyclictest3704557-21smartctl09:55:292
3644343993217,11cyclictest3704557-21smartctl09:55:292
3644343993217,10cyclictest3716586-21open_files10:30:262
3644343993216,11cyclictest904-21systemd-logind12:28:502
364434399321,28cyclictest2197014-21PK-Backend11:02:362
3644343993118,10cyclictest3708560-21seq10:09:332
364433699313,11cyclictest141rcu_preempt09:10:040
364434699301,26cyclictest3710270-21chrt10:14:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional