You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-28 - 23:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Dec 28, 2025 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
67795625080,9sleep30-21swapper/307:05:203
6781472970,5sleep14497-21PK-Backend07:05:211
7374492840,6sleep30-21swapper/309:55:023
7747572800,5sleep24497-21PK-Backend11:40:212
67788227646,21sleep20-21swapper/207:05:202
7630242700,7sleep1251rcuc/111:05:311
7451642700,20sleep342-21ksoftirqd/310:15:193
7094042660,23sleep013-21ksoftirqd/008:35:140
67744826645,15sleep00-21swapper/007:05:100
679076996117,11cyclictest7697542/usr/sbin/munin11:25:240
7543372600,7sleep10-21swapper/110:40:271
197399600,9rtkit-daemon1972-21rtkit-daemon09:43:503
7151982540,6sleep30-21swapper/308:50:233
6851622520,6sleep10-21swapper/107:25:131
7187152510,7chrt4497-21PK-Backend09:00:213
6970762510,6sleep10-21swapper/108:00:021
7697532490,4sleep20-21swapper/211:25:242
7083092470,14sleep3421ksoftirqd/308:30:273
7145372460,5chrt4497-21PK-Backend08:50:131
7272762440,5sleep30-21swapper/309:25:163
679080994428,12cyclictest787870-21apt-get12:20:011
679088994316,7cyclictest141rcu_preempt07:30:163
679084994315,9cyclictest34-21ksoftirqd/209:00:312
7743562420,5sleep30-21swapper/311:40:143
679088994224,5cyclictest42-21ksoftirqd/311:54:143
197399420,29rtkit-daemon4497-21PK-Backend09:06:260
197399420,28rtkit-daemon4497-21PK-Backend08:51:260
6892022410,6sleep10-21swapper/107:35:271
679084994126,11cyclictest744259-21/usr/sbin/munin10:15:002
679084994122,6cyclictest34-21ksoftirqd/212:15:012
197399410,27rtkit-daemon0-21swapper/007:34:380
197399410,26rtkit-daemon0-21swapper/011:10:100
197399410,26rtkit-daemon0-21swapper/011:10:090
679088994017,12cyclictest42-21ksoftirqd/311:40:003
679084994024,11cyclictest438-20systemd-journal12:09:552
679084994022,14cyclictest34-21ksoftirqd/209:15:402
679084994020,16cyclictest697779-21sed08:00:252
679084994011,6cyclictest141rcu_preempt08:50:112
679080994022,5cyclictest26-21ksoftirqd/107:20:181
679076994021,16cyclictest758632-21/usr/sbin/munin10:55:250
197399400,28rtkit-daemon4497-21PK-Backend08:18:160
679088993928,7cyclictest1137-21runrttasks11:33:103
679088993928,7cyclictest1137-21runrttasks11:33:103
679088993922,5cyclictest42-21ksoftirqd/312:00:223
679084993924,11cyclictest765299-21cron11:15:002
679084993924,11cyclictest765299-21cron11:15:002
67908499392,32cyclictest793800-21grep12:35:172
679084993922,5cyclictest34-21ksoftirqd/208:10:182
679084993922,13cyclictest34-21ksoftirqd/210:35:332
679080993923,11cyclictest438-20systemd-journal09:30:281
679080993922,13cyclictest707965-21smartctl08:30:201
679080993922,13cyclictest26-21ksoftirqd/110:33:481
679080993922,12cyclictest438-20systemd-journal11:54:131
679080993922,12cyclictest438-20systemd-journal09:20:001
679076993929,6cyclictest707291-21dump-pmu-power08:30:000
679076993928,6cyclictest719646-21chrt09:04:270
679076993923,12cyclictest733271-21python309:40:280
197399390,25rtkit-daemon0-21swapper/011:47:320
7891692380,5sleep34497-21PK-Backend12:20:263
7891692380,5sleep34497-21PK-Backend12:20:263
7429592380,8chrt0-21swapper/010:10:160
67908899384,30cyclictest683976-21sh07:20:303
679088993822,12cyclictest923-21dbus-daemon11:05:013
679088993822,11cyclictest712466-21cron08:45:003
679088993820,14cyclictest703238-21python308:15:293
679088993813,7cyclictest792975-21timerwakeupswit12:30:323
67908499389,6cyclictest141rcu_preempt11:34:292
67908499389,6cyclictest141rcu_preempt11:34:292
67908499382,9cyclictest0-21swapper/208:55:132
679084993823,11cyclictest1136-21gdbus07:30:142
679084993822,11cyclictest790509-21cat12:25:172
679084993820,14cyclictest34-21ksoftirqd/209:36:022
679084993820,14cyclictest34-21ksoftirqd/208:09:532
679080993823,11cyclictest26-21ksoftirqd/110:17:061
679080993822,12cyclictest749357-21sshd10:25:281
679080993822,11cyclictest438-20systemd-journal11:20:291
679080993820,14cyclictest26-21ksoftirqd/108:19:021
679080993820,14cyclictest26-21ksoftirqd/107:15:041
67908099381,32cyclictest759533-21sshd10:55:281
679080993812,6cyclictest141rcu_preempt09:50:001
679080993810,6cyclictest141rcu_preempt07:30:161
679076993822,11cyclictest438-20systemd-journal11:20:410
679076993822,11cyclictest13-21ksoftirqd/012:05:140
679076993821,11cyclictest693448-21idleruntime-cro07:50:000
679076993814,6cyclictest141rcu_preempt07:55:020
197399380,27rtkit-daemon4497-21PK-Backend11:07:380
197399380,27rtkit-daemon4497-21PK-Backend10:51:200
197399380,26rtkit-daemon4497-21PK-Backend10:01:470
197399380,25rtkit-daemon0-21swapper/009:58:060
197399380,25rtkit-daemon0-21swapper/009:58:060
6796262370,5chrt0-21swapper/007:10:150
67908899379,24cyclictest0-21swapper/310:25:143
679088993722,11cyclictest791346-21sshd12:26:283
679088993722,11cyclictest738543-21taskset09:55:293
679088993722,11cyclictest738543-21taskset09:55:293
67908899372,14cyclictest743156-21df_inode10:10:193
679088993721,12cyclictest779916-21if_enp2s011:55:193
679088993721,12cyclictest779916-21if_enp2s011:55:183
679088993721,11cyclictest438-20systemd-journal10:49:553
679088993721,11cyclictest438-20systemd-journal10:49:543
679088993720,12cyclictest753187-21cron10:40:003
67908499375,28cyclictest739917-21missed_timers10:00:242
679084993726,7cyclictest702510-21cpuspeed_turbos08:15:152
679084993722,11cyclictest761836-21cron11:05:002
679084993722,11cyclictest1033-21gdbus10:30:012
679084993721,11cyclictest438-20systemd-journal07:29:592
679084993719,14cyclictest34-21ksoftirqd/209:05:392
679084993711,7cyclictest7576212netstat10:50:242
67908499371,12cyclictest924-21NetworkManager09:55:292
67908499371,12cyclictest924-21NetworkManager09:55:292
67908499371,12cyclictest789077-21head12:20:222
67908499371,12cyclictest789077-21head12:20:222
679080993726,7cyclictest761850-21sh11:05:001
679080993721,12cyclictest923-21dbus-daemon09:55:011
679080993721,11cyclictest701161-21sort08:10:201
679080993721,11cyclictest26-21ksoftirqd/109:07:311
679080993720,13cyclictest740762-21seq10:04:571
679080993719,14cyclictest26-21ksoftirqd/112:22:011
679080993719,14cyclictest26-21ksoftirqd/112:22:001
679080993719,14cyclictest26-21ksoftirqd/109:28:591
679080993718,14cyclictest26-21ksoftirqd/112:08:361
679076993723,10cyclictest753197-21munin-plugin-st10:40:000
679076993719,13cyclictest13-21ksoftirqd/010:20:010
679076993716,17cyclictest729551-21switchtime09:30:300
67907699371,18cyclictest438-20systemd-journal10:10:000
197399370,30rtkit-daemon0-21swapper/309:37:473
197399370,26rtkit-daemon4497-21PK-Backend08:43:380
197399370,26rtkit-daemon4497-21PK-Backend08:06:530
197399370,26rtkit-daemon0-21swapper/007:24:490
197399370,25rtkit-daemon4497-21PK-Backend10:31:140
197399370,24rtkit-daemon0-21swapper/011:41:280
67908899367,9cyclictest0-21swapper/310:05:133
679088993625,6cyclictest42-21ksoftirqd/307:55:003
679088993624,8cyclictest747633-21users10:20:313
679088993624,8cyclictest747633-21users10:20:303
679088993622,11cyclictest42-21ksoftirqd/311:05:243
67908899362,13cyclictest776336-21df_inode11:45:143
679088993621,11cyclictest726615-21cron09:25:003
679088993614,17cyclictest719938-21NetworkManager09:05:043
67908499366,8cyclictest34-21ksoftirqd/208:45:432
67908499366,8cyclictest34-21ksoftirqd/208:45:422
67908499363,11cyclictest34-21ksoftirqd/210:58:272
67908499363,11cyclictest34-21ksoftirqd/208:28:372
679084993622,10cyclictest683600-21diskstats07:20:222
67908499362,11cyclictest779101-21systemd-udevd11:54:142
679084993619,12cyclictest727054-21grep09:25:142
679084993619,12cyclictest34-21ksoftirqd/209:31:112
679084993618,14cyclictest766609-21cat11:15:282
679084993618,14cyclictest34-21ksoftirqd/211:22:092
679084993611,5cyclictest141rcu_preempt11:55:302
679084993611,5cyclictest141rcu_preempt11:55:302
679084993610,5cyclictest141rcu_preempt12:01:342
67908099366,9cyclictest26-21ksoftirqd/111:35:371
67908099362,9cyclictest438-20systemd-journal08:10:001
679080993621,11cyclictest26-21ksoftirqd/109:13:471
679080993620,12cyclictest733933-21date09:45:021
679080993619,12cyclictest774760-21wc11:40:201
679080993619,12cyclictest26-21ksoftirqd/112:39:481
67907699363,28cyclictest682283-21wc07:15:260
679076993620,11cyclictest685680-21cat07:25:230
679076993619,11cyclictest923-21dbus-daemon08:55:050
679076993618,13cyclictest713496-21ntp_states08:45:260
679076993618,13cyclictest713496-21ntp_states08:45:260
6977762350,8sleep34497-21PK-Backend08:00:253
67908899352,29cyclictest695846-21hddtemp_smartct07:55:203
679088993520,11cyclictest787659-21sshd12:17:283
679088993517,14cyclictest740059-21ntpq10:00:253
679088993517,13cyclictest438-20systemd-journal07:44:503
67908899351,12cyclictest438-20systemd-journal08:25:013
67908499359,5cyclictest141rcu_preempt09:47:082
67908499354,10cyclictest34-21ksoftirqd/210:17:352
679084993527,5cyclictest689894-21file07:39:482
679084993525,7cyclictest1137-21runrttasks09:11:232
679084993520,11cyclictest736590-21grep09:50:232
679084993519,12cyclictest690985-21ntp_states07:40:252
679084993518,13cyclictest773423-21cat11:35:282
679080993521,10cyclictest706138-21expr08:25:141
679080993520,11cyclictest711490-21interrupts08:40:191
679080993520,10cyclictest742960-21expr10:10:161
679080993519,12cyclictest1-21systemd12:35:011
679080993518,13cyclictest26-21ksoftirqd/107:53:341
679080993518,12cyclictest438-20systemd-journal09:00:001
679080993517,14cyclictest26-21ksoftirqd/110:06:351
679080993516,6cyclictest26-21ksoftirqd/108:35:321
67908099351,29cyclictest752036-21expr10:35:171
679076993521,10cyclictest700788-21grep08:10:120
679076993520,11cyclictest778413-21tr11:50:210
679076993519,12cyclictest725226-21awk09:20:140
679076993519,12cyclictest705108-21sed08:20:310
679076993518,13cyclictest697921-21ntp_states08:00:270
679076993518,13cyclictest689442-21grep07:35:320
679076993518,12cyclictest736154-21cat09:50:140
679076993518,12cyclictest1074-21lldpd09:29:470
679076993517,13cyclictest438-20systemd-journal10:40:290
67907699351,12cyclictest4497-21PK-Backend08:35:010
679088993422,8cyclictest716867-21hddtemp_smartct08:55:173
679088993420,10cyclictest757953-21grep10:50:313
679088993420,10cyclictest710152-21gzip08:35:283
679088993419,11cyclictest793958-21expr12:35:163
679088993419,11cyclictest767034-21cron11:20:003
679088993419,11cyclictest764781-21cat11:10:293
679088993419,11cyclictest764781-21cat11:10:293
67908899341,30cyclictest4497-21PK-Backend09:48:593
67908899341,29cyclictest4497-21PK-Backend09:31:473
67908899341,12cyclictest714300-21cut08:50:013
67908899341,12cyclictest714300-21cut08:50:013
67908899341,12cyclictest692835-21python307:45:283
679084993423,7cyclictest753636-21cpuspeed_turbos10:40:142
679084993423,7cyclictest693719-21cpu07:50:122
67908499342,16cyclictest34-21ksoftirqd/211:46:292
679084993419,11cyclictest787878-21ls12:20:012
679084993418,11cyclictest762207-21cpuspeed11:05:152
679084993418,11cyclictest750172-21expr10:30:132
679084993416,14cyclictest34-21ksoftirqd/208:40:252
67908499341,5cyclictest4497-21PK-Backend09:22:022
679084993415,14cyclictest34-21ksoftirqd/207:56:532
679084993415,14cyclictest34-21ksoftirqd/207:15:322
679080993422,9cyclictest719494-21sshd09:02:291
679080993417,5cyclictest26-21ksoftirqd/111:57:351
679080993417,5cyclictest26-21ksoftirqd/111:57:351
679080993417,13cyclictest781968-21ntp_states12:00:271
679080993417,13cyclictest704850-21ntp_states08:20:271
679076993422,8cyclictest755868-21head10:45:220
679076993422,8cyclictest755868-21head10:45:220
679076993419,10cyclictest780976-21pluginstate12:00:010
679076993419,10cyclictest780976-21pluginstate12:00:010
679076993417,12cyclictest724287-21awk09:15:260
679076993416,8cyclictest722801-21vmstat09:10:300
679076993416,14cyclictest773593-21users11:35:300
679076993416,13cyclictest760838-21iwlist11:00:220
67907699341,29cyclictest0-21swapper/012:01:270
197399340,28rtkit-daemon0-21swapper/307:13:083
197399340,23rtkit-daemon0-21swapper/012:30:570
679088993323,6cyclictest42-21ksoftirqd/308:09:183
679088993320,9cyclictest681458-21/usr/sbin/munin07:15:143
67908899331,4cyclictest4497-21PK-Backend08:26:443
679088993314,15cyclictest42-21ksoftirqd/310:45:013
67908899331,29cyclictest4497-21PK-Backend12:13:273
67908899331,29cyclictest0-21swapper/311:22:293
67908899331,28cyclictest750244-21cat10:30:143
67908899331,28cyclictest723832-21hddtemp_smartct09:15:183
679084993318,11cyclictest747618-21cat10:20:312
679084993318,11cyclictest747618-21cat10:20:312
679084993318,11cyclictest710226-21cat08:35:302
679084993317,12cyclictest732922-21wc09:40:212
679084993317,12cyclictest708884-21sshd08:33:302
679084993316,12cyclictest704077-21find08:20:112
67908099337,7cyclictest0-21swapper/107:40:221
67908099337,5cyclictest141rcu_preempt07:45:381
679080993320,9cyclictest26-21ksoftirqd/109:35:301
679080993319,10cyclictest725338-21grep09:20:151
679080993319,10cyclictest713101-21expr08:45:141
679080993319,10cyclictest713101-21expr08:45:141
67908099331,5cyclictest0-21swapper/111:15:171
679080993315,13cyclictest697160-21cat08:00:131
67908099331,29cyclictest4497-21PK-Backend10:23:411
67908099331,29cyclictest4497-21PK-Backend10:23:401
67908099331,29cyclictest0-21swapper/110:50:351
67908099331,29cyclictest0-21swapper/110:45:531
67908099331,29cyclictest0-21swapper/110:45:531
67908099331,29cyclictest0-21swapper/107:12:561
67907699335,5cyclictest141rcu_preempt11:15:160
679076993324,5cyclictest4497-21PK-Backend09:35:090
679076993318,11cyclictest690461-21expr07:40:120
679076993317,12cyclictest734426-21expr09:45:130
679076993315,13cyclictest790172-21apt12:25:140
67907699331,28cyclictest4497-21PK-Backend10:27:380
67908899322,11cyclictest141rcu_preempt12:09:233
679084993217,11cyclictest692493-21ls07:45:212
679084993217,11cyclictest34-21ksoftirqd/212:30:362
67908499321,28cyclictest0-21swapper/207:11:412
67908099328,5cyclictest141rcu_preempt09:55:371
67908099328,5cyclictest141rcu_preempt09:55:361
67908099322,11cyclictest771415-21missed_timers11:30:231
67908099322,11cyclictest771415-21missed_timers11:30:231
679080993217,11cyclictest785216-21sed12:10:211
679080993216,11cyclictest777221-21sshd11:47:281
67908099321,4cyclictest4497-21PK-Backend12:26:341
67908099321,4cyclictest0-21swapper/111:29:021
679076993217,11cyclictest788455-21awk12:20:140
679076993217,11cyclictest788455-21awk12:20:140
67907699321,5cyclictest141rcu_preempt11:30:300
67907699321,5cyclictest141rcu_preempt11:30:290
67907699321,28cyclictest0-21swapper/012:15:130
67908899312,7cyclictest689036-21memory07:35:243
679088993125,4cyclictest0-21swapper/311:25:563
679088993118,10cyclictest723321-21latency_hist09:15:013
67908899311,25cyclictest684945-21apt07:25:113
679084993115,11cyclictest438-20systemd-journal10:09:282
67908499311,4cyclictest0-21swapper/210:49:372
67908499311,4cyclictest0-21swapper/210:49:372
679080993116,11cyclictest764632-21pluginstate11:10:271
679080993116,11cyclictest764632-21pluginstate11:10:271
679076993117,11cyclictest785278-21sed12:10:210
67907699311,27cyclictest0-21swapper/007:59:240
67907699301,4cyclictest0-21swapper/012:35:310
679088992922,4cyclictest4497-21PK-Backend08:10:283
679076992918,7cyclictest746671-21tr10:20:150
679076992918,7cyclictest746671-21tr10:20:140
67908899271,10cyclictest4497-21PK-Backend10:56:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional