You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-22 - 16:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Mar 22, 2025 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
209479121430,7sleep234-21ksoftirqd/207:05:282
209479121430,7sleep234-21ksoftirqd/207:05:272
1623991040,18rtkit-daemon0-21swapper/008:48:023
162399820,65rtkit-daemon0-21swapper/007:08:280
162399820,65rtkit-daemon0-21swapper/007:08:280
21401652800,6sleep013-21ksoftirqd/009:20:250
162399750,5rtkit-daemon0-21swapper/109:36:541
209526327339,28sleep10-21swapper/107:08:541
209526327339,28sleep10-21swapper/107:08:541
209509826948,14sleep30-21swapper/307:06:463
209509826948,14sleep30-21swapper/307:06:463
162399680,9rtkit-daemon1622-21rtkit-daemon09:53:392
162399670,9rtkit-daemon1622-21rtkit-daemon08:55:402
21233092660,7chrt2197014-21PK-Backend08:30:170
162399660,9rtkit-daemon1622-21rtkit-daemon08:50:362
162399590,9rtkit-daemon1622-21rtkit-daemon10:25:172
162399580,9rtkit-daemon1622-21rtkit-daemon09:09:422
162399560,9rtkit-daemon1622-21rtkit-daemon09:49:352
162399560,8rtkit-daemon1622-21rtkit-daemon11:26:170
2095494995329,20cyclictest1753-21gnome-shell10:27:433
2095481995340,9cyclictest13-21ksoftirqd/012:35:010
162399530,9rtkit-daemon1622-21rtkit-daemon10:13:372
162399520,8rtkit-daemon1622-21rtkit-daemon09:04:482
2095494995128,19cyclictest1753-21gnome-shell11:40:333
2095481995114,33cyclictest13-21ksoftirqd/011:00:350
21619552490,6sleep0161rcuc/010:35:150
2095491994929,15cyclictest34-21ksoftirqd/210:20:282
2095486994936,10cyclictest26-21ksoftirqd/111:25:401
2095481994930,11cyclictest13-21ksoftirqd/007:21:260
2095481994924,14cyclictest13-21ksoftirqd/010:14:350
2095481994923,15cyclictest13-21ksoftirqd/011:09:170
209548199491,6cyclictest141rcu_preempt10:53:560
2095491994823,20cyclictest34-21ksoftirqd/210:40:002
2095481994824,15cyclictest13-21ksoftirqd/008:08:070
2095481994821,23cyclictest2103359-40apt-check07:30:190
2095481994821,14cyclictest13-21ksoftirqd/009:06:200
2095494994724,19cyclictest1753-21gnome-shell09:33:213
2095494994723,20cyclictest1753-21gnome-shell07:12:103
2095481994736,8cyclictest13-21ksoftirqd/010:00:020
2095481994722,14cyclictest13-21ksoftirqd/007:48:260
2095491994635,6cyclictest34-21ksoftirqd/212:00:002
2095491994635,6cyclictest34-21ksoftirqd/212:00:002
2095491994626,7cyclictest34-21ksoftirqd/207:20:292
2095486994616,12cyclictest2155676-21date10:15:021
2095481994621,12cyclictest13-21ksoftirqd/010:59:040
2095481994621,11cyclictest438-20systemd-journal08:22:590
2095481994620,14cyclictest13-21ksoftirqd/012:26:450
162399460,33rtkit-daemon1753-21gnome-shell12:35:502
2095481994523,13cyclictest13-21ksoftirqd/007:35:490
2095481994521,12cyclictest13-21ksoftirqd/012:10:040
2095481994521,12cyclictest13-21ksoftirqd/010:40:460
2095481994521,12cyclictest13-21ksoftirqd/010:20:100
2095481994521,12cyclictest13-21ksoftirqd/010:18:210
2095481994521,12cyclictest13-21ksoftirqd/009:42:020
2095481994521,12cyclictest13-21ksoftirqd/007:54:030
2095481994520,13cyclictest13-21ksoftirqd/011:44:190
2095481994520,13cyclictest13-21ksoftirqd/009:39:180
2095481994520,12cyclictest13-21ksoftirqd/010:25:220
2095481994519,13cyclictest13-21ksoftirqd/011:33:060
2095481994519,13cyclictest13-21ksoftirqd/011:33:050
2095481994423,17cyclictest13-21ksoftirqd/007:58:200
2095481994423,12cyclictest13-21ksoftirqd/008:56:060
2095481994422,14cyclictest13-21ksoftirqd/011:56:500
2095481994422,14cyclictest13-21ksoftirqd/011:56:500
2095481994421,14cyclictest13-21ksoftirqd/009:00:110
2095481994421,13cyclictest13-21ksoftirqd/007:40:460
2095481994420,14cyclictest13-21ksoftirqd/012:19:450
2095481994420,14cyclictest13-21ksoftirqd/012:19:450
2095481994420,13cyclictest13-21ksoftirqd/011:15:110
2095481994420,12cyclictest13-21ksoftirqd/011:54:540
2095481994420,12cyclictest13-21ksoftirqd/011:54:540
2095481994420,12cyclictest13-21ksoftirqd/011:21:420
2095481994420,12cyclictest13-21ksoftirqd/010:09:020
2095481994420,12cyclictest13-21ksoftirqd/009:48:400
2095481994420,12cyclictest13-21ksoftirqd/008:36:270
2095481994420,12cyclictest13-21ksoftirqd/008:29:400
2095481994420,12cyclictest13-21ksoftirqd/007:28:500
2095481994420,12cyclictest13-21ksoftirqd/007:16:430
2095481994417,14cyclictest13-21ksoftirqd/011:45:280
162399440,31rtkit-daemon1753-21gnome-shell07:46:091
162399440,30rtkit-daemon1753-21gnome-shell09:28:120
2095491994326,7cyclictest1753-21gnome-shell08:03:082
2095491994325,13cyclictest2103359-40apt-check07:30:142
2095491994324,6cyclictest34-21ksoftirqd/208:20:002
2095491994324,14cyclictest34-21ksoftirqd/210:05:012
2095481994332,8cyclictest13-21ksoftirqd/010:00:010
2095481994319,12cyclictest13-21ksoftirqd/012:09:520
2095481994319,12cyclictest13-21ksoftirqd/008:03:240
2095481994318,21cyclictest2141748-21apt09:30:090
162399430,34rtkit-daemon2095475-21cyclictest11:32:301
162399430,34rtkit-daemon2095475-21cyclictest11:32:301
162399430,32rtkit-daemon2197014-21PK-Backend11:43:052
2095491994234,5cyclictest0-21swapper/212:11:312
2095491994233,6cyclictest34-21ksoftirqd/210:20:002
2095491994231,7cyclictest1753-21gnome-shell07:16:412
2095491994224,6cyclictest34-21ksoftirqd/210:30:592
2095491994224,6cyclictest34-21ksoftirqd/207:25:272
2095491994224,14cyclictest34-21ksoftirqd/211:25:002
2095491994224,14cyclictest34-21ksoftirqd/208:35:162
2095491994219,17cyclictest1-21systemd09:35:012
2095486994216,22cyclictest1897929-21snapd12:09:471
2095481994231,7cyclictest1753-21gnome-shell10:33:480
2095481994223,7cyclictest13-21ksoftirqd/011:39:530
2095481994223,7cyclictest13-21ksoftirqd/008:16:540
2095481994222,7cyclictest13-21ksoftirqd/009:15:130
2095481994221,13cyclictest13-21ksoftirqd/012:22:170
162399420,30rtkit-daemon2197014-21PK-Backend09:12:050
209549499418,14cyclictest884-21gdbus11:05:023
209549499411,36cyclictest2174328-21/usr/sbin/munin11:15:273
2095491994123,6cyclictest34-21ksoftirqd/208:34:392
2095491994123,13cyclictest34-21ksoftirqd/209:25:012
2095491994123,13cyclictest34-21ksoftirqd/207:40:152
2095491994123,13cyclictest2164392-21uptime10:40:312
2095491994122,6cyclictest34-21ksoftirqd/207:10:142
2095491994122,15cyclictest34-21ksoftirqd/209:44:362
2095481994123,6cyclictest13-21ksoftirqd/010:45:530
2095481994123,6cyclictest13-21ksoftirqd/010:45:520
2095481994123,6cyclictest13-21ksoftirqd/008:46:210
2095481994122,7cyclictest13-21ksoftirqd/008:40:030
2095481994122,6cyclictest13-21ksoftirqd/008:51:460
2095481994121,7cyclictest13-21ksoftirqd/007:13:470
2095481994120,12cyclictest13-21ksoftirqd/012:36:340
209549499401,9cyclictest2197014-21PK-Backend09:02:273
2095491994029,7cyclictest1753-21gnome-shell11:39:442
2095491994023,5cyclictest34-21ksoftirqd/207:35:292
2095491994022,6cyclictest34-21ksoftirqd/207:55:262
2095491994021,6cyclictest34-21ksoftirqd/209:30:022
2095491994021,14cyclictest34-21ksoftirqd/211:30:022
2095486994023,12cyclictest438-20systemd-journal11:54:561
2095486994023,12cyclictest438-20systemd-journal11:54:561
2095486994019,17cyclictest2198389-21gdbus12:36:281
2095486994017,19cyclictest2103359-31apt-check07:30:121
2095486994015,20cyclictest2157213-21cron10:19:591
2095486994014,22cyclictest2187687-21ntp_kernel_err12:00:231
209548699401,35cyclictest2179399-21find11:35:121
2095481994028,8cyclictest2116225-21acpi08:10:100
2095481994022,6cyclictest13-21ksoftirqd/012:00:080
2095481994022,6cyclictest13-21ksoftirqd/009:54:100
162399400,31rtkit-daemon26-21ksoftirqd/111:14:281
162399400,31rtkit-daemon26-21ksoftirqd/111:14:281
162399400,26rtkit-daemon0-21swapper/109:26:221
21447552390,5sleep2331rcuc/209:35:312
2095494993924,11cyclictest2161143-21sshd10:32:593
209549499392,33cyclictest1753-21gnome-shell07:36:513
209549499392,32cyclictest1753-21gnome-shell11:05:063
2095494993921,14cyclictest3242253-21nautilus11:50:023
2095491993928,7cyclictest1753-21gnome-shell10:58:532
2095491993922,5cyclictest34-21ksoftirqd/211:53:462
2095491993922,5cyclictest34-21ksoftirqd/211:53:462
2095491993920,6cyclictest34-21ksoftirqd/212:30:222
2095491993920,6cyclictest34-21ksoftirqd/212:28:312
2095486993912,22cyclictest2131743-21timerwakeupswit08:55:321
2095481993919,12cyclictest13-21ksoftirqd/011:13:110
2095481993919,12cyclictest13-21ksoftirqd/011:13:110
162399390,33rtkit-daemon970-21ntpd08:49:292
162399390,28rtkit-daemon2197014-21PK-Backend12:17:113
162399390,28rtkit-daemon2197014-21PK-Backend12:17:103
162399390,27rtkit-daemon0-21swapper/212:07:042
162399390,26rtkit-daemon0-21swapper/110:59:211
209549499387,13cyclictest1753-21gnome-shell11:54:593
209549499387,13cyclictest1753-21gnome-shell11:54:593
209549499382,32cyclictest1753-21gnome-shell07:21:493
2095494993823,11cyclictest2185522-21ls11:55:183
2095494993823,11cyclictest2185522-21ls11:55:183
2095494993823,11cyclictest2138453-21proc_pri09:15:293
2095494993822,11cyclictest3242253-21nautilus08:30:013
209549199385,29cyclictest1069-21runrttasks11:07:432
2095491993827,7cyclictest1753-21gnome-shell11:45:192
2095491993827,7cyclictest1753-21gnome-shell08:24:432
2095491993827,7cyclictest1753-21gnome-shell08:09:122
2095491993826,8cyclictest1753-21gnome-shell12:15:212
2095491993826,8cyclictest1753-21gnome-shell12:15:212
2095491993824,9cyclictest34-21ksoftirqd/211:00:202
2095491993822,13cyclictest34-21ksoftirqd/212:25:022
2095491993821,6cyclictest34-21ksoftirqd/211:15:232
2095491993821,13cyclictest34-21ksoftirqd/210:00:002
209548699382,32cyclictest1753-21gnome-shell10:09:481
2095486993821,13cyclictest2114837-21expr08:00:311
2095486993818,15cyclictest1632-21dbus-daemon11:00:031
209548699381,33cyclictest1753-21gnome-shell11:22:391
162399380,5rtkit-daemon0-21swapper/212:02:042
162399380,27rtkit-daemon0-21swapper/110:49:021
162399380,27rtkit-daemon0-21swapper/110:49:021
162399380,26rtkit-daemon0-21swapper/108:31:461
162399380,26rtkit-daemon0-21swapper/108:26:531
162399380,25rtkit-daemon0-21swapper/109:33:301
2095494993723,10cyclictest2155153-21sshd10:10:593
2095494993722,11cyclictest2167663-21expr10:55:153
2095494993722,11cyclictest2148134-21expr09:50:103
2095494993720,13cyclictest855-21dbus-daemon08:40:003
2095494993720,13cyclictest2178801-21seq11:30:453
2095494993720,13cyclictest2178801-21seq11:30:453
2095494993718,15cyclictest2097207-21munin-node07:15:063
209549499371,16cyclictest1753-21gnome-shell10:18:043
2095491993727,6cyclictest860997-21snmpd07:47:572
2095491993726,7cyclictest1753-21gnome-shell08:28:522
2095491993720,6cyclictest34-21ksoftirqd/210:50:012
2095491993720,6cyclictest34-21ksoftirqd/210:50:002
2095491993720,13cyclictest34-21ksoftirqd/208:41:132
2095486993725,9cyclictest2147617-21dump-pmu-power09:50:001
2095486993723,10cyclictest2158620-21sshd10:20:591
209548699372,30cyclictest2195548-21diskstats12:30:131
2095486993720,12cyclictest2193566-21sh12:20:291
2095486993720,12cyclictest2129443-21if_err_enp2s008:50:151
2095486993711,10cyclictest2105195-21cpu07:35:131
162399370,28rtkit-daemon1622-21rtkit-daemon09:10:183
162399370,26rtkit-daemon0-21swapper/109:16:031
162399370,25rtkit-daemon0-21swapper/107:42:471
2095494993624,8cyclictest1753-21gnome-shell07:56:183
2095494993623,9cyclictest1753-21gnome-shell09:24:103
2095494993621,11cyclictest2172217-21idleruntime11:10:163
2095494993621,11cyclictest2172217-21idleruntime11:10:153
2095494993620,12cyclictest2130486-21pool-org.gnome.08:55:013
2095494993620,12cyclictest2124615-21cron08:35:003
2095494993619,12cyclictest2104006-21missed_timers07:30:273
2095494993616,16cyclictest1753-21gnome-shell12:01:003
2095491993625,7cyclictest1753-21gnome-shell11:10:062
2095491993625,7cyclictest1753-21gnome-shell11:10:052
2095491993625,7cyclictest1753-21gnome-shell08:11:022
2095491993622,10cyclictest34-21ksoftirqd/209:10:522
2095491993619,6cyclictest34-21ksoftirqd/211:33:502
2095491993619,6cyclictest34-21ksoftirqd/211:33:492
2095486993622,10cyclictest26-21ksoftirqd/108:21:371
2095486993621,11cyclictest26-21ksoftirqd/109:59:011
2095486993621,11cyclictest2151898-21expr10:00:161
209548699361,9cyclictest0-21swapper/107:54:521
2095486993618,13cyclictest438-20systemd-journal08:15:001
162399360,26rtkit-daemon0-21swapper/312:28:013
162399360,25rtkit-daemon0-21swapper/107:10:391
209549499357,6cyclictest141rcu_preempt10:41:093
2095494993522,9cyclictest1753-21gnome-shell09:39:023
209549499352,29cyclictest1753-21gnome-shell07:27:403
2095494993521,11cyclictest42-21ksoftirqd/307:40:303
2095494993520,10cyclictest2147805-21cat09:50:033
2095494993518,13cyclictest1753-21gnome-shell10:35:103
209549199354,11cyclictest34-21ksoftirqd/207:54:322
2095491993524,6cyclictest34-21ksoftirqd/209:15:102
2095491993523,7cyclictest34-21ksoftirqd/210:54:512
209549199351,17cyclictest1753-21gnome-shell10:05:172
2095486993525,6cyclictest1753-21gnome-shell09:10:231
2095486993520,11cyclictest2113587-21sshd07:59:591
2095486993518,12cyclictest438-20systemd-journal08:45:011
2095486993517,13cyclictest2190207-21python312:10:241
209548699351,14cyclictest2197014-21PK-Backend10:30:081
2095486993511,20cyclictest2135637-21cron09:10:011
2095494993422,8cyclictest1753-21gnome-shell09:44:033
2095494993422,8cyclictest1753-21gnome-shell09:29:553
2095494993420,10cyclictest2189211-21sed12:10:013
2095494993419,11cyclictest42-21ksoftirqd/311:28:583
2095494993419,11cyclictest2193303-21cut12:20:253
2095494993419,11cyclictest2149889-21expr09:55:153
2095494993419,10cyclictest42-21ksoftirqd/309:05:323
2095494993417,12cyclictest2152152-21wc10:00:233
209549499341,30cyclictest0-21swapper/308:10:183
209548699345,13cyclictest2148363-21irqstats09:50:161
2095486993423,7cyclictest2185400-21tr11:55:171
2095486993423,7cyclictest2185400-21tr11:55:161
2095486993420,10cyclictest26-21ksoftirqd/112:17:171
2095486993420,10cyclictest26-21ksoftirqd/112:17:161
2095486993420,10cyclictest26-21ksoftirqd/108:45:361
2095486993419,11cyclictest2125910-21cat08:35:311
2095486993418,12cyclictest2145696-21grep09:40:121
2095486993418,12cyclictest2098797-21python307:15:291
2095486993418,11cyclictest2115794-21sshd08:08:001
2095486993417,13cyclictest3242254-21gmain07:25:011
2095486993416,13cyclictest2182234-21users11:40:321
162399340,24rtkit-daemon2197014-21PK-Backend12:32:483
209549499336,10cyclictest42-21ksoftirqd/308:55:043
209549499333,8cyclictest1753-21gnome-shell11:21:403
209549499333,26cyclictest2111885-21turbostat07:55:003
209549499332,9cyclictest0-21swapper/310:07:413
2095494993325,5cyclictest0-21swapper/310:53:333
2095494993321,8cyclictest1753-21gnome-shell08:40:223
2095494993319,10cyclictest2158976-21systemd-run10:25:003
2095494993318,11cyclictest2109202-21grep07:45:213
2095494993312,17cyclictest42-21ksoftirqd/308:00:123
2095486993320,9cyclictest1753-21gnome-shell10:39:491
2095486993319,10cyclictest2171305-21seq11:07:071
2095486993318,11cyclictest2166280-21cat10:50:171
2095486993318,11cyclictest2119044-21cat08:15:371
2095486993317,12cyclictest918-21gdbus09:21:551
2095486993316,12cyclictest2101817-21cat07:25:211
209548699331,29cyclictest0-21swapper/111:47:591
209548699331,15cyclictest2173918-21df_inode11:15:151
2095494993225,4cyclictest2197014-21PK-Backend08:07:493
209549499321,4cyclictest2197014-21PK-Backend08:24:043
209549499321,4cyclictest0-21swapper/311:38:353
209549499321,28cyclictest0-21swapper/308:15:193
209548699325,10cyclictest26-21ksoftirqd/110:30:001
209548699322,13cyclictest2132834-21diskstats09:00:171
2095486993218,10cyclictest2163518-21expr10:40:141
209549499311,4cyclictest2197014-21PK-Backend12:35:113
209549499311,4cyclictest0-21swapper/310:47:183
209549499311,4cyclictest0-21swapper/310:47:173
2095494993114,13cyclictest2189545-21cat12:10:093
2095486993117,10cyclictest26-21ksoftirqd/112:29:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional