You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 15, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991050,8rtkit-daemon1622-21rtkit-daemon00:30:030
22920672970,5sleep2331rcuc/219:45:342
162399900,8rtkit-daemon1622-21rtkit-daemon20:03:073
23302802870,5sleep10-21swapper/121:35:181
162399850,6rtkit-daemon1622-21rtkit-daemon19:51:353
227776228239,21sleep20-21swapper/219:09:182
227798399782,47cyclictest1753-21gnome-shell22:31:213
227748227644,26sleep30-21swapper/319:05:453
22978562740,5sleep00-21swapper/020:05:010
227798399742,36cyclictest1753-21gnome-shell00:05:043
227798399722,49cyclictest1753-21gnome-shell19:59:143
227798399722,48cyclictest1753-21gnome-shell23:10:183
227798399721,49cyclictest2354747-21sed22:45:223
227798399721,41cyclictest2309024-21sed20:35:243
227798399721,41cyclictest2309024-21sed20:35:243
227752227240,9sleep00-21swapper/019:06:140
227798399712,50cyclictest1753-21gnome-shell21:34:243
227798399709,40cyclictest0-21swapper/323:50:203
227798399704,47cyclictest2304624-21NetworkManager20:23:413
227798399702,47cyclictest1753-21gnome-shell21:19:353
227798399692,47cyclictest1753-21gnome-shell21:29:193
227798399692,46cyclictest1753-21gnome-shell00:32:103
227798399692,42cyclictest2388002-21cat00:20:133
227798399692,37cyclictest1753-21gnome-shell21:05:273
227798399691,37cyclictest2301255-21taskset20:14:573
162399690,59rtkit-daemon0-21swapper/119:05:581
227798399663,33cyclictest1753-21gnome-shell21:45:183
227798399662,45cyclictest1753-21gnome-shell21:02:113
227798399662,44cyclictest1753-21gnome-shell23:33:393
227798399662,44cyclictest1753-21gnome-shell20:32:393
227798399661,37cyclictest1753-21gnome-shell00:35:303
227798399652,43cyclictest1753-21gnome-shell19:48:053
227798399652,36cyclictest1753-21gnome-shell19:39:573
227798399652,36cyclictest1753-21gnome-shell19:39:563
227798399652,34cyclictest1753-21gnome-shell22:05:253
227798399651,37cyclictest2357602-21runrttasks22:55:023
2277983996428,32cyclictest1753-21gnome-shell23:25:243
227798399642,38cyclictest2631682-21snapd21:55:593
227798399642,36cyclictest1753-21gnome-shell20:25:253
227798399642,34cyclictest1753-21gnome-shell21:20:223
227798399632,38cyclictest1753-21gnome-shell21:42:053
227798399632,34cyclictest1753-21gnome-shell19:29:253
227798399632,34cyclictest1753-21gnome-shell19:11:323
227798399632,33cyclictest1753-21gnome-shell19:42:423
227798399622,34cyclictest1753-21gnome-shell23:42:233
227798399621,35cyclictest2372645-21/usr/sbin/munin23:35:313
2277974996211,48cyclictest1622-21rtkit-daemon23:32:481
227798399612,36cyclictest1753-21gnome-shell00:18:383
227798399611,35cyclictest1753-21gnome-shell20:18:033
227798399602,34cyclictest1753-21gnome-shell23:56:383
162399600,9rtkit-daemon1622-21rtkit-daemon22:28:573
227798399592,35cyclictest1753-21gnome-shell22:54:573
227798399592,35cyclictest1753-21gnome-shell22:00:133
227798399592,32cyclictest0-21swapper/320:51:223
227798399592,32cyclictest0-21swapper/320:51:223
227798399591,33cyclictest0-21swapper/321:15:013
227798399582,33cyclictest1753-21gnome-shell19:15:253
227798399582,32cyclictest1753-21gnome-shell23:24:333
227798399582,32cyclictest1753-21gnome-shell22:44:103
227798399582,32cyclictest1753-21gnome-shell22:14:433
227798399582,31cyclictest1753-21gnome-shell22:22:393
227798399582,31cyclictest1753-21gnome-shell20:45:053
227798399581,33cyclictest854-21avahi-daemon20:59:223
2277970995847,7cyclictest13-21ksoftirqd/023:50:360
162399580,8rtkit-daemon1622-21rtkit-daemon23:08:003
227798399572,33cyclictest1753-21gnome-shell21:35:493
227798399571,32cyclictest0-21swapper/322:15:323
2277974995719,19cyclictest26-21ksoftirqd/122:58:121
227798399562,32cyclictest1753-21gnome-shell21:53:183
227798399562,29cyclictest1753-21gnome-shell23:02:123
227798399561,50cyclictest1753-21gnome-shell19:32:223
2277979995618,13cyclictest1632-21dbus-daemon21:25:012
2277974995617,18cyclictest26-21ksoftirqd/123:16:231
227798399552,31cyclictest1753-21gnome-shell23:45:043
227798399552,31cyclictest1753-21gnome-shell00:14:373
227798399551,31cyclictest2310192-21/usr/sbin/munin20:40:193
227798399551,30cyclictest0-21swapper/323:16:253
2277979995519,11cyclictest2357555-21cat22:55:022
2277979995517,12cyclictest2303788-21cut20:20:222
2277979995516,34cyclictest2360200-40lsb_release23:00:232
2277974995521,18cyclictest26-21ksoftirqd/123:58:581
2277974995428,8cyclictest26-21ksoftirqd/123:50:011
2277974995418,19cyclictest26-21ksoftirqd/121:31:061
162399540,8rtkit-daemon1622-21rtkit-daemon19:20:463
23735272530,4sleep00-21swapper/023:40:020
227798399530,49cyclictest1622-21rtkit-daemon22:35:493
2277979995319,8cyclictest2367067-21ls23:20:212
2277979995315,11cyclictest2296415-21cat20:00:142
2277974995330,20cyclictest26-21ksoftirqd/123:50:361
2277974995319,18cyclictest26-21ksoftirqd/120:18:461
2277974995319,18cyclictest26-21ksoftirqd/119:46:581
2277979995216,11cyclictest2283669-21systemd19:25:012
2277970995229,8cyclictest13-21ksoftirqd/023:58:370
2277979995115,11cyclictest2330191-21expr21:35:162
2277979995114,11cyclictest2358603-21expr22:55:312
227797499519,20cyclictest26-21ksoftirqd/120:55:231
2277974995114,16cyclictest26-21ksoftirqd/120:14:231
2277974995113,17cyclictest26-21ksoftirqd/120:10:011
2277974995110,13cyclictest26-21ksoftirqd/100:35:221
23472052500,6sleep234-21ksoftirqd/222:25:062
22957652500,15chrt13-21ksoftirqd/019:56:160
2277983995016,31cyclictest3242253-21nautilus00:05:013
2277979995019,10cyclictest34-21ksoftirqd/221:10:022
2277979995017,8cyclictest1069-21runrttasks00:31:482
227797499509,19cyclictest26-21ksoftirqd/122:15:021
227797499508,20cyclictest26-21ksoftirqd/121:10:251
2277974995026,7cyclictest26-21ksoftirqd/123:01:461
227797499501,22cyclictest1753-21gnome-shell00:27:241
2277974995010,20cyclictest26-21ksoftirqd/119:25:151
2277970995012,20cyclictest13-21ksoftirqd/023:40:130
2277979994916,8cyclictest2372351-21uname23:35:242
2277979994915,12cyclictest1-21systemd20:10:432
2277979994913,11cyclictest2281288-21switchtime19:15:302
2277979994912,14cyclictest884-21gdbus23:11:452
2277979994910,18cyclictest1753-21gnome-shell00:19:402
2277979994910,16cyclictest1753-21gnome-shell22:49:032
227797499496,39cyclictest1622-21rtkit-daemon23:40:501
227797499496,20cyclictest26-21ksoftirqd/100:01:381
227797499492,12cyclictest26-21ksoftirqd/100:20:311
2277974994920,9cyclictest26-21ksoftirqd/119:35:011
227797099492,8cyclictest13-21ksoftirqd/020:27:430
2277970994910,22cyclictest13-21ksoftirqd/023:00:040
162399490,37rtkit-daemon2197014-21PK-Backend21:46:390
23911682480,6sleep02197014-21PK-Backend00:27:270
23262812480,5sleep02197014-21PK-Backend21:25:030
22967372480,6sleep12197014-21PK-Backend20:00:191
2277979994819,7cyclictest1753-21gnome-shell20:55:142
2277979994817,10cyclictest2329593-21sshd21:34:442
2277979994816,11cyclictest34-21ksoftirqd/200:10:202
2277979994810,18cyclictest1753-21gnome-shell22:33:192
2277974994837,7cyclictest251rcuc/123:30:011
227797499481,22cyclictest26-21ksoftirqd/121:00:261
227797099488,6cyclictest141rcu_preempt23:24:560
227797099488,6cyclictest141rcu_preempt21:41:250
2277983994716,26cyclictest2389389-21gpgv00:25:033
227798399471,42cyclictest1753-21gnome-shell20:09:593
2277979994719,8cyclictest1753-21gnome-shell21:42:312
2277979994717,8cyclictest2197014-21PK-Backend22:21:072
2277979994716,10cyclictest34-21ksoftirqd/219:51:282
2277979994716,10cyclictest34-21ksoftirqd/200:03:202
2277979994715,8cyclictest468-21systemd-udevd23:50:362
2277979994713,11cyclictest2318415-21grep21:00:312
227797499479,12cyclictest26-21ksoftirqd/120:30:301
227797499471,12cyclictest26-21ksoftirqd/120:40:171
227797099474,18cyclictest13-21ksoftirqd/021:15:000
227797099471,20cyclictest13-21ksoftirqd/022:55:310
227797999468,18cyclictest1753-21gnome-shell21:19:082
227797999462,23cyclictest1753-21gnome-shell23:08:392
2277979994616,9cyclictest1753-21gnome-shell23:49:302
2277979994616,10cyclictest34-21ksoftirqd/223:32:072
227797999461,22cyclictest2344200-21cat22:15:232
2277979994612,12cyclictest2342406-21df22:10:152
2277979994610,16cyclictest1753-21gnome-shell23:26:182
227797499466,17cyclictest1753-21gnome-shell22:30:211
227797499464,19cyclictest26-21ksoftirqd/100:19:391
227797499463,21cyclictest26-21ksoftirqd/122:15:341
2277974994615,10cyclictest26-21ksoftirqd/100:10:001
227797499461,21cyclictest26-21ksoftirqd/119:15:121
227797099461,9cyclictest2307623-21chrt20:30:290
2277970994612,9cyclictest13-21ksoftirqd/022:25:170
23886522450,7sleep02197014-21PK-Backend00:20:260
22953502450,7sleep22197014-21PK-Backend19:55:252
227797999459,16cyclictest1753-21gnome-shell21:58:092
227797999458,16cyclictest1753-21gnome-shell20:33:192
2277979994518,9cyclictest1753-21gnome-shell00:35:192
2277979994518,6cyclictest1069-21runrttasks22:09:402
2277979994511,8cyclictest0-21swapper/220:15:132
2277979994510,16cyclictest1753-21gnome-shell19:26:382
2277979994510,15cyclictest1753-21gnome-shell23:57:522
2277979994510,15cyclictest1753-21gnome-shell19:40:112
227797499457,9cyclictest26-21ksoftirqd/122:50:251
227797499452,9cyclictest26-21ksoftirqd/123:10:021
2277974994513,10cyclictest26-21ksoftirqd/119:11:481
227797499451,16cyclictest2367060-21smartctl23:20:201
2277970994517,9cyclictest13-21ksoftirqd/020:45:440
162399450,9rtkit-daemon1753-21gnome-shell19:10:310
162399450,34rtkit-daemon2197014-21PK-Backend21:50:172
227797999447,18cyclictest1753-21gnome-shell22:00:072
2277974994421,11cyclictest2364339-21systemd-udevd23:11:421
2277974994417,11cyclictest26-21ksoftirqd/121:09:171
2277970994424,8cyclictest13-21ksoftirqd/019:50:270
23435882430,5sleep0161rcuc/022:15:020
227797999432,18cyclictest2288841-21munin-run19:40:012
227797999432,18cyclictest2288841-21munin-run19:40:012
2277979994317,7cyclictest2197014-21PK-Backend20:06:002
2277974994332,8cyclictest251rcuc/120:20:311
227797499431,19cyclictest1753-21gnome-shell20:51:231
227797499431,19cyclictest1753-21gnome-shell20:51:231
2277974994311,15cyclictest26-21ksoftirqd/122:08:161
2277970994332,8cyclictest13-21ksoftirqd/022:41:440
2277970994325,14cyclictest2291078-40apt-check19:45:190
227797099431,9cyclictest2386125-21taskset00:15:070
2277970994315,8cyclictest13-21ksoftirqd/020:40:270
227797999422,21cyclictest1753-21gnome-shell21:10:182
227797999422,14cyclictest2315124-40lsb_release20:51:232
227797999422,14cyclictest2315124-40lsb_release20:51:222
227797499427,5cyclictest141rcu_preempt21:25:031
227797499424,9cyclictest26-21ksoftirqd/121:55:511
2277974994217,11cyclictest141rcu_preempt21:54:541
227797499421,16cyclictest26-21ksoftirqd/100:30:261
227797499421,13cyclictest26-21ksoftirqd/123:35:361
227797499421,12cyclictest26-21ksoftirqd/121:15:311
227797499421,11cyclictest0-21swapper/120:25:141
227797099426,6cyclictest141rcu_preempt21:36:090
227797099425,22cyclictest13-21ksoftirqd/000:06:260
2277970994224,14cyclictest2298836-21ntpq20:05:300
2277970994224,14cyclictest13-21ksoftirqd/022:38:500
2277970994215,7cyclictest141rcu_preempt22:45:230
2277970994214,8cyclictest13-21ksoftirqd/000:00:130
2277970994211,15cyclictest13-21ksoftirqd/020:10:300
2277970994210,7cyclictest141rcu_preempt19:37:220
2277970994210,7cyclictest141rcu_preempt19:37:220
23207182410,5sleep0161rcuc/021:10:010
227797999411,9cyclictest0-21swapper/219:10:302
227797499418,16cyclictest26-21ksoftirqd/119:40:201
227797499412,10cyclictest0-21swapper/122:45:191
227797499411,26cyclictest26-21ksoftirqd/121:45:241
227797499411,20cyclictest26-21ksoftirqd/100:10:181
227797499411,19cyclictest1753-21gnome-shell20:36:151
227797499411,19cyclictest1753-21gnome-shell20:36:151
2277970994129,8cyclictest13-21ksoftirqd/022:15:140
2277970994117,9cyclictest13-21ksoftirqd/020:40:010
2277970994117,9cyclictest13-21ksoftirqd/020:40:010
2277970994110,10cyclictest13-21ksoftirqd/019:44:440
227797999402,18cyclictest2327620-40apt-check21:27:162
227797499407,10cyclictest26-21ksoftirqd/119:50:191
227797499402,11cyclictest1753-21gnome-shell19:20:421
2277974994014,9cyclictest26-21ksoftirqd/119:57:311
227797499401,15cyclictest26-21ksoftirqd/122:30:011
227797499401,13cyclictest26-21ksoftirqd/122:22:341
2277974994011,10cyclictest26-21ksoftirqd/120:46:141
227797099408,16cyclictest13-21ksoftirqd/022:01:060
2277970994029,8cyclictest13-21ksoftirqd/023:11:420
2277970994015,8cyclictest13-21ksoftirqd/000:35:340
2277970994013,9cyclictest13-21ksoftirqd/021:20:020
2277970994011,15cyclictest13-21ksoftirqd/019:17:300
23509272390,7chrt1753-21gnome-shell22:35:151
22882002390,5chrt0-21swapper/119:35:281
22882002390,5chrt0-21swapper/119:35:281
227797499391,12cyclictest26-21ksoftirqd/121:20:261
227797099398,17cyclictest2197014-21PK-Backend22:22:080
227797099397,15cyclictest2302988-21dump-pmu-power20:19:560
2277970993928,7cyclictest13-21ksoftirqd/019:28:360
227797099392,33cyclictest1753-21gnome-shell23:28:450
227797099392,20cyclictest13-21ksoftirqd/000:12:160
2277970993914,6cyclictest141rcu_preempt20:21:030
2277970993911,17cyclictest141rcu_preempt22:33:490
2277970993910,10cyclictest13-21ksoftirqd/019:22:330
2277979993820,14cyclictest2305374-21df20:25:162
2277979993814,10cyclictest2389392-21pool-org.gnome.00:25:012
2277970993823,11cyclictest1-21systemd21:55:010
2277970993822,6cyclictest13-21ksoftirqd/022:54:490
2277970993820,6cyclictest13-21ksoftirqd/023:16:520
2277970993817,16cyclictest2315003-21cat20:50:310
2277970993817,16cyclictest2315003-21cat20:50:310
227797099381,16cyclictest2375930-21cat23:45:230
2277979993720,12cyclictest438-20systemd-journal00:06:282
227797499376,16cyclictest26-21ksoftirqd/121:40:311
227797099377,7cyclictest141rcu_preempt21:20:070
2277970993721,5cyclictest13-21ksoftirqd/022:08:440
227797099372,13cyclictest1617-10pipewire-media-19:35:000
2277970993720,7cyclictest13-21ksoftirqd/023:09:050
2277970993720,7cyclictest13-21ksoftirqd/021:56:150
2277970993720,7cyclictest13-21ksoftirqd/021:30:110
2277970993710,14cyclictest13-21ksoftirqd/021:00:060
227797999361,31cyclictest1753-21gnome-shell21:48:332
227797499364,16cyclictest26-21ksoftirqd/122:00:271
2277974993619,14cyclictest2352652-21cstates22:40:171
2277970993621,5cyclictest13-21ksoftirqd/020:56:320
2277970993614,5cyclictest141rcu_preempt23:31:530
2277979993417,12cyclictest2310977-21awk20:40:272
227797999341,29cyclictest2197014-21PK-Backend20:35:232
227797999341,29cyclictest2197014-21PK-Backend20:35:222
23745502330,5sleep234-21ksoftirqd/223:40:322
2277979993316,13cyclictest2390973-21users00:25:332
227797999331,28cyclictest2197014-21PK-Backend23:15:112
227797999331,28cyclictest2197014-21PK-Backend20:45:052
227797999319,18cyclictest1753-21gnome-shell22:37:202
227797999311,4cyclictest2197014-21PK-Backend22:43:362
227797999311,15cyclictest2197014-21PK-Backend19:31:502
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional