You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 22:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jun 28, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
236292824920,9sleep32197014-21PK-Backend07:05:173
237724221010,6sleep3411rcuc/307:45:183
24181622980,8sleep32197014-21PK-Backend09:40:213
236388329066,15sleep10-21swapper/107:08:501
162399880,11rtkit-daemon2365797-21idleruntime-cro07:15:012
236376927842,28sleep20-21swapper/207:07:232
162399770,6rtkit-daemon2197014-21PK-Backend08:02:252
162399770,5rtkit-daemon0-21swapper/208:40:352
236411599702,48cyclictest1753-21gnome-shell08:15:153
162399690,59rtkit-daemon0-21swapper/007:07:080
236411599684,37cyclictest1753-21gnome-shell11:05:063
236411599684,37cyclictest1753-21gnome-shell11:05:053
236411599682,7cyclictest1753-21gnome-shell12:37:263
236411599662,7cyclictest2462154-21diskstats11:45:203
236411599662,39cyclictest2467738-40lsb_release12:00:243
236411599661,43cyclictest2477392-21turbostat12:30:013
236411599652,7cyclictest1753-21gnome-shell07:54:413
236411599652,40cyclictest1753-21gnome-shell08:04:213
236411599651,38cyclictest854-21avahi-daemon07:16:293
236411599642,8cyclictest1753-21gnome-shell12:23:563
236411599642,8cyclictest1753-21gnome-shell10:18:413
236411599642,8cyclictest1753-21gnome-shell08:13:503
236411599642,8cyclictest1753-21gnome-shell07:36:473
236411599632,8cyclictest1753-21gnome-shell09:56:073
236411599632,8cyclictest1753-21gnome-shell08:41:133
236411599632,7cyclictest1753-21gnome-shell11:50:253
2364115996325,18cyclictest42-21ksoftirqd/309:53:423
236411599622,8cyclictest1753-21gnome-shell08:05:473
236411599622,7cyclictest1753-21gnome-shell09:01:373
236411599622,11cyclictest1753-21gnome-shell08:35:503
162399620,9rtkit-daemon1622-21rtkit-daemon11:14:133
236411599612,8cyclictest1753-21gnome-shell09:45:583
236411599612,7cyclictest1753-21gnome-shell12:16:233
236411599612,7cyclictest1753-21gnome-shell09:24:423
236411599602,9cyclictest1753-21gnome-shell10:46:093
236411599602,8cyclictest1753-21gnome-shell10:42:303
236411599602,7cyclictest1753-21gnome-shell12:34:423
236411599602,7cyclictest1753-21gnome-shell11:15:113
236411599592,7cyclictest1753-21gnome-shell07:42:493
236411599581,11cyclictest860997-21snmpd07:58:483
236410799585,9cyclictest26-21ksoftirqd/109:05:011
162399570,8rtkit-daemon1622-21rtkit-daemon09:05:133
162399550,9rtkit-daemon1622-21rtkit-daemon11:00:392
162399550,9rtkit-daemon1622-21rtkit-daemon10:28:153
162399550,9rtkit-daemon1622-21rtkit-daemon10:07:372
162399550,8rtkit-daemon1622-21rtkit-daemon11:36:152
2364115995415,34cyclictest2391312-21basename08:25:233
2364115995414,34cyclictest2181-21gdbus09:10:033
2364107995425,24cyclictest438-20systemd-journal09:10:001
162399540,8rtkit-daemon1622-21rtkit-daemon09:15:202
162399540,8rtkit-daemon1622-21rtkit-daemon09:08:092
24129722530,8chrt2197014-21PK-Backend09:25:240
24089512530,5chrt0-21swapper/109:15:151
2364115995323,26cyclictest2466952-21rm12:00:023
162399530,9rtkit-daemon1622-21rtkit-daemon10:40:222
162399530,9rtkit-daemon1622-21rtkit-daemon09:36:483
162399530,9rtkit-daemon1622-21rtkit-daemon09:36:473
162399530,9rtkit-daemon1622-21rtkit-daemon09:30:083
162399530,8rtkit-daemon1622-21rtkit-daemon11:20:133
24638472520,5sleep2331rcuc/211:50:202
2364107995211,20cyclictest26-21ksoftirqd/111:25:361
2364107995210,22cyclictest26-21ksoftirqd/108:00:351
162399520,9rtkit-daemon1622-21rtkit-daemon10:03:402
23945942510,7sleep22197014-21PK-Backend08:35:202
2364115995123,24cyclictest2402755-21awk09:00:003
2364115995114,33cyclictest2461542-21latency_hist11:45:023
162399510,9rtkit-daemon1622-21rtkit-daemon09:56:432
162399510,8rtkit-daemon1622-21rtkit-daemon12:21:582
162399510,8rtkit-daemon1622-21rtkit-daemon11:04:143
162399510,8rtkit-daemon1622-21rtkit-daemon10:38:262
236411599502,9cyclictest1753-21gnome-shell11:27:563
236411599502,8cyclictest1753-21gnome-shell08:45:183
236411599502,44cyclictest1753-21gnome-shell07:24:063
236411599502,43cyclictest1753-21gnome-shell10:31:313
2364107995030,15cyclictest26-21ksoftirqd/107:55:431
2364107995029,5cyclictest141rcu_preempt09:43:311
2364105995024,14cyclictest141rcu_preempt07:37:080
2364105995023,8cyclictest13-21ksoftirqd/011:35:320
162399500,8rtkit-daemon1622-21rtkit-daemon11:31:232
162399500,8rtkit-daemon1622-21rtkit-daemon09:52:192
236411599492,8cyclictest1753-21gnome-shell07:35:013
236411599492,8cyclictest1753-21gnome-shell07:35:013
236411599492,43cyclictest2409780-21grep09:15:323
2364107994925,8cyclictest26-21ksoftirqd/108:15:421
2364107994924,8cyclictest26-21ksoftirqd/111:44:521
2364107994923,9cyclictest26-21ksoftirqd/112:37:351
2364107994913,8cyclictest26-21ksoftirqd/107:25:021
2364105994919,16cyclictest13-21ksoftirqd/011:01:110
24662222480,5sleep22197014-21PK-Backend11:55:312
236411599482,9cyclictest1753-21gnome-shell11:00:013
236411599482,8cyclictest1753-21gnome-shell10:08:043
236411599482,7cyclictest1753-21gnome-shell10:36:123
236411599482,42cyclictest1753-21gnome-shell12:07:573
236411599482,42cyclictest1753-21gnome-shell12:07:573
236411599482,41cyclictest1753-21gnome-shell08:30:043
236411599482,18cyclictest1800-21upowerd10:21:473
2364115994819,25cyclictest1753-21gnome-shell10:13:403
236410799488,23cyclictest26-21ksoftirqd/110:14:591
236410799484,21cyclictest141rcu_preempt11:45:151
2364107994826,5cyclictest141rcu_preempt09:23:281
2364107994817,10cyclictest26-21ksoftirqd/107:35:241
236411599472,7cyclictest1753-21gnome-shell07:13:113
236411599472,41cyclictest2962348-21rs:main3
2364115994713,30cyclictest2442423-21dpkg10:50:173
2364112994719,23cyclictest438-20systemd-journal12:19:552
2364107994714,7cyclictest26-21ksoftirqd/107:45:191
236410799471,13cyclictest26-21ksoftirqd/109:25:291
236410799471,11cyclictest2476810-21/usr/sbin/munin12:25:341
2364107994710,10cyclictest26-21ksoftirqd/108:30:331
2364105994718,7cyclictest13-21ksoftirqd/012:27:450
236411599469,32cyclictest1753-21gnome-shell08:20:043
236411599462,7cyclictest1753-21gnome-shell07:25:283
2364115994619,23cyclictest1069-21runrttasks09:25:163
236410799466,11cyclictest26-21ksoftirqd/108:55:261
236410799464,18cyclictest26-21ksoftirqd/110:50:001
2364107994620,7cyclictest26-21ksoftirqd/110:15:191
2364107994618,7cyclictest26-21ksoftirqd/112:21:171
2364107994613,10cyclictest26-21ksoftirqd/112:00:441
236410599469,19cyclictest13-21ksoftirqd/010:10:120
2364105994622,19cyclictest2366457-21hddtemp_smartct07:15:070
2364105994616,14cyclictest13-21ksoftirqd/012:35:010
2364105994615,17cyclictest13-21ksoftirqd/011:22:080
236411599459,32cyclictest1753-21gnome-shell11:30:253
2364115994523,18cyclictest1753-21gnome-shell12:14:343
236411299455,36cyclictest1622-21rtkit-daemon10:56:572
2364107994535,7cyclictest251rcuc/109:12:211
2364107994518,8cyclictest26-21ksoftirqd/109:45:161
2364107994516,7cyclictest26-21ksoftirqd/108:44:151
2364107994516,6cyclictest141rcu_preempt07:50:201
2364107994512,10cyclictest26-21ksoftirqd/109:55:331
2364107994510,10cyclictest26-21ksoftirqd/108:35:331
236410599459,12cyclictest141rcu_preempt11:16:270
2364105994515,17cyclictest13-21ksoftirqd/010:48:500
2364105994511,17cyclictest13-21ksoftirqd/007:41:400
236410799444,21cyclictest26-21ksoftirqd/111:11:051
2364107994415,9cyclictest26-21ksoftirqd/111:05:261
2364107994415,9cyclictest26-21ksoftirqd/111:05:261
2364107994414,6cyclictest141rcu_preempt08:05:191
2364107994412,6cyclictest141rcu_preempt11:57:091
2364107994412,10cyclictest26-21ksoftirqd/110:05:321
236410799441,17cyclictest26-21ksoftirqd/111:00:331
2364107994410,10cyclictest26-21ksoftirqd/110:20:251
2364105994424,5cyclictest141rcu_preempt08:36:350
2364105994422,6cyclictest141rcu_preempt08:57:570
2364105994419,9cyclictest13-21ksoftirqd/009:05:010
2364105994418,6cyclictest141rcu_preempt08:34:550
2364105994417,15cyclictest141rcu_preempt09:31:150
236411599438,31cyclictest2400437-21switchtime08:50:323
2364112994322,16cyclictest34-21ksoftirqd/207:40:152
236410799438,9cyclictest26-21ksoftirqd/108:20:171
236410799433,13cyclictest26-21ksoftirqd/110:40:321
2364107994323,5cyclictest141rcu_preempt10:02:031
2364107994322,17cyclictest2445877-21grep11:00:021
236410799432,12cyclictest26-21ksoftirqd/108:25:251
2364107994310,9cyclictest26-21ksoftirqd/111:31:041
2364107994310,7cyclictest26-21ksoftirqd/110:38:471
2364105994326,12cyclictest2440399-21cron10:45:000
2364105994324,14cyclictest438-20systemd-journal07:30:410
2364105994324,14cyclictest438-20systemd-journal07:30:410
2364105994321,9cyclictest13-21ksoftirqd/012:35:290
236410599432,19cyclictest13-21ksoftirqd/011:35:000
2364105994321,8cyclictest13-21ksoftirqd/011:10:400
2364105994321,8cyclictest13-21ksoftirqd/011:05:240
2364105994321,8cyclictest13-21ksoftirqd/011:05:230
2364105994321,5cyclictest141rcu_preempt12:15:180
162399430,13rtkit-daemon141rcu_preempt11:10:242
2364115994214,24cyclictest2425841-21vmstat10:00:373
236411299421,13cyclictest2378711-21/usr/sbin/munin07:50:172
236410799426,9cyclictest26-21ksoftirqd/112:35:001
236410799426,9cyclictest26-21ksoftirqd/111:52:051
236410799426,14cyclictest2360272-21kworker/1:107:12:531
236410799424,17cyclictest26-21ksoftirqd/111:36:011
2364107994215,7cyclictest26-21ksoftirqd/107:34:031
2364107994215,7cyclictest26-21ksoftirqd/107:34:031
2364107994215,6cyclictest141rcu_preempt07:29:431
2364105994231,8cyclictest161rcuc/010:00:270
2364105994224,14cyclictest2379469-21ntpq07:50:310
2364105994220,6cyclictest141rcu_preempt11:50:100
2364105994217,9cyclictest13-21ksoftirqd/012:10:010
2364105994217,9cyclictest13-21ksoftirqd/012:10:010
2364105994215,6cyclictest141rcu_preempt12:00:280
2364105994213,8cyclictest13-21ksoftirqd/009:22:130
2364105994210,7cyclictest13-21ksoftirqd/008:13:350
162399420,35rtkit-daemon2197014-21PK-Backend10:45:222
162399420,30rtkit-daemon2197014-21PK-Backend07:19:552
236411299413,11cyclictest141rcu_preempt09:25:332
236411299412,19cyclictest1753-21gnome-shell08:57:582
236410799417,10cyclictest26-21ksoftirqd/109:30:261
2364107994125,12cyclictest2367990-21cron07:20:001
2364107994114,15cyclictest26-21ksoftirqd/112:16:151
236410799411,16cyclictest1938-21gsd-housekeepin11:17:251
236410799411,12cyclictest1753-21gnome-shell09:51:361
236410799411,11cyclictest2376011-21runrttasks07:40:281
236410799411,10cyclictest26-21ksoftirqd/111:20:341
236410599418,6cyclictest141rcu_preempt09:40:010
236410599418,6cyclictest141rcu_preempt09:40:000
236410599414,15cyclictest13-21ksoftirqd/011:48:200
2364105994130,7cyclictest13-21ksoftirqd/009:42:410
2364105994123,7cyclictest2406339-21systemd-udevd09:07:590
2364105994114,6cyclictest141rcu_preempt09:18:150
2364105994112,6cyclictest141rcu_preempt08:08:330
2364105994111,8cyclictest13-21ksoftirqd/011:55:170
236410599411,15cyclictest2197014-21PK-Backend10:31:330
2364105994110,6cyclictest141rcu_preempt07:46:070
2364105994110,5cyclictest141rcu_preempt10:50:200
23746692401,17sleep234-21ksoftirqd/207:36:482
2364112994023,13cyclictest34-21ksoftirqd/209:30:552
2364112994020,15cyclictest34-21ksoftirqd/211:45:012
236411299401,34cyclictest1753-21gnome-shell10:10:162
2364107994014,7cyclictest26-21ksoftirqd/108:12:331
2364107994013,5cyclictest141rcu_preempt10:30:571
2364107994012,8cyclictest26-21ksoftirqd/108:46:551
236410599409,6cyclictest141rcu_preempt10:35:150
236410599407,6cyclictest141rcu_preempt10:17:210
236410599405,17cyclictest13-21ksoftirqd/010:20:470
236410599405,16cyclictest13-21ksoftirqd/008:44:560
2364105994025,11cyclictest2460099-21sort11:40:140
2364105994024,12cyclictest2388767-21sshd08:18:420
2364105994020,5cyclictest141rcu_preempt09:10:140
2364105994020,5cyclictest141rcu_preempt08:00:350
2364105994020,16cyclictest855-21dbus-daemon07:55:440
2364105994018,11cyclictest2421396-21cat09:50:160
2364105994014,9cyclictest13-21ksoftirqd/009:57:050
236410599401,19cyclictest1753-21gnome-shell07:21:180
2364105994010,17cyclictest13-21ksoftirqd/007:12:350
2364112993922,13cyclictest34-21ksoftirqd/211:50:022
2364112993922,12cyclictest438-20systemd-journal09:24:592
2364112993921,14cyclictest34-21ksoftirqd/207:20:192
236411299390,35cyclictest1622-21rtkit-daemon12:34:292
236410799393,8cyclictest1753-21gnome-shell10:25:191
2364107993911,6cyclictest141rcu_preempt12:08:401
2364107993911,6cyclictest141rcu_preempt12:08:391
2364107993911,6cyclictest141rcu_preempt08:53:181
236410799391,14cyclictest2417284-21unin-run09:40:011
236410799391,14cyclictest2417284-21unin-run09:40:011
2364107993910,6cyclictest141rcu_preempt10:53:151
236410599399,6cyclictest141rcu_preempt08:51:430
236410599398,6cyclictest141rcu_preempt10:28:260
236410599398,6cyclictest141rcu_preempt08:20:170
236410599395,16cyclictest13-21ksoftirqd/007:25:510
236410599392,33cyclictest1753-21gnome-shell10:08:170
236410599391,16cyclictest2456308-21munin-run11:30:000
236410599391,10cyclictest2397820-21apt08:45:150
236411299389,16cyclictest1753-21gnome-shell08:13:222
2364112993827,7cyclictest2468740-21grep12:04:592
236411299382,32cyclictest1753-21gnome-shell12:27:142
236411299382,31cyclictest2418660-21python309:40:322
2364112993823,11cyclictest2407852-21open_inodes09:10:292
2364112993820,6cyclictest34-21ksoftirqd/208:15:352
2364112993820,13cyclictest34-21ksoftirqd/207:25:212
2364112993819,15cyclictest34-21ksoftirqd/208:05:302
2364112993814,5cyclictest141rcu_preempt09:45:262
2364112993814,11cyclictest141rcu_preempt10:23:592
236411299381,11cyclictest141rcu_preempt11:05:252
236411299381,11cyclictest141rcu_preempt11:05:252
236410799388,11cyclictest141rcu_preempt12:10:281
236410599388,6cyclictest141rcu_preempt12:20:410
236410599386,7cyclictest141rcu_preempt12:10:180
162399380,27rtkit-daemon0-21swapper/208:29:082
236411299371,5cyclictest141rcu_preempt12:15:002
2364112993714,6cyclictest141rcu_preempt07:57:022
236410599375,17cyclictest13-21ksoftirqd/009:49:220
2364105993721,12cyclictest13-21ksoftirqd/010:59:180
236410599371,32cyclictest161rcuc/008:27:160
236411299366,7cyclictest2197014-21PK-Backend08:32:162
2364112993620,13cyclictest34-21ksoftirqd/210:19:342
2364112993618,14cyclictest34-21ksoftirqd/209:37:542
2364112993618,14cyclictest34-21ksoftirqd/209:37:542
2364112993525,5cyclictest34-21ksoftirqd/207:47:052
2364112993521,11cyclictest34-21ksoftirqd/207:33:472
2364112993521,11cyclictest34-21ksoftirqd/207:33:472
2364112993520,11cyclictest2469818-21cut12:05:312
2364112993520,11cyclictest2469818-21cut12:05:312
236411299351,9cyclictest0-21swapper/212:38:152
236411299341,9cyclictest2197014-21PK-Backend10:29:272
236411299341,8cyclictest0-21swapper/209:03:142
2364112993416,6cyclictest34-21ksoftirqd/211:15:352
2364112993318,11cyclictest2398227-21idleruntime08:45:232
236411299331,28cyclictest2197014-21PK-Backend11:25:382
2364112993312,6cyclictest141rcu_preempt10:55:012
2364112993311,5cyclictest141rcu_preempt10:30:312
236411299321,28cyclictest2197014-21PK-Backend08:20:302
236411299317,6cyclictest141rcu_preempt08:50:542
236411299313,11cyclictest34-21ksoftirqd/211:21:072
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional