You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-22 - 13:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Wed Jan 22, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
332249922340,9sleep2341ksoftirqd/219:09:182
332580121250,6sleep11753-21gnome-shell19:15:241
33705662910,5sleep30-21swapper/321:30:013
34136192830,8sleep02197014-21PK-Backend23:55:130
162399800,66rtkit-daemon0-21swapper/319:08:083
33376362750,4sleep00-21swapper/019:50:170
332248327459,9sleep10-21swapper/119:09:061
332182427144,16sleep00-21swapper/019:05:220
162399710,5rtkit-daemon0-21swapper/320:17:363
332269599621,36cyclictest3329761-21sshd19:26:311
332269599621,35cyclictest3365707-21cat21:10:341
332269599612,37cyclictest0-21swapper/119:42:371
332269599581,33cyclictest3337086-21taskset19:47:411
332269599581,30cyclictest3333700-21seq19:38:421
332269599581,30cyclictest3333700-21seq19:38:421
332269599571,50cyclictest3401920-21grep23:15:131
332269599571,34cyclictest0-21swapper/120:37:071
332269599562,29cyclictest1753-21gnome-shell20:34:031
162399560,33rtkit-daemon0-21swapper/322:56:483
33610562550,4sleep0181irq_work/021:00:010
3322697995518,33cyclictest3414961-40mandb00:00:022
332269599552,49cyclictest1753-21gnome-shell00:15:301
162399550,9rtkit-daemon1622-21rtkit-daemon19:28:383
33709702540,6sleep00-21swapper/021:30:100
3322695995425,25cyclictest3367959-21cron21:20:001
332269599541,47cyclictest3394853-21cat22:50:191
3322695995320,29cyclictest3422809-21sh00:25:011
3322695995319,29cyclictest438-20systemd-journal20:18:201
332269599531,36cyclictest3386886-21taskset22:21:181
332269599521,46cyclictest3415579-21df00:00:141
332269799511,8cyclictest1753-21gnome-shell20:45:122
3322697995118,28cyclictest3369050-21cut21:20:262
3322697995118,28cyclictest3341887-21python320:00:292
3322697995117,30cyclictest3383768-21ls22:15:012
3322697995115,32cyclictest884-21gdbus22:45:022
3322697995112,34cyclictest3367983-21/usr/sbin/munin21:20:002
3322695995117,29cyclictest855-21dbus-daemon23:10:011
332269599511,46cyclictest3403819-21cstates23:20:121
33520602500,7sleep32197014-21PK-Backend20:30:213
3322697995021,25cyclictest3323707-21processes19:10:282
3322697995019,26cyclictest3421354-21apt00:20:132
3322697995018,29cyclictest3395729-21pool-org.gnome.22:55:002
332269599502,43cyclictest1753-21gnome-shell23:30:211
332269599502,43cyclictest1753-21gnome-shell23:30:211
3322695995019,26cyclictest3400118-21ls23:10:101
3322695995019,26cyclictest3400118-21ls23:10:091
3322695995018,26cyclictest438-20systemd-journal23:40:211
3322695995016,29cyclictest3383281-21cat22:10:411
332269599501,44cyclictest3342648-21cat20:05:031
332269599501,44cyclictest3327557-21netstat19:20:231
332269599501,33cyclictest3351330-21wc20:30:021
162399500,5rtkit-daemon0-21swapper/321:40:363
332269799499,17cyclictest1753-21gnome-shell19:26:542
3322697994919,25cyclictest1069-21runrttasks21:12:102
3322697994917,28cyclictest1069-21runrttasks22:01:562
3322697994915,29cyclictest3419688-21cpuspeed_turbos00:15:122
3322697994914,31cyclictest3330989-21needreboot19:30:272
3322697994912,32cyclictest3423959-21cat00:30:092
3322695994919,26cyclictest3376022-21sshd21:48:211
332269599491,44cyclictest3392589-21grep22:40:231
332269599491,43cyclictest3373531-21chrt21:36:451
332269599491,43cyclictest3368347-21grep21:20:121
332269599491,43cyclictest3337953-21awk19:50:221
332269599491,42cyclictest438-20systemd-journal23:27:211
3322697994824,20cyclictest1753-21gnome-shell22:38:232
3322697994818,26cyclictest3411991-21ls23:50:112
3322697994818,26cyclictest3411991-21ls23:50:102
3322697994813,31cyclictest3349838-21cpuspeed_turbos20:25:132
3322697994813,30cyclictest3378331-21cpuspeed_turbos21:55:142
3322697994811,32cyclictest2962348-21rs:main2
3322697994810,17cyclictest1753-21gnome-shell20:35:132
332269599482,42cyclictest3242253-21nautilus00:00:011
332269599482,41cyclictest3384400-21smartctl22:15:201
332269599481,42cyclictest3374046-21/usr/sbin/munin21:40:221
332269599481,42cyclictest3340175-21switchtime19:55:311
332269599481,30cyclictest3359449-21latency_hist20:55:021
3322695994811,32cyclictest0-21swapper/100:30:091
3322697994719,24cyclictest3344691-21cstates20:10:172
3322697994718,25cyclictest3389552-21apt-get22:35:002
3322697994718,25cyclictest1069-21runrttasks23:29:192
3322697994717,26cyclictest34-21ksoftirqd/219:46:162
3322697994715,27cyclictest3361311-21apt21:00:122
3322697994715,27cyclictest3361311-21apt21:00:112
3322697994714,29cyclictest3348459-21cut20:20:242
332269799471,20cyclictest3356892-21apt-config20:45:032
332269599478,21cyclictest1753-21gnome-shell23:47:111
332269599472,41cyclictest3393954-21cron22:50:011
332269599472,40cyclictest3370408-21latency_hist21:30:011
332269599472,40cyclictest1753-21gnome-shell22:55:111
332269599471,41cyclictest3362167-21/usr/sbin/munin21:00:261
332269599471,41cyclictest3362167-21/usr/sbin/munin21:00:261
332269599471,30cyclictest3376802-21df21:50:111
3322695994712,30cyclictest3381516-21wc22:05:021
332269799469,18cyclictest1753-21gnome-shell21:08:592
332269799469,18cyclictest1753-21gnome-shell19:56:262
332269799469,18cyclictest1753-21gnome-shell19:40:382
332269799469,18cyclictest1753-21gnome-shell00:12:072
332269799469,17cyclictest1753-21gnome-shell23:45:042
332269799469,16cyclictest1753-21gnome-shell22:20:432
3322697994620,21cyclictest3408044-21idleruntime23:35:182
3322697994619,23cyclictest1069-21runrttasks21:51:402
3322697994615,27cyclictest3352027-40lsb_release20:30:192
3322697994615,26cyclictest1069-21runrttasks23:17:132
332269799461,25cyclictest1753-21gnome-shell22:05:032
332269799461,25cyclictest1753-21gnome-shell22:05:032
3322697994611,31cyclictest3397831-21cpuspeed_turbos23:00:132
3322697994610,18cyclictest1753-21gnome-shell21:31:392
3322697994610,17cyclictest1753-21gnome-shell23:23:262
3322697994610,16cyclictest1753-21gnome-shell23:40:542
3322697994610,16cyclictest1753-21gnome-shell23:09:382
332269599462,28cyclictest3390843-21ntp_states22:35:241
3322695994616,25cyclictest3363143-21cpuspeed_turbos21:05:141
3322695994615,26cyclictest3378442-21cat21:55:141
332269599461,41cyclictest3418522-21sed00:10:251
332269599461,40cyclictest3356621-21chrt20:44:551
332269599461,40cyclictest3330584-21cstates19:30:181
3322700994526,14cyclictest3388465-21diskstats22:30:133
3322700994515,25cyclictest42-21ksoftirqd/321:55:313
332269799457,17cyclictest1753-21gnome-shell23:13:382
332269799457,17cyclictest1753-21gnome-shell23:13:382
332269799457,17cyclictest1753-21gnome-shell21:40:142
332269799455,17cyclictest1753-21gnome-shell19:21:172
3322697994518,23cyclictest1069-21runrttasks19:38:212
3322697994518,23cyclictest1069-21runrttasks19:38:212
332269799451,40cyclictest1753-21gnome-shell21:26:472
3322697994513,28cyclictest3360835-21taskset20:57:302
3322697994512,28cyclictest1753-21gnome-shell20:50:172
3322697994510,17cyclictest1753-21gnome-shell00:27:382
3322697994510,16cyclictest1753-21gnome-shell19:15:362
3322697994510,16cyclictest1753-21gnome-shell00:00:312
332269599453,37cyclictest3345467-21timerandwakeup20:10:341
332269599452,28cyclictest3417090-21sshd00:07:211
3322695994516,25cyclictest3361060-21basename21:00:011
3322695994515,26cyclictest3343352-21missed_timers20:05:221
332269599451,40cyclictest1753-21gnome-shell21:35:011
332269599451,29cyclictest1753-21gnome-shell22:28:201
332269599451,29cyclictest1753-21gnome-shell22:28:191
34201412440,5sleep0161rcuc/000:15:210
33447952440,5sleep00-21swapper/020:10:180
3322700994426,14cyclictest42-21ksoftirqd/300:00:243
3322700994420,11cyclictest42-21ksoftirqd/323:13:423
3322700994420,11cyclictest42-21ksoftirqd/323:13:423
332269799449,15cyclictest1753-21gnome-shell22:46:032
332269799447,17cyclictest1753-21gnome-shell22:59:572
332269799446,15cyclictest1753-21gnome-shell21:39:362
332269599444,35cyclictest3388556-21cat22:30:151
3322695994420,20cyclictest1069-21runrttasks00:25:151
332269599441,39cyclictest3349268-21taskset20:23:261
3322695994412,28cyclictest3409067-21seq23:39:551
3322691994428,11cyclictest3331789-21apt-get19:35:000
162399440,27rtkit-daemon42-21ksoftirqd/320:03:503
332269799437,13cyclictest3406094-21cat23:30:102
332269799437,13cyclictest3406094-21cat23:30:102
332269799436,17cyclictest1753-21gnome-shell21:48:122
3322695994317,22cyclictest3323443-21latency19:10:241
162399430,5rtkit-daemon0-21swapper/320:20:323
162399430,31rtkit-daemon0-21swapper/019:24:250
3322700994213,24cyclictest42-21ksoftirqd/323:40:213
332270099421,23cyclictest1753-21gnome-shell22:38:143
332269799427,11cyclictest3346508-21cat20:15:152
332269599421,37cyclictest3413187-21turbostat23:55:011
332269599421,37cyclictest3413187-21turbostat23:55:001
332269599421,36cyclictest3398665-21ls23:00:291
332269599421,30cyclictest3357190-40apt-check20:45:241
3322691994224,13cyclictest438-20systemd-journal21:19:560
332269199421,16cyclictest2197014-21PK-Backend20:39:410
3322700994123,14cyclictest42-21ksoftirqd/300:26:293
332269599411,36cyclictest3426424-21sed00:35:301
332269599411,35cyclictest3381811-21taskset22:07:521
332269599411,35cyclictest3381811-21taskset22:07:511
332269199412,34cyclictest1753-21gnome-shell22:27:250
332269199412,34cyclictest1753-21gnome-shell22:27:250
162399410,7rtkit-daemon3358430-21memory20:50:183
3322700994025,11cyclictest42-21ksoftirqd/320:40:293
332270099401,8cyclictest2197014-21PK-Backend20:36:163
3322697994014,22cyclictest2197014-21PK-Backend00:07:102
332269799401,12cyclictest3343698-21find20:05:292
3322691994024,12cyclictest3401307-21sshd23:12:210
3322691994024,12cyclictest3401307-21sshd23:12:210
332269199402,34cyclictest1753-21gnome-shell20:16:350
332269199401,14cyclictest1753-21gnome-shell00:09:440
162399400,27rtkit-daemon0-21swapper/321:34:103
3322700993924,11cyclictest3340778-21taskset20:00:013
3322700993924,11cyclictest3324345-21cron19:15:003
3322700993923,12cyclictest860997-21snmpd23:56:243
3322700993920,14cyclictest42-21ksoftirqd/319:40:223
3322700993920,14cyclictest3242253-21nautilus00:15:013
332270099391,15cyclictest33257982head19:15:253
3322697993915,20cyclictest311irq_work/219:50:062
332269199397,6cyclictest141rcu_preempt00:11:560
3322691993922,13cyclictest3394984-21sshd22:50:210
3322691993922,12cyclictest3425409-21date00:35:030
3322691993921,13cyclictest438-20systemd-journal20:35:010
3322691993921,13cyclictest3340037-21ntp_states19:55:260
162399390,27rtkit-daemon0-21swapper/319:50:433
332270099387,28cyclictest2197014-21PK-Backend23:08:553
3322700993826,8cyclictest1753-21gnome-shell22:00:563
332270099382,32cyclictest1753-21gnome-shell19:38:393
332270099382,32cyclictest1753-21gnome-shell19:38:393
3322700993822,5cyclictest42-21ksoftirqd/300:25:003
3322700993821,12cyclictest3376483-21awk21:50:093
3322700993820,14cyclictest3367216-21ntp_kernel_err21:15:253
3322700993820,13cyclictest438-20systemd-journal20:45:273
3322700993816,18cyclictest3382962-21netstat22:10:193
332269799381,11cyclictest3385401-21chrt22:19:272
332269199389,24cyclictest1753-21gnome-shell23:36:530
332269199386,11cyclictest13-21ksoftirqd/021:41:020
3322691993827,7cyclictest1069-21runrttasks23:07:350
3322691993820,13cyclictest3403721-21cut23:20:110
3322691993819,14cyclictest468-21systemd-udevd19:46:490
3322691993816,18cyclictest1753-21gnome-shell21:10:090
3322691993815,18cyclictest855-21dbus-daemon22:50:020
332269199381,13cyclictest2197014-21PK-Backend21:58:410
162399380,26rtkit-daemon0-21swapper/321:24:103
162399380,26rtkit-daemon0-21swapper/320:29:563
3322700993726,7cyclictest3359636-21cpuspeed20:55:113
3322700993723,10cyclictest3411442-21cron23:50:003
3322700993721,11cyclictest438-20systemd-journal21:36:213
3322700993720,6cyclictest42-21ksoftirqd/323:30:093
3322700993720,6cyclictest42-21ksoftirqd/323:30:093
3322700993720,13cyclictest1753-21gnome-shell21:10:593
3322700993719,6cyclictest42-21ksoftirqd/323:18:493
332270099371,32cyclictest3343073-21cstates20:05:153
332270099371,17cyclictest1753-21gnome-shell22:43:243
332269199375,11cyclictest13-21ksoftirqd/020:21:440
332269199375,11cyclictest13-21ksoftirqd/019:10:090
332269199372,5cyclictest0-21swapper/022:06:290
332269199372,5cyclictest0-21swapper/022:06:280
3322691993725,9cyclictest855-21dbus-daemon20:50:000
332269199372,31cyclictest1753-21gnome-shell20:01:530
332269199372,30cyclictest1753-21gnome-shell00:04:290
3322691993722,11cyclictest3344219-21/usr/sbin/munin20:10:010
3322700993623,9cyclictest1753-21gnome-shell22:51:523
3322700993620,12cyclictest3409076-21cron23:39:593
3322700993620,11cyclictest1617-10pipewire-media-20:11:413
3322700993619,6cyclictest42-21ksoftirqd/323:54:223
3322700993619,6cyclictest42-21ksoftirqd/323:54:223
3322700993618,14cyclictest1753-21gnome-shell00:39:073
332270099361,30cyclictest1753-21gnome-shell19:30:273
3322700993612,20cyclictest391irq_work/321:05:313
332269199368,15cyclictest1069-21runrttasks22:43:190
332269199367,7cyclictest141rcu_preempt00:25:320
3322691993625,7cyclictest1069-21runrttasks22:38:090
332269199362,30cyclictest0-21swapper/023:43:190
332269199362,30cyclictest0-21swapper/019:16:380
3322691993620,12cyclictest3425749-21expr00:35:140
3322691993620,11cyclictest3363757-21ntp_kernel_err21:05:240
3322691993617,14cyclictest438-20systemd-journal21:29:210
332269199361,31cyclictest1753-21gnome-shell21:45:440
3322700993524,7cyclictest3385881-21cpuspeed_turbos22:20:133
3322700993520,11cyclictest3336055-21expr19:45:153
3322700993519,11cyclictest3404492-21expr23:20:283
3322700993517,14cyclictest3362216-21sed21:00:293
3322700993517,14cyclictest3362216-21sed21:00:283
3322700993517,13cyclictest438-20systemd-journal00:10:013
3322700993517,13cyclictest3419770-21cstates00:15:133
3322691993525,7cyclictest1069-21runrttasks21:53:210
332269199352,29cyclictest0-21swapper/023:45:480
332269199352,28cyclictest1753-21gnome-shell21:01:010
332269199352,28cyclictest1753-21gnome-shell21:01:000
332269199352,28cyclictest1753-21gnome-shell20:26:490
332269199352,11cyclictest13-21ksoftirqd/023:53:310
332269199352,11cyclictest13-21ksoftirqd/023:53:310
3322691993520,11cyclictest3388832-21ntp_states22:30:220
3322691993520,10cyclictest3395967-21apt22:55:110
3322691993518,13cyclictest3329143-21sed19:25:210
3322691993516,13cyclictest3402546-21ntpq23:15:260
332269199351,29cyclictest2197014-21PK-Backend00:20:170
162399350,6rtkit-daemon0-21swapper/019:37:340
162399350,6rtkit-daemon0-21swapper/019:37:330
162399350,24rtkit-daemon0-21swapper/322:15:333
3322700993423,7cyclictest3423877-21tr00:30:103
3322700993420,10cyclictest3405514-21seq23:28:443
3322700993417,13cyclictest3397779-21cat23:00:133
3322700993416,6cyclictest42-21ksoftirqd/319:21:023
332269799341,8cyclictest33874392sleep222:25:152
332269799341,8cyclictest33874392sleep222:25:152
3322691993425,5cyclictest2197014-21PK-Backend22:02:520
3322691993419,11cyclictest855-21dbus-daemon23:01:560
3322691993419,11cyclictest3386541-21ps22:20:320
3322691993419,11cyclictest3384233-21cat22:15:140
332270099332,26cyclictest1753-21gnome-shell22:48:463
3322700993316,12cyclictest438-20systemd-journal22:09:213
3322700993316,12cyclictest438-20systemd-journal22:09:213
332270099331,12cyclictest141rcu_preempt21:50:023
332269199335,5cyclictest141rcu_preempt22:10:190
332269199332,8cyclictest1753-21gnome-shell20:53:190
332269199332,7cyclictest2197014-21PK-Backend21:21:100
332269199332,27cyclictest1753-21gnome-shell21:36:060
3322691993322,6cyclictest13-21ksoftirqd/023:31:180
3322691993322,6cyclictest13-21ksoftirqd/023:31:180
3322700993218,11cyclictest3387848-21awk22:30:003
3322700993218,11cyclictest3387848-21awk22:30:003
3322691993215,6cyclictest13-21ksoftirqd/019:40:320
3322691993210,18cyclictest1753-21gnome-shell23:25:030
332269199311,4cyclictest2197014-21PK-Backend20:41:420
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional