You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 01:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 02, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
351531624520,9sleep21753-21gnome-shell07:05:232
351556822520,9sleep10-21swapper/107:05:261
35745602820,7sleep21753-21gnome-shell09:55:122
35849442810,4sleep20-21swapper/210:25:142
35849442810,4sleep20-21swapper/210:25:142
162399760,6rtkit-daemon0-21swapper/111:16:301
3516098997523,32cyclictest854-21avahi-daemon08:14:253
351569227439,9sleep00-21swapper/007:06:310
162399740,6rtkit-daemon0-21swapper/110:22:491
162399740,5rtkit-daemon0-21swapper/110:34:581
162399740,5rtkit-daemon0-21swapper/110:28:231
162399740,5rtkit-daemon0-21swapper/110:28:231
162399720,5rtkit-daemon0-21swapper/110:16:521
162399700,8rtkit-daemon1622-21rtkit-daemon09:30:423
3516096996855,9cyclictest34-21ksoftirqd/209:25:012
3516096996855,9cyclictest34-21ksoftirqd/209:25:012
3516092996813,34cyclictest26-21ksoftirqd/107:55:021
351568026747,14sleep30-21swapper/307:06:223
162399650,8rtkit-daemon1622-21rtkit-daemon08:34:342
3516092996226,19cyclictest26-21ksoftirqd/107:25:381
35959082610,6sleep00-21swapper/010:55:150
351609699610,57cyclictest1622-21rtkit-daemon08:24:562
351609699610,56cyclictest1622-21rtkit-daemon09:40:452
162399600,9rtkit-daemon1622-21rtkit-daemon09:27:283
351609699590,55cyclictest1622-21rtkit-daemon07:38:092
162399590,8rtkit-daemon1622-21rtkit-daemon09:25:532
162399590,8rtkit-daemon1622-21rtkit-daemon08:41:102
351609699570,54cyclictest1622-21rtkit-daemon10:02:382
35684272560,9chrt0-21swapper/009:35:300
351609899569,42cyclictest3619490-21/usr/sbin/munin12:00:233
351609699560,52cyclictest1622-21rtkit-daemon07:10:572
35542132550,14sleep126-21ksoftirqd/108:55:221
351609699550,51cyclictest1622-21rtkit-daemon11:22:352
3516092995542,10cyclictest26-21ksoftirqd/110:48:151
351609299551,50cyclictest251rcuc/109:25:011
351609299551,50cyclictest251rcuc/109:25:011
351609299551,31cyclictest3604267-21/usr/sbin/munin11:20:111
3516092995512,18cyclictest26-21ksoftirqd/112:05:021
351608999551,5cyclictest141rcu_preempt07:34:350
3516098995424,25cyclictest904-21systemd-logind07:25:383
3516098995424,25cyclictest438-20systemd-journal09:01:403
351609299545,20cyclictest26-21ksoftirqd/107:30:281
3516092995426,9cyclictest26-21ksoftirqd/111:40:021
3516092995426,9cyclictest26-21ksoftirqd/111:40:021
351609299541,29cyclictest26-21ksoftirqd/111:10:001
162399540,9rtkit-daemon1622-21rtkit-daemon08:40:263
162399540,8rtkit-daemon1622-21rtkit-daemon07:17:312
36130062530,6sleep02197014-21PK-Backend11:45:020
351609899532,10cyclictest1753-21gnome-shell10:44:063
3516098995319,28cyclictest438-20systemd-journal09:49:403
351609299533,22cyclictest26-21ksoftirqd/111:51:201
162399530,8rtkit-daemon1622-21rtkit-daemon07:40:062
351609899526,41cyclictest3628281-21tr12:25:153
3516098995225,23cyclictest3537521-21unin-run08:10:003
351609899522,45cyclictest1632-21dbus-daemon09:25:013
351609899522,45cyclictest1632-21dbus-daemon09:25:013
3516098995217,30cyclictest3543675-21if_enp2s008:25:213
3516096995210,6cyclictest141rcu_preempt10:25:002
351609299528,20cyclictest26-21ksoftirqd/111:05:001
351609299528,19cyclictest26-21ksoftirqd/110:50:241
351609299528,19cyclictest26-21ksoftirqd/110:50:241
351609299528,19cyclictest26-21ksoftirqd/108:25:231
351609299525,10cyclictest26-21ksoftirqd/109:15:011
162399520,9rtkit-daemon1622-21rtkit-daemon11:40:383
162399520,9rtkit-daemon1622-21rtkit-daemon07:48:193
162399520,8rtkit-daemon1622-21rtkit-daemon09:38:072
162399520,32rtkit-daemon2197014-21PK-Backend12:08:012
351609899519,29cyclictest1753-21gnome-shell11:25:123
351609899511,12cyclictest3604705-21basename11:20:193
162399510,8rtkit-daemon1622-21rtkit-daemon09:55:013
35743912500,7sleep02197014-21PK-Backend09:55:020
351609899502,43cyclictest1753-21gnome-shell10:07:533
351609899502,43cyclictest1753-21gnome-shell09:14:293
3516098995020,25cyclictest3616495-21sshd11:51:403
3516098995014,31cyclictest3532883-21expr07:55:163
351609299505,21cyclictest26-21ksoftirqd/110:40:301
351609299501,4cyclictest141rcu_preempt12:29:161
351609899498,17cyclictest1753-21gnome-shell09:18:243
351609899492,42cyclictest1753-21gnome-shell12:18:013
351609899492,42cyclictest1753-21gnome-shell11:38:543
351609899492,42cyclictest1753-21gnome-shell11:38:533
351609899492,42cyclictest1753-21gnome-shell10:32:103
351609899492,42cyclictest1753-21gnome-shell10:16:433
3516098994916,28cyclictest1632-21dbus-daemon08:50:003
3516098994915,30cyclictest3614144-21sensors_temp11:45:333
3516098994915,30cyclictest1069-21runrttasks12:30:423
3516098994914,30cyclictest3554651-21timerandwakeup08:55:303
3516098994913,31cyclictest3547376-21cat08:35:233
351609299499,21cyclictest26-21ksoftirqd/112:15:251
351609299499,18cyclictest26-21ksoftirqd/109:27:171
351609299498,18cyclictest1753-21gnome-shell11:45:341
351609299497,19cyclictest26-21ksoftirqd/107:55:151
3516092994910,19cyclictest26-21ksoftirqd/112:08:081
3516092994910,19cyclictest26-21ksoftirqd/109:40:251
351609899489,35cyclictest1753-21gnome-shell12:24:503
351609899489,34cyclictest1753-21gnome-shell10:01:043
351609899482,8cyclictest1753-21gnome-shell07:30:163
351609899482,42cyclictest1753-21gnome-shell10:11:053
351609899482,41cyclictest1753-21gnome-shell08:19:013
351609899482,41cyclictest1753-21gnome-shell08:19:013
351609899482,41cyclictest1753-21gnome-shell08:00:313
3516098994822,22cyclictest1753-21gnome-shell11:04:173
3516098994820,24cyclictest3617014-21/usr/sbin/munin11:55:153
3516098994816,27cyclictest1069-21runrttasks07:15:123
351609899481,41cyclictest1753-21gnome-shell12:35:143
3516098994812,31cyclictest391irq_work/309:55:153
3516098994810,33cyclictest1753-21gnome-shell12:14:293
3516096994825,14cyclictest34-21ksoftirqd/208:35:262
351609299483,22cyclictest26-21ksoftirqd/108:35:291
3516092994825,5cyclictest141rcu_preempt10:08:331
3516092994821,23cyclictest3529747-21tr07:45:221
3516089994828,7cyclictest13-21ksoftirqd/009:20:130
3516089994828,7cyclictest13-21ksoftirqd/009:20:130
3516089994825,14cyclictest13-21ksoftirqd/011:51:170
351609899478,16cyclictest1753-21gnome-shell07:20:253
351609899472,40cyclictest1753-21gnome-shell08:30:453
3516098994722,21cyclictest1753-21gnome-shell07:37:433
3516098994721,21cyclictest42-21ksoftirqd/309:05:203
351609299472,23cyclictest26-21ksoftirqd/111:55:141
35579972460,5sleep02197014-21PK-Backend09:05:270
351609899469,33cyclictest1753-21gnome-shell10:26:273
351609899469,33cyclictest1753-21gnome-shell10:26:273
3516098994618,25cyclictest3593778-21latency_hist10:50:023
3516098994615,26cyclictest1753-21gnome-shell08:50:143
3516098994614,28cyclictest3584660-21taskset10:25:013
3516098994613,28cyclictest3603478-21switchtime11:15:313
3516098994612,30cyclictest3599548-21if_err_enp2s011:05:213
3516096994628,14cyclictest1753-21gnome-shell09:45:092
3516096994627,15cyclictest1753-21gnome-shell07:21:182
3516096994623,14cyclictest34-21ksoftirqd/211:14:322
3516096994623,14cyclictest34-21ksoftirqd/208:15:142
3516096994623,14cyclictest34-21ksoftirqd/208:15:132
3516096994622,20cyclictest1753-21gnome-shell10:39:412
351609299462,22cyclictest26-21ksoftirqd/112:21:211
351609299462,22cyclictest26-21ksoftirqd/109:45:281
351609299462,22cyclictest26-21ksoftirqd/108:41:571
351609299461,23cyclictest26-21ksoftirqd/109:05:261
162399460,6rtkit-daemon1622-21rtkit-daemon09:42:573
351609899459,31cyclictest1753-21gnome-shell07:50:293
351609899456,12cyclictest3567967-21cat09:35:203
351609899454,36cyclictest3528292-21wc07:40:393
3516098994517,24cyclictest3621479-21sed12:05:293
3516098994514,26cyclictest3542135-21cat08:20:323
3516096994523,13cyclictest34-21ksoftirqd/212:30:132
3516096994518,23cyclictest3613645-40lsb_release11:45:232
3516096994512,10cyclictest34-21ksoftirqd/212:29:162
351609299452,22cyclictest26-21ksoftirqd/108:20:221
3516089994533,8cyclictest13-21ksoftirqd/010:48:140
3516089994531,5cyclictest13-21ksoftirqd/012:05:290
3516089994518,8cyclictest13-21ksoftirqd/010:30:250
35898582440,12chrt251rcuc/110:36:531
35769372440,7chrt13-21ksoftirqd/010:00:250
351609899442,8cyclictest1753-21gnome-shell10:55:493
3516098994417,22cyclictest1753-21gnome-shell11:31:283
3516098994414,26cyclictest3594010-21cpuspeed_turbos10:50:143
3516098994414,26cyclictest3594010-21cpuspeed_turbos10:50:143
351609899441,38cyclictest1753-21gnome-shell11:11:283
3516096994425,5cyclictest34-21ksoftirqd/209:00:312
3516096994422,6cyclictest34-21ksoftirqd/209:20:012
3516096994422,14cyclictest34-21ksoftirqd/210:10:292
3516096994413,12cyclictest34-21ksoftirqd/209:15:012
351609299448,18cyclictest26-21ksoftirqd/107:15:071
351609299446,10cyclictest26-21ksoftirqd/108:15:151
351609299446,10cyclictest26-21ksoftirqd/108:15:141
351609299442,21cyclictest26-21ksoftirqd/108:15:011
351609299441,22cyclictest26-21ksoftirqd/109:05:001
351609299441,18cyclictest3521118-21/usr/sbin/munin07:20:281
351608999445,9cyclictest3581094-21sshd10:14:400
3516096994332,7cyclictest1753-21gnome-shell11:42:072
3516096994331,8cyclictest1753-21gnome-shell12:15:262
3516096994324,7cyclictest34-21ksoftirqd/212:15:022
351609299434,20cyclictest26-21ksoftirqd/111:45:001
351609299433,13cyclictest26-21ksoftirqd/112:31:101
351608999434,11cyclictest141rcu_preempt11:26:040
3516089994324,15cyclictest13-21ksoftirqd/007:50:110
3516098994217,21cyclictest3517773-21perf07:15:003
3516098994214,25cyclictest2197014-21PK-Backend10:40:003
3516096994227,10cyclictest3603825-21sshd11:17:412
351609699422,21cyclictest34-21ksoftirqd/212:01:492
351609299427,16cyclictest26-21ksoftirqd/110:00:231
3516092994224,13cyclictest438-20systemd-journal12:15:001
3516089994222,13cyclictest13-21ksoftirqd/011:55:320
3516089994220,18cyclictest1753-21gnome-shell12:16:210
3516089994211,6cyclictest141rcu_preempt08:25:000
351609699419,16cyclictest34-21ksoftirqd/208:49:372
351609699419,16cyclictest141rcu_preempt12:20:162
351609699418,19cyclictest34-21ksoftirqd/207:55:412
351609699418,17cyclictest34-21ksoftirqd/211:54:402
3516096994130,7cyclictest1753-21gnome-shell11:07:332
3516096994123,13cyclictest34-21ksoftirqd/207:50:062
3516096994121,15cyclictest3618533-21pool-org.gnome.12:00:022
351609299412,20cyclictest26-21ksoftirqd/107:41:081
3516089994125,12cyclictest3623752-21apt-get12:15:020
3516089994116,12cyclictest3629487-21systemd-udevd12:29:170
162399410,34rtkit-daemon2197014-21PK-Backend11:33:112
35562702400,7chrt2197014-21PK-Backend09:00:280
351609699405,6cyclictest141rcu_preempt10:50:012
3516096994024,5cyclictest34-21ksoftirqd/208:57:462
3516096994024,12cyclictest855-21dbus-daemon08:01:562
3516096994022,14cyclictest34-21ksoftirqd/210:50:262
3516096994022,14cyclictest34-21ksoftirqd/210:50:262
3516096994022,14cyclictest3242253-21nautilus08:30:012
351609299406,10cyclictest26-21ksoftirqd/110:10:261
351609299405,19cyclictest26-21ksoftirqd/109:51:001
351609299403,13cyclictest26-21ksoftirqd/108:45:281
351609299401,9cyclictest26-21ksoftirqd/108:32:301
351609299401,21cyclictest3607452-21runrttasks11:29:441
351609299401,21cyclictest3575732-21seq09:57:351
351609299401,18cyclictest26-21ksoftirqd/108:00:151
351608999409,17cyclictest1753-21gnome-shell09:42:090
3516089994022,13cyclictest13-21ksoftirqd/010:25:310
3516089994022,13cyclictest13-21ksoftirqd/010:25:310
3516089994020,12cyclictest13-21ksoftirqd/008:35:020
3516089994018,13cyclictest13-21ksoftirqd/011:15:150
3516089994013,13cyclictest141rcu_preempt10:18:440
3516089994013,13cyclictest141rcu_preempt09:30:060
351608999401,10cyclictest2197014-21PK-Backend11:20:230
3516096993928,8cyclictest34-21ksoftirqd/212:40:012
3516096993928,7cyclictest1753-21gnome-shell11:28:352
3516096993928,7cyclictest1753-21gnome-shell10:05:102
3516096993927,8cyclictest1753-21gnome-shell08:12:262
351609699392,17cyclictest1753-21gnome-shell10:42:502
3516096993921,13cyclictest34-21ksoftirqd/210:30:202
351609699391,9cyclictest1753-21gnome-shell10:16:052
351609699391,15cyclictest3522954-21xargs07:25:312
351609699391,10cyclictest34-21ksoftirqd/211:05:012
351609299394,9cyclictest0-21swapper/109:18:071
351609299393,12cyclictest26-21ksoftirqd/109:35:211
3516092993917,7cyclictest26-21ksoftirqd/108:07:311
3516092993912,10cyclictest26-21ksoftirqd/111:14:431
351608999398,6cyclictest141rcu_preempt10:05:450
351608999394,5cyclictest141rcu_preempt08:46:390
351608999392,5cyclictest141rcu_preempt08:59:180
3516089993921,13cyclictest3548865-21hddtemp_smartct08:40:190
3516089993921,13cyclictest13-21ksoftirqd/012:21:200
3516089993921,13cyclictest13-21ksoftirqd/008:05:310
3516089993918,18cyclictest1753-21gnome-shell11:35:020
351608999391,7cyclictest13-21ksoftirqd/011:14:430
3516089993912,12cyclictest141rcu_preempt09:18:310
351608999391,13cyclictest141rcu_preempt10:36:420
162399390,32rtkit-daemon34-21ksoftirqd/209:50:592
3516096993827,7cyclictest1753-21gnome-shell09:06:112
3516096993827,7cyclictest1753-21gnome-shell07:45:502
3516096993821,6cyclictest34-21ksoftirqd/207:30:212
3516096993813,10cyclictest34-21ksoftirqd/208:05:212
3516092993827,8cyclictest26-21ksoftirqd/109:32:281
3516092993826,8cyclictest251rcuc/112:35:221
3516092993819,14cyclictest3596345-21ntpq10:55:261
3516092993815,20cyclictest251rcuc/108:51:501
351608999383,5cyclictest141rcu_preempt08:53:230
3516089993826,7cyclictest13-21ksoftirqd/008:27:140
3516089993826,7cyclictest13-21ksoftirqd/008:10:130
3516089993826,7cyclictest13-21ksoftirqd/007:35:130
3516089993821,13cyclictest13-21ksoftirqd/007:40:190
3516089993820,6cyclictest13-21ksoftirqd/009:30:010
3516089993820,6cyclictest13-21ksoftirqd/007:15:280
3516089993813,12cyclictest141rcu_preempt12:33:300
3516089993811,11cyclictest141rcu_preempt08:01:190
351609699379,5cyclictest141rcu_preempt11:40:002
351609699379,5cyclictest141rcu_preempt11:40:002
3516096993726,7cyclictest1753-21gnome-shell10:59:072
351609299371,31cyclictest3516958-21needreboot07:10:231
351609299371,18cyclictest3526943-21taskset07:39:061
3516089993725,7cyclictest13-21ksoftirqd/010:40:140
3516089993725,7cyclictest13-21ksoftirqd/010:20:300
3516089993722,11cyclictest3559743-21swap09:10:290
3516089993721,6cyclictest13-21ksoftirqd/008:20:020
3516089993721,6cyclictest13-21ksoftirqd/008:20:020
3516089993721,5cyclictest13-21ksoftirqd/007:20:200
3516089993720,12cyclictest13-21ksoftirqd/012:35:050
3516089993716,8cyclictest141rcu_preempt07:13:060
351608999371,4cyclictest141rcu_preempt11:40:020
351608999371,4cyclictest141rcu_preempt11:40:020
3516089993713,15cyclictest13-21ksoftirqd/009:45:430
3516096993625,7cyclictest1753-21gnome-shell08:50:552
3516089993626,7cyclictest3598046-21cut11:00:250
351608999362,31cyclictest0-21swapper/009:58:110
351608999361,11cyclictest3533929-21runrttasks07:59:030
351608999361,10cyclictest141rcu_preempt11:45:150
3516096993525,7cyclictest1753-21gnome-shell09:31:132
3516089993525,6cyclictest13-21ksoftirqd/007:47:560
3516089993524,6cyclictest13-21ksoftirqd/012:00:190
3516089993519,12cyclictest13-21ksoftirqd/007:25:380
3516089993517,7cyclictest141rcu_preempt08:35:580
351608999351,13cyclictest2197014-21PK-Backend10:51:290
351608999351,13cyclictest2197014-21PK-Backend10:51:290
3516092993411,8cyclictest0-21swapper/111:30:141
3516089993414,15cyclictest141rcu_preempt11:05:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional