You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Jul 06, 2025 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
272532523400,8sleep00-21swapper/007:10:000
272441122520,8sleep3411rcuc/307:05:283
1623991240,22rtkit-daemon707415-21gmain08:45:273
1623991010,10rtkit-daemon1622-21rtkit-daemon07:50:163
162399900,9rtkit-daemon1622-21rtkit-daemon10:41:131
162399900,9rtkit-daemon1622-21rtkit-daemon10:41:131
28052502850,6sleep00-21swapper/010:55:150
28200092770,7sleep00-21swapper/011:35:320
272488327349,16sleep10-21swapper/107:08:451
162399700,5rtkit-daemon0-21swapper/309:34:123
162399700,5rtkit-daemon0-21swapper/309:34:123
28211722680,7sleep126-21ksoftirqd/111:40:161
28211722680,7sleep126-21ksoftirqd/111:40:161
162399650,9rtkit-daemon1622-21rtkit-daemon11:14:211
272472326439,19sleep20-21swapper/207:06:422
162399610,9rtkit-daemon1622-21rtkit-daemon09:46:560
162399590,9rtkit-daemon1622-21rtkit-daemon07:57:263
162399590,12rtkit-daemon1753-21gnome-shell11:03:132
28408002580,13sleep1251rcuc/112:35:301
162399580,8rtkit-daemon1622-21rtkit-daemon07:21:572
162399570,8rtkit-daemon1622-21rtkit-daemon08:06:132
2725430995620,19cyclictest34-21ksoftirqd/210:05:272
28279082550,7sleep02197014-21PK-Backend12:00:160
27950932540,14sleep0161rcuc/010:25:290
27321802540,8sleep22197014-21PK-Backend07:25:272
27261072530,14sleep2331rcuc/207:10:222
2725430995213,21cyclictest34-21ksoftirqd/207:16:372
162399520,8rtkit-daemon1622-21rtkit-daemon10:48:260
2725430995141,6cyclictest34-21ksoftirqd/212:33:392
162399510,10rtkit-daemon1753-21gnome-shell08:35:263
2725429995022,10cyclictest2834418-21awk12:20:001
2725434994922,13cyclictest2801774-21expr10:45:163
2725430994937,9cyclictest34-21ksoftirqd/209:28:382
28236862480,17chrt13-21ksoftirqd/011:45:590
28165032470,5sleep0161rcuc/011:25:340
2725430994726,17cyclictest1753-21gnome-shell11:06:582
2725429994730,13cyclictest2802922-21/usr/sbin/munin10:50:011
2725424994716,15cyclictest2797663-21/usr/sbin/munin10:35:280
272542499471,24cyclictest2817418-21seq11:30:130
162399470,34rtkit-daemon2197014-21PK-Backend11:18:192
2725430994627,15cyclictest1753-21gnome-shell11:56:422
272543099461,16cyclictest27617712sort08:50:252
272543099459,13cyclictest438-20systemd-journal09:53:222
2725429994516,12cyclictest26-21ksoftirqd/110:25:121
27646892440,5sleep00-21swapper/009:00:120
2725434994414,5cyclictest141rcu_preempt10:40:223
2725434994414,5cyclictest141rcu_preempt10:40:223
2725430994433,7cyclictest1753-21gnome-shell08:32:322
2725429994417,11cyclictest2824378-21cpuspeed_turbos11:50:181
27821562430,18sleep013-21ksoftirqd/009:50:150
2725434994321,16cyclictest42-21ksoftirqd/311:45:333
2725430994330,9cyclictest1753-21gnome-shell08:28:112
2725430994324,6cyclictest34-21ksoftirqd/211:40:222
2725430994324,6cyclictest34-21ksoftirqd/211:40:222
2725430994321,13cyclictest34-21ksoftirqd/212:25:292
2725430994314,10cyclictest34-21ksoftirqd/208:50:012
272542999432,26cyclictest2803568-21diskstats10:50:181
272542499432,10cyclictest0-21swapper/012:05:410
28370512420,5chrt0-21swapper/012:25:230
272543499426,6cyclictest141rcu_preempt07:24:053
272543499423,5cyclictest141rcu_preempt10:05:273
2725430994231,7cyclictest1753-21gnome-shell12:05:562
2725430994231,7cyclictest1753-21gnome-shell09:20:252
2725430994231,7cyclictest1753-21gnome-shell07:40:402
2725430994231,7cyclictest1753-21gnome-shell07:40:402
2725430994224,13cyclictest34-21ksoftirqd/210:15:022
2725430994224,13cyclictest34-21ksoftirqd/210:15:022
2725429994225,14cyclictest2726106-21/usr/sbin/munin07:10:221
272542499429,10cyclictest1069-21runrttasks12:35:200
272542499422,9cyclictest0-21swapper/011:22:410
272542499421,10cyclictest0-21swapper/007:56:130
162399420,6rtkit-daemon0-21swapper/209:05:282
162399420,5rtkit-daemon0-21swapper/308:42:323
162399420,35rtkit-daemon3242254-21gmain08:01:212
162399420,35rtkit-daemon3242254-21gmain08:01:212
272543499412,6cyclictest141rcu_preempt09:28:403
2725434994121,15cyclictest42-21ksoftirqd/310:35:193
2725434994111,11cyclictest2181-21gdbus11:20:013
2725430994130,7cyclictest1753-21gnome-shell09:19:092
2725430994130,7cyclictest1753-21gnome-shell07:46:492
2725430994129,8cyclictest1753-21gnome-shell09:38:022
2725430994129,8cyclictest1753-21gnome-shell09:30:282
2725430994129,8cyclictest1753-21gnome-shell09:30:272
2725430994122,14cyclictest855-21dbus-daemon07:55:002
2725430994122,14cyclictest34-21ksoftirqd/210:55:232
2725430994122,14cyclictest2755785-21awk08:35:132
2725429994128,9cyclictest231irq_work/108:40:011
272542999412,35cyclictest1753-21gnome-shell08:12:381
2725424994130,7cyclictest2733531-21df_inode07:30:170
2725424994125,11cyclictest438-20systemd-journal11:10:010
2725424994123,13cyclictest438-20systemd-journal09:25:000
272543499403,33cyclictest2787070-21ls10:05:013
2725430994029,7cyclictest1753-21gnome-shell11:37:362
2725430994029,7cyclictest1753-21gnome-shell11:27:362
2725430994023,5cyclictest34-21ksoftirqd/209:00:192
2725430994022,7cyclictest34-21ksoftirqd/208:42:372
2725430994022,14cyclictest34-21ksoftirqd/209:42:502
2725430994022,13cyclictest34-21ksoftirqd/212:00:052
272543099402,13cyclictest1753-21gnome-shell11:23:572
2725430994012,10cyclictest34-21ksoftirqd/209:10:142
2725429994021,16cyclictest2799288-21latency_hist10:40:021
272542999401,9cyclictest438-20systemd-journal08:20:001
2725429994013,11cyclictest2779556-21timerandwakeup09:40:331
272542999401,22cyclictest2818207-21cut11:30:271
272542999401,21cyclictest2729130-21fido_id07:16:391
272542499405,10cyclictest2786413-21python310:00:400
272542499402,9cyclictest2775507-21diskstats09:30:150
272542499402,9cyclictest2775507-21diskstats09:30:150
2725424994025,12cyclictest2745083-21awk08:05:000
2725424994025,12cyclictest2745083-21awk08:05:000
272542499401,11cyclictest2197014-21PK-Backend08:35:190
28298522390,6chrt0-21swapper/312:05:183
272543499394,16cyclictest1753-21gnome-shell12:02:153
2725434993925,10cyclictest42-21ksoftirqd/311:50:103
2725434993923,12cyclictest42-21ksoftirqd/309:37:223
2725434993922,6cyclictest42-21ksoftirqd/311:45:013
2725434993922,6cyclictest42-21ksoftirqd/311:45:013
2725434993922,13cyclictest1-21systemd10:30:003
2725434993920,13cyclictest42-21ksoftirqd/309:20:263
2725434993915,6cyclictest391irq_work/307:17:273
2725434993912,14cyclictest42-21ksoftirqd/310:20:223
272543099392,9cyclictest1753-21gnome-shell10:46:232
2725430993928,7cyclictest1753-21gnome-shell12:19:212
2725430993928,7cyclictest1753-21gnome-shell10:21:002
2725430993925,10cyclictest2751212-21sshd08:20:212
2725430993923,6cyclictest34-21ksoftirqd/208:56:142
2725430993922,13cyclictest34-21ksoftirqd/207:33:542
2725430993921,5cyclictest34-21ksoftirqd/211:45:302
2725430993921,14cyclictest34-21ksoftirqd/211:10:142
2725430993918,18cyclictest34-21ksoftirqd/211:33:052
2725430993916,7cyclictest34-21ksoftirqd/210:15:242
272542999397,15cyclictest141rcu_preempt10:15:001
272542999397,15cyclictest141rcu_preempt10:15:001
2725429993925,10cyclictest2827127-21sshd11:55:431
2725429993922,12cyclictest438-20systemd-journal11:15:381
2725429993915,20cyclictest2775714-21memory09:30:211
2725429993915,20cyclictest2775714-21memory09:30:211
272542999390,35cyclictest1622-21rtkit-daemon09:48:261
272542499392,9cyclictest0-21swapper/009:19:090
272542499392,9cyclictest0-21swapper/008:34:250
272542499392,8cyclictest2803943-21ntp_states10:50:270
27680622380,17sleep013-21ksoftirqd/009:10:020
272543499382,32cyclictest2809846-21sh11:10:013
2725434993822,11cyclictest2725613-21unattended-upgr07:10:233
2725434993819,13cyclictest42-21ksoftirqd/311:25:303
272543499381,33cyclictest1753-21gnome-shell12:33:353
272543499381,18cyclictest141rcu_preempt11:35:283
2725430993827,7cyclictest1753-21gnome-shell10:32:262
2725430993827,7cyclictest1753-21gnome-shell10:25:572
2725430993827,7cyclictest1753-21gnome-shell10:00:182
2725430993827,7cyclictest1753-21gnome-shell07:55:182
2725430993827,7cyclictest1753-21gnome-shell07:38:172
2725430993825,9cyclictest2799162-21apt10:40:002
2725430993821,6cyclictest34-21ksoftirqd/212:35:142
2725430993821,13cyclictest34-21ksoftirqd/211:53:252
2725430993821,13cyclictest34-21ksoftirqd/210:44:402
2725430993821,13cyclictest34-21ksoftirqd/210:44:402
2725430993821,13cyclictest34-21ksoftirqd/208:13:362
2725430993820,6cyclictest34-21ksoftirqd/208:15:432
272542999382,8cyclictest1753-21gnome-shell09:27:221
2725429993823,11cyclictest2813813-21runrttasks11:20:071
2725429993822,11cyclictest438-20systemd-journal12:05:001
2725429993821,13cyclictest2792981-21awk10:20:231
2725429993821,13cyclictest1632-21dbus-daemon09:25:001
2725429993821,12cyclictest2788013-21systemd-udevd10:05:271
2725429993818,16cyclictest2829834-21expr12:05:161
2725429993817,17cyclictest2732882-21latency_hist07:30:011
272542499382,7cyclictest1753-21gnome-shell08:17:330
2725424993819,8cyclictest1069-21runrttasks07:22:570
27839492370,7sleep32197014-21PK-Backend09:55:133
27839492370,7sleep32197014-21PK-Backend09:55:133
2725434993729,5cyclictest0-21swapper/311:11:353
2725434993726,7cyclictest1753-21gnome-shell12:25:163
2725434993724,9cyclictest42-21ksoftirqd/311:55:053
2725434993724,9cyclictest1753-21gnome-shell07:31:383
2725434993722,11cyclictest42-21ksoftirqd/307:36:143
2725434993720,5cyclictest42-21ksoftirqd/310:56:223
272543499371,8cyclictest2197014-21PK-Backend09:50:263
2725434993717,15cyclictest42-21ksoftirqd/310:30:203
2725434993716,9cyclictest141rcu_preempt08:26:493
2725434993716,8cyclictest141rcu_preempt12:20:313
2725434993711,22cyclictest0-21swapper/308:06:153
2725430993720,5cyclictest34-21ksoftirqd/209:55:232
2725430993720,5cyclictest34-21ksoftirqd/209:55:222
2725430993719,6cyclictest34-21ksoftirqd/212:20:212
2725429993727,6cyclictest2769615-21dump-pmu-power09:15:001
272542999372,31cyclictest0-21swapper/108:04:311
272542999372,31cyclictest0-21swapper/108:04:301
2725429993723,10cyclictest2743040-21sshd07:56:221
2725429993720,13cyclictest2806804-21expr11:00:141
2725429993720,12cyclictest904-21systemd-logind10:58:501
2725429993717,16cyclictest2796169-21cstates10:30:151
2725429993717,16cyclictest2751077-21latency08:20:201
2725429993716,17cyclictest2730177-21irqstats07:20:221
2725429993715,5cyclictest141rcu_preempt08:50:011
2725429993714,9cyclictest2770731-21processes09:15:301
272542499372,7cyclictest1753-21gnome-shell10:06:430
272542499372,7cyclictest1753-21gnome-shell08:49:300
272542499372,5cyclictest0-21swapper/008:26:410
272542499372,31cyclictest1753-21gnome-shell11:52:040
272542499372,10cyclictest0-21swapper/011:44:080
272542499372,10cyclictest0-21swapper/011:44:080
2725424993715,18cyclictest1753-21gnome-shell08:50:210
272542499371,21cyclictest2835191-21latency12:20:230
272542499371,11cyclictest141rcu_preempt12:15:240
272543499364,8cyclictest2197014-21PK-Backend08:50:253
272543499362,6cyclictest2833683-21ntp_states12:15:263
2725434993624,6cyclictest42-21ksoftirqd/310:11:513
2725434993624,6cyclictest42-21ksoftirqd/310:11:513
2725434993622,9cyclictest1753-21gnome-shell11:31:043
2725434993621,11cyclictest2754326-21cat08:30:153
2725434993619,13cyclictest2739131-21awk07:45:253
2725434993617,7cyclictest141rcu_preempt12:35:213
2725434993617,7cyclictest141rcu_preempt07:40:343
2725434993617,7cyclictest141rcu_preempt07:40:343
2725434993616,7cyclictest141rcu_preempt08:10:493
2725430993620,12cyclictest2781095-21seq09:45:282
2725430993619,6cyclictest34-21ksoftirqd/210:50:262
2725429993622,10cyclictest2764670-21sshd09:00:081
2725429993621,11cyclictest2809782-21sshd11:09:221
2725429993621,11cyclictest2754151-21expr08:30:141
2725429993619,12cyclictest2820175-21users11:35:341
2725429993615,6cyclictest2785342-21latency_hist10:00:021
2725429993615,6cyclictest2785342-21latency_hist10:00:021
272542999361,18cyclictest2736893-21cat07:40:181
272542999361,18cyclictest2736893-21cat07:40:171
272542499365,15cyclictest141rcu_preempt12:30:200
2725424993628,5cyclictest0-21swapper/007:47:500
272542499362,5cyclictest0-21swapper/011:19:320
272542499362,5cyclictest0-21swapper/011:11:270
2725424993624,8cyclictest1069-21runrttasks10:14:380
2725424993624,8cyclictest1069-21runrttasks10:14:380
2725424993621,11cyclictest2807249-21grep11:00:240
2725424993620,12cyclictest1632-21dbus-daemon12:00:010
2725424993619,12cyclictest2774700-21systemd-udevd09:28:390
2725424993619,12cyclictest2768805-21cat09:10:230
2725424993618,13cyclictest2832273-21cat12:10:340
272543499352,28cyclictest1753-21gnome-shell10:15:053
2725434993521,9cyclictest1753-21gnome-shell10:53:003
2725434993521,10cyclictest2832689-21awk12:15:003
2725434993520,11cyclictest42-21ksoftirqd/308:55:233
2725434993516,7cyclictest141rcu_preempt08:00:293
2725434993516,7cyclictest141rcu_preempt08:00:293
272543499351,31cyclictest42-21ksoftirqd/309:00:563
272543499351,29cyclictest2197014-21PK-Backend09:15:193
2725434993510,20cyclictest42-21ksoftirqd/309:45:203
2725430993517,6cyclictest34-21ksoftirqd/212:10:272
272542999356,11cyclictest26-21ksoftirqd/109:05:181
2725429993523,9cyclictest2791162-21hddtemp_smartct10:15:201
2725429993521,10cyclictest2834674-21apt12:20:151
2725429993520,10cyclictest2824021-21/usr/sbin/munin11:50:011
2725429993519,12cyclictest2832269-21cat12:10:331
2725429993515,8cyclictest26-21ksoftirqd/108:59:041
272542499352,5cyclictest0-21swapper/008:06:250
2725424993524,6cyclictest13-21ksoftirqd/008:10:320
2725424993523,8cyclictest1753-21gnome-shell07:25:440
2725424993523,7cyclictest2738135-21dump-pmu-power07:45:000
2725424993523,7cyclictest2738135-21dump-pmu-power07:45:000
272542499352,30cyclictest0-21swapper/010:33:170
272542499352,30cyclictest0-21swapper/010:17:190
272542499352,11cyclictest13-21ksoftirqd/008:58:200
2725424993521,10cyclictest2725358-21kworker/0:107:16:370
272542499351,9cyclictest1753-21gnome-shell07:10:140
2725424993519,12cyclictest2757701-21acpi08:40:110
2725424993519,11cyclictest2735649-21ntpq07:35:270
2725424993517,13cyclictest2800588-21sshd10:41:220
2725424993517,13cyclictest2800588-21sshd10:41:220
2725424993517,13cyclictest2779664-21cat09:40:330
2725424993516,15cyclictest2777576-21awk09:35:250
2725434993419,11cyclictest2778572-21/usr/sbin/munin09:40:163
2725434993416,14cyclictest1753-21gnome-shell09:12:033
2725434993416,13cyclictest2813998-21cut11:20:153
2725434993413,18cyclictest42-21ksoftirqd/311:00:223
2725429993423,7cyclictest26-21ksoftirqd/112:28:311
2725429993420,10cyclictest2739653-21sshd07:47:221
2725429993418,12cyclictest2752895-21cat08:25:231
2725429993417,13cyclictest2745706-21sh08:05:161
2725429993417,12cyclictest2816349-21ntpq11:25:291
2725429993416,13cyclictest438-20systemd-journal08:52:221
272542999341,5cyclictest0-21swapper/109:37:531
2725429993414,6cyclictest26-21ksoftirqd/107:30:231
272542499342,5cyclictest0-21swapper/008:21:370
2725424993418,12cyclictest2784532-21ntpq09:55:240
2725424993418,12cyclictest2784532-21ntpq09:55:240
272542499341,10cyclictest0-21swapper/010:20:430
272543499332,9cyclictest1753-21gnome-shell08:15:223
2725434993317,11cyclictest2732012-21load07:25:243
272542999337,8cyclictest26-21ksoftirqd/107:54:081
272542999335,7cyclictest2827520-21kworker/1:112:31:011
272542999334,12cyclictest26-21ksoftirqd/108:43:101
2725429993319,10cyclictest2782665-21sort09:50:241
2725429993311,11cyclictest141rcu_preempt10:02:251
272542499335,7cyclictest141rcu_preempt07:50:170
272543499323,4cyclictest141rcu_preempt09:10:013
2725434993214,5cyclictest141rcu_preempt08:24:483
2725429993216,11cyclictest2734948-21cpuspeed07:35:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional