You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri May 02, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
351038422540,7sleep00-21swapper/007:05:240
351051529344,16sleep30-21swapper/307:05:283
162399850,63rtkit-daemon0-21swapper/207:09:402
35647832750,4chrt3564784-21munin-node09:40:121
351089127540,29sleep10-21swapper/107:07:271
3511226997019,35cyclictest1069-21runrttasks11:37:323
351122699692,48cyclictest1753-21gnome-shell09:26:303
351122699692,44cyclictest1753-21gnome-shell10:25:173
3511226996921,30cyclictest1753-21gnome-shell08:40:323
351122699682,37cyclictest1753-21gnome-shell10:50:183
351122699682,36cyclictest1753-21gnome-shell12:30:223
351122699672,34cyclictest1753-21gnome-shell10:11:273
351122699672,32cyclictest1753-21gnome-shell11:08:483
3511226996714,29cyclictest3610190-21seq11:47:023
351121899670,63cyclictest1622-21rtkit-daemon10:58:270
351122699662,44cyclictest1753-21gnome-shell11:01:033
351122699662,38cyclictest3564309-21chrt09:37:163
351122699662,36cyclictest1753-21gnome-shell10:00:233
351122699662,34cyclictest1753-21gnome-shell12:20:193
351122699662,32cyclictest1753-21gnome-shell09:48:433
351122699662,31cyclictest1753-21gnome-shell11:10:293
351122699652,45cyclictest1753-21gnome-shell08:24:253
351122699652,44cyclictest1753-21gnome-shell12:26:403
351122699652,44cyclictest1753-21gnome-shell10:21:513
351122699652,44cyclictest1753-21gnome-shell09:20:113
351122699652,44cyclictest1753-21gnome-shell08:47:563
351122699652,44cyclictest1753-21gnome-shell07:22:293
351122699652,42cyclictest1753-21gnome-shell09:43:273
351122699652,42cyclictest1753-21gnome-shell09:31:393
351122699652,42cyclictest1753-21gnome-shell08:56:283
351122699652,41cyclictest1753-21gnome-shell11:20:063
351122699652,26cyclictest1753-21gnome-shell07:39:363
351122699651,35cyclictest3574743-21taskset10:08:023
351122699651,35cyclictest3547200-21expr08:50:143
351122699651,35cyclictest3547200-21expr08:50:143
351122699642,45cyclictest1753-21gnome-shell12:18:323
351122699642,44cyclictest1753-21gnome-shell12:02:393
351122699642,44cyclictest1753-21gnome-shell08:08:073
351122699642,43cyclictest1753-21gnome-shell12:07:353
351122699642,43cyclictest1753-21gnome-shell09:11:213
351122699642,42cyclictest1753-21gnome-shell10:34:573
351122699642,42cyclictest1753-21gnome-shell07:46:043
351122699642,41cyclictest1753-21gnome-shell11:19:493
351122699641,32cyclictest1753-21gnome-shell11:32:483
351122699632,43cyclictest1753-21gnome-shell11:52:293
351122699632,43cyclictest1753-21gnome-shell07:10:563
351122699632,42cyclictest1753-21gnome-shell08:34:463
351122699632,39cyclictest1753-21gnome-shell08:37:213
351122699632,34cyclictest1753-21gnome-shell10:40:263
351122699632,33cyclictest1753-21gnome-shell08:17:543
351122699632,30cyclictest1753-21gnome-shell10:19:573
351122699632,29cyclictest1753-21gnome-shell09:15:213
351122699632,28cyclictest3591246-21irqstats10:55:233
351122699631,44cyclictest1753-21gnome-shell07:16:393
351122699631,37cyclictest3553115-21sed09:05:323
351122699631,37cyclictest3553115-21sed09:05:323
3511226996310,27cyclictest3603056-21cat11:30:023
36058022620,5sleep20-21swapper/211:35:242
351122699622,43cyclictest1753-21gnome-shell08:00:533
351122699622,43cyclictest1753-21gnome-shell07:42:593
351122699622,42cyclictest1753-21gnome-shell10:49:563
351122699622,41cyclictest1753-21gnome-shell07:57:033
3511226996212,29cyclictest3518247-21runrttasks07:26:003
351122699612,42cyclictest1753-21gnome-shell12:12:373
351122699612,41cyclictest1753-21gnome-shell11:41:453
351122699612,32cyclictest1753-21gnome-shell08:14:123
351122699612,32cyclictest1753-21gnome-shell08:14:123
3511224996149,9cyclictest34-21ksoftirqd/208:35:012
162399610,9rtkit-daemon1622-21rtkit-daemon08:25:570
351122699602,32cyclictest1753-21gnome-shell10:39:443
351122699602,32cyclictest1753-21gnome-shell08:29:143
351122699592,27cyclictest3527291-21munin-run07:55:013
351122699591,33cyclictest860997-21snmpd09:53:093
351122699591,33cyclictest860997-21snmpd09:53:083
351122699582,32cyclictest860997-21snmpd09:03:183
351121899580,54cyclictest1622-21rtkit-daemon10:35:300
162399580,8rtkit-daemon1622-21rtkit-daemon08:12:430
162399580,8rtkit-daemon1622-21rtkit-daemon08:12:430
351122699571,34cyclictest860997-21snmpd07:33:553
36165792560,7sleep22197014-21PK-Backend12:05:232
351122699552,29cyclictest1753-21gnome-shell12:36:573
351122699551,29cyclictest860997-21snmpd11:55:083
351122699541,29cyclictest860997-21snmpd09:59:113
162399540,9rtkit-daemon1622-21rtkit-daemon10:34:520
162399530,8rtkit-daemon1622-21rtkit-daemon08:29:491
351122199520,48cyclictest1622-21rtkit-daemon12:05:321
162399520,9rtkit-daemon1622-21rtkit-daemon07:23:030
3511224995124,5cyclictest141rcu_preempt09:36:572
3511224995124,5cyclictest141rcu_preempt08:35:312
3511224995123,5cyclictest141rcu_preempt10:31:542
351122499507,5cyclictest141rcu_preempt09:58:462
36154752490,20sleep126-21ksoftirqd/112:03:171
35149772490,6chrt13-21ksoftirqd/007:18:270
3511221994926,19cyclictest3517167-21awk07:25:161
351121899491,32cyclictest141rcu_preempt08:46:030
351122499481,16cyclictest3562658-21chrt09:35:022
351122499477,16cyclictest141rcu_preempt10:29:282
3511224994718,8cyclictest34-21ksoftirqd/212:22:432
3511218994722,22cyclictest3541994-21/usr/sbin/munin08:35:260
3511224994628,5cyclictest141rcu_preempt09:14:442
3511224994623,8cyclictest34-21ksoftirqd/209:15:212
3511221994629,13cyclictest2181-21gdbus09:05:001
351122199462,6cyclictest1753-21gnome-shell09:55:161
3511218994634,8cyclictest1753-21gnome-shell11:04:570
3511224994513,8cyclictest34-21ksoftirqd/209:20:272
3511221994524,17cyclictest3576172-21timerandwakeup10:10:331
3511218994533,8cyclictest1753-21gnome-shell07:47:430
3511218994523,13cyclictest13-21ksoftirqd/011:35:270
3511224994431,9cyclictest3590489-21munin-run10:55:002
3511221994424,15cyclictest3242253-21nautilus10:35:011
351121899448,32cyclictest13-21ksoftirqd/009:12:020
3511218994425,6cyclictest13-21ksoftirqd/010:15:200
351122499438,8cyclictest34-21ksoftirqd/209:26:062
351122499436,4cyclictest141rcu_preempt12:32:032
3511224994331,8cyclictest34-21ksoftirqd/211:30:182
3511221994316,14cyclictest3529250-21/usr/sbin/munin08:00:141
3511218994325,14cyclictest13-21ksoftirqd/010:10:170
3511218994324,7cyclictest13-21ksoftirqd/011:48:460
3511218994324,7cyclictest13-21ksoftirqd/011:14:050
3511218994324,7cyclictest13-21ksoftirqd/009:36:290
3511218994324,6cyclictest13-21ksoftirqd/009:25:130
3511218994323,8cyclictest13-21ksoftirqd/010:00:160
3511218994313,9cyclictest13-21ksoftirqd/008:16:020
35504952420,8chrt2197014-21PK-Backend09:00:162
351122499428,5cyclictest141rcu_preempt07:50:012
351122499427,19cyclictest34-21ksoftirqd/208:45:272
351122499424,5cyclictest141rcu_preempt11:15:212
3511224994228,10cyclictest3595713-21gpgv11:10:002
3511224994223,5cyclictest141rcu_preempt10:21:292
3511224994217,9cyclictest34-21ksoftirqd/211:47:382
3511224994217,8cyclictest34-21ksoftirqd/208:24:302
351122499421,14cyclictest3569561-21taskset09:53:052
351122499421,14cyclictest3569561-21taskset09:53:052
3511218994231,7cyclictest1753-21gnome-shell09:30:080
3511218994231,7cyclictest1753-21gnome-shell07:30:210
3511218994225,6cyclictest13-21ksoftirqd/011:30:210
3511218994224,6cyclictest13-21ksoftirqd/012:25:340
3511218994224,6cyclictest13-21ksoftirqd/009:20:250
3511218994224,6cyclictest13-21ksoftirqd/007:25:200
3511218994224,14cyclictest13-21ksoftirqd/007:39:420
3511218994224,13cyclictest13-21ksoftirqd/008:40:310
3511218994223,6cyclictest13-21ksoftirqd/009:50:160
3511218994223,6cyclictest13-21ksoftirqd/009:50:160
3511218994223,15cyclictest3567293-21ntp_states09:45:320
3511218994223,14cyclictest13-21ksoftirqd/010:50:160
3511218994222,7cyclictest13-21ksoftirqd/008:50:110
3511218994222,7cyclictest13-21ksoftirqd/008:50:110
351121899421,19cyclictest35296932idleruntime08:00:200
351121899421,14cyclictest3613896-21turbostat12:00:000
35650062410,6chrt0-21swapper/009:40:150
351122499415,5cyclictest141rcu_preempt11:10:032
3511224994113,14cyclictest34-21ksoftirqd/210:07:332
3511221994113,13cyclictest3552675-21wc09:05:241
3511221994113,13cyclictest3552675-21wc09:05:241
3511218994130,7cyclictest1753-21gnome-shell08:23:100
3511218994129,8cyclictest1753-21gnome-shell12:22:410
3511218994124,5cyclictest13-21ksoftirqd/010:50:010
3511218994124,13cyclictest13-21ksoftirqd/012:07:050
3511218994123,6cyclictest13-21ksoftirqd/012:15:260
3511218994123,6cyclictest13-21ksoftirqd/011:25:310
3511218994123,6cyclictest13-21ksoftirqd/007:10:190
351121899412,35cyclictest1753-21gnome-shell08:33:380
3511218994123,13cyclictest13-21ksoftirqd/010:25:170
3511218994122,7cyclictest13-21ksoftirqd/011:40:210
3511218994122,14cyclictest13-21ksoftirqd/010:40:260
3511218994121,15cyclictest13-21ksoftirqd/007:40:160
351122499409,16cyclictest1753-21gnome-shell11:25:462
351122499408,12cyclictest141rcu_preempt07:20:302
351122499404,8cyclictest0-21swapper/208:50:122
351122499404,8cyclictest0-21swapper/208:50:122
3511224994023,12cyclictest3608400-21apt-get11:45:012
3511224994017,12cyclictest34-21ksoftirqd/207:12:032
3511224994015,9cyclictest34-21ksoftirqd/208:25:252
3511224994014,4cyclictest141rcu_preempt12:18:342
3511224994013,13cyclictest141rcu_preempt10:42:082
351122499401,17cyclictest34-21ksoftirqd/209:40:592
351122499401,17cyclictest2197014-21PK-Backend08:17:542
351122499401,16cyclictest2197014-21PK-Backend12:40:012
351122499401,16cyclictest2197014-21PK-Backend10:10:282
3511221994022,14cyclictest3584199-21expr10:35:171
3511221994022,14cyclictest3242253-21nautilus11:10:011
3511221994020,18cyclictest3604220-40lsb_release11:30:281
3511218994023,6cyclictest13-21ksoftirqd/012:12:290
3511218994022,6cyclictest13-21ksoftirqd/012:30:210
3511218994022,13cyclictest13-21ksoftirqd/011:05:460
3511218994022,13cyclictest13-21ksoftirqd/009:10:020
3511218994022,13cyclictest13-21ksoftirqd/009:10:020
3511218994019,16cyclictest3599687-21/usr/sbin/munin11:20:130
351122499393,19cyclictest34-21ksoftirqd/208:10:262
351122499393,19cyclictest34-21ksoftirqd/208:10:262
3511224993922,5cyclictest141rcu_preempt12:27:562
3511224993917,9cyclictest34-21ksoftirqd/212:00:102
351122499391,21cyclictest34-21ksoftirqd/209:50:012
351122499391,16cyclictest2197014-21PK-Backend08:55:222
351122199398,8cyclictest26-21ksoftirqd/108:45:151
3511221993929,7cyclictest1069-21runrttasks07:14:051
351122199392,33cyclictest1753-21gnome-shell07:17:211
3511221993922,13cyclictest3611794-21vmstat11:50:321
3511221993913,9cyclictest26-21ksoftirqd/109:25:031
3511218993928,7cyclictest1753-21gnome-shell08:05:110
3511218993923,5cyclictest13-21ksoftirqd/011:15:160
3511218993923,13cyclictest13-21ksoftirqd/010:07:320
3511218993921,6cyclictest13-21ksoftirqd/012:03:060
3511218993921,6cyclictest13-21ksoftirqd/010:20:260
3511218993921,6cyclictest13-21ksoftirqd/008:55:210
3511218993918,17cyclictest3550331-21/usr/sbin/munin09:00:230
351121899391,7cyclictest1358-1kworker/u9:107:50:200
36126842380,6chrt251rcuc/111:55:161
36069042380,4sleep10-21swapper/111:40:021
351122499388,5cyclictest141rcu_preempt07:15:032
351122499387,5cyclictest141rcu_preempt11:00:022
351122499381,21cyclictest3518484-21chrt07:29:042
3511224993811,8cyclictest34-21ksoftirqd/208:07:062
351122499381,15cyclictest2197014-21PK-Backend11:20:132
351122499381,15cyclictest2197014-21PK-Backend08:45:002
351122499381,13cyclictest2197014-21PK-Backend10:15:222
351122199382,8cyclictest1753-21gnome-shell09:51:501
351122199382,8cyclictest1753-21gnome-shell09:51:501
3511221993822,12cyclictest3598633-21needreboot11:15:241
3511221993821,13cyclictest3609432-21irqstats11:45:221
3511221993821,12cyclictest438-20systemd-journal08:31:341
3511221993812,5cyclictest141rcu_preempt12:14:351
3511218993827,7cyclictest1753-21gnome-shell09:57:090
3511218993822,12cyclictest13-21ksoftirqd/009:17:210
3511218993821,13cyclictest13-21ksoftirqd/012:35:270
3511218993820,6cyclictest13-21ksoftirqd/007:58:340
3511218993818,6cyclictest141rcu_preempt11:51:380
351122499373,8cyclictest34-21ksoftirqd/211:00:592
351122499372,32cyclictest0-21swapper/207:58:362
3511224993720,13cyclictest34-21ksoftirqd/207:38:482
3511224993720,12cyclictest34-21ksoftirqd/210:00:232
351122499371,16cyclictest3553538-21latency_hist09:10:002
351122499371,16cyclictest3553538-21latency_hist09:10:002
351122499371,15cyclictest2197014-21PK-Backend07:40:182
3511224993710,5cyclictest141rcu_preempt10:39:182
351122199375,28cyclictest3572534-21/usr/sbin/munin10:00:291
351122199374,5cyclictest141rcu_preempt10:50:331
3511221993725,8cyclictest1753-21gnome-shell09:22:311
3511221993723,10cyclictest3588708-21sshd10:49:331
3511221993722,11cyclictest3520964-21forks07:35:221
3511221993721,12cyclictest3528415-21sensors_temp07:55:311
3511221993721,11cyclictest438-20systemd-journal10:59:551
3511221993720,12cyclictest438-20systemd-journal12:35:001
3511221993716,9cyclictest26-21ksoftirqd/107:45:021
3511221993716,8cyclictest3578744-21tr10:20:141
3511221993716,17cyclictest3556309-21open_inodes09:15:261
3511221993715,18cyclictest3592631-21cpuspeed_turbos11:00:151
3511221993710,11cyclictest3566969-21basename09:45:241
3511221993710,11cyclictest3548455-21cat08:55:031
3511221993710,11cyclictest3548455-21cat08:55:021
3511224993619,13cyclictest34-21ksoftirqd/208:01:042
3511224993614,10cyclictest141rcu_preempt11:51:262
351122499361,20cyclictest34-21ksoftirqd/211:55:032
3511221993627,5cyclictest3548628-21apt08:55:141
3511221993624,8cyclictest1753-21gnome-shell09:34:191
3511221993622,10cyclictest26-21ksoftirqd/108:24:021
3511221993619,13cyclictest3574040-21missed_timers10:05:241
3511221993610,12cyclictest26-21ksoftirqd/108:12:071
3511221993610,12cyclictest26-21ksoftirqd/108:12:061
162399360,25rtkit-daemon2197014-21PK-Backend07:33:102
35857282350,9chrt3585731-21cpuspeed_turbos10:40:171
351122499351,19cyclictest34-21ksoftirqd/212:12:202
3511221993519,12cyclictest3627605-21sshd12:38:331
3511221993519,12cyclictest3535979-21date08:20:011
3511221993519,12cyclictest3519642-21which07:30:281
351122199351,8cyclictest906-21gmain10:15:311
3511221993518,13cyclictest855-21dbus-daemon07:25:011
3511221993518,13cyclictest3563140-21cat09:35:151
3511221993518,12cyclictest438-20systemd-journal08:39:331
3511221993518,12cyclictest3601544-21cpuspeed_turbos11:25:161
3511221993517,14cyclictest3580869-21kernelversion10:25:231
3511221993516,7cyclictest141rcu_preempt11:20:091
351122199351,21cyclictest141rcu_preempt07:50:101
3511221993511,6cyclictest141rcu_preempt12:29:131
35270572340,4sleep22197014-21PK-Backend07:52:222
3511224993415,5cyclictest141rcu_preempt10:47:432
3511221993422,8cyclictest1753-21gnome-shell08:40:161
3511221993419,10cyclictest3553773-21date09:10:031
3511221993417,13cyclictest3597311-21grep11:11:321
3511221993417,12cyclictest438-20systemd-journal12:23:331
351122199341,21cyclictest1753-21gnome-shell07:47:531
3511221993322,8cyclictest0-21swapper/111:36:551
3511221993319,10cyclictest3619822-21cat12:15:211
3511221993216,12cyclictest3531214-21cstates08:05:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional