You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-08 - 14:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 08, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
225942238166,59sleep20-21swapper/207:09:472
224652235163,24sleep30-21swapper/307:08:113
224112205167,25sleep10-21swapper/107:07:291
225422204167,24sleep00-21swapper/007:09:080
1712521850,4sleep22271299cyclictest09:20:202
99212610,2sleep20-21swapper/207:50:242
119312570,3sleep22271299cyclictest07:55:222
298552550,3sleep22271299cyclictest11:05:012
261672540,1sleep10-21swapper/109:40:271
2271699451,31cyclictest18077-21basename09:25:003
2271699441,26cyclictest28230-21cat12:15:013
2271699406,24cyclictest14537-21/usr/sbin/munin10:30:253
2271699391,34cyclictest814-21sshd07:11:413
2271699391,27cyclictest9388-21grep07:50:173
2271699391,26cyclictest30527-21sendmail_mailqu09:50:253
2271699391,26cyclictest1141-21sed07:30:263
2271699381,24cyclictest4470-21awk08:50:273
2271699381,22cyclictest4607-21ntp_states10:05:213
2271699381,22cyclictest0-21swapper/311:50:273
2271699371,25cyclictest18708-21tune2fs09:25:153
2271699371,22cyclictest27732-21/usr/sbin/munin11:00:233
2271699371,21cyclictest26094-21wc12:10:013
2271699371,21cyclictest0-21swapper/310:10:293
2271699363,18cyclictest0-21swapper/309:39:373
2271699362,22cyclictest12050irq/123-ahci[0012:35:013
2271699361,32cyclictest1976-21/usr/sbin/munin11:15:143
2271699361,25cyclictest21420-21chrt11:58:463
2271699361,24cyclictest12745-21munin-run11:40:013
2271699361,21cyclictest5648-21gunzip07:40:243
2271699361,21cyclictest14942-21turbostat11:49:563
2271699361,20cyclictest30605-21latency_hist08:40:013
2271699361,20cyclictest15427-21/usr/sbin/munin08:05:133
2271699361,20cyclictest0-21swapper/308:45:253
2271699361,18cyclictest11479-21diskmemload12:03:103
2271699352,20cyclictest20515-21users10:40:273
2271699351,30cyclictest25338-21grep08:25:253
2271699351,30cyclictest25338-21grep08:25:253
2271699351,24cyclictest28717-21sshd09:45:413
2271699351,21cyclictest25952-21grep09:40:223
2271699351,21cyclictest10364-21chrt11:32:523
2271699351,20cyclictest17934-21processes10:35:243
2271699351,20cyclictest17934-21processes10:35:243
2271699351,18cyclictest0-21swapper/311:25:223
2271699342,28cyclictest1303-40dnf10:00:103
2271699341,30cyclictest780-21rngd07:28:253
2271699341,29cyclictest25794-21/usr/sbin/munin07:15:133
2271699341,29cyclictest13900-21/usr/sbin/munin09:15:243
2271699341,20cyclictest8427-21tune2fs10:15:163
2271699341,20cyclictest31776-21expr09:55:133
2271699341,19cyclictest0-21swapper/310:47:513
22700993424,2cyclictest30533-21kworker/0:108:49:320
2271699332,26cyclictest32489-21diskstats11:10:153
2271699332,20cyclictest31032-21expr08:40:133
2271699331,19cyclictest20878-21fschecks_count09:30:153
2271699331,19cyclictest20878-21fschecks_count09:30:153
2271699331,18cyclictest4346-21cat07:40:013
2271699330,5cyclictest0-21swapper/312:40:013
2271699322,17cyclictest25297-21latency_hist10:55:013
2271699322,15cyclictest0-21swapper/308:15:393
2271699321,28cyclictest17630-21/usr/sbin/munin08:10:233
2271699321,28cyclictest10137-21/usr/sbin/munin10:20:163
2271699321,27cyclictest27992-21/usr/sbin/munin07:20:163
2271699321,15cyclictest0-21swapper/311:24:463
2270099321,21cyclictest5614-21grep10:10:020
22716993114,3cyclictest25688-21tr10:55:153
2271699311,27cyclictest30305-21/usr/sbin/munin11:05:153
2271699311,27cyclictest12972-21/usr/sbin/munin11:40:193
2271699311,27cyclictest12972-21/usr/sbin/munin11:40:193
2271699311,26cyclictest29636-21smart_sda12:15:233
22700993120,2cyclictest17092-21kworker/0:208:11:080
2270099310,22cyclictest10421-21cat10:20:140
2271699305,16cyclictest780-21rngd08:56:193
2271699303,19cyclictest86850irq/129-enp1s0-08:30:143
2271699302,25cyclictest30924-21diskstats12:20:153
2271699301,26cyclictest7293-21/usr/sbin/munin09:00:233
2271699301,26cyclictest6619-21/usr/sbin/munin07:45:223
2271699301,26cyclictest307-21/usr/sbin/munin12:25:163
2271699301,26cyclictest12338-21/usr/sbin/munin10:25:253
2271699301,26cyclictest11020-21/usr/sbin/munin07:55:213
2271699300,27cyclictest11703-21/usr/sbin/munin09:10:183
2270099300,21cyclictest0-21swapper/012:35:180
2271699291,25cyclictest9563-21/usr/sbin/munin09:05:123
2271699291,25cyclictest22019-21/usr/sbin/munin08:20:203
2271699291,25cyclictest13218-21/usr/sbin/munin08:00:123
2271299292,2cyclictest9978-21fschecks_time09:05:152
2271299291,11cyclictest0-21swapper/212:15:232
2271299291,11cyclictest0-21swapper/207:49:372
2271299290,14cyclictest780-21rngd08:46:352
2270099292,16cyclictest29793-21grep08:35:250
2270099291,4cyclictest20670-21kworker/0:210:51:290
22700992910,9cyclictest86850irq/129-enp1s0-10:16:360
22700992910,8cyclictest8569-21/usr/sbin/munin11:30:120
22700992910,16cyclictest2383-21munin-run12:30:000
22712992816,3cyclictest5523-21kworker/2:212:04:592
2271299281,12cyclictest0-21swapper/211:32:492
2271299280,11cyclictest0-21swapper/209:40:152
2270599286,11cyclictest14634-21snmpd08:50:291
2270099286,2cyclictest30533-21kworker/0:108:50:320
22700992818,2cyclictest20221-21kworker/0:009:45:150
2271299271,12cyclictest0-21swapper/212:39:392
2271299271,11cyclictest0-21swapper/212:10:252
2271299271,10cyclictest0-21swapper/211:07:132
2270099279,2cyclictest8416-21kworker/0:108:05:060
2270099278,8cyclictest30533-21kworker/0:108:40:220
2270099273,3cyclictest8416-21kworker/0:107:50:240
2270099273,14cyclictest4424-21/usr/sbin/munin07:40:250
2270099270,18cyclictest0-21swapper/011:09:530
2271299261,13cyclictest0-21swapper/211:43:032
2271299261,13cyclictest0-21swapper/211:43:032
2271299261,10cyclictest23357-21sed08:20:252
2271299261,10cyclictest0-21swapper/209:34:032
2271299261,10cyclictest0-21swapper/209:34:032
2271299261,10cyclictest0-21swapper/208:14:352
2270599261,17cyclictest25268-21taskset10:54:451
2270599261,11cyclictest7479-21ls09:00:151
2270099268,15cyclictest6160-21perf11:25:000
2270099264,10cyclictest20002-21kworker/u8:110:00:010
2270099263,16cyclictest20002-21kworker/u8:109:34:590
2270099263,16cyclictest20002-21kworker/u8:109:34:590
2270099262,18cyclictest0-21swapper/010:10:290
22700992612,4cyclictest19822-21/usr/sbin/munin08:15:170
2270099261,15cyclictest7985-21kworker/0:311:45:350
2271299251,10cyclictest0-21swapper/208:28:352
2271299251,10cyclictest0-21swapper/208:28:352
2270599259,8cyclictest21546-21kworker/u8:008:20:061
2270599251,21cyclictest4958-21grep12:35:151
2270599251,16cyclictest0-21swapper/109:52:281
2270099253,4cyclictest23600-21kworker/0:212:04:590
2270099253,13cyclictest88150irq/134-enp2s0-08:01:120
2270099252,15cyclictest21730-21kworker/0:108:28:160
2270099252,15cyclictest21730-21kworker/0:108:28:160
2270099251,13cyclictest0-21swapper/011:35:250
2270099251,11cyclictest16152-21chrt10:30:390
2270099250,14cyclictest5092-21/usr/sbin/munin08:55:210
23332240,1sleep00-21swapper/007:35:140
2271299241,9cyclictest32425-21unixbench_singl08:41:032
2271299241,9cyclictest0-21swapper/207:26:412
2271299241,8cyclictest0-21swapper/211:10:192
2271299241,7cyclictest0-21swapper/210:07:232
2271299241,6cyclictest5344-21ps11:20:252
2271299241,6cyclictest0-21swapper/209:36:462
2271299241,10cyclictest0-21swapper/209:00:012
2271299241,10cyclictest0-21swapper/208:04:192
2270599244,5cyclictest29334-40dnf11:00:341
2270599241,9cyclictest0-21swapper/108:06:511
2270599241,4cyclictest5774-21hwlatdetect08:55:191
2270599241,16cyclictest0-21swapper/110:58:511
2270099249,7cyclictest12338-21/usr/sbin/munin10:25:220
22700992414,2cyclictest88150irq/134-enp2s0-12:16:090
2270099241,17cyclictest0-21swapper/009:37:030
2270099241,14cyclictest22019-21/usr/sbin/munin08:20:240
2270099240,13cyclictest0-21swapper/009:28:350
2270099240,11cyclictest14473-21cut09:15:150
2271299239,5cyclictest30-21rcuc/212:10:012
2271299239,3cyclictest162182awk11:45:212
2271299233,3cyclictest30-21rcuc/212:20:262
2271299231,12cyclictest0-21swapper/211:52:192
2271299231,10cyclictest0-21swapper/210:25:032
2270599230,5cyclictest0-21swapper/111:56:181
2270599230,18cyclictest86850irq/129-enp1s0-09:11:271
2270099236,8cyclictest0-21swapper/011:10:160
2270099236,12cyclictest18630-21swap11:50:260
2270099236,11cyclictest0-21swapper/011:03:540
2270099233,9cyclictest0-21swapper/009:40:250
2270099233,2cyclictest31188-21kworker/0:110:01:370
2270099232,10cyclictest22906-21sshd10:47:410
2270099231,9cyclictest11247-21kworker/0:009:11:510
2270099231,7cyclictest317-21kworker/0:209:00:340
2270099231,15cyclictest0-21swapper/011:44:420
2270099231,15cyclictest0-21swapper/011:44:420
22700992311,3cyclictest87950irq/132-enp2s0-12:12:410
2270099230,17cyclictest0-21swapper/010:37:440
2270099230,17cyclictest0-21swapper/010:37:440
2270099230,11cyclictest28269-21kworker/0:308:34:280
2271299229,5cyclictest86550irq/126-enp1s0-08:30:242
2271299225,4cyclictest2213-21latency10:00:192
2271299221,8cyclictest0-21swapper/208:36:512
2271299221,7cyclictest0-21swapper/209:50:192
2271299221,14cyclictest0-21swapper/209:55:162
2270599223,3cyclictest19743-21/usr/sbin/munin10:40:201
2270599223,16cyclictest31702-21grep07:25:271
2270599223,10cyclictest88050irq/133-enp2s0-11:51:501
2270599222,17cyclictest7517-21grep11:25:271
2270599222,12cyclictest23321-21kworker/u8:011:42:161
2270599222,12cyclictest23321-21kworker/u8:011:42:161
2270599222,11cyclictest1372-21/usr/sbin/munin10:00:131
22705992217,2cyclictest258472sleep112:08:521
2270599221,6cyclictest10319-21cstates10:20:151
2270599221,5cyclictest0-21swapper/112:27:421
22705992214,3cyclictest0-21swapper/110:39:231
22705992214,3cyclictest0-21swapper/110:39:221
2270599221,3cyclictest0-21swapper/112:21:491
2270599221,2cyclictest10488-21strings09:05:211
22705992210,3cyclictest22829-21ntp_states12:00:231
2270099229,2cyclictest32061-21kworker/0:007:29:360
2270099226,12cyclictest1003-21cut07:30:250
2270099224,6cyclictest86850irq/129-enp1s0-11:15:160
2270099223,3cyclictest88050irq/133-enp2s0-07:24:150
2270099223,2cyclictest13518-21kworker/0:109:22:490
2270099222,2cyclictest28931-21kworker/0:209:50:170
2270099221,9cyclictest8416-21kworker/0:107:49:200
22700992211,5cyclictest86750irq/128-enp1s0-12:05:130
2271299211,9cyclictest0-21swapper/207:17:032
2271299211,10cyclictest0-21swapper/209:25:072
2270599212,11cyclictest6415-21hddtemp_smartct10:10:181
2270599212,10cyclictest23011-21cpuspeed_turbos07:10:131
2270599211,5cyclictest0-21swapper/112:32:481
22705992114,2cyclictest0-21swapper/108:44:381
2270599211,3cyclictest2648-21hddtemp_smartct11:15:181
2270599211,18cyclictest0-21swapper/111:06:321
2270599211,18cyclictest0-21swapper/108:39:091
2270599211,17cyclictest32135-21runrttasks11:10:101
2270599211,11cyclictest20794-21expr09:30:151
2270599211,11cyclictest20794-21expr09:30:151
2270599211,11cyclictest15348-21irqstats10:30:201
2270099215,3cyclictest0-21swapper/011:25:180
2270099213,5cyclictest27578-21taskset07:17:470
2270099211,5cyclictest0-21swapper/010:40:280
2270099211,14cyclictest0-21swapper/010:55:230
2270099210,12cyclictest0-21swapper/007:10:220
6142201,2sleep10-21swapper/107:30:211
2271299206,7cyclictest30-21rcuc/210:40:192
2271299206,3cyclictest30-21rcuc/211:56:192
2271299205,5cyclictest12050irq/123-ahci[0010:55:012
2271299204,8cyclictest12050irq/123-ahci[0010:20:002
2271299204,5cyclictest22499-21kworker/2:009:45:152
2271299203,9cyclictest11479-21diskmemload10:20:282
2271299201,9cyclictest0-21swapper/210:48:542
2271299201,4cyclictest0-21swapper/211:39:132
22712992014,3cyclictest6399-21sed10:10:162
2270599207,11cyclictest31847-21kworker/u8:108:15:021
2270599206,4cyclictest11242-21fschecks_time11:35:171
2270599205,7cyclictest22687-21cyclictest12:16:031
2270599205,7cyclictest11479-21diskmemload09:25:241
2270599203,3cyclictest23803-21/usr/sbin/munin09:35:271
2270599202,3cyclictest12931-40dnf07:59:341
22705992016,3cyclictest0-21swapper/110:06:201
22705992016,3cyclictest0-21swapper/107:48:341
22705992013,2cyclictest0-21swapper/110:46:281
22705992012,3cyclictest0-21swapper/109:56:181
22705992012,3cyclictest0-21swapper/109:45:031
2270599201,18cyclictest0-21swapper/110:26:081
2270599201,18cyclictest0-21swapper/107:35:221
2270599201,17cyclictest0-21swapper/111:45:141
2270599201,15cyclictest0-21swapper/111:33:471
2270599200,19cyclictest0-21swapper/108:32:481
2270599200,15cyclictest0-21swapper/108:47:441
2270099209,2cyclictest88150irq/134-enp2s0-09:08:110
22700992011,4cyclictest0-21swapper/012:33:140
2270099201,10cyclictest30575-21/usr/sbin/munin12:20:190
22700992010,5cyclictest12938-21taskset07:59:340
2271299199,2cyclictest86750irq/128-enp1s0-09:15:202
2271299198,3cyclictest86550irq/126-enp1s0-08:05:142
2271299197,5cyclictest86850irq/129-enp1s0-10:35:192
2271299197,5cyclictest86850irq/129-enp1s0-10:35:182
2271299197,5cyclictest30-21rcuc/208:19:562
2271299191,9cyclictest5523-21kworker/2:210:30:272
2271299191,6cyclictest4187-21turbostat.cron07:44:562
2271299191,6cyclictest1273-21smartctl07:30:282
22712991915,3cyclictest32-21ksoftirqd/212:25:232
22712991913,3cyclictest24422-21grep07:10:272
2271299191,10cyclictest0-21swapper/208:51:232
2270599192,9cyclictest27992-21/usr/sbin/munin07:20:121
2270599192,3cyclictest13893-21hddtemp_smartct08:00:171
2270599191,8cyclictest16850-21if_enp2s009:20:181
22705991916,2cyclictest0-21swapper/107:18:081
22705991915,3cyclictest0-21swapper/108:10:461
2270599191,2cyclictest0-21swapper/112:12:291
2270599191,17cyclictest0-21swapper/111:20:271
2270599191,17cyclictest0-21swapper/109:19:261
2270599191,17cyclictest0-21swapper/107:50:041
2270599191,17cyclictest0-21swapper/107:44:031
2270099193,8cyclictest19563-21/usr/sbin/munin11:55:140
2271299187,2cyclictest88050irq/133-enp2s0-12:33:292
2271299185,3cyclictest5523-21kworker/2:210:55:152
2271299183,8cyclictest0-21swapper/207:20:272
2271299181,9cyclictest0-21swapper/211:25:172
2271299181,5cyclictest7293-21/usr/sbin/munin09:00:202
22712991813,3cyclictest30-21rcuc/207:37:442
2271299181,11cyclictest0-21swapper/211:15:152
2271299181,11cyclictest0-21swapper/209:10:092
22705991815,2cyclictest0-21swapper/108:27:081
22705991815,2cyclictest0-21swapper/108:27:081
22705991812,3cyclictest8267-21df10:15:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional