You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 22:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Nov 21, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
120772238163,24sleep30-21swapper/307:07:363
121672235164,57sleep00-21swapper/007:08:450
121922206169,24sleep20-21swapper/207:09:042
121152204167,24sleep10-21swapper/107:08:061
1666921310,4sleep01235599cyclictest07:15:230
321252680,3sleep132123-21cat07:50:291
163532540,4sleep316355-21fschecks_time12:10:163
12367995147,3cyclictest0-21swapper/212:30:292
227842480,4sleep31237299cyclictest07:30:153
275372450,2sleep10-21swapper/112:35:151
12367994239,2cyclictest0-21swapper/210:06:152
12367993935,3cyclictest0-21swapper/208:18:172
143702360,9sleep07-21ksoftirqd/012:05:190
170092330,1sleep30-21swapper/307:15:263
1235599313,13cyclictest21615-21munin-run11:10:010
1235599302,16cyclictest18059-21munin-run08:35:010
12367992911,9cyclictest0-21swapper/210:30:212
12362992917,8cyclictest0-21swapper/112:12:521
12355992916,4cyclictest0-21swapper/009:40:100
12362992818,3cyclictest0-21swapper/110:19:201
12362992818,3cyclictest0-21swapper/110:19:191
12362992816,6cyclictest0-21swapper/111:35:421
12362992719,3cyclictest0-21swapper/110:32:271
12362992718,3cyclictest0-21swapper/111:17:131
1236299271,4cyclictest4932-21/usr/sbin/munin11:45:131
28902260,5sleep339-21rcuc/311:40:133
1236799263,6cyclictest16507-40dnf07:15:242
1236799261,6cyclictest0-21swapper/211:17:132
1236799261,6cyclictest0-21swapper/210:37:422
1236799261,5cyclictest0-21swapper/209:40:102
1236799261,3cyclictest0-21swapper/209:52:212
1236299268,3cyclictest836-21runrttasks12:09:501
1235599261,10cyclictest0-21swapper/011:25:220
12367992522,2cyclictest0-21swapper/207:38:092
1236799251,5cyclictest0-21swapper/211:35:422
1236799251,4cyclictest0-21swapper/209:17:092
1236299252,2cyclictest86850irq/129-enp1s0-09:10:541
12362992512,11cyclictest0-21swapper/110:09:441
1236299251,19cyclictest0-21swapper/109:26:231
12355992516,3cyclictest0-21swapper/009:17:090
87792240,4sleep3401ktimersoftd/309:25:173
1237299248,3cyclictest10855-21kworker/3:007:46:573
12367992421,2cyclictest0-21swapper/211:22:242
12367992411,3cyclictest86550irq/126-enp1s0-10:25:202
12362992414,3cyclictest0-21swapper/109:47:571
1236299241,3cyclictest0-21swapper/108:42:521
1236299241,20cyclictest27952-21cpuspeed_turbos10:10:121
1236299241,11cyclictest0-21swapper/111:04:121
1236299240,5cyclictest0-21swapper/112:32:371
1236299240,5cyclictest0-21swapper/110:23:271
1236299240,4cyclictest0-21swapper/110:29:051
1236299240,17cyclictest0-21swapper/108:00:231
1236299240,17cyclictest0-21swapper/108:00:231
1235599241,15cyclictest0-21swapper/012:14:570
1235599241,15cyclictest0-21swapper/009:38:200
1236799239,3cyclictest32-21ksoftirqd/211:30:012
12367992310,2cyclictest86750irq/128-enp1s0-08:00:242
12367992310,2cyclictest86750irq/128-enp1s0-08:00:242
1236799230,5cyclictest0-21swapper/210:19:192
1236799230,5cyclictest0-21swapper/210:19:192
12362992317,3cyclictest13386-21diskstats10:50:161
1236299231,3cyclictest9033-21latency_hist11:55:021
1236299231,3cyclictest22754-21memory11:10:181
1236299231,17cyclictest0-21swapper/110:45:451
12362992311,3cyclictest0-21swapper/109:56:391
1236299231,11cyclictest0-21swapper/109:31:121
1236299231,10cyclictest30811-21cat07:50:011
1236299230,17cyclictest0-21swapper/111:05:331
1236299230,17cyclictest0-21swapper/109:36:041
1236299230,16cyclictest0-21swapper/111:32:541
1236299230,15cyclictest0-21swapper/112:20:311
1236299230,15cyclictest0-21swapper/112:20:301
1236299230,15cyclictest0-21swapper/111:42:011
1235599234,6cyclictest780-21rngd08:10:200
12355992314,3cyclictest0-21swapper/010:30:280
12355992312,3cyclictest0-21swapper/011:30:390
1235599230,9cyclictest0-21swapper/011:55:180
1237299222,6cyclictest14815-1kworker/u9:110:02:263
1237299221,8cyclictest0-21swapper/310:47:583
1237299221,6cyclictest0-21swapper/312:25:023
1237299221,6cyclictest0-21swapper/309:20:253
1237299221,4cyclictest0-21swapper/311:39:153
1236799226,4cyclictest22753-21cat08:45:132
1236799224,5cyclictest8467-21timerwakeupswit11:50:252
1236799221,7cyclictest0-21swapper/207:20:512
1236799221,11cyclictest0-21swapper/211:31:262
1236799220,5cyclictest0-21swapper/211:59:012
1236299229,2cyclictest26775-21sshd08:54:181
12362992218,3cyclictest0-21swapper/110:59:281
12362992216,3cyclictest16442-21taskset09:43:291
12362992211,7cyclictest0-21swapper/109:22:271
1236299221,10cyclictest19071-21timerwakeupswit07:20:231
12362992210,3cyclictest0-21swapper/111:28:541
12362992210,2cyclictest22554-21cpuspeed_turbos12:25:111
1236299220,4cyclictest0-21swapper/110:37:171
1236299220,20cyclictest0-21swapper/107:25:201
1236299220,15cyclictest0-21swapper/112:02:541
1235599228,8cyclictest0-21swapper/011:46:550
261752210,5sleep31237299cyclictest08:50:243
1237299216,12cyclictest11460-21/usr/sbin/munin12:00:133
1237299213,5cyclictest14815-1kworker/u9:109:32:063
1237299211,5cyclictest14815-1kworker/u9:110:36:453
1237299211,4cyclictest14815-1kworker/u9:112:37:223
1236799217,3cyclictest86550irq/126-enp1s0-08:30:272
1236799215,5cyclictest32-21ksoftirqd/212:20:242
1236799215,5cyclictest32-21ksoftirqd/212:20:242
1236799211,6cyclictest0-21swapper/211:06:482
1236799211,6cyclictest0-21swapper/209:55:152
1236799211,5cyclictest0-21swapper/212:12:522
1236799211,4cyclictest0-21swapper/212:28:062
1236799211,4cyclictest0-21swapper/212:16:562
1236799210,5cyclictest0-21swapper/211:11:252
12362992116,3cyclictest26280-21cat07:40:021
1236299211,4cyclictest0-21swapper/109:51:081
1236299211,2cyclictest836-21runrttasks12:17:521
1236299211,18cyclictest0-21swapper/108:55:191
1236299211,10cyclictest13881-21/usr/sbin/munin08:25:121
1236299210,15cyclictest0-21swapper/110:04:561
1236299210,14cyclictest0-21swapper/111:20:051
1235599217,9cyclictest0-21swapper/010:08:270
1235599214,2cyclictest0-21swapper/007:53:210
1235599211,6cyclictest0-21swapper/011:16:360
12355992115,1cyclictest0-21swapper/010:15:140
12355992115,1cyclictest0-21swapper/010:15:140
1235599211,10cyclictest0-21swapper/012:38:160
1235599211,10cyclictest0-21swapper/010:21:490
1235599210,6cyclictest0-21swapper/011:41:020
45872200,2sleep30-21swapper/310:30:173
1237299207,8cyclictest759-21dbus-daemon09:01:383
1237299206,4cyclictest86550irq/126-enp1s0-10:15:123
1237299206,4cyclictest86550irq/126-enp1s0-10:15:123
1237299202,5cyclictest14815-1kworker/u9:111:20:083
1237299201,8cyclictest0-21swapper/308:30:173
1237299201,6cyclictest0-21swapper/309:36:203
12372992014,3cyclictest5495-21grep08:05:123
1237299201,11cyclictest20276-21/usr/sbin/munin12:20:203
1237299201,11cyclictest20276-21/usr/sbin/munin12:20:203
1237299201,11cyclictest19846-21/usr/sbin/munin07:25:153
1237299201,10cyclictest3786-21memory08:00:193
1237299201,10cyclictest3786-21memory08:00:193
1236799208,3cyclictest0-21swapper/210:50:022
1236799205,3cyclictest86750irq/128-enp1s0-09:10:212
1236799204,4cyclictest32-21ksoftirqd/212:05:192
1236799204,4cyclictest18767-21switchtime11:00:322
1236799203,4cyclictest0-21swapper/210:50:222
1236799202,6cyclictest28037-21users07:40:292
1236799202,5cyclictest3586-21memory11:40:192
1236799201,6cyclictest30838-21/usr/sbin/munin07:50:162
1236799201,6cyclictest29845-21perl09:00:152
1236799201,6cyclictest0-21swapper/209:32:262
1236799201,4cyclictest0-21swapper/210:40:452
1236799201,10cyclictest0-21swapper/210:10:052
1236299204,9cyclictest0-21swapper/111:55:221
1236299202,8cyclictest0-21swapper/109:16:241
1236299201,6cyclictest1660-21strings07:55:181
12362992014,3cyclictest8450-21wc08:10:221
1236299201,18cyclictest0-21swapper/108:37:451
1236299201,18cyclictest0-21swapper/107:41:091
1236299201,18cyclictest0-21swapper/107:17:341
1236299201,17cyclictest0-21swapper/108:17:051
1236299200,19cyclictest0-21swapper/107:33:501
1235599207,3cyclictest0-21swapper/012:15:270
1235599207,2cyclictest0-21swapper/010:55:160
1235599207,10cyclictest4556-21sshd08:01:210
1235599207,10cyclictest4556-21sshd08:01:210
1235599206,7cyclictest0-21swapper/009:50:500
1235599206,3cyclictest0-21swapper/010:53:320
1235599205,6cyclictest1116-21hddtemp_smartct11:35:160
1235599205,3cyclictest86750irq/128-enp1s0-10:10:220
1235599205,3cyclictest0-21swapper/009:20:210
1235599202,9cyclictest780-21rngd12:25:590
1235599201,9cyclictest0-21swapper/009:13:560
1235599201,8cyclictest0-21swapper/010:40:460
1235599201,7cyclictest13881-21/usr/sbin/munin08:25:200
12355992015,3cyclictest23604-21grep07:30:250
12355992013,2cyclictest0-21swapper/012:00:140
1235599201,13cyclictest0-21swapper/010:49:110
1235599201,13cyclictest0-21swapper/009:59:580
1235599201,12cyclictest0-21swapper/011:00:310
12355992011,2cyclictest0-21swapper/009:05:280
1235599201,10cyclictest18567-21memory07:20:170
12355992010,3cyclictest18281-21/usr/sbin/munin08:35:200
265862191,2sleep00-21swapper/012:34:300
1237299198,3cyclictest14634-21snmpd08:42:473
1237299198,3cyclictest12559-21/usr/sbin/munin07:10:243
1237299195,7cyclictest780-21rngd10:27:453
1237299194,5cyclictest0-21swapper/307:55:233
1237299192,6cyclictest10608-21munin-run10:45:013
1237299191,8cyclictest0-21swapper/311:05:373
1237299191,7cyclictest0-21swapper/311:33:063
1237299191,7cyclictest0-21swapper/309:15:163
1237299191,5cyclictest10996-21sed08:15:283
1237299191,4cyclictest0-21swapper/310:58:303
1237299191,10cyclictest24671-21/usr/sbin/munin12:30:173
1237299191,10cyclictest18074-21/usr/sbin/munin12:15:203
1237299191,10cyclictest14602-21/usr/sbin/munin09:40:153
1237299190,7cyclictest21993-21kworker/u8:210:15:013
1236799194,7cyclictest0-21swapper/210:01:462
1236799194,4cyclictest2660-21awk08:00:012
1236799194,4cyclictest1332-21unixbench_multi10:20:262
1236799191,8cyclictest27002-21cat08:55:012
1236799191,8cyclictest0-21swapper/212:00:292
1236799191,6cyclictest28752-21chrt12:39:052
12367991913,4cyclictest13653-21dump-pmu-power08:25:012
12367991912,4cyclictest21962-21sed08:40:262
1236299192,4cyclictest23742-21ntp_states08:45:221
12362991915,3cyclictest0-21swapper/110:40:161
12362991912,3cyclictest12069-21fschecks_count08:20:171
1236299191,17cyclictest0-21swapper/108:32:081
1236299191,17cyclictest0-21swapper/107:12:221
1235599197,3cyclictest0-21swapper/011:14:540
1235599197,3cyclictest0-21swapper/008:41:500
1235599196,8cyclictest0-21swapper/009:33:140
1235599196,3cyclictest17871-21chrt09:45:210
1235599195,8cyclictest0-21swapper/011:50:080
1235599193,3cyclictest86750irq/128-enp1s0-10:35:210
1235599191,9cyclictest0-21swapper/009:26:130
1235599191,9cyclictest0-21swapper/007:59:000
12355991914,3cyclictest2037-21/usr/sbin/munin10:25:140
12355991912,4cyclictest12017-21/usr/sbin/munin08:20:140
1235599190,9cyclictest0-21swapper/008:46:210
1235599190,17cyclictest0-21swapper/011:20:480
61562180,4sleep230-21rcuc/208:05:192
60852181,3sleep00-21swapper/008:05:180
1237299188,2cyclictest7065-21/usr/sbin/munin11:50:163
1237299187,3cyclictest14634-21snmpd07:37:343
1237299185,8cyclictest0-21swapper/307:20:153
1237299182,6cyclictest14349-21hddtemp_smartct12:05:183
1237299181,9cyclictest14634-21snmpd08:12:213
1237299181,9cyclictest0-21swapper/311:10:173
1237299181,8cyclictest0-21swapper/311:25:373
1237299181,8cyclictest0-21swapper/311:15:153
1237299181,6cyclictest15856-21perf08:30:013
1237299181,6cyclictest0-21swapper/310:06:453
1237299181,5cyclictest0-21swapper/311:57:083
1237299181,5cyclictest0-21swapper/309:11:293
12372991812,4cyclictest20245-21crond08:40:013
1236799185,3cyclictest86850irq/129-enp1s0-09:25:142
1236799183,3cyclictest0-21swapper/210:55:292
1236799182,6cyclictest18389-21users09:45:252
1236799181,8cyclictest0-21swapper/209:21:382
1236799181,6cyclictest0-21swapper/211:45:272
1236799181,6cyclictest0-21swapper/209:06:452
1236799181,6cyclictest0-21swapper/207:34:462
1236799181,5cyclictest14526-21latency_hist07:15:022
1236799181,5cyclictest13945-21unixbench_singl09:35:262
12362991812,3cyclictest6667-21who08:05:231
1236299181,16cyclictest0-21swapper/109:05:091
1236299181,16cyclictest0-21swapper/109:02:071
1235599185,3cyclictest86750irq/128-enp1s0-12:20:220
1235599185,3cyclictest86750irq/128-enp1s0-12:20:220
1235599181,6cyclictest0-21swapper/008:15:210
1235599181,4cyclictest24873-21/usr/sbin/munin08:50:240
1235599181,11cyclictest0-21swapper/010:00:210
1235599181,11cyclictest0-21swapper/007:10:160
215722170,3sleep00-21swapper/007:27:000
1237299176,3cyclictest14815-1kworker/u9:110:21:443
1237299175,3cyclictest13935-21latency10:50:203
1237299173,12cyclictest41-21ksoftirqd/311:45:203
1237299171,9cyclictest0-21swapper/309:50:123
1237299171,7cyclictest31660-21grep09:05:163
1237299171,6cyclictest0-21swapper/311:02:493
1237299171,6cyclictest0-21swapper/309:47:503
1237299171,6cyclictest0-21swapper/308:20:273
1237299171,4cyclictest24156-21unixbench-2d08:45:273
1237299171,3cyclictest0-21swapper/307:41:263
1237299171,2cyclictest0-21swapper/309:57:123
12372991711,3cyclictest31606-21if_err_enp1s007:50:183
12367991712,3cyclictest18617-21sh08:35:132
12367991712,3cyclictest14197-21ls08:25:132
1236799171,10cyclictest0-21swapper/208:55:202
1235599171,3cyclictest27050-21grep07:40:190
1235599171,3cyclictest0-21swapper/009:03:360
1235599171,2cyclictest29022-21grep07:45:130
1237299169,4cyclictest27440-21fschecks_count08:55:153
1236799161,5cyclictest30-21rcuc/208:10:172
1236799161,5cyclictest30193-21unixbench_singl07:45:242
12367991610,3cyclictest19996-21grep07:25:102
12355991612,2cyclictest0-21swapper/007:35:230
12355991610,3cyclictest27881-21awk08:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional