You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 22:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 01, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
228062236163,24sleep30-21swapper/307:09:083
228102232162,57sleep20-21swapper/207:09:112
225582227154,60sleep00-21swapper/007:05:550
226592207169,25sleep10-21swapper/107:07:131
1119721600,4sleep12296999cyclictest11:35:121
1040021350,3sleep12296999cyclictest09:05:191
22974999793,3cyclictest0-21swapper/210:51:082
185572610,7sleep1221ktimersoftd/108:10:151
67742530,4sleep36775-21unixbench-2d08:55:253
272942510,1sleep00-21swapper/007:15:230
22962993910,12cyclictest7011-21basename12:40:010
251712370,3sleep10-21swapper/108:25:171
141432360,1sleep20-21swapper/210:25:282
2297499341,23cyclictest29814-21unixbench_singl07:20:262
2296999341,13cyclictest0-21swapper/107:30:191
2296999331,16cyclictest0-21swapper/108:20:221
2296999331,15cyclictest22192-21latency10:45:191
2296999331,12cyclictest0-21swapper/111:15:241
2296999321,13cyclictest30738-21cat12:20:021
2296999311,15cyclictest0-21swapper/110:20:171
22979993012,15cyclictest14516-21idleruntime-cro10:30:013
2296999308,3cyclictest0-21swapper/109:32:281
2296999305,3cyclictest3198-40dnf10:00:591
2296999292,12cyclictest7934-21munin-run10:15:001
2296999291,2cyclictest0-21swapper/111:20:301
2296999287,19cyclictest8154-21sed09:00:161
2296999287,19cyclictest8154-21sed09:00:161
22969992811,3cyclictest28328-21netstat09:45:211
22969992811,3cyclictest16196-21latency_hist09:20:021
2296999281,12cyclictest0-21swapper/112:24:021
2296999281,12cyclictest0-21swapper/111:09:101
2296999281,12cyclictest0-21swapper/111:09:101
2296999281,12cyclictest0-21swapper/111:00:061
2296999281,12cyclictest0-21swapper/110:30:271
2296999281,11cyclictest0-21swapper/110:36:031
2296999281,11cyclictest0-21swapper/109:37:491
240712270,3sleep224076-21pmu-power09:35:222
2296999271,9cyclictest0-21swapper/112:03:401
22969992712,4cyclictest496-21jbd2/dm-0-810:55:231
2296299271,13cyclictest27722-21munin-run11:00:010
2296999269,3cyclictest4031-21memory08:50:191
2296999268,3cyclictest12566-21expr09:10:161
2296999261,3cyclictest0-21swapper/112:28:361
2296999261,12cyclictest0-21swapper/111:41:341
2296999261,11cyclictest0-21swapper/109:24:101
2296999261,10cyclictest0-21swapper/112:38:161
2296299267,16cyclictest8560-21munin-run11:30:000
2296299261,13cyclictest20491-21munin-run09:30:020
2297999251,16cyclictest0-21swapper/310:05:053
2297499254,8cyclictest0-21swapper/211:45:242
2296999251,12cyclictest0-21swapper/107:25:131
2296999251,10cyclictest0-21swapper/111:12:501
2296999251,10cyclictest0-21swapper/108:15:301
2296299256,16cyclictest27102-21munin-run09:45:010
2296299253,9cyclictest2227250kworker/u8:109:10:350
2296299251,5cyclictest0-21swapper/011:49:230
2296299251,4cyclictest0-21swapper/012:12:230
2296299251,4cyclictest0-21swapper/012:12:230
22979992420,3cyclictest0-21swapper/308:19:573
2297999241,3cyclictest0-21swapper/311:40:293
2297499247,3cyclictest0-21swapper/212:05:282
2297499245,6cyclictest88250irq/135-enp2s0-07:15:162
2296999242,5cyclictest0-21swapper/112:05:221
2296999241,7cyclictest0-21swapper/109:26:441
2296999241,13cyclictest0-21swapper/110:28:211
2296999241,10cyclictest0-21swapper/108:42:131
2296299241,6cyclictest0-21swapper/012:26:100
2296299241,6cyclictest0-21swapper/010:14:020
2296299241,5cyclictest0-21swapper/011:50:030
2296299241,11cyclictest9507-21awk09:05:010
2296299241,11cyclictest9507-21awk09:05:010
2296299241,10cyclictest25328-21cut12:05:220
2297999234,4cyclictest24525-21awk09:35:273
2297999231,9cyclictest0-21swapper/309:55:183
22979992318,3cyclictest29379-21if_err_enp1s012:15:193
2297999231,6cyclictest0-21swapper/310:51:233
2297999231,11cyclictest23659-40dnf12:01:153
2297999231,10cyclictest0-21swapper/308:39:433
2297999230,10cyclictest0-21swapper/309:20:193
2297499238,4cyclictest11861-21cat09:10:012
2297499233,9cyclictest0-21swapper/212:01:162
2296999234,3cyclictest0-21swapper/111:56:401
2296999233,3cyclictest0-21swapper/111:45:081
2296999231,6cyclictest0-21swapper/110:42:571
22969992315,3cyclictest0-21swapper/107:20:411
2296999231,16cyclictest0-21swapper/107:41:061
2296999231,14cyclictest10352-21unixbench_singl11:30:281
2296999231,12cyclictest0-21swapper/108:45:261
2296999231,11cyclictest0-21swapper/110:15:131
2296999231,11cyclictest0-21swapper/108:35:171
2296999231,10cyclictest0-21swapper/111:25:181
2296299233,10cyclictest30126-21gltestperf09:50:160
2296299231,6cyclictest0-21swapper/011:22:250
123152231,4sleep312554-21gltestperf09:10:163
44592220,2sleep20-21swapper/211:20:132
300072220,5sleep230010-21pmu-power08:35:292
2297999222,12cyclictest0-21swapper/309:32:483
2297999221,4cyclictest0-21swapper/311:56:293
2297999221,15cyclictest0-21swapper/310:37:263
2297999221,15cyclictest0-21swapper/309:40:203
2297999221,10cyclictest0-21swapper/309:50:183
22979992210,2cyclictest15776-21kworker/3:111:00:103
2297499225,8cyclictest780-21rngd08:10:112
2297499225,6cyclictest0-21swapper/210:48:402
2296999223,4cyclictest27473-21ntp_states12:10:221
2296999223,4cyclictest27473-21ntp_states12:10:221
2296999221,4cyclictest0-21swapper/109:50:371
22969992211,2cyclictest0-21swapper/110:05:151
2296999221,12cyclictest0-21swapper/107:50:221
2296999221,10cyclictest0-21swapper/112:32:461
2296999221,10cyclictest0-21swapper/110:54:141
2296999220,6cyclictest0-21swapper/111:54:201
2296999220,17cyclictest0-21swapper/108:33:511
2296299224,9cyclictest7-21ksoftirqd/008:40:150
2296299223,4cyclictest0-21swapper/012:33:500
2296299222,4cyclictest0-21swapper/010:33:360
2296299221,6cyclictest0-21swapper/012:01:530
2296299221,6cyclictest0-21swapper/011:09:210
2296299221,6cyclictest0-21swapper/011:09:200
2296299221,6cyclictest0-21swapper/010:38:450
2296299221,6cyclictest0-21swapper/009:36:200
2296299221,5cyclictest0-21swapper/010:26:080
2296299221,5cyclictest0-21swapper/009:20:410
2296299221,4cyclictest0-21swapper/010:18:010
22962992212,2cyclictest6862-21sed07:45:010
2296299220,5cyclictest22636-21kworker/u8:009:55:060
42082211,2sleep20-21swapper/208:50:242
2297999218,3cyclictest0-21swapper/307:22:353
2297999218,11cyclictest41-21ksoftirqd/310:47:183
2297999216,3cyclictest0-21swapper/308:44:203
2297999214,2cyclictest86550irq/126-enp1s0-11:16:033
2297999211,9cyclictest20572cat12:25:263
2297999211,7cyclictest0-21swapper/310:04:033
2297999211,6cyclictest0-21swapper/312:05:233
2297999211,5cyclictest0-21swapper/311:45:153
2297999211,14cyclictest0-21swapper/310:22:043
2297999211,10cyclictest0-21swapper/312:10:243
2297999211,10cyclictest0-21swapper/312:10:243
2297999211,10cyclictest0-21swapper/311:11:123
2297999210,15cyclictest0-21swapper/311:20:363
2297499213,3cyclictest88250irq/135-enp2s0-09:25:192
2297499211,8cyclictest5029-21/usr/sbin/munin12:35:182
22974992112,5cyclictest15677-21systemd-journal08:05:072
2296999216,3cyclictest0-21swapper/107:46:211
2296999214,3cyclictest0-21swapper/108:05:061
22969992114,4cyclictest11890-21grep07:55:181
22969992114,4cyclictest11890-21grep07:55:181
2296999211,15cyclictest0-21swapper/108:58:151
2296999211,15cyclictest0-21swapper/108:02:571
2296999211,15cyclictest0-21swapper/107:10:221
2296999211,11cyclictest0-21swapper/107:36:391
2296999211,11cyclictest0-21swapper/107:15:181
2296999210,6cyclictest0-21swapper/109:42:581
2296299214,15cyclictest30855-21munin-run08:40:020
2296299211,8cyclictest5278-21unixbench_multi10:05:280
2296299211,4cyclictest0-21swapper/011:37:570
22962992112,3cyclictest86750irq/128-enp1s0-08:15:180
2297999203,2cyclictest88150irq/134-enp2s0-07:15:143
2297999202,15cyclictest20067-21memory10:40:243
2297999201,9cyclictest9722-21/usr/sbin/munin09:05:153
2297999201,6cyclictest0-21swapper/312:34:213
2297999201,6cyclictest0-21swapper/311:51:403
22979992015,3cyclictest14703-21/usr/sbin/munin09:15:163
22979992013,4cyclictest8116-21grep09:00:163
22979992013,4cyclictest8116-21grep09:00:163
2297999201,2cyclictest88050irq/133-enp2s0-07:13:513
2297999201,14cyclictest0-21swapper/310:11:283
2297999201,13cyclictest0-21swapper/312:37:563
2297999201,13cyclictest0-21swapper/310:16:323
2297999200,13cyclictest0-21swapper/310:55:363
2297499203,6cyclictest0-21swapper/211:07:392
2297499203,6cyclictest0-21swapper/211:07:392
2297499202,7cyclictest0-21swapper/209:48:292
2297499201,9cyclictest0-21swapper/210:41:132
2297499201,7cyclictest0-21swapper/211:40:282
2297499201,7cyclictest0-21swapper/210:21:472
2297499201,7cyclictest0-21swapper/208:42:562
2297499201,7cyclictest0-21swapper/207:58:072
2297499201,7cyclictest0-21swapper/207:58:072
2297499201,6cyclictest0-21swapper/208:25:132
2297499201,13cyclictest0-21swapper/209:55:172
2296999201,11cyclictest31900-21cstates09:55:121
2296299208,3cyclictest1492-21unixbench_multi11:10:260
2296299203,6cyclictest13735-1kworker/u9:011:44:320
2296299202,9cyclictest836450kworker/u8:112:21:230
2296299202,7cyclictest11-21rcuc/009:15:270
2296299202,10cyclictest30457-21/usr/sbin/munin07:25:190
2296299201,5cyclictest0-21swapper/011:55:400
2296299201,5cyclictest0-21swapper/011:33:440
2296299201,3cyclictest964-21kworker/u8:110:00:220
2296299201,11cyclictest0-21swapper/010:45:260
2296299201,11cyclictest0-21swapper/008:45:220
2296299201,10cyclictest28278-21cpu07:20:110
83992190,4sleep230-21rcuc/209:00:172
83992190,4sleep230-21rcuc/209:00:172
2297999193,3cyclictest88150irq/134-enp2s0-08:05:303
2297999192,2cyclictest86650irq/127-enp1s0-08:00:563
2297999191,9cyclictest0-21swapper/312:22:423
2297999191,9cyclictest0-21swapper/311:28:433
2297999191,6cyclictest0-21swapper/308:47:553
22979991916,2cyclictest0-21swapper/307:34:523
22979991914,3cyclictest23281-21grep08:20:213
2297999191,3cyclictest0-21swapper/309:45:573
22979991913,3cyclictest5131-21cut07:40:153
22979991913,3cyclictest25764-21swap08:25:253
22979991912,4cyclictest9431-21sed11:30:183
2297999191,15cyclictest15094-21df_inode10:30:163
2297999191,10cyclictest0-21swapper/311:09:193
2297999191,10cyclictest0-21swapper/311:09:193
2297999190,14cyclictest0-21swapper/309:29:253
2297499196,3cyclictest0-21swapper/208:16:072
2297499196,2cyclictest30-21rcuc/207:40:122
2297499195,3cyclictest86850irq/129-enp1s0-11:55:262
2297499195,3cyclictest86850irq/129-enp1s0-07:35:112
2297499195,3cyclictest0-21swapper/210:00:562
2297499193,5cyclictest0-21swapper/209:20:532
2297499193,3cyclictest86850irq/129-enp1s0-08:30:122
2297499192,6cyclictest4093-21/usr/sbin/munin12:30:252
2297499192,5cyclictest0-21swapper/211:03:302
2297499192,4cyclictest0-21swapper/211:31:142
2297499191,7cyclictest0-21swapper/207:33:002
2297499191,6cyclictest6593-21/usr/sbin/munin11:25:202
2297499191,6cyclictest10983-21/usr/sbin/munin11:35:222
2297499191,6cyclictest0-21swapper/212:21:462
2297499191,6cyclictest0-21swapper/211:51:082
2297499191,6cyclictest0-21swapper/211:10:332
2297499191,6cyclictest0-21swapper/209:18:262
2297499191,6cyclictest0-21swapper/209:12:362
22974991914,3cyclictest5730-21grep08:55:142
22974991914,3cyclictest15886-21sed10:30:252
2297499191,3cyclictest0-21swapper/211:17:282
2297499191,11cyclictest0-21swapper/209:30:122
2296299199,2cyclictest0-21swapper/008:30:230
2296299196,4cyclictest29430-21unixbench-2d11:00:270
2296299196,3cyclictest12050irq/123-ahci[0009:00:010
2296299193,6cyclictest11-21rcuc/007:45:210
2296299191,9cyclictest4042-21unixbench_singl07:35:250
2296299191,9cyclictest26000-21grep08:25:260
2296299191,8cyclictest9682-21grep07:50:170
2296299191,6cyclictest0-21swapper/008:00:150
2296299191,4cyclictest0-21swapper/009:31:560
2296299191,10cyclictest1538-21smart_sda07:30:250
2296299191,10cyclictest14634-21snmpd08:08:040
2296299190,5cyclictest28980-21kworker/u8:210:52:510
2297999186,3cyclictest0-21swapper/308:50:203
2297999182,11cyclictest86550irq/126-enp1s0-11:35:183
2297999180,9cyclictest0-21swapper/308:31:363
2297999180,9cyclictest0-21swapper/307:36:353
2297999180,17cyclictest0-21swapper/307:29:513
2297499183,3cyclictest86850irq/129-enp1s0-08:00:552
2297499181,6cyclictest894-21/usr/sbin/munin08:45:152
2297499181,6cyclictest0-21swapper/210:08:352
2297499181,4cyclictest23166-21/usr/sbin/munin07:10:132
2297499181,4cyclictest0-21swapper/210:12:512
22974991813,3cyclictest32-21ksoftirqd/210:15:162
22974991812,3cyclictest27455-21ntpq12:10:212
22974991812,3cyclictest27455-21ntpq12:10:212
2296299188,2cyclictest11-21rcuc/009:45:210
2296299187,3cyclictest19154-21/usr/sbin/munin10:40:260
2296299187,3cyclictest0-21swapper/009:05:190
2296299185,7cyclictest0-21swapper/007:55:230
2296299185,7cyclictest0-21swapper/007:55:230
2296299182,6cyclictest3706-21grep11:15:260
2296299181,7cyclictest3115-21/usr/sbin/munin08:50:230
120272181,2sleep00-21swapper/010:21:180
22979991712,3cyclictest18662-21if_err_enp1s008:10:173
2297999171,15cyclictest0-21swapper/307:47:313
2297499173,3cyclictest86750irq/128-enp1s0-10:55:272
2297499173,3cyclictest86550irq/126-enp1s0-07:45:242
2297499171,6cyclictest0-21swapper/208:20:202
2297499171,4cyclictest1528-21memory12:25:192
22974991712,3cyclictest30782-21expr07:25:152
2297499171,14cyclictest0-21swapper/212:15:292
2297499171,13cyclictest10618-21/usr/sbin/munin07:50:262
2296299175,2cyclictest23280-21needreboot08:20:210
2296299171,6cyclictest24157-21needreboot07:10:210
2296299171,5cyclictest8364-21kworker/u8:112:16:320
2297999160,4cyclictest0-21swapper/307:50:263
2297499162,6cyclictest0-21swapper/209:40:042
2297499161,6cyclictest0-21swapper/209:51:232
22974991610,3cyclictest17578-21grep10:35:172
2296299161,6cyclictest0-21swapper/008:10:220
22979991510,3cyclictest11752-21fschecks_time07:55:173
22979991510,3cyclictest11752-21fschecks_time07:55:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional