You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 15:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 15, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218112236162,25sleep20-21swapper/207:08:062
216972234162,59sleep30-21swapper/307:06:383
216742205166,25sleep10-21swapper/107:06:191
218142202164,25sleep00-21swapper/007:08:070
1230721120,5sleep112313-21gltestperf10:25:191
303532610,5sleep02204799cyclictest11:05:230
75302530,1sleep00-21swapper/010:15:130
71222490,1sleep00-21swapper/010:15:010
22057994028,3cyclictest25361-21kworker/2:011:15:502
133182400,1sleep30-21swapper/308:00:163
22057993722,3cyclictest25361-21kworker/2:008:57:302
22057993620,3cyclictest25361-21kworker/2:009:53:342
22057993620,3cyclictest25361-21kworker/2:009:53:342
22057993525,2cyclictest25361-21kworker/2:010:46:462
22057993518,3cyclictest25361-21kworker/2:010:30:542
22057993418,3cyclictest25361-21kworker/2:012:13:502
2205799341,26cyclictest25361-21kworker/2:011:52:442
2205799332,20cyclictest25361-21kworker/2:009:08:282
2205799332,20cyclictest25361-21kworker/2:008:52:182
2205799331,25cyclictest25361-21kworker/2:011:31:362
2205799331,25cyclictest25361-21kworker/2:010:20:582
2205799331,22cyclictest25361-21kworker/2:008:46:222
2205799322,24cyclictest25361-21kworker/2:010:04:362
22057993219,2cyclictest25361-21kworker/2:009:42:302
2205799321,22cyclictest25361-21kworker/2:009:35:202
2205799321,22cyclictest23105-21kworker/2:108:15:342
2205799321,21cyclictest25361-21kworker/2:011:29:502
2204799327,12cyclictest31516-21munin-run07:30:000
2204799324,12cyclictest14257-21idleruntime-cro11:45:000
22057993115,2cyclictest25361-21kworker/2:008:28:462
2205799302,12cyclictest25361-21kworker/2:011:49:242
2205799302,12cyclictest23105-21kworker/2:107:21:422
22057993019,3cyclictest25361-21kworker/2:011:56:222
22057993019,3cyclictest25361-21kworker/2:009:17:302
22057993017,7cyclictest25361-21kworker/2:012:21:262
22057993013,3cyclictest23105-21kworker/2:108:15:022
2205799301,23cyclictest25361-21kworker/2:011:42:362
2205799301,21cyclictest25361-21kworker/2:009:45:362
2205799301,19cyclictest25361-21kworker/2:012:07:382
2205799300,21cyclictest25361-21kworker/2:008:37:342
2204799301,14cyclictest23350-21munin-run08:25:010
2205799292,17cyclictest25361-21kworker/2:012:37:282
2205799292,10cyclictest25361-21kworker/2:011:36:302
22057992916,3cyclictest25361-21kworker/2:009:59:342
2205799291,11cyclictest23105-21kworker/2:107:17:532
2204799295,12cyclictest3568-21perf07:40:000
2204799295,11cyclictest1995-21idleruntime-cro08:50:000
2204799293,13cyclictest13621-21munin-run10:30:010
22047992910,16cyclictest8600-21awk09:05:010
22064992810,3cyclictest0-21swapper/307:14:423
2205799282,12cyclictest25361-21kworker/2:010:07:222
2205799282,10cyclictest25361-21kworker/2:011:04:362
22057992818,2cyclictest25361-21kworker/2:010:19:262
2205799280,18cyclictest23105-21kworker/2:108:21:542
2205399289,10cyclictest6280-21/usr/sbin/munin10:10:321
2205399281,17cyclictest20602-21cut10:45:151
2205799278,2cyclictest23105-21kworker/2:108:00:022
2205799272,9cyclictest25361-21kworker/2:010:55:222
2205799272,9cyclictest25361-21kworker/2:010:55:222
2205799272,11cyclictest25361-21kworker/2:012:28:262
22057992716,2cyclictest25361-21kworker/2:011:23:382
22057992714,2cyclictest25361-21kworker/2:010:35:422
2205799271,19cyclictest25361-21kworker/2:009:13:002
2205799271,19cyclictest23105-21kworker/2:107:32:042
2205799271,13cyclictest25361-21kworker/2:009:23:122
2205799271,13cyclictest25361-21kworker/2:008:40:482
2205799271,10cyclictest25361-21kworker/2:012:34:042
2205799271,10cyclictest25361-21kworker/2:010:12:382
2204799275,11cyclictest1678-21crond12:30:010
193482270,4sleep339-21rcuc/308:15:133
22057992614,3cyclictest25361-21kworker/2:011:11:282
2205799261,18cyclictest25361-21kworker/2:010:42:042
22057992610,3cyclictest23105-21kworker/2:108:02:462
2206499257,9cyclictest0-21swapper/308:44:073
2206499253,11cyclictest23576-21/usr/sbin/munin08:25:263
2206499252,5cyclictest0-21swapper/311:09:383
2206499252,11cyclictest27354-21/usr/sbin/munin07:20:273
2206499251,3cyclictest16043-21/usr/sbin/munin10:35:253
2206499251,11cyclictest14634-21snmpd07:55:273
22057992519,2cyclictest25361-21kworker/2:012:00:142
22057992519,2cyclictest25361-21kworker/2:012:00:142
22057992517,2cyclictest25361-21kworker/2:010:53:262
2205799251,11cyclictest25361-21kworker/2:009:31:422
2205799251,11cyclictest25361-21kworker/2:009:25:302
2205799251,11cyclictest23105-21kworker/2:107:50:002
2205399255,5cyclictest780-21rngd10:35:221
2205399254,9cyclictest0-21swapper/112:35:271
2205399252,4cyclictest8586-21munin-run09:05:021
2204799250,12cyclictest24621-21idleruntime-cro10:55:000
14092250,3sleep011-21rcuc/010:00:200
2206499242,13cyclictest780-21rngd08:24:073
2206499241,9cyclictest0-21swapper/311:55:203
2205799247,3cyclictest23105-21kworker/2:107:26:242
2205799242,9cyclictest25361-21kworker/2:011:07:342
22057992418,2cyclictest23105-21kworker/2:107:12:372
2205799241,14cyclictest23105-21kworker/2:107:52:142
22057992411,2cyclictest23105-21kworker/2:107:35:162
22057992410,3cyclictest25361-21kworker/2:008:33:342
22053992416,2cyclictest0-21swapper/112:14:081
2205399241,20cyclictest18815-40dnf08:13:311
2205399241,18cyclictest0-21swapper/111:04:181
2204799245,16cyclictest16445-21munin-run11:50:000
65342231,18sleep16658-21smartctl07:45:191
283322230,2sleep10-21swapper/112:15:171
225022231,19sleep022660-21users12:00:280
225022231,19sleep022660-21users12:00:280
2206499237,4cyclictest780-21rngd12:20:183
2206499231,3cyclictest814-21sshd11:26:493
2206499231,3cyclictest14634-21snmpd09:38:143
2205799239,2cyclictest25361-21kworker/2:009:00:282
2205799231,11cyclictest23105-21kworker/2:107:40:262
2205799231,11cyclictest23105-21kworker/2:107:40:262
22057992310,3cyclictest25361-21kworker/2:012:16:582
2205399231,8cyclictest31630-21kernelversion09:55:201
22053992313,3cyclictest0-21swapper/110:31:511
2205399231,17cyclictest0-21swapper/110:07:441
2205399231,17cyclictest0-21swapper/109:14:011
2204799235,15cyclictest25553-21munin-run08:30:010
2204799230,11cyclictest321-21perf10:00:000
2206499228,6cyclictest39-21rcuc/310:44:583
2206499226,7cyclictest0-21swapper/309:24:463
2206499225,10cyclictest0-21swapper/308:56:093
2206499221,2cyclictest207522sleep309:30:203
2206499221,13cyclictest0-21swapper/310:21:093
2206499220,9cyclictest0-21swapper/310:31:353
2205799228,3cyclictest23105-21kworker/2:108:09:422
2205399229,6cyclictest31829-21kworker/u8:212:28:191
2205399229,4cyclictest12047-21awk11:40:011
2205399222,9cyclictest0-21swapper/111:28:361
2205399222,11cyclictest30231-21diskstats12:20:141
2205399221,8cyclictest0-21swapper/111:05:041
22053992217,3cyclictest21782-21perf09:35:011
22053992214,3cyclictest0-21swapper/110:53:351
2205399221,3cyclictest32547-21cat07:30:191
2205399221,16cyclictest0-21swapper/111:34:131
2205399221,16cyclictest0-21swapper/109:41:281
2205399221,15cyclictest0-21swapper/110:58:191
2205399221,15cyclictest0-21swapper/110:58:191
2205399221,12cyclictest10791-21awk09:10:021
2205399220,6cyclictest0-21swapper/109:54:481
2205399220,6cyclictest0-21swapper/109:54:481
2204799225,15cyclictest29950-21munin-run08:40:000
2204799221,15cyclictest0-21swapper/012:06:050
2206499219,3cyclictest12050irq/123-ahci[0012:30:023
2206499211,6cyclictest0-21swapper/308:32:093
22064992115,3cyclictest13877-21gltestperf09:15:193
2206499211,12cyclictest24212-21/usr/sbin/munin09:40:223
2206499211,12cyclictest0-21swapper/309:47:583
2206499211,10cyclictest14775-21/usr/sbin/munin08:05:253
2205799211,10cyclictest25361-21kworker/2:010:29:162
2205399212,5cyclictest780-21rngd09:45:241
22053992117,3cyclictest0-21swapper/107:55:201
2205399211,6cyclictest0-21swapper/110:01:581
22053992114,4cyclictest13190-21memory11:40:191
2205399211,2cyclictest21415-21kworker/1:107:10:161
2205399211,19cyclictest0-21swapper/111:49:201
2205399211,18cyclictest0-21swapper/108:30:101
2205399211,18cyclictest0-21swapper/108:06:331
2205399211,17cyclictest0-21swapper/109:15:241
2205399211,15cyclictest0-21swapper/111:11:261
2205399211,15cyclictest0-21swapper/110:16:121
2205399211,12cyclictest86850irq/129-enp1s0-12:31:171
2205399210,20cyclictest0-21swapper/107:29:251
2205399210,15cyclictest0-21swapper/110:41:251
2205399210,13cyclictest86850irq/129-enp1s0-08:25:341
2204799216,6cyclictest0-21swapper/009:15:570
2204799216,2cyclictest86550irq/126-enp1s0-10:06:200
2204799213,10cyclictest2210-21/usr/sbin/munin08:50:150
2204799211,7cyclictest0-21swapper/011:18:310
2204799211,3cyclictest26106-21smartctl10:55:260
2204799211,3cyclictest26106-21smartctl10:55:260
22047992111,6cyclictest10782-21diskmemload09:20:230
17602211,2sleep10-21swapper/108:47:211
272942201,3sleep30-21swapper/312:13:083
2206499208,3cyclictest22641-21/usr/sbin/munin10:50:253
2206499207,4cyclictest41-21ksoftirqd/309:15:013
2206499207,3cyclictest780-21rngd09:05:293
2206499206,2cyclictest12050irq/123-ahci[0012:40:013
2206499205,7cyclictest780-21rngd12:02:353
2206499205,7cyclictest780-21rngd12:02:353
2206499205,7cyclictest780-21rngd07:46:573
2206499204,3cyclictest0-21swapper/311:15:223
2206499201,9cyclictest401ktimersoftd/312:15:253
22064992011,7cyclictest32478-21kworker/u8:010:01:083
2206499201,11cyclictest842-21lldpd12:30:163
2206499201,11cyclictest14634-21snmpd09:57:483
2205399206,12cyclictest0-21swapper/109:20:091
2205399201,8cyclictest0-21swapper/111:50:131
22053992015,3cyclictest31395-21rt-features08:40:251
2205399201,2cyclictest0-21swapper/111:23:281
2205399201,18cyclictest0-21swapper/109:25:181
2205399201,18cyclictest0-21swapper/108:50:181
2205399201,17cyclictest0-21swapper/108:17:171
2205399201,15cyclictest0-21swapper/109:39:081
2205399201,13cyclictest0-21swapper/111:55:311
2205399201,10cyclictest14540-21latency_hist08:05:021
2205399200,18cyclictest20227-21kworker/u8:108:55:201
2205399200,14cyclictest0-21swapper/110:20:121
2204799209,3cyclictest3782-21/usr/sbin/munin07:40:180
2204799209,3cyclictest3782-21/usr/sbin/munin07:40:180
2204799203,14cyclictest8545-21cut11:30:200
2204799201,6cyclictest0-21swapper/010:40:070
22047992013,4cyclictest18965-21tr11:55:120
2204799201,11cyclictest10782-21diskmemload12:31:020
2204799201,10cyclictest3636-21cut11:20:120
2204799201,10cyclictest26709-21unixbench_singl07:15:270
2204799201,10cyclictest11016-21/usr/sbin/munin09:10:190
320692191,3sleep00-21swapper/008:44:140
2206499198,4cyclictest779-21ntpd07:40:323
2206499198,4cyclictest779-21ntpd07:40:323
2206499198,3cyclictest5051-21/usr/sbin/munin10:10:313
2206499198,3cyclictest0-21swapper/309:01:473
2206499196,2cyclictest0-21swapper/308:35:293
2206499195,2cyclictest0-21swapper/307:25:133
2206499194,3cyclictest780-21rngd10:20:013
2206499194,3cyclictest28838-21kworker/u8:208:49:533
2206499193,3cyclictest0-21swapper/311:13:123
22064991912,4cyclictest15542-21ntp_states11:45:243
2206499191,10cyclictest17610-21/usr/sbin/munin09:25:273
2206499191,10cyclictest14634-21snmpd11:43:403
2205399198,3cyclictest29942-21latency_hist08:40:021
22053991915,3cyclictest0-21swapper/107:21:561
22053991913,3cyclictest22056-21hddtemp_smartct08:20:181
22053991912,4cyclictest2341-21ntpq11:15:211
2205399191,17cyclictest0-21swapper/112:03:051
2205399191,17cyclictest0-21swapper/112:03:051
2205399191,17cyclictest0-21swapper/107:15:301
2205399190,6cyclictest0-21swapper/112:07:181
2204799196,4cyclictest0-21swapper/012:23:230
2204799196,4cyclictest0-21swapper/011:25:390
2204799192,3cyclictest12053-21munin-run11:40:010
2204799191,7cyclictest0-21swapper/011:10:160
2204799191,5cyclictest0-21swapper/012:19:460
2204799191,10cyclictest5979-21/usr/sbin/munin07:45:170
2204799191,10cyclictest0-21swapper/010:30:330
2206499186,3cyclictest3483-21/usr/sbin/munin11:20:193
2206499185,3cyclictest0-21swapper/311:30:323
2206499185,3cyclictest0-21swapper/307:51:073
2206499184,4cyclictest780-21rngd08:50:153
2206499182,5cyclictest0-21swapper/311:54:103
2206499181,7cyclictest0-21swapper/310:05:133
2206499181,6cyclictest0-21swapper/312:07:553
2206499181,3cyclictest25841-21hwlatdetect07:15:163
22064991813,3cyclictest29778-21ps09:50:233
22064991813,3cyclictest29778-21ps09:50:233
22064991812,3cyclictest12336-21hwlatdetect10:25:183
2206499181,15cyclictest0-21swapper/310:45:083
2206499180,9cyclictest0-21swapper/311:35:153
22053991814,3cyclictest0-21swapper/107:50:161
2205399181,16cyclictest0-21swapper/107:41:101
2205399181,16cyclictest0-21swapper/107:41:101
2205399181,16cyclictest0-21swapper/107:36:161
2204799186,3cyclictest0-21swapper/009:36:100
2204799183,12cyclictest4409-21/usr/sbin/munin08:55:240
2204799181,4cyclictest28210-21proc_pri11:00:270
22047991812,3cyclictest10964-21grep10:20:280
2204799181,10cyclictest0-21swapper/009:47:290
2204799180,5cyclictest0-21swapper/012:35:140
182992181,2sleep00-21swapper/011:50:480
2206499173,6cyclictest401ktimersoftd/307:30:013
2206499171,3cyclictest26550-21sshd10:56:483
2206499171,3cyclictest26550-21sshd10:56:483
22064991712,3cyclictest2112-21grep07:35:163
22064991712,3cyclictest16988-21basename08:10:123
2204799175,4cyclictest20722-21memory09:30:200
2204799172,7cyclictest27354-21/usr/sbin/munin07:20:170
2204799171,8cyclictest14634-21snmpd07:57:550
2204799171,2cyclictest25743-21sed09:40:290
22047991712,3cyclictest23741-21unixbench_multi07:10:260
2204799171,15cyclictest0-21swapper/009:06:130
22047991711,3cyclictest26214-21cut08:30:170
22047991711,3cyclictest20612-21sort08:15:260
2204799171,13cyclictest18531-21grep08:10:260
2204799171,13cyclictest0-21swapper/009:25:240
2204799171,11cyclictest0-21swapper/010:45:250
141782171,3sleep00-21swapper/008:00:340
2206499161,6cyclictest0-21swapper/311:00:313
2204799162,12cyclictest29223-21gltestperf09:50:160
2204799162,12cyclictest29223-21gltestperf09:50:160
2204799161,9cyclictest0-21swapper/012:10:070
2204799161,6cyclictest14634-21snmpd07:32:500
2204799161,14cyclictest0-21swapper/007:53:000
22047991610,4cyclictest15584-21irqstats08:05:180
2204799160,10cyclictest0-21swapper/010:35:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional