You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 15:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 28, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269012233163,57sleep20-21swapper/207:05:092
286752232165,54sleep30-21swapper/307:07:403
287122204166,26sleep00-21swapper/007:08:080
287132202166,24sleep10-21swapper/107:08:101
28957998376,2cyclictest0-21swapper/210:09:252
94262590,3sleep00-21swapper/011:15:260
310012560,4sleep081ktimersoftd/012:05:180
296452530,3sleep02894499cyclictest10:50:130
115612530,2sleep3401ktimersoftd/312:35:153
294192510,4sleep07-21ksoftirqd/009:35:120
222412510,3sleep230-21rcuc/210:30:252
310822450,2sleep230-21rcuc/210:50:272
272402450,1sleep20-21swapper/210:45:022
2895799381,14cyclictest6869-21tail07:30:162
2895799381,14cyclictest6869-21tail07:30:162
2895799374,7cyclictest11902-40dnf07:40:262
2894499379,13cyclictest17042-21idleruntime-cro07:55:010
28944993712,12cyclictest6651-21munin-run08:45:010
28944993712,12cyclictest6651-21munin-run08:45:010
2896199361,26cyclictest0-21swapper/309:24:313
2895799365,5cyclictest17650-21diskmemload11:38:252
2895799361,10cyclictest15772-21ntp_states11:30:212
28961993516,13cyclictest0-21swapper/309:08:353
2896199351,21cyclictest0-21swapper/309:28:113
2896199351,20cyclictest0-21swapper/311:26:273
28957993513,19cyclictest1-21systemd09:00:012
28957993411,3cyclictest6536-21/usr/sbin/munin12:25:162
28957993411,3cyclictest21657-21/usr/sbin/munin08:05:162
28961993317,9cyclictest0-21swapper/310:05:073
2896199331,21cyclictest0-21swapper/309:04:513
2896199331,20cyclictest0-21swapper/311:12:243
2896199331,20cyclictest0-21swapper/310:14:163
2895799337,4cyclictest24118-21/usr/sbin/munin10:35:222
2895799332,3cyclictest10693-21crond12:35:022
2895799332,2cyclictest87350irq/126-enp1s0-09:30:242
28957993313,17cyclictest13936-21grep09:00:152
2896199321,20cyclictest0-21swapper/311:56:233
2896199321,19cyclictest0-21swapper/310:30:483
2895799329,3cyclictest4766-21idleruntime07:25:172
2895799329,2cyclictest32637-21df12:10:122
2895799329,2cyclictest32637-21df12:10:122
2895799321,4cyclictest27129-21timerwakeupswit11:55:262
28957993211,3cyclictest17796-21cat09:10:012
28957993210,2cyclictest28241-21/usr/sbin/munin08:20:182
2895099329,15cyclictest827-21runrttasks08:50:391
2894499326,12cyclictest5082-21perf09:55:000
28961993119,3cyclictest0-21swapper/312:06:513
28961993119,3cyclictest0-21swapper/309:13:393
28961993117,7cyclictest0-21swapper/311:33:033
28961993116,9cyclictest0-21swapper/307:45:043
2896199311,20cyclictest0-21swapper/311:23:273
2896199311,13cyclictest0-21swapper/309:15:433
2896199311,12cyclictest0-21swapper/310:47:523
2895799319,3cyclictest827-21runrttasks12:24:282
2895799319,2cyclictest28852-21/usr/sbin/munin09:35:152
2895799318,4cyclictest23138-21/usr/sbin/munin09:20:182
2895799317,3cyclictest0-21swapper/210:59:252
2895799317,3cyclictest0-21swapper/210:59:252
2895799315,3cyclictest87350irq/126-enp1s0-09:15:192
28961993019,3cyclictest0-21swapper/307:54:283
2896199301,20cyclictest0-21swapper/310:51:243
2896199301,20cyclictest0-21swapper/308:18:083
2896199301,20cyclictest0-21swapper/307:40:403
2896199301,19cyclictest0-21swapper/311:38:073
2896199301,19cyclictest0-21swapper/309:53:113
2896199301,13cyclictest2833-21cat07:20:193
2896199301,13cyclictest0-21swapper/310:59:443
2896199301,13cyclictest0-21swapper/310:59:443
2896199301,13cyclictest0-21swapper/310:23:363
2896199301,12cyclictest0-21swapper/307:11:563
2896199301,11cyclictest0-21swapper/312:21:483
2895799309,3cyclictest12428-21fschecks_time10:10:132
2895799308,3cyclictest9675-21cat10:05:022
2895799308,3cyclictest4206-21cut11:05:152
2895799308,3cyclictest31145-21ls09:40:112
2895799305,4cyclictest29826-21sshd12:04:292
2895799304,5cyclictest27471-21expr10:45:132
2895799304,3cyclictest25448-21chrt11:54:532
2895799302,3cyclictest87350irq/126-enp1s0-11:20:252
2895799301,4cyclictest3442-21diskstats09:50:142
28957993012,2cyclictest0-21swapper/208:40:092
28957993012,2cyclictest0-21swapper/208:40:092
28957993011,3cyclictest15257-21cat07:50:132
28957993010,3cyclictest9648-21processes07:35:232
2894499303,12cyclictest22669-21idleruntime-cro10:35:000
2894499302,13cyclictest12901-21munin-run12:40:010
28961992914,8cyclictest0-21swapper/310:25:483
28961992914,10cyclictest0-21swapper/312:14:313
28961992914,10cyclictest0-21swapper/312:14:313
2896199291,17cyclictest0-21swapper/309:47:553
2896199291,17cyclictest0-21swapper/309:47:553
2896199291,12cyclictest0-21swapper/308:56:313
2896199291,11cyclictest0-21swapper/312:33:283
2896199291,11cyclictest0-21swapper/312:28:123
2896199291,11cyclictest0-21swapper/311:53:593
2896199291,10cyclictest0-21swapper/311:02:203
2895799299,4cyclictest0-21swapper/211:45:172
2895799299,3cyclictest12351-21sed12:35:242
2895799299,2cyclictest20514-21sort11:40:242
2895799298,3cyclictest24438-21gltestperf08:10:142
2895799292,11cyclictest9415-21/usr/sbin/munin11:15:252
2895799291,3cyclictest87350irq/126-enp1s0-09:27:002
28957992910,3cyclictest648-21ntp_offset07:15:202
28961992817,3cyclictest0-21swapper/309:38:033
28961992815,7cyclictest0-21swapper/311:40:393
2896199281,17cyclictest0-21swapper/312:02:473
2896199281,16cyclictest0-21swapper/309:31:153
2896199281,11cyclictest0-21swapper/311:08:323
2896199281,11cyclictest0-21swapper/308:04:443
2895799288,3cyclictest20864-21sort08:00:252
2895799285,3cyclictest0-21swapper/207:20:152
2895799282,3cyclictest808-21snmpd11:14:212
2895799281,4cyclictest20452-21turbostat.cron10:30:022
2894499283,12cyclictest8845-21munin-run08:50:020
323962270,3sleep02894499cyclictest07:15:150
28961992718,3cyclictest0-21swapper/307:32:483
28961992718,3cyclictest0-21swapper/307:32:473
28961992715,3cyclictest0-21swapper/308:08:523
2896199271,18cyclictest0-21swapper/311:17:513
2896199271,18cyclictest0-21swapper/310:36:203
2896199271,12cyclictest0-21swapper/310:17:483
2896199271,11cyclictest0-21swapper/312:16:163
2896199271,11cyclictest0-21swapper/308:33:323
2896199271,10cyclictest0-21swapper/309:43:273
28961992710,3cyclictest0-21swapper/310:02:153
2895799279,3cyclictest961-21cpuspeed_turbos09:45:132
2895799279,3cyclictest961-21cpuspeed_turbos09:45:132
2895799279,3cyclictest17851-21gltestperf07:55:152
2895799278,3cyclictest1696-21cstates11:00:112
2895799271,6cyclictest0-21swapper/209:10:112
2895799271,4cyclictest0-21swapper/209:55:222
2895799271,3cyclictest87350irq/126-enp1s0-12:05:512
224082270,4sleep081ktimersoftd/009:20:110
156702271,22sleep015671-21/usr/sbin/munin09:05:120
2896199261,9cyclictest0-21swapper/310:42:443
28961992614,7cyclictest0-21swapper/308:45:483
2896199261,18cyclictest0-21swapper/308:37:443
2896199261,18cyclictest0-21swapper/308:21:203
2896199261,11cyclictest0-21swapper/309:57:113
2895799262,8cyclictest3010-21cut08:35:152
2895799261,6cyclictest0-21swapper/210:15:302
2895799261,2cyclictest0-21swapper/212:15:252
2895799261,14cyclictest0-21swapper/208:29:362
2895799261,12cyclictest0-21swapper/208:49:252
2895799261,11cyclictest0-21swapper/211:29:412
2896199255,9cyclictest0-21swapper/307:56:403
28961992517,3cyclictest0-21swapper/307:19:323
28961992516,3cyclictest0-21swapper/307:28:513
2896199251,21cyclictest9958-21sed08:50:193
2896199251,17cyclictest0-21swapper/308:25:483
28961992510,3cyclictest0-21swapper/308:44:163
28961992510,3cyclictest0-21swapper/308:44:163
2895799251,6cyclictest27550-21unixbench_multi08:15:242
2895799251,4cyclictest29144-21/usr/sbin/munin07:10:172
2895099251,18cyclictest0-21swapper/110:01:151
104522250,4sleep00-21swapper/008:50:260
2896199246,9cyclictest0-21swapper/307:37:323
28961992416,3cyclictest0-21swapper/308:13:243
2895799247,3cyclictest14824-21latency_hist07:50:022
2895799241,4cyclictest17650-21diskmemload10:25:022
2895099242,19cyclictest23274-21munin-run11:50:011
28950992412,2cyclictest0-21swapper/108:05:121
2895099241,16cyclictest0-21swapper/110:33:281
49882230,1sleep10-21swapper/112:20:151
28961992312,3cyclictest0-21swapper/311:45:593
2895099232,15cyclictest0-21swapper/108:48:301
2895099231,20cyclictest0-21swapper/110:45:481
2895099231,17cyclictest0-21swapper/109:23:301
2895099231,16cyclictest0-21swapper/108:43:431
2895099231,16cyclictest0-21swapper/108:43:431
28944992311,3cyclictest6360-21ntp_states09:55:200
241142230,5sleep12895099cyclictest11:50:181
2895099223,5cyclictest0-21swapper/110:12:161
2895099222,15cyclictest0-21swapper/108:30:431
2895099222,12cyclictest0-21swapper/110:41:301
2895099222,11cyclictest9081-21diskstats12:30:131
2895099222,11cyclictest12634-21munin-run07:45:011
28950992214,3cyclictest0-21swapper/107:24:501
2895099221,2cyclictest87650irq/129-enp1s0-12:00:271
2895099220,15cyclictest0-21swapper/108:10:191
2894499226,6cyclictest0-21swapper/009:28:100
156132220,1sleep00-21swapper/011:30:190
2895799218,3cyclictest10224-21/usr/sbin/munin08:50:232
2895099217,3cyclictest5310-21users11:05:281
2895099211,18cyclictest26958-21smart_sda11:55:251
2895099211,18cyclictest0-21swapper/110:57:211
2895099211,18cyclictest0-21swapper/110:57:211
2895099211,16cyclictest0-21swapper/110:36:101
2895099211,15cyclictest0-21swapper/112:35:101
2895099211,15cyclictest0-21swapper/108:22:541
2895099211,11cyclictest19849-21munin-run09:15:021
2895099211,11cyclictest11613-21diskstats08:55:141
2895099211,10cyclictest15893-21kernelversion07:50:171
28950992110,3cyclictest17581-21smartctl11:35:171
28950992110,3cyclictest10251-21perl10:05:161
2895099210,10cyclictest0-21swapper/111:22:561
2894499219,3cyclictest9470-21munin-run10:05:010
2894499214,7cyclictest11-21rcuc/011:50:280
2894499214,3cyclictest0-21swapper/007:55:190
2894499212,11cyclictest26509-21irqstats11:55:180
2894499211,6cyclictest0-21swapper/012:30:370
2894499211,5cyclictest0-21swapper/012:17:440
2894499211,11cyclictest4338-21/usr/sbin/munin12:20:110
2894499210,11cyclictest0-21swapper/012:10:130
2894499210,11cyclictest0-21swapper/012:10:130
2895799202,4cyclictest2404-21cat08:35:012
2895099209,3cyclictest0-21swapper/109:30:141
2895099204,9cyclictest0-21swapper/112:05:161
2895099204,13cyclictest1-21systemd11:30:021
2895099202,16cyclictest0-21swapper/111:41:241
2895099201,6cyclictest0-21swapper/110:25:391
28950992016,3cyclictest0-21swapper/110:23:091
28950992016,3cyclictest0-21swapper/109:26:281
28950992016,2cyclictest21-21rcuc/109:35:131
28950992013,4cyclictest7450-21memory12:25:181
28950992013,2cyclictest0-21swapper/107:10:081
2895099201,18cyclictest0-21swapper/107:15:301
2895099201,14cyclictest0-21swapper/107:57:071
2895099201,11cyclictest4821-21if_enp2s007:25:171
2895099201,11cyclictest26268-21expr08:15:121
2895099201,11cyclictest19800-21diskstats08:00:131
2895099201,11cyclictest15694-21cpu09:05:111
2895099201,11cyclictest14728-21swap09:00:251
2895099201,10cyclictest4010-21memory09:50:191
2895099200,2cyclictest87650irq/129-enp1s0-10:16:121
2895099200,14cyclictest0-21swapper/108:25:191
2894499209,3cyclictest13466-21/usr/sbin/munin09:00:150
2894499205,3cyclictest0-21swapper/009:40:180
2894499202,4cyclictest0-21swapper/011:08:300
2894499201,9cyclictest0-21swapper/010:25:330
28944992013,4cyclictest3070-21grep08:35:160
2894499201,12cyclictest0-21swapper/012:00:270
2894499201,11cyclictest28241-21/usr/sbin/munin08:20:190
2894499201,11cyclictest16286-21/usr/sbin/munin10:20:110
2894499201,11cyclictest12857-21/usr/sbin/munin07:45:170
2894499201,11cyclictest0-21swapper/011:42:410
2894499201,10cyclictest6905-21head11:10:200
2894499201,10cyclictest2041-21pmu-power09:45:220
2894499201,10cyclictest2041-21pmu-power09:45:220
2894499201,10cyclictest0-21swapper/009:18:100
25032201,2sleep10-21swapper/109:45:301
25032201,2sleep10-21swapper/109:45:301
2895099199,5cyclictest0-21swapper/112:14:011
2895099199,5cyclictest0-21swapper/112:14:011
2895099192,3cyclictest0-21swapper/109:55:411
2895099192,16cyclictest0-21swapper/110:51:391
2895099192,16cyclictest0-21swapper/109:16:181
2895099192,15cyclictest0-21swapper/111:31:281
28950991915,3cyclictest0-21swapper/111:15:241
2895099191,17cyclictest0-21swapper/112:15:401
2895099191,14cyclictest0-21swapper/107:33:471
2895099191,14cyclictest0-21swapper/107:33:471
2895099191,10cyclictest6435-21grep11:10:141
2895099191,10cyclictest4452-21munin-run08:40:001
2895099191,10cyclictest32315-21/usr/sbin/munin09:40:231
2895099190,14cyclictest0-21swapper/107:45:341
2894499196,6cyclictest0-21swapper/010:06:550
2894499196,4cyclictest0-21swapper/010:58:340
2894499196,4cyclictest0-21swapper/010:58:340
2894499196,2cyclictest17010-21tr11:35:130
2894499195,2cyclictest0-21swapper/007:28:080
2894499194,4cyclictest0-21swapper/011:24:050
2894499194,12cyclictest7-21ksoftirqd/010:40:160
2894499191,6cyclictest0-21swapper/010:36:330
2894499191,3cyclictest0-21swapper/010:17:120
2894499191,10cyclictest27555-21unixbench_multi08:15:250
2894499191,10cyclictest0-21swapper/009:32:290
2895099181,2cyclictest0-21swapper/107:37:471
2895099180,9cyclictest3559-21latency_hist11:05:011
2894499188,3cyclictest25825-21awk08:15:020
2894499188,2cyclictest28326-21awk10:45:200
2894499186,4cyclictest0-21swapper/012:25:180
2894499184,5cyclictest0-21swapper/011:27:550
2894499183,3cyclictest0-21swapper/011:48:440
2894499183,12cyclictest11-21rcuc/008:00:180
2894499182,2cyclictest0-21swapper/010:10:520
2894499182,13cyclictest11-21rcuc/007:20:160
2894499181,3cyclictest0-21swapper/011:02:020
28944991813,3cyclictest603-21grep08:30:140
2894499181,10cyclictest7255-21strings07:30:180
2894499181,10cyclictest7255-21strings07:30:180
2894499180,10cyclictest11261-21gltestperf07:40:140
2894499172,3cyclictest0-21swapper/009:10:120
2894499171,14cyclictest0-21swapper/008:25:100
28944991710,4cyclictest29299-21cpuspeed_turbos07:10:100
2894499161,6cyclictest8461-21/usr/sbin/munin07:35:200
2894499161,13cyclictest11265-21/usr/sbin/munin08:55:150
2894499150,14cyclictest0-21swapper/008:05:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional