You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 17:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jun 23, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
280402239166,21sleep20-21swapper/207:09:112
280492204166,25sleep10-21swapper/107:09:191
279112203167,23sleep30-21swapper/307:07:353
277712199162,25sleep00-21swapper/007:05:440
3140121830,4sleep32820899cyclictest09:40:213
2820199104101,2cyclictest0-21swapper/209:54:432
28201999389,3cyclictest0-21swapper/209:23:282
167022600,4sleep30-21swapper/311:35:153
77032540,2sleep20-21swapper/210:00:192
315672520,1sleep20-21swapper/207:15:132
265052520,5sleep126508-21unixbench_multi11:55:231
9932510,2sleep20-21swapper/209:45:182
283982490,2sleep00-21swapper/009:35:130
260662480,2sleep10-21swapper/109:30:121
317612470,2sleep20-21swapper/208:30:022
296992470,2sleep00-21swapper/012:05:140
2820899442,32cyclictest23104-21/usr/sbin/munin08:10:243
28201993917,3cyclictest0-21swapper/210:38:182
28201993917,3cyclictest0-21swapper/210:38:182
2820899385,4cyclictest12050irq/123-ahci[0009:05:013
2820899385,3cyclictest87650irq/129-enp1s0-10:50:233
2820899371,6cyclictest20293-21/usr/sbin/munin10:30:153
28208993712,22cyclictest32512-21/usr/sbin/munin09:45:143
2820899364,5cyclictest22785-40dnf08:09:053
2820899361,3cyclictest194622cpuspeed_turbos09:15:123
2820899361,10cyclictest7706-21/usr/sbin/munin07:35:163
2820899359,3cyclictest25301-21/usr/sbin/munin08:15:113
2820899355,3cyclictest87650irq/129-enp1s0-11:40:233
2820899351,7cyclictest3749-21cpuspeed_turbos12:20:153
2820899351,6cyclictest24042-21expr09:25:143
2820899351,6cyclictest24042-21expr09:25:143
2819199359,12cyclictest26308-21munin-run10:45:000
100472350,1sleep10-21swapper/107:40:121
2820899349,3cyclictest16498-21/usr/sbin/munin07:55:233
2820899344,3cyclictest23488-21crond09:25:013
2820899337,2cyclictest0-21swapper/310:27:473
2820899331,7cyclictest0-21swapper/310:59:103
2820899331,5cyclictest7190-21ntp_states08:45:223
2820899331,4cyclictest6825-21sort07:30:253
2820899331,4cyclictest18193-21ntp_states09:10:223
2820899331,29cyclictest20598-21/usr/sbin/munin11:45:243
28208993310,20cyclictest6744-21/usr/sbin/munin10:00:133
2820199331,20cyclictest0-21swapper/208:45:042
2819599331,4cyclictest0-21swapper/110:42:391
2819199337,13cyclictest9686-21munin-run07:40:020
2820899329,2cyclictest11487-21df10:10:133
2820899323,6cyclictest5806-21smart_sda09:55:233
2820899321,6cyclictest26335-21timerandwakeup11:55:233
28208993216,13cyclictest29227-21ls09:35:223
2820899321,28cyclictest839-21lldpd09:33:263
28208993212,17cyclictest31896-21/usr/sbin/munin08:30:153
2820899321,10cyclictest87350irq/126-enp1s0-10:35:153
2820899321,10cyclictest87350irq/126-enp1s0-10:35:153
28208993210,19cyclictest5449-21unixbench_multi08:40:253
2820199321,19cyclictest0-21swapper/211:43:172
2820199321,19cyclictest0-21swapper/207:20:092
2820899316,4cyclictest9197-21/usr/sbin/munin08:50:183
2820899314,3cyclictest87350irq/126-enp1s0-09:10:013
2820899313,5cyclictest30663-21smart_sda12:05:263
2820899312,26cyclictest1097-21/usr/sbin/munin07:20:243
2820899312,25cyclictest10953-21ntpq07:40:203
28208993117,12cyclictest3419-21awk12:20:013
2820899311,5cyclictest16898-21diskmemload12:35:113
2820899311,4cyclictest6384-21perl12:25:153
28208993110,18cyclictest827-21runrttasks08:22:273
28208993110,18cyclictest827-21runrttasks08:22:273
28208993110,18cyclictest827-21runrttasks07:45:473
2820899310,9cyclictest0-21swapper/310:15:153
28201993127,3cyclictest0-21swapper/207:12:152
2820199311,14cyclictest0-21swapper/210:59:422
2820899308,4cyclictest0-21swapper/311:50:223
2820899308,3cyclictest2973-21/usr/sbin/munin11:05:143
2820899307,3cyclictest1819-21awk11:00:213
2820899306,13cyclictest87350irq/126-enp1s0-07:30:023
2820899305,3cyclictest87350irq/126-enp1s0-12:35:003
2820899302,4cyclictest27536-21diskstats12:00:143
2820899301,4cyclictest11785-21/usr/sbin/munin11:25:133
2820899301,26cyclictest16898-21diskmemload10:45:203
28208993010,17cyclictest2663-21sort08:35:213
28201993016,9cyclictest0-21swapper/212:07:492
2820199301,13cyclictest0-21swapper/208:16:282
2820199301,12cyclictest0-21swapper/211:28:012
28201993010,2cyclictest0-21swapper/207:55:252
2819599301,20cyclictest0-21swapper/111:37:361
2820899299,3cyclictest25578-21sendmail_mailst10:40:223
2820899297,4cyclictest763-21dbus-daemon09:50:203
2820899293,5cyclictest15372-21ntp_states07:50:213
2820899292,8cyclictest32415-21head07:15:233
2820899292,6cyclictest28390-21/usr/sbin/munin07:10:283
2820899291,5cyclictest14350-21diskstats11:30:143
2820899291,4cyclictest0-21swapper/311:10:173
2820899291,13cyclictest0-21swapper/308:02:393
28208992911,15cyclictest10003-21ntp_offset10:05:203
2820199291,19cyclictest0-21swapper/212:25:572
2820199291,12cyclictest0-21swapper/209:38:492
2820199291,11cyclictest0-21swapper/210:13:002
2820899289,3cyclictest10521-21acpi08:55:113
2820899289,2cyclictest827-21runrttasks11:24:563
2820899288,3cyclictest87350irq/126-enp1s0-11:15:173
2820899284,4cyclictest0-21swapper/312:10:213
28208992811,14cyclictest30440-21if_enp1s008:25:173
2820199281,13cyclictest0-21swapper/211:05:202
2820199281,12cyclictest24083-21forks09:25:152
2820199281,12cyclictest24083-21forks09:25:152
2820199281,12cyclictest0-21swapper/210:49:402
28201992810,6cyclictest0-21swapper/209:01:302
2820899271,4cyclictest17511-21latency_hist10:25:013
28201992713,10cyclictest0-21swapper/211:04:022
2820199271,13cyclictest4574-21sed08:40:162
2820199271,13cyclictest0-21swapper/211:51:112
2820199271,13cyclictest0-21swapper/211:24:432
2820199271,12cyclictest0-21swapper/209:15:192
108532270,3sleep09-21rcu_preempt07:40:190
2820199268,9cyclictest0-21swapper/209:10:312
28201992615,3cyclictest0-21swapper/212:31:172
28201992611,3cyclictest9913-21users08:50:262
2820199261,13cyclictest0-21swapper/211:10:122
2820199261,11cyclictest0-21swapper/212:12:352
28201992610,9cyclictest0-21swapper/210:41:382
2820199259,9cyclictest0-21swapper/211:55:232
2820199258,3cyclictest0-21swapper/207:37:572
28201992513,3cyclictest0-21swapper/209:57:062
28201992512,3cyclictest0-21swapper/210:50:182
2820199251,13cyclictest0-21swapper/211:34:072
2820199251,12cyclictest0-21swapper/210:22:322
2820199251,11cyclictest0-21swapper/210:29:402
2820199251,11cyclictest0-21swapper/209:30:172
28201992510,3cyclictest0-21swapper/212:04:252
2819599256,8cyclictest0-21swapper/109:45:541
2819599251,19cyclictest0-21swapper/107:29:331
2819199251,11cyclictest29487-21awk08:25:000
2819199251,11cyclictest29487-21awk08:25:000
2820199247,6cyclictest0-21swapper/208:03:122
2820199247,3cyclictest0-21swapper/208:38:062
2820199241,12cyclictest0-21swapper/207:54:172
28201992410,3cyclictest0-21swapper/211:38:032
2819599245,8cyclictest0-21swapper/111:08:181
2820199231,11cyclictest0-21swapper/211:17:322
2820199231,11cyclictest0-21swapper/207:26:412
2820199230,16cyclictest0-21swapper/212:35:252
2819599234,13cyclictest87550irq/128-enp1s0-07:22:471
2819599233,7cyclictest0-21swapper/110:55:251
2819599231,9cyclictest0-21swapper/110:24:311
2819599231,12cyclictest0-21swapper/112:25:171
28195992310,11cyclictest629-21kworker/u8:108:29:111
2819599230,13cyclictest0-21swapper/110:12:031
2819599230,12cyclictest0-21swapper/112:03:271
28191992318,2cyclictest11-21rcuc/009:50:020
2819199230,10cyclictest87350irq/126-enp1s0-10:30:000
242022230,7sleep1221ktimersoftd/108:10:221
219022231,2sleep00-21swapper/010:35:010
306792221,17sleep030912-21gltestperf09:40:160
2820199228,9cyclictest0-21swapper/209:08:462
2820199228,8cyclictest0-21swapper/210:06:462
2820199228,3cyclictest0-21swapper/211:46:312
2820199226,3cyclictest0-21swapper/212:17:492
2820199226,3cyclictest0-21swapper/208:55:222
2820199225,10cyclictest0-21swapper/208:30:382
2820199221,9cyclictest0-21swapper/208:14:142
28201992213,3cyclictest0-21swapper/207:30:212
2819599229,3cyclictest0-21swapper/111:03:291
2819599222,8cyclictest0-21swapper/112:14:081
2819599222,18cyclictest0-21swapper/109:01:581
2819599222,18cyclictest0-21swapper/108:20:191
2819599222,18cyclictest0-21swapper/108:20:191
2819599221,3cyclictest15725-21interrupts09:05:141
2819599221,2cyclictest0-21swapper/111:45:041
2819599221,19cyclictest0-21swapper/108:50:121
2819599221,18cyclictest10222-21/usr/sbin/munin12:35:191
2819599221,15cyclictest0-21swapper/111:22:331
2819599221,11cyclictest1028-21cat07:20:011
2819599221,10cyclictest19961-21grep10:30:121
2819599220,15cyclictest87650irq/129-enp1s0-10:39:541
2819599220,15cyclictest87650irq/129-enp1s0-10:39:531
2819199226,8cyclictest31179-21fw_packets07:15:030
2819199223,12cyclictest0-21swapper/011:53:530
2819199223,12cyclictest0-21swapper/011:12:310
2819199222,13cyclictest0-21swapper/011:26:440
2819199220,13cyclictest0-21swapper/011:57:590
2820199214,11cyclictest808-21snmpd08:09:062
2820199211,3cyclictest0-21swapper/210:33:422
2820199211,14cyclictest0-21swapper/212:20:372
2820199211,12cyclictest0-21swapper/207:46:132
28201992110,3cyclictest11301-21sed07:40:252
2819599212,18cyclictest0-21swapper/107:31:001
2819599211,8cyclictest0-21swapper/109:51:551
2819599211,8cyclictest0-21swapper/109:43:241
2819599211,6cyclictest0-21swapper/112:08:551
2819599211,4cyclictest0-21swapper/110:09:021
2819599211,18cyclictest0-21swapper/107:38:411
2819599211,13cyclictest0-21swapper/110:00:081
2819599211,12cyclictest0-21swapper/109:58:201
2819599211,11cyclictest19766-21ntp_states08:00:221
2819599211,10cyclictest7160-21tr08:45:211
2819599211,10cyclictest30515-21munin-node07:15:021
2819599211,10cyclictest1771-21df12:15:131
2819599210,16cyclictest0-21swapper/111:51:481
2819199217,4cyclictest13572-21grep07:45:260
2819199216,5cyclictest0-21swapper/012:39:160
2819199214,10cyclictest11382-21kworker/u8:011:35:370
2819199212,14cyclictest0-21swapper/010:36:570
2819199212,14cyclictest0-21swapper/010:36:570
2819199212,12cyclictest0-21swapper/010:01:540
2819199211,5cyclictest6514-21sh10:00:020
2819199211,5cyclictest0-21swapper/008:25:140
2819199211,13cyclictest0-21swapper/011:48:580
2819199211,13cyclictest0-21swapper/010:17:380
2819199211,10cyclictest16898-21diskmemload11:24:110
2819199211,10cyclictest0-21swapper/011:40:200
321862200,3sleep02819199cyclictest08:30:110
2820199209,7cyclictest0-21swapper/210:16:382
2820199207,3cyclictest0-21swapper/209:41:532
2819599206,6cyclictest0-21swapper/109:36:041
2819599206,3cyclictest0-21swapper/109:13:541
2819599204,4cyclictest0-21swapper/112:34:451
2819599202,16cyclictest0-21swapper/108:36:261
2819599201,6cyclictest11859-21grep11:25:111
28195992016,3cyclictest0-21swapper/110:48:051
2819599201,3cyclictest808-21snmpd11:17:191
2819599201,18cyclictest0-21swapper/107:48:061
2819599201,17cyclictest0-21swapper/110:51:331
2819599201,16cyclictest20419-21open_inodes09:15:221
2819599201,13cyclictest0-21swapper/110:25:121
2819599201,11cyclictest16338-21sort07:55:011
2819599201,10cyclictest4275-21diskstats08:40:141
2819599201,10cyclictest21874-21entropy09:20:131
2819599200,7cyclictest629-21kworker/u8:109:00:011
2819199207,3cyclictest16898-21diskmemload10:59:040
2819199205,9cyclictest16775-21sshd09:08:390
2819199203,15cyclictest12498-21munin-run09:00:010
2819199202,11cyclictest3147-21kworker/u8:012:17:240
2819199202,11cyclictest21721-21kworker/u8:210:46:300
2819199201,7cyclictest0-21swapper/009:30:070
2819199201,5cyclictest0-21swapper/012:25:210
28191992013,4cyclictest3091-21cut11:05:100
2819199201,11cyclictest8322-21/usr/sbin/munin08:50:150
2819199201,11cyclictest21521-21/usr/sbin/munin09:20:250
2819199201,11cyclictest0-21swapper/012:10:180
2819199201,10cyclictest3773-21grep12:20:150
2819199200,3cyclictest28492-21kworker/u8:011:00:200
2819199200,10cyclictest0-21swapper/009:12:060
2819599191,8cyclictest0-21swapper/110:17:531
28195991916,2cyclictest0-21swapper/108:30:101
2819599191,5cyclictest4707-21ntp_states12:20:241
28195991915,3cyclictest0-21swapper/108:19:251
28195991915,3cyclictest0-21swapper/108:06:251
2819599191,17cyclictest0-21swapper/111:43:111
2819599191,17cyclictest0-21swapper/111:12:161
2819599191,16cyclictest0-21swapper/111:32:021
2819199194,6cyclictest0-21swapper/009:53:540
2819199191,9cyclictest115662fschecks_count10:10:150
2819199191,5cyclictest0-21swapper/010:07:430
2819199191,13cyclictest0-21swapper/011:17:250
2819199191,11cyclictest9099-21kworker/u8:211:31:290
2819199191,11cyclictest0-21swapper/012:03:490
2819199191,10cyclictest808-21snmpd08:17:130
2819199190,13cyclictest0-21swapper/008:45:250
2820199182,2cyclictest87450irq/127-enp1s0-08:23:582
2820199182,2cyclictest87450irq/127-enp1s0-08:23:582
2819599181,16cyclictest0-21swapper/109:28:261
2819599181,16cyclictest0-21swapper/109:28:261
2819599181,16cyclictest0-21swapper/107:59:201
2819199188,5cyclictest0-21swapper/007:24:560
2819199188,2cyclictest16148-21head10:20:150
2819199187,3cyclictest827-21runrttasks08:00:060
2819199186,3cyclictest15447-21grep07:50:230
2819199182,8cyclictest9092-21ntp_states12:30:220
2819199181,9cyclictest5895-21latency_hist08:45:010
2819199181,9cyclictest30411-21munin-node07:15:020
2819199181,10cyclictest0-21swapper/010:53:110
2819199180,4cyclictest0-21swapper/008:10:210
2819199180,11cyclictest0-21swapper/009:27:500
2819199180,11cyclictest0-21swapper/009:27:500
146672180,3sleep00-21swapper/009:04:440
2819199175,4cyclictest20930-21cpu08:05:120
2819199175,3cyclictest5537-21cpu07:30:120
2819199172,9cyclictest0-21swapper/007:25:150
2819199171,3cyclictest12050irq/123-ahci[0008:00:010
2819199170,7cyclictest19921-21gltestperf09:15:160
2819199161,2cyclictest0-21swapper/008:39:580
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional