You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 04:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Sep 14, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
248872236163,24sleep30-21swapper/319:06:443
248092232162,57sleep20-21swapper/219:05:432
248222228161,55sleep00-21swapper/019:05:530
249122203168,24sleep10-21swapper/119:07:021
25240999689,4cyclictest21144-21munin-run22:40:012
164342690,1sleep20-21swapper/221:15:132
25234993410,11cyclictest0-21swapper/122:35:141
25234993314,8cyclictest31340-21grep23:00:241
2524099323,12cyclictest3546-21latency_hist22:00:012
2523499318,6cyclictest12962-21latency_hist23:35:031
2523499293,20cyclictest87550irq/128-enp1s0-00:15:251
2524099287,12cyclictest17928-21df_inode23:45:132
2524099283,12cyclictest0-21swapper/221:30:252
2523499284,16cyclictest24514-21wc22:45:191
2523499281,21cyclictest0-21swapper/123:18:451
2524099272,6cyclictest30601-21awk00:15:012
25234992713,12cyclictest24068-21kworker/u8:021:08:591
2523499270,8cyclictest0-21swapper/121:16:361
2524099268,4cyclictest0-21swapper/219:45:302
2524099268,4cyclictest0-21swapper/219:45:302
2524099266,7cyclictest11698-21/usr/sbin/munin19:50:152
2524099261,3cyclictest27137-21awk21:40:012
2523499263,10cyclictest11731-21latency_hist21:05:011
25234992614,3cyclictest0-21swapper/123:25:181
2523499261,17cyclictest0-21swapper/121:47:421
2523499261,13cyclictest2936-21munin-run20:45:011
2523499260,17cyclictest87550irq/128-enp1s0-22:16:431
2523099261,13cyclictest13940-21munin-run21:10:010
2523099261,12cyclictest24937-21idleruntime-cro21:35:010
2524099251,11cyclictest4397-21fgrep20:45:242
2523499258,10cyclictest0-21swapper/121:40:141
2523499251,17cyclictest0-21swapper/100:04:191
2523499251,12cyclictest1-21systemd20:00:011
2523499251,10cyclictest5389-21/usr/sbin/munin00:30:151
2523499250,12cyclictest24917-21sh21:35:011
2523099256,16cyclictest10158-21munin-run22:15:010
2523099252,12cyclictest12353-21munin-run22:20:010
25230992518,4cyclictest5089-21diskstats19:35:140
2523099251,13cyclictest26416-21kworker/0:020:50:010
80142240,2sleep20-21swapper/200:35:192
2524099247,2cyclictest17707-21latency_hist20:05:012
2524099244,10cyclictest22955-21users22:40:282
2524099243,10cyclictest20509-21runrttasks21:24:572
2524099242,6cyclictest5011-21cat00:30:012
2524099242,11cyclictest763-21dbus-daemon23:20:212
25240992415,6cyclictest17395-21latency_hist23:45:012
2524099241,11cyclictest0-21swapper/221:10:022
2523499249,3cyclictest0-21swapper/122:20:131
2523499248,9cyclictest0-21swapper/100:12:091
2523499247,3cyclictest26410-21needreboot19:10:221
2523499247,10cyclictest0-21swapper/123:11:181
2523499245,11cyclictest0-21swapper/121:10:221
2523499244,9cyclictest25394-21expr21:35:121
2523499243,15cyclictest19201-21kworker/u8:200:24:171
2523499243,15cyclictest19201-21kworker/u8:200:24:171
2523499242,15cyclictest21023-21kworker/u8:023:05:111
2523499242,12cyclictest23341-21munin-run22:45:011
2523499241,17cyclictest0-21swapper/121:53:541
2523499241,16cyclictest0-21swapper/122:29:221
2523499241,15cyclictest0-21swapper/100:36:191
2523499241,15cyclictest0-21swapper/100:05:151
2523499241,11cyclictest24639-21cpuspeed_turbos20:20:131
2523499241,11cyclictest0-21swapper/123:45:191
2523499240,18cyclictest29605-21kworker/u8:123:50:251
2524799238,12cyclictest827-21runrttasks22:35:193
2524099234,10cyclictest14780-21/usr/sbin/munin22:25:212
2523499238,9cyclictest0-21swapper/119:20:181
2523499238,10cyclictest0-21swapper/122:50:111
2523499234,10cyclictest22271-21cat20:15:021
25234992319,3cyclictest0-21swapper/119:47:411
25234992319,3cyclictest0-21swapper/119:47:411
25234992318,3cyclictest0-21swapper/120:53:351
2523499231,16cyclictest1076-21kworker/u8:122:05:021
2523499231,16cyclictest1076-21kworker/u8:122:05:021
2523499231,16cyclictest0-21swapper/121:23:341
2523499231,16cyclictest0-21swapper/120:35:311
2523499231,16cyclictest0-21swapper/119:32:401
2523499231,15cyclictest0-21swapper/122:11:031
2523499231,13cyclictest0-21swapper/100:29:191
2523499231,12cyclictest13931-21diskmemload21:29:291
2523499231,11cyclictest0-21swapper/120:00:141
2523499231,11cyclictest0-21swapper/119:15:131
2523499231,10cyclictest1732-21tr21:55:131
2523499230,16cyclictest0-21swapper/122:30:261
2524799221,12cyclictest9829-21nfs4_client23:25:203
2524799221,12cyclictest27802-21ntp_states20:25:213
2524799221,11cyclictest0-21swapper/300:15:283
2524099228,3cyclictest9027-21expr23:25:132
2524099228,3cyclictest519-21rt-features21:50:262
2524099228,2cyclictest9818-21taskset22:10:482
2524099227,3cyclictest19787-21/usr/sbin/munin23:50:102
2524099226,3cyclictest31615-21date21:50:022
2524099225,4cyclictest21415-21gltestperf21:25:152
2524099223,6cyclictest937-21/usr/sbin/munin20:40:242
2524099223,6cyclictest579-21systemd-journal00:22:282
2524099223,6cyclictest579-21systemd-journal00:22:282
2524099222,11cyclictest14549-21latency_hist22:25:012
2523499227,9cyclictest0-21swapper/119:35:231
2523499223,10cyclictest13209-21bc23:35:111
25234992214,2cyclictest0-21swapper/119:44:101
2523499221,18cyclictest0-21swapper/120:30:171
2523499221,16cyclictest0-21swapper/122:02:341
2523499221,15cyclictest0-21swapper/120:47:571
2523499221,15cyclictest0-21swapper/120:28:541
2523499221,15cyclictest0-21swapper/120:05:021
2523499221,10cyclictest0-21swapper/122:55:151
2523099228,3cyclictest164352cat21:15:130
2523099221,10cyclictest10770-21sadc23:30:010
2524799212,16cyclictest23051-21ntp_states23:55:193
25247992118,2cyclictest5750irq/9-acpi21:56:263
2524799211,13cyclictest0-21swapper/320:55:283
2524799211,13cyclictest0-21swapper/320:55:283
2524799211,12cyclictest29547-21/usr/sbin/munin21:45:203
2524799211,12cyclictest0-21swapper/322:22:233
25247992111,2cyclictest0-21swapper/320:38:563
2524799211,10cyclictest28520-21ps21:40:233
2524099214,4cyclictest10371-21grep21:00:152
2524099211,9cyclictest0-21swapper/220:11:502
2524099211,6cyclictest28319-21/usr/sbin/munin19:15:232
2524099211,5cyclictest0-21swapper/222:59:422
2524099211,12cyclictest0-21swapper/223:55:082
2524099210,11cyclictest87350irq/126-enp1s0-19:40:012
2523499214,5cyclictest11731-21fschecks_count19:50:151
2523499214,4cyclictest16457-21ntp_states23:40:211
2523499214,14cyclictest7908-21diskstats20:55:141
2523499214,14cyclictest7908-21diskstats20:55:141
25234992117,2cyclictest22680-21diskstats20:15:151
2523499211,15cyclictest0-21swapper/119:26:581
2523499211,11cyclictest0-21swapper/123:55:131
2523499210,16cyclictest0-21swapper/123:22:011
2523099212,11cyclictest20750-21/usr/sbin/munin21:25:120
2523099211,12cyclictest28618-21/usr/sbin/munin00:10:300
2524799209,3cyclictest28327-21forks22:55:143
2524799209,3cyclictest26738-21df00:05:143
2524799201,7cyclictest0-21swapper/322:15:193
2524799201,5cyclictest0-21swapper/321:25:133
2524799201,11cyclictest2433-21cstates23:10:123
2524799201,11cyclictest22334-21/usr/sbin/munin20:15:203
2524799201,11cyclictest0-21swapper/319:35:243
2524799201,11cyclictest0-21swapper/300:26:243
2524799201,10cyclictest4751-21df_inode23:15:133
25247992010,8cyclictest2644-21sshd20:41:283
25247992010,7cyclictest7277-21sshd20:54:283
2524099207,2cyclictest808-21snmpd23:31:452
2524099206,3cyclictest3430-21/usr/sbin/munin23:10:222
2524099206,2cyclictest31449-21grep23:00:262
2524099205,2cyclictest87550irq/128-enp1s0-22:00:132
2524099204,2cyclictest5362-21/usr/sbin/munin20:50:242
2524099203,4cyclictest12050irq/123-ahci[0020:10:012
2524099201,7cyclictest395-21cat00:20:012
2524099201,7cyclictest0-21swapper/222:19:522
2524099201,6cyclictest26905-21fschecks_time00:05:172
2524099201,16cyclictest27908-21proc_pri20:25:222
25240992011,5cyclictest25778-21diskstats19:10:172
2524099201,11cyclictest32152-21perf23:09:562
2524099201,10cyclictest0-21swapper/219:27:512
2523099208,3cyclictest28935-21/usr/sbin/munin20:30:180
2523099202,15cyclictest1260-21cut00:20:150
2523099202,15cyclictest1260-21cut00:20:150
2523099201,7cyclictest0-21swapper/021:42:100
2523099201,6cyclictest0-21swapper/022:58:380
2523099201,11cyclictest31428-21memory19:20:200
25230992010,3cyclictest87550irq/128-enp1s0-00:25:200
2524799196,4cyclictest0-21swapper/321:11:393
2524799195,6cyclictest39-21rcuc/320:45:153
2524799195,5cyclictest0-21swapper/323:00:243
2524799193,3cyclictest0-21swapper/322:00:563
2524799191,9cyclictest0-21swapper/322:29:283
2524799191,8cyclictest0-21swapper/323:21:163
2524799191,13cyclictest0-21swapper/300:12:333
2524799191,11cyclictest0-21swapper/323:05:063
2524799191,10cyclictest808-21snmpd19:47:033
2524799191,10cyclictest808-21snmpd19:47:033
2524799191,10cyclictest7242-21/usr/sbin/munin00:35:243
2524799191,10cyclictest20135-21/usr/sbin/munin20:10:183
2524799191,10cyclictest17653-21hddtemp_smartct22:30:203
2524799191,10cyclictest12054-21ntpq23:30:213
2524799191,10cyclictest0-21swapper/323:37:573
2524099194,3cyclictest659-21awk20:40:012
2524099194,3cyclictest24461-21cat20:20:022
2524099194,2cyclictest9239-21sshd20:56:282
2524099194,2cyclictest9239-21sshd20:56:282
2524099192,7cyclictest87350irq/126-enp1s0-21:44:362
2524099191,9cyclictest0-21swapper/222:35:012
2524099191,5cyclictest6324-21df22:05:152
2524099191,5cyclictest6324-21df22:05:152
2524099191,10cyclictest0-21swapper/219:24:242
2524099190,8cyclictest5042-21/usr/sbin/munin00:30:172
25230991914,3cyclictest24222-21cut22:45:160
25230991912,4cyclictest31180-21/usr/sbin/munin00:15:150
2523099191,11cyclictest15394-21/usr/sbin/munin23:40:260
2523099191,11cyclictest0-21swapper/019:40:150
288892180,11sleep339-21rcuc/319:15:153
2524799189,3cyclictest12013-21sh19:50:163
2524799187,3cyclictest23290-21df_inode21:30:143
2524799183,7cyclictest0-21swapper/321:23:333
2524799182,13cyclictest5042-21/usr/sbin/munin00:30:183
2524799181,9cyclictest0-21swapper/323:51:273
2524799181,8cyclictest0-21swapper/319:32:053
2524799181,6cyclictest0-21swapper/322:47:083
2524799181,3cyclictest26631-21unixbench-2d21:35:243
2524799181,10cyclictest6941-21/usr/sbin/munin19:40:213
2524799181,10cyclictest23252-21kworker/u8:219:58:333
2524799181,10cyclictest0-21swapper/322:51:283
2524799181,10cyclictest0-21swapper/322:06:343
2524799181,10cyclictest0-21swapper/322:06:343
2524799180,5cyclictest0-21swapper/321:52:033
2524799180,16cyclictest0-21swapper/322:40:283
2524099188,2cyclictest87350irq/126-enp1s0-22:45:122
2524099183,6cyclictest87550irq/128-enp1s0-23:35:252
2524099181,5cyclictest25769-21/usr/sbin/munin22:50:142
2524099181,4cyclictest0-21swapper/200:04:432
2524099181,3cyclictest0-21swapper/223:16:172
2524099181,15cyclictest0-21swapper/221:05:352
2524099181,15cyclictest0-21swapper/219:30:472
2524099180,6cyclictest7289-21diskstats19:40:162
2523099188,3cyclictest2333-21awk19:30:010
2523099188,2cyclictest25189-21cut00:00:240
2523099187,3cyclictest26374-21sshd20:23:280
2523099185,6cyclictest755-21rngd20:15:230
2523099184,4cyclictest30806-21smart_sda21:45:240
2523099182,13cyclictest20386-21grep23:50:140
2523099181,9cyclictest808-21snmpd20:26:090
2523099181,7cyclictest25562-21cut21:35:140
2523099181,5cyclictest0-21swapper/023:47:210
2523099181,5cyclictest0-21swapper/021:14:190
25230991812,3cyclictest14196-21cut19:55:150
2523099181,14cyclictest32255-21sed20:35:210
2523099181,10cyclictest0-21swapper/023:06:410
2523099180,7cyclictest0-21swapper/019:52:080
2523099180,6cyclictest7242-21/usr/sbin/munin00:35:220
164702181,2sleep30-21swapper/323:40:233
43462171,2sleep00-21swapper/019:32:590
284202170,4sleep028421-21cpuspeed_turbos19:15:120
2524799171,7cyclictest9163-21needreboot22:10:183
2524799171,6cyclictest29568-21gltestperf20:30:173
2524799171,5cyclictest25193-21cut00:00:243
2524799171,5cyclictest0-21swapper/319:20:363
2524799171,3cyclictest32709-21/usr/sbin/munin19:25:193
2524799171,13cyclictest17936-21diskstats23:45:143
25247991711,3cyclictest1299-21hddtemp_smartct00:20:153
25247991711,3cyclictest1299-21hddtemp_smartct00:20:153
2524799171,10cyclictest6626-21kworker/u8:121:16:043
2524799170,9cyclictest0-21swapper/320:20:223
2524099171,4cyclictest30951-21sed20:35:012
2524099171,14cyclictest0-21swapper/219:57:152
2523099176,3cyclictest18126-21pmu-power22:30:270
2523099176,3cyclictest11059-21sshd19:49:280
2523099176,3cyclictest11059-21sshd19:49:280
2523099174,3cyclictest20162-21grep22:35:210
2523099172,4cyclictest810-21sshd21:02:280
2523099171,9cyclictest808-21snmpd23:21:440
2523099171,6cyclictest0-21swapper/023:35:400
2523099171,5cyclictest0-21swapper/022:02:270
2523099171,4cyclictest0-21swapper/000:33:330
2523099171,3cyclictest808-21snmpd23:56:020
2523099171,3cyclictest0-21swapper/021:57:000
25230991712,3cyclictest19162-21expr21:20:180
2523099171,13cyclictest26414-21sed22:50:140
25230991711,3cyclictest22432-21ntp_states22:40:230
2523099171,10cyclictest0-21swapper/023:19:370
25230991710,4cyclictest32694-21/usr/sbin/munin21:50:210
119552170,2sleep30-21swapper/321:05:083
2524799166,8cyclictest87350irq/126-enp1s0-20:00:173
2524799161,5cyclictest9816-21cpu21:00:123
25247991612,3cyclictest0-21swapper/319:12:423
25247991611,3cyclictest19012-21ntpq20:05:223
2524099167,6cyclictest24529-21/usr/sbin/munin20:20:162
2523099161,8cyclictest7122-21runrttasks20:52:290
2523099161,6cyclictest27415-21perf19:15:010
2523099161,6cyclictest0-21swapper/023:30:290
2523099161,6cyclictest0-21swapper/022:25:180
2523099161,5cyclictest31160-21head23:00:220
2523099161,4cyclictest0-21swapper/022:22:500
2523099161,4cyclictest0-21swapper/022:09:010
2523099161,4cyclictest0-21swapper/022:09:010
2523099161,3cyclictest17938-21/usr/sbin/munin20:05:200
25230991612,3cyclictest0-21swapper/020:40:100
25230991611,3cyclictest2208-21/usr/sbin/munin23:10:170
25230991610,4cyclictest6626-21kworker/u8:120:55:150
25230991610,4cyclictest6626-21kworker/u8:120:55:140
2523099152,10cyclictest15850-21grep20:00:140
2523099151,5cyclictest26501-21grep00:05:130
2523099151,11cyclictest21187-21tr20:10:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional