You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 16:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 30, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
188022204168,24sleep30-21swapper/307:07:063
189702202166,24sleep20-21swapper/207:09:182
186702202166,24sleep10-21swapper/107:05:241
188662201164,25sleep00-21swapper/007:07:560
2379421420,4sleep31912799cyclictest07:15:253
1703621410,4sleep01910999cyclictest10:40:210
49221380,4sleep01910999cyclictest11:15:240
19121999592,2cyclictest0-21swapper/208:19:372
19121999592,2cyclictest0-21swapper/208:19:372
19121996763,3cyclictest0-21swapper/212:16:042
325962540,4sleep132601-21pmu-power11:15:211
19109994910,12cyclictest13811-21idleruntime-cro08:10:000
273172480,1sleep127316-21egrep08:40:111
1910999486,12cyclictest18673-21basename12:00:010
1910999471,16cyclictest21633-21munin-run09:40:010
19109994710,13cyclictest19438-21munin-run09:35:010
1910999466,12cyclictest32629-21munin-run10:05:020
19121994036,3cyclictest0-21swapper/208:11:182
19116993510,12cyclictest0-21swapper/111:55:131
1910999347,12cyclictest9408-21perf08:00:010
19109993410,11cyclictest27478-21awk12:20:020
1910999338,11cyclictest17246-21awk09:30:010
1910999338,11cyclictest17246-21awk09:30:010
1910999336,12cyclictest16007-21awk08:15:000
1912199301,2cyclictest0-21swapper/212:33:242
187242300,5sleep21912199cyclictest12:00:012
1912799285,9cyclictest5007-21awk07:50:003
1911699281,12cyclictest0-21swapper/109:50:481
314022270,4sleep232-21ksoftirqd/208:45:022
1912799272,18cyclictest0-21swapper/311:34:563
1912799270,19cyclictest0-21swapper/311:16:233
1911699270,9cyclictest0-21swapper/111:33:411
31492261,21sleep03182-21smart_sda12:35:230
1912799268,4cyclictest22189-21/usr/sbin/munin12:05:213
19127992616,3cyclictest852-21cpuspeed_turbos07:40:123
1912199261,10cyclictest0-21swapper/212:35:202
19109992611,2cyclictest7626-21kworker/0:211:35:010
110612260,3sleep121-21rcuc/110:27:381
1912799253,14cyclictest8584-21hostname11:35:183
1912799253,13cyclictest0-21swapper/309:25:203
1912799253,13cyclictest0-21swapper/309:25:203
1912799252,12cyclictest27690-21/usr/sbin/munin12:20:153
19127992517,4cyclictest30729-21sshd08:43:543
1912799250,15cyclictest0-21swapper/307:20:133
1912799250,11cyclictest17577-21tr09:30:113
1912199259,3cyclictest0-21swapper/208:05:562
1912199251,11cyclictest0-21swapper/210:12:152
1910999252,10cyclictest6062-21kworker/0:109:09:570
1910999251,17cyclictest13395-21kworker/0:210:51:210
1910999251,12cyclictest2469-21munin-run10:10:020
1910999251,12cyclictest1221-21perf08:50:010
1912799249,6cyclictest0-21swapper/310:55:273
1912799249,4cyclictest31558-21memory10:00:193
1912799247,5cyclictest30339-21ntp_states11:10:223
1912799243,18cyclictest23298-21/usr/sbin/munin12:10:163
1912799243,13cyclictest20414-21/usr/sbin/munin10:50:163
1912799242,9cyclictest14268-21munin-plugin-st11:50:013
1912799242,12cyclictest2692-21/usr/sbin/munin10:10:273
1912799241,14cyclictest0-21swapper/310:19:353
1912799241,10cyclictest10261-21missed_timers10:25:183
1912199241,11cyclictest0-21swapper/211:36:242
1911699248,4cyclictest0-21swapper/109:38:191
1911699241,16cyclictest0-21swapper/110:14:141
1911699241,12cyclictest27783-21sort12:20:111
1910999249,2cyclictest18538-21kworker/0:012:05:200
1910999242,10cyclictest15253-21kworker/0:109:50:540
19109992411,2cyclictest7626-21kworker/0:211:35:260
1912799239,3cyclictest0-21swapper/309:49:523
1912799237,8cyclictest0-21swapper/308:05:143
1912799235,3cyclictest28450-21/usr/sbin/munin09:55:203
1912799234,11cyclictest28795-21/usr/sbin/munin07:30:193
1912799234,10cyclictest2462-21latency_hist10:10:013
1912799233,9cyclictest20268-21sed10:50:013
1912799230,16cyclictest0-21swapper/309:15:193
1912199238,3cyclictest0-21swapper/209:18:042
1912199231,2cyclictest0-21swapper/209:50:012
1912199230,10cyclictest0-21swapper/210:39:082
1912199230,10cyclictest0-21swapper/210:39:082
1911699239,12cyclictest4483-21kworker/u8:211:24:161
1911699239,12cyclictest3723-21kworker/u8:207:41:561
1911699239,12cyclictest22266-21kworker/u8:009:17:191
1911699237,13cyclictest26496-21systemd-journal11:00:471
1911699231,16cyclictest0-21swapper/109:43:181
1911699230,1cyclictest0-21swapper/107:17:451
1910999239,3cyclictest8898-21kworker/0:110:27:540
1910999239,3cyclictest15253-21kworker/0:109:55:540
1910999239,3cyclictest15253-21kworker/0:109:48:460
1910999239,2cyclictest7626-21kworker/0:211:41:500
1910999236,10cyclictest19386-21kworker/0:108:24:330
1910999232,9cyclictest24845-21kworker/0:208:36:490
1910999232,10cyclictest6510-21kworker/0:210:17:300
1910999232,10cyclictest30895-21kworker/0:011:14:170
1910999231,8cyclictest15953-21kworker/0:110:45:090
1910999231,10cyclictest18538-21kworker/0:012:03:180
19109992310,3cyclictest26044-21kworker/0:007:29:140
70512220,2sleep20-21swapper/211:30:242
226802220,5sleep31912799cyclictest08:30:023
204132221,3sleep120553-21cpuspeed_turbos10:50:111
1912799228,2cyclictest18891-21acpi12:00:113
1912799227,12cyclictest3382-21ls11:25:013
1912799222,6cyclictest22848-21cpu08:30:123
1912799222,16cyclictest2330-21cut12:35:143
1912799222,11cyclictest26600-21/usr/sbin/munin07:25:253
1912799221,7cyclictest810-21sshd08:21:463
1912799221,5cyclictest0-21swapper/310:35:163
1912799221,5cyclictest0-21swapper/310:35:163
1912799221,13cyclictest0-21swapper/309:52:493
1912199227,4cyclictest31864-21awk12:30:012
1912199226,4cyclictest1659-21pmu-power10:05:362
1912199221,9cyclictest0-21swapper/209:09:562
19121992215,4cyclictest28648-21grep11:05:262
1912199220,11cyclictest0-21swapper/210:50:122
1911699228,5cyclictest0-21swapper/110:02:271
1911699225,9cyclictest11828-21/usr/sbin/munin08:05:231
1911699221,9cyclictest0-21swapper/111:50:001
1911699221,3cyclictest8246-21diskstats11:35:151
1911699221,3cyclictest0-21swapper/111:53:501
1911699221,3cyclictest0-21swapper/111:53:501
1911699221,18cyclictest25124-21ntp_states09:45:211
1910999229,2cyclictest22415-21kworker/0:311:04:330
1910999229,2cyclictest11859-21kworker/0:011:46:420
1910999228,2cyclictest3811-21kworker/0:208:58:170
1910999222,9cyclictest29462-21kworker/0:312:30:500
1910999222,10cyclictest29462-21kworker/0:312:29:140
1912799216,6cyclictest0-21swapper/312:33:023
1912799215,8cyclictest24868-21/usr/sbin/munin11:00:283
1912799215,3cyclictest0-21swapper/309:05:223
1912799214,9cyclictest15726-21NetworkManager08:11:423
1912799213,5cyclictest0-21swapper/311:40:143
1912799213,11cyclictest0-21swapper/310:42:593
1912799213,11cyclictest0-21swapper/309:44:123
1912799211,13cyclictest0-21swapper/310:22:133
1912799211,12cyclictest3851-21diskstats11:25:143
1912799211,11cyclictest31853-21latency_hist12:30:013
1912799210,13cyclictest5226-21/usr/sbin/munin07:50:163
1912199215,11cyclictest87550irq/128-enp1s0-09:54:352
1912199214,8cyclictest1281650kworker/u8:109:30:292
19121992111,2cyclictest32630-21awk10:05:012
1912199210,9cyclictest0-21swapper/211:47:112
1911699219,3cyclictest24962-21cat08:35:021
1911699217,6cyclictest0-21swapper/110:15:171
1911699217,12cyclictest22266-21kworker/u8:009:20:051
1911699211,14cyclictest0-21swapper/112:27:531
1911699211,14cyclictest0-21swapper/112:16:311
1911699211,11cyclictest22162-21ntp_states12:05:211
1911699211,11cyclictest0-21swapper/112:34:151
19116992110,9cyclictest0-21swapper/111:25:141
19116992110,3cyclictest16074-21irqstats09:25:181
19116992110,3cyclictest16074-21irqstats09:25:181
1910999219,3cyclictest2260-21kworker/0:310:11:220
1910999219,2cyclictest29462-21kworker/0:312:24:340
1910999217,3cyclictest10574-21kworker/0:009:18:130
1910999216,4cyclictest30486-21kworker/0:107:34:380
1910999212,2cyclictest87550irq/128-enp1s0-10:33:550
1910999212,10cyclictest19386-21kworker/0:108:25:410
1910999211,17cyclictest25269-21munin-run12:15:020
1910999211,10cyclictest3142-21kworker/0:211:23:490
1910999211,10cyclictest2991-21kworker/0:011:26:580
1912799206,6cyclictest25825-21df12:15:103
1912799205,5cyclictest790-21in:imjournal11:10:023
1912799204,8cyclictest0-21swapper/311:59:503
1912799204,11cyclictest0-21swapper/310:32:143
1912799203,5cyclictest32056-21grep08:45:163
1912799202,6cyclictest0-21swapper/311:50:133
1912799202,6cyclictest0-21swapper/311:50:133
19127992013,4cyclictest16642-21cut08:15:143
19127992013,4cyclictest16642-21cut08:15:143
1912199208,3cyclictest0-21swapper/208:56:162
1912199206,4cyclictest25644-21/usr/sbin/munin07:20:222
1912199202,8cyclictest0-21swapper/210:43:092
1912199201,9cyclictest29470-21hddtemp_smartct07:30:162
1911699208,10cyclictest0-21swapper/110:36:431
1911699208,10cyclictest0-21swapper/110:36:431
1911699207,3cyclictest0-21swapper/112:11:041
1911699206,3cyclictest11222-21vmstat08:00:271
19116992016,3cyclictest0-21swapper/110:21:111
1911699201,17cyclictest0-21swapper/110:41:331
1911699201,17cyclictest0-21swapper/108:35:131
1911699201,11cyclictest6611-21cpuspeed_turbos09:05:121
1911699201,11cyclictest5419-21proc_pri09:00:221
1911699201,11cyclictest4585-21unixbench_singl07:45:251
1911699201,11cyclictest2234-21cstates12:35:131
1911699201,10cyclictest30413-21latency_hist10:00:021
1911699201,10cyclictest14148-21cpuspeed_turbos08:10:111
1911699201,10cyclictest10428-21df11:40:111
1911699201,10cyclictest0-21swapper/109:32:321
1910999206,3cyclictest21083-21kworker/0:207:19:290
1910999206,2cyclictest4938-21kworker/0:307:11:260
1910999206,2cyclictest10574-21kworker/0:009:21:570
1910999203,3cyclictest273162egrep08:40:100
1910999201,11cyclictest9621-21/usr/sbin/munin08:00:130
1910999201,10cyclictest24402-21/usr/sbin/munin07:20:190
1912799194,12cyclictest9732-21ntp_states09:10:213
19127991912,4cyclictest13070-21/usr/sbin/munin09:20:303
1912799191,15cyclictest25246-21cpuspeed_turbos08:35:133
1912799191,10cyclictest0-21swapper/307:55:243
1912199196,3cyclictest0-21swapper/211:00:162
1912199194,3cyclictest0-21swapper/209:00:162
1912199191,10cyclictest23016-21df_inode10:55:152
1912199190,11cyclictest0-21swapper/211:11:082
1911699196,4cyclictest19226-21df12:00:131
19116991915,3cyclictest0-21swapper/110:55:351
19116991915,3cyclictest0-21swapper/107:20:101
19116991914,3cyclictest31605-21grep07:35:151
19116991912,4cyclictest18576-21cut10:45:151
1911699191,16cyclictest0-21swapper/110:30:441
1911699191,16cyclictest0-21swapper/110:07:511
1911699191,15cyclictest20376-21ntp_states07:10:181
1911699191,10cyclictest8727-21cpu09:10:101
1911699191,10cyclictest5948-21if_enp1s007:50:161
1911699191,10cyclictest29516-21/usr/sbin/munin07:30:171
1911699191,10cyclictest0-21swapper/111:10:241
1911699190,13cyclictest87550irq/128-enp1s0-11:08:541
1910999198,3cyclictest13678-21/usr/sbin/munin10:35:180
1910999198,3cyclictest13678-21/usr/sbin/munin10:35:180
1910999198,2cyclictest22415-21kworker/0:311:09:010
1910999197,3cyclictest26044-21kworker/0:007:45:380
1910999197,3cyclictest1080-21kworker/0:009:04:290
1910999194,4cyclictest6062-21kworker/0:109:12:450
1910999194,2cyclictest11859-21kworker/0:011:51:420
1910999194,2cyclictest11859-21kworker/0:011:51:420
1912799185,3cyclictest0-21swapper/309:37:013
1912799182,9cyclictest0-21swapper/307:35:153
19127991812,3cyclictest10547-21sort08:00:193
19127991810,5cyclictest4610-21diskstats09:00:143
1912199188,3cyclictest87550irq/128-enp1s0-11:54:132
1912199188,3cyclictest87550irq/128-enp1s0-11:54:132
1912199183,9cyclictest0-21swapper/209:24:002
1912199183,4cyclictest0-21swapper/210:30:062
1912199182,4cyclictest1179-21sort07:40:142
1912199181,8cyclictest0-21swapper/210:20:212
1912199181,7cyclictest32426-21/usr/sbin/munin08:45:182
1912199181,7cyclictest0-21swapper/211:40:242
1912199181,7cyclictest0-21swapper/210:46:012
1912199181,14cyclictest22518-21cut09:40:162
19116991814,3cyclictest0-21swapper/108:55:551
1911699181,16cyclictest0-21swapper/108:46:441
1911699181,16cyclictest0-21swapper/108:25:301
1911699181,16cyclictest0-21swapper/107:57:191
1911699181,16cyclictest0-21swapper/107:28:051
1911699181,15cyclictest0-21swapper/108:51:071
1911699181,15cyclictest0-21swapper/108:20:501
1910999186,3cyclictest13737-21kworker/0:208:16:450
1910999186,3cyclictest13737-21kworker/0:208:16:450
1910999186,2cyclictest15253-21kworker/0:109:40:540
1910999185,3cyclictest1080-21kworker/0:008:50:170
1910999182,7cyclictest22672-21/usr/sbin/munin10:55:140
247372171,3sleep00-21swapper/008:33:240
1912799170,5cyclictest87350irq/126-enp1s0-07:10:173
1912199173,8cyclictest0-21swapper/212:03:582
1912199173,3cyclictest5750irq/9-acpi11:20:192
1912199171,6cyclictest0-21swapper/210:18:002
1912199171,4cyclictest29277-21users12:20:242
1912199171,4cyclictest10504-21sort08:00:192
1912199171,2cyclictest0-21swapper/209:55:112
19121991712,3cyclictest3027-21/usr/sbin/munin07:45:232
19121991712,3cyclictest28590-21cat07:30:012
1912199171,15cyclictest0-21swapper/208:37:002
1912199171,15cyclictest0-21swapper/208:20:432
19121991711,3cyclictest21632-21munin-plugin-st09:40:002
1912199171,10cyclictest0-21swapper/211:15:582
1912199171,10cyclictest0-21swapper/210:27:372
19121991710,4cyclictest3640-21grep11:25:122
1911699170,16cyclictest0-21swapper/108:15:271
1911699170,16cyclictest0-21swapper/108:15:271
1910999172,4cyclictest32695-21sshd07:36:450
1910999171,6cyclictest0-21swapper/010:20:150
19109991712,3cyclictest1495-21hwlatdetect07:40:170
1910999170,9cyclictest4826-21kworker/0:207:51:280
1912799169,4cyclictest2739-21hddtemp_smartct08:55:143
1912199165,3cyclictest32-21ksoftirqd/209:26:182
1912199165,3cyclictest32-21ksoftirqd/209:26:172
1912199162,4cyclictest500-21sh07:40:012
1912199161,9cyclictest0-21swapper/207:50:102
1912199161,6cyclictest0-21swapper/212:10:182
1912199161,2cyclictest0-21swapper/208:31:402
19121991612,3cyclictest0-21swapper/207:17:082
19121991611,3cyclictest7423-21/usr/sbin/munin07:55:162
19121991611,3cyclictest22372-21swap12:05:232
19121991611,3cyclictest21382-21awk08:25:172
19121991610,3cyclictest9007-21df09:10:122
19121991610,3cyclictest20563-21sensors_temp07:10:202
1912799150,14cyclictest0-21swapper/308:51:273
1912199142,3cyclictest12050irq/123-ahci[0008:55:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional