You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-18 - 18:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Dec 18, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
200742280182,23sleep20-21swapper/219:06:172
201872231163,56sleep30-21swapper/319:07:483
200872201164,25sleep00-21swapper/019:06:270
200162201165,24sleep10-21swapper/119:05:311
9142590,6sleep339-21rcuc/300:30:163
26282590,3sleep12044699cyclictest23:20:191
298422540,1sleep30-21swapper/323:10:103
77882520,3sleep232-21ksoftirqd/219:50:242
240652510,1sleep20-21swapper/200:10:122
178812510,1sleep00-21swapper/020:15:130
99092500,3sleep39908-21fschecks_time21:10:143
20446994440,3cyclictest0-21swapper/121:15:191
278452340,18sleep327850-21unixbench_multi20:35:273
2045799340,26cyclictest0-21swapper/319:19:523
2043899337,12cyclictest0-21swapper/022:30:010
2043899327,12cyclictest30194-21munin-run00:25:010
2043899317,11cyclictest9149-21diskmemload21:20:010
2043899317,11cyclictest23938-21idleruntime-cro20:30:010
2043899307,10cyclictest988-21idleruntime-cro22:05:010
2043899302,13cyclictest4428-21munin-run00:40:010
2043899302,13cyclictest4428-21munin-run00:40:010
2045799291,11cyclictest0-21swapper/322:25:213
2043899294,12cyclictest1594-21munin-run23:20:010
2043899291,13cyclictest9144-21idleruntime-cro21:10:010
264832280,3sleep30-21swapper/300:15:153
2045799281,9cyclictest0-21swapper/323:02:043
2045799281,11cyclictest0-21swapper/320:10:573
2045799281,10cyclictest0-21swapper/300:38:283
2045799281,10cyclictest0-21swapper/300:38:283
2045299287,3cyclictest0-21swapper/220:45:002
20452992811,4cyclictest84202grep22:20:152
250602270,4sleep121-21rcuc/100:10:241
2045799270,13cyclictest0-21swapper/323:23:323
2045799270,13cyclictest0-21swapper/321:03:283
2045799270,12cyclictest0-21swapper/320:02:373
2044699271,11cyclictest18548-21latency_hist22:45:011
2043899272,13cyclictest27367-21munin-run23:05:000
2045799261,10cyclictest0-21swapper/321:25:283
2045799261,10cyclictest0-21swapper/319:30:043
2044699261,5cyclictest0-21swapper/120:46:071
2044699261,17cyclictest0-21swapper/121:40:141
2043899268,15cyclictest6961-21munin-run21:05:010
2045799251,8cyclictest0-21swapper/319:55:043
2045799251,8cyclictest0-21swapper/319:55:043
2045799251,10cyclictest0-21swapper/320:19:333
2045799250,11cyclictest0-21swapper/320:09:573
20446992521,3cyclictest0-21swapper/121:54:451
204022250,2sleep020448-21/usr/sbin/munin00:00:200
2045799241,9cyclictest0-21swapper/319:42:523
2045799241,10cyclictest0-21swapper/319:35:563
2045799240,10cyclictest0-21swapper/319:25:283
2044699247,10cyclictest17338-21timerandwakeup21:25:261
2044699244,3cyclictest0-21swapper/122:33:081
20446992420,2cyclictest0-21swapper/123:56:231
20446992419,3cyclictest0-21swapper/121:04:101
2044699241,15cyclictest0-21swapper/100:30:161
2044699241,10cyclictest25522-21idleruntime-cro19:20:011
2044699241,10cyclictest24805-21sed20:30:151
20446992410,3cyclictest19180-40dnf00:00:031
2045799230,12cyclictest0-21swapper/320:25:123
2045299235,5cyclictest87150irq/126-enp1s0-21:59:152
2045299233,5cyclictest23607-21munin-run00:10:012
20446992320,2cyclictest0-21swapper/121:59:561
2044699231,20cyclictest0-21swapper/100:20:141
2044699231,17cyclictest0-21swapper/119:51:281
2044699231,16cyclictest0-21swapper/100:39:041
2044699231,16cyclictest0-21swapper/100:39:041
2044699231,11cyclictest12938-21awk20:05:011
2044699231,11cyclictest10877-21cat20:00:011
2044699231,11cyclictest10877-21cat20:00:011
2044699231,10cyclictest869-21sssd_nss19:35:141
2044699231,10cyclictest29360-21irqstats20:40:181
2044699231,10cyclictest21915-21latency_hist20:25:021
2044699230,21cyclictest0-21swapper/123:11:091
2044699230,16cyclictest0-21swapper/121:45:111
2043899231,16cyclictest10742-21kworker/0:220:24:540
2045799222,9cyclictest20757-21awk22:50:013
2045799221,12cyclictest0-21swapper/320:52:323
2045799220,8cyclictest0-21swapper/322:35:293
2045799220,12cyclictest0-21swapper/300:20:403
2045799220,10cyclictest0-21swapper/323:15:363
2045299226,4cyclictest4126-21sh19:45:012
2045299222,6cyclictest20751-21sh22:50:012
20452992218,3cyclictest0-21swapper/221:22:342
20452992216,3cyclictest26123-21ls20:35:012
2044699226,4cyclictest31412-21head00:25:201
2044699222,3cyclictest0-21swapper/119:42:251
2044699222,15cyclictest0-21swapper/119:29:351
2044699222,15cyclictest0-21swapper/119:22:391
20446992219,2cyclictest0-21swapper/123:05:061
20446992218,2cyclictest5988-21sadc23:30:011
20446992218,2cyclictest5988-21sadc23:30:011
20446992218,2cyclictest0-21swapper/123:15:131
20446992215,4cyclictest23853-21sed22:55:151
2044699221,15cyclictest0-21swapper/119:48:391
2044699221,10cyclictest4387-21needreboot22:10:191
2044699221,10cyclictest2271-21ntp_states22:05:211
2044699221,10cyclictest15756-21idleruntime-cro21:25:011
2044699221,10cyclictest10632-21/usr/sbin/munin21:10:241
2044699221,10cyclictest0-21swapper/122:25:111
2044699220,20cyclictest0-21swapper/123:50:031
2043899221,12cyclictest0-21swapper/019:54:110
89902210,3sleep02043899cyclictest19:55:110
89902210,3sleep02043899cyclictest19:55:110
5532210,4sleep121-21rcuc/122:01:011
244462211,2sleep20-21swapper/221:44:022
2045799216,8cyclictest0-21swapper/323:40:203
2045799215,3cyclictest0-21swapper/320:40:083
2045799212,10cyclictest18168-21/usr/sbin/munin21:30:173
2045799211,9cyclictest0-21swapper/320:46:043
2045799211,7cyclictest0-21swapper/319:10:323
2045799211,13cyclictest0-21swapper/323:07:163
2045799210,6cyclictest0-21swapper/322:40:043
2045799210,10cyclictest0-21swapper/320:30:283
2045299216,4cyclictest13466-21expr20:05:132
2045299212,6cyclictest0-21swapper/219:30:152
2044699219,3cyclictest3743-21missed_timers20:55:191
2044699212,17cyclictest0-21swapper/122:51:131
2044699212,10cyclictest0-21swapper/121:39:381
2044699211,8cyclictest0-21swapper/122:15:111
2044699211,8cyclictest0-21swapper/122:15:111
2044699211,7cyclictest10607-21/usr/sbin/munin23:40:181
20446992114,4cyclictest21964-21df_inode00:05:151
20446992114,2cyclictest0-21swapper/120:37:591
2044699211,3cyclictest0-21swapper/120:18:431
2044699211,18cyclictest15433-21fschecks_count23:50:141
2044699211,18cyclictest0-21swapper/123:39:371
2044699211,18cyclictest0-21swapper/122:45:161
2044699211,15cyclictest0-21swapper/120:11:121
2044699211,14cyclictest0-21swapper/120:08:481
2044699211,11cyclictest0-21swapper/121:30:111
2044699211,10cyclictest9047-21awk22:20:241
2043899213,10cyclictest4361-21/usr/sbin/munin19:45:250
2043899211,6cyclictest0-21swapper/022:05:150
2043899210,11cyclictest29573-21perf23:10:000
317852201,3sleep00-21swapper/019:31:080
2045799209,3cyclictest20985-21/usr/sbin/munin22:50:133
2045799208,4cyclictest23833-21/usr/sbin/munin00:10:173
2045799204,3cyclictest0-21swapper/322:12:333
2045799203,6cyclictest31052-40dnf21:59:143
2045799201,11cyclictest0-21swapper/300:29:443
2045299205,4cyclictest87450irq/129-enp1s0-00:30:132
2045299205,4cyclictest12050irq/123-ahci[0019:20:012
2045299203,4cyclictest31766-21awk23:15:012
2045299202,6cyclictest1826-21/usr/sbin/munin23:20:232
2045299201,7cyclictest2777-21/usr/sbin/munin20:55:222
2045299201,5cyclictest576-21systemd-journal00:20:002
2045299201,5cyclictest14393-21users23:45:262
2044699205,3cyclictest87350irq/128-enp1s0-19:10:181
2044699201,17cyclictest0-21swapper/123:30:151
2044699201,14cyclictest0-21swapper/120:26:581
2044699201,10cyclictest7410-21ls21:05:141
2044699201,10cyclictest0-21swapper/100:17:351
2043899204,4cyclictest3592-21kworker/0:223:25:190
2043899204,4cyclictest3592-21kworker/0:223:25:190
2043899203,6cyclictest22898-21kworker/0:223:00:020
2043899202,10cyclictest3592-21kworker/0:223:47:340
20438992015,3cyclictest25577-21sed20:30:250
2043899201,11cyclictest24763-21/usr/sbin/munin21:45:240
2043899201,11cyclictest23543-21/usr/sbin/munin19:15:170
2043899201,11cyclictest16586-21/usr/sbin/munin22:40:240
2043899201,11cyclictest0-21swapper/020:45:100
20438992010,3cyclictest16371-21awk22:40:010
2045799192,8cyclictest11576-21/usr/sbin/munin21:15:153
2045799191,9cyclictest800-21snmpd19:54:233
2045799191,9cyclictest0-21swapper/300:05:243
2045799191,8cyclictest869-21sssd_nss22:55:483
2045799191,12cyclictest0-21swapper/321:09:123
2045799191,10cyclictest5757-21sshd23:27:113
2045799191,10cyclictest5757-21sshd23:27:113
2045799191,10cyclictest31361-21/usr/sbin/munin22:00:253
2045299196,3cyclictest87450irq/129-enp1s0-20:50:202
2045299195,3cyclictest87450irq/129-enp1s0-20:00:162
2045299195,3cyclictest87350irq/128-enp1s0-22:35:272
2045299195,3cyclictest12050irq/123-ahci[0000:00:022
2045299194,3cyclictest30-21rcuc/222:15:142
2045299194,3cyclictest30-21rcuc/222:15:142
2045299194,3cyclictest21003-21sshd19:10:102
2045299194,3cyclictest19796-21fschecks_count00:00:152
2045299193,4cyclictest27438-21open_files20:35:232
2045299193,4cyclictest2002-21if_err_enp2s022:05:172
2045299192,4cyclictest3449-21grep00:35:212
2045299192,4cyclictest3449-21grep00:35:212
2045299191,6cyclictest32328-21/usr/sbin/munin19:35:182
2045299191,6cyclictest25387-21/usr/sbin/munin23:00:242
2045299191,4cyclictest87150irq/126-enp1s0-23:40:012
20452991914,3cyclictest4020-21grep22:10:152
20452991913,3cyclictest28879-21gltestperf00:20:162
20452991911,5cyclictest31585-21/usr/sbin/munin20:45:172
2044699192,8cyclictest0-21swapper/120:50:231
2044699192,16cyclictest0-21swapper/119:30:031
2044699191,9cyclictest0-21swapper/122:35:121
2044699191,10cyclictest0-21swapper/123:03:301
2043899199,3cyclictest32619-21/usr/sbin/munin00:30:250
2043899198,3cyclictest14997-21/usr/sbin/munin23:50:220
2043899195,2cyclictest87350irq/128-enp1s0-22:55:020
2043899193,3cyclictest0-21swapper/021:35:220
20438991914,3cyclictest22421-21interrupts00:05:190
20438991913,3cyclictest30852-21ntp_states23:10:190
2043899191,15cyclictest22622-21munin-run19:15:010
2043899191,12cyclictest27725-21kworker/0:100:19:340
2043899191,10cyclictest2167-21/usr/sbin/munin19:40:200
2043899191,10cyclictest13775-21/usr/sbin/munin21:20:180
2043899190,13cyclictest0-21swapper/021:50:150
146592191,3sleep30-21swapper/323:48:433
279252181,2sleep00-21swapper/019:25:060
206222181,2sleep00-21swapper/022:48:290
2045799187,8cyclictest22717-21sa121:40:113
2045799181,9cyclictest0-21swapper/319:49:483
2045799181,15cyclictest0-21swapper/300:00:143
2045799181,11cyclictest0-21swapper/320:23:443
2045299184,3cyclictest87450irq/129-enp1s0-20:20:222
2045299184,3cyclictest87450irq/129-enp1s0-20:10:192
2045299183,3cyclictest5005-21cpu21:00:122
2045299182,4cyclictest13273-21awk22:30:212
2045299181,7cyclictest87150irq/126-enp1s0-23:40:102
2045299181,6cyclictest9149-21diskmemload23:53:152
2045299181,5cyclictest27936-21/usr/sbin/munin19:25:282
2045299181,5cyclictest24763-21/usr/sbin/munin21:45:122
2045299181,5cyclictest0-21swapper/223:33:112
2045299181,4cyclictest22010-21egrep20:25:122
2045299181,4cyclictest21284-21memory21:35:192
20452991813,3cyclictest19130-21grep20:15:262
20452991812,4cyclictest27253-21sed19:20:272
20452991811,4cyclictest9349-21grep19:55:142
20452991811,4cyclictest9349-21grep19:55:142
2045299181,14cyclictest28141-21cut21:50:232
2043899181,9cyclictest800-21snmpd23:56:320
2043899181,9cyclictest0-21swapper/000:25:110
2043899181,8cyclictest25742-21/usr/sbin/munin19:20:260
20438991813,3cyclictest9382-21/usr/sbin/munin21:10:190
20438991813,3cyclictest17931-21idleruntime-cro21:30:010
20438991811,4cyclictest1225-21head20:50:150
321112170,2sleep20-21swapper/200:26:402
2045799179,5cyclictest21631-21smart_sda21:35:243
2045799173,2cyclictest88350irq/132-enp2s0-21:46:183
2045799171,9cyclictest7877-21sshd23:31:113
2045799171,4cyclictest17199-21/usr/sbin/munin23:55:143
20457991711,3cyclictest7067-21grep22:15:263
20457991711,3cyclictest7067-21grep22:15:263
2045299174,3cyclictest87150irq/126-enp1s0-23:05:202
2045299173,3cyclictest12050irq/123-ahci[0023:15:172
2045299171,6cyclictest11473-21grep22:25:252
2045299171,5cyclictest19647-21unixbench-2d21:30:252
2045299171,5cyclictest15970-21/usr/sbin/munin21:25:222
2045299171,5cyclictest0-21swapper/222:41:112
2045299171,4cyclictest4602-21gltestperf23:25:152
2045299171,4cyclictest4602-21gltestperf23:25:152
2045299171,14cyclictest0-21swapper/221:05:072
20452991711,3cyclictest11576-21/usr/sbin/munin21:15:112
20452991711,3cyclictest10147-21if_enp2s021:10:162
2043899173,11cyclictest6219-21/usr/sbin/munin23:30:160
2043899171,9cyclictest10742-21kworker/0:220:35:070
2043899171,8cyclictest12545-21NetworkManager23:44:430
20438991712,3cyclictest28853-21expr20:40:130
20438991712,3cyclictest19708-21grep21:30:250
2043899171,15cyclictest0-21swapper/000:11:330
2043899171,13cyclictest7801-21/usr/sbin/munin22:20:190
20438991711,3cyclictest28655-21kworker/0:321:55:100
20438991711,3cyclictest23376-21interrupts21:40:160
20438991711,3cyclictest12192-21/usr/sbin/munin22:30:150
2043899170,10cyclictest0-21swapper/020:14:110
2045799165,3cyclictest12050irq/123-ahci[0021:55:013
2045799161,7cyclictest12050irq/123-ahci[0019:25:013
2045799161,6cyclictest1208-21/usr/sbin/munin22:05:233
2045799161,6cyclictest0-21swapper/320:55:223
20457991612,3cyclictest0-21swapper/323:37:273
2045799161,13cyclictest0-21swapper/322:30:113
2045799161,12cyclictest87150irq/126-enp1s0-23:50:113
2045799161,10cyclictest0-21swapper/322:24:573
2045799160,8cyclictest0-21swapper/321:24:503
2045299163,3cyclictest87350irq/128-enp1s0-19:45:132
2045299161,3cyclictest31361-21/usr/sbin/munin22:00:132
2045299161,14cyclictest0-21swapper/222:56:022
2045299160,5cyclictest0-21swapper/222:50:152
2043899169,4cyclictest1957-21grep23:20:150
2043899161,9cyclictest0-21swapper/022:15:190
2043899161,9cyclictest0-21swapper/022:15:190
20438991611,3cyclictest513-21head19:35:150
20438991611,3cyclictest13184-21cpu20:05:130
2043899161,13cyclictest12302-21grep20:00:260
20438991610,4cyclictest9447-21ntp_kernel_err23:35:230
2043899160,10cyclictest0-21swapper/022:11:110
2043899151,3cyclictest2183-21kworker/0:020:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional