You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 18:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 15, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
282482237162,26sleep20-21swapper/207:08:092
280442206168,26sleep00-21swapper/007:05:300
282772201164,25sleep10-21swapper/107:08:321
280472201165,24sleep30-21swapper/307:05:333
2751621520,2sleep32848699cyclictest10:45:173
170212560,4sleep1221ktimersoftd/107:55:111
28476994844,3cyclictest0-21swapper/111:24:071
28476994844,3cyclictest0-21swapper/111:24:071
214912480,1sleep011-21rcuc/008:05:130
2846699388,14cyclictest17185-21munin-run09:10:010
28466993810,13cyclictest7778-21munin-run07:35:010
28476993633,2cyclictest0-21swapper/109:50:481
2846699356,13cyclictest4605-21munin-run09:55:010
2846699334,13cyclictest22203-21munin-run10:35:010
2846699314,17cyclictest8382-21munin-run08:50:010
2846699302,13cyclictest767-21awk11:00:010
49372290,8sleep232-21ksoftirqd/209:55:112
2848299291,12cyclictest0-21swapper/210:08:462
2848299291,12cyclictest0-21swapper/210:08:462
28486992712,3cyclictest87450irq/129-enp1s0-08:35:233
2846699279,15cyclictest25978-21munin-run09:30:010
2848699261,8cyclictest0-21swapper/311:40:503
2846699261,12cyclictest22812-21munin-run11:50:010
28466992610,14cyclictest26602-21perf10:45:010
2847699251,4cyclictest23361-21grep09:20:241
2848699241,11cyclictest2168-21sshd09:47:073
2848699241,11cyclictest10188-21sshd07:40:083
2848299243,4cyclictest26026-21latency_hist09:30:022
2846699242,19cyclictest6809-21munin-run10:00:010
156932240,5sleep32848699cyclictest11:30:233
2848699238,3cyclictest3565-21sshd12:19:083
2848699237,3cyclictest11664-21sshd11:23:083
2848699237,3cyclictest11664-21sshd11:23:083
2848299231,12cyclictest0-21swapper/212:22:022
2847699232,3cyclictest449-21ntp_states12:10:211
2847699232,17cyclictest5236-21munin-run11:10:011
2847699231,16cyclictest0-21swapper/107:53:061
2847699231,15cyclictest17192-21diskmemload10:02:171
2846699238,8cyclictest18090-21kworker/0:112:07:180
28466992315,4cyclictest23550-21tr08:10:110
2846699230,11cyclictest25019-21awk11:55:010
58172220,1sleep10-21swapper/109:55:171
2848699225,3cyclictest0-21swapper/311:26:563
2848699224,4cyclictest19683-21sshd10:26:083
2848699221,10cyclictest28672-21/usr/sbin/munin07:10:143
2848699220,9cyclictest0-21swapper/308:27:423
2848299229,5cyclictest87450irq/129-enp1s0-11:40:132
2848299229,5cyclictest87450irq/129-enp1s0-08:50:242
2848299228,3cyclictest0-21swapper/212:03:542
28476992214,3cyclictest0-21swapper/107:17:081
2847699221,2cyclictest20854-21mailstats09:15:221
2847699221,18cyclictest19981-21seq11:40:241
2847699221,16cyclictest0-21swapper/112:38:091
2847699221,16cyclictest0-21swapper/112:03:161
2847699221,16cyclictest0-21swapper/109:49:281
2846699222,8cyclictest32439-21kworker/0:008:54:400
2846699222,10cyclictest25087-21kworker/0:008:20:200
2846699221,6cyclictest14024-21turbostat11:30:010
27132221,2sleep00-21swapper/008:35:160
2848699216,2cyclictest0-21swapper/309:40:143
2848699214,7cyclictest10468-21/usr/sbin/munin12:35:183
2848699212,6cyclictest11438-21/usr/sbin/munin10:10:133
2848699211,5cyclictest11873-21ntp_states08:55:213
2848299219,3cyclictest17258-21irqstats11:35:182
2848299217,5cyclictest12050irq/123-ahci[0012:35:012
2848299211,12cyclictest0-21swapper/210:45:422
2847699211,3cyclictest4772-21proc_pri07:25:221
2847699211,3cyclictest0-21swapper/111:15:241
28476992113,3cyclictest0-21swapper/110:36:211
28476992112,2cyclictest11778-21sh10:10:131
28476992111,9cyclictest0-21swapper/108:14:301
2847699211,15cyclictest0-21swapper/112:15:591
2847699211,15cyclictest0-21swapper/112:06:141
2847699211,15cyclictest0-21swapper/111:26:391
2847699211,15cyclictest0-21swapper/111:01:281
2847699211,15cyclictest0-21swapper/110:42:511
2847699211,15cyclictest0-21swapper/109:06:151
2847699211,15cyclictest0-21swapper/108:47:071
2847699211,15cyclictest0-21swapper/108:09:561
2847699211,14cyclictest0-21swapper/109:27:031
2847699211,14cyclictest0-21swapper/109:11:051
2847699211,11cyclictest12549-21tr07:45:101
2847699211,11cyclictest12549-21tr07:45:101
2847699210,16cyclictest0-21swapper/112:25:141
2846699211,8cyclictest30153-21kworker/0:209:40:120
40852200,4sleep29-21rcu_preempt07:25:142
2848699207,2cyclictest24307-21sshd10:39:073
2848699206,3cyclictest0-21swapper/308:54:023
2848699205,4cyclictest87450irq/129-enp1s0-09:50:203
2848699202,5cyclictest6527-21ntp_states11:10:203
2848699202,15cyclictest24621-21/usr/sbin/munin10:40:223
2848699201,7cyclictest3259-21/usr/sbin/munin11:05:223
2848699201,6cyclictest0-21swapper/307:19:393
2848699201,5cyclictest13631-21/usr/sbin/munin10:15:133
2848699200,10cyclictest0-21swapper/311:54:163
2848299209,7cyclictest23417-21systemd-journal09:20:312
2848299207,5cyclictest12050irq/123-ahci[0008:05:022
2847699209,3cyclictest4565-21diskstats08:40:141
2847699201,5cyclictest800-21snmpd10:50:171
28476992013,2cyclictest0-21swapper/110:06:431
28476992013,2cyclictest0-21swapper/110:06:431
28476992013,2cyclictest0-21swapper/109:38:101
28476992013,2cyclictest0-21swapper/108:01:481
2847699201,2cyclictest0-21swapper/108:28:371
28476992012,3cyclictest0-21swapper/109:04:291
2847699201,17cyclictest0-21swapper/110:21:301
2847699201,15cyclictest0-21swapper/112:34:301
2847699201,15cyclictest0-21swapper/110:16:581
2847699201,14cyclictest0-21swapper/112:22:041
2847699201,14cyclictest0-21swapper/111:38:051
2847699201,14cyclictest0-21swapper/111:12:091
2847699201,11cyclictest30793-21munin-node08:55:271
2847699201,11cyclictest20390-21cpuspeed_turbos10:30:131
2847699200,14cyclictest0-21swapper/109:30:031
2846699209,3cyclictest19609-21/usr/sbin/munin09:15:170
2846699208,2cyclictest23709-21kworker/0:009:47:240
2846699206,2cyclictest1166-21kworker/0:012:17:520
2846699205,2cyclictest4895-21kworker/0:211:10:560
2846699202,11cyclictest13631-21/usr/sbin/munin10:15:200
2846699202,10cyclictest9854-21/usr/sbin/munin11:20:230
2846699202,10cyclictest9854-21/usr/sbin/munin11:20:220
2846699202,10cyclictest29980-21/usr/sbin/munin08:25:220
2846699202,10cyclictest11438-21/usr/sbin/munin10:10:230
2846699201,9cyclictest30354-21kworker/0:107:55:070
2846699201,8cyclictest30354-21kworker/0:107:54:110
28466992014,3cyclictest29934-21memory10:50:200
2846699201,16cyclictest12173-21munin-run07:45:010
2846699201,11cyclictest1046-21/usr/sbin/munin11:00:230
284012200,4sleep02846699cyclictest10:45:260
115872201,2sleep20-21swapper/211:22:092
115872201,2sleep20-21swapper/211:22:092
2848699198,2cyclictest0-21swapper/309:00:153
2848699195,3cyclictest3876-21/usr/sbin/munin12:20:183
2848699191,5cyclictest1407-21/usr/sbin/munin07:20:213
2848699191,5cyclictest0-21swapper/308:30:123
2848699191,4cyclictest15063-21grep07:50:143
2848699191,2cyclictest31842-21/usr/sbin/munin12:10:143
2848699191,15cyclictest5502-21head09:55:153
2848299198,4cyclictest87150irq/126-enp1s0-08:25:582
2848299193,5cyclictest30-21rcuc/210:55:132
2848299192,7cyclictest32-21ksoftirqd/207:30:112
2848299191,9cyclictest87450irq/129-enp1s0-11:15:172
2848299191,8cyclictest12050irq/123-ahci[0009:35:022
2848299191,6cyclictest3259-21/usr/sbin/munin11:05:132
28482991914,3cyclictest12274-21grep08:55:252
2848299191,3cyclictest18159-21cpuspeed_turbos10:25:122
28482991912,4cyclictest26287-21ntpq11:55:202
2848299191,10cyclictest0-21swapper/209:08:462
28476991913,2cyclictest0-21swapper/107:30:351
28476991912,2cyclictest0-21swapper/109:43:111
28476991912,2cyclictest0-21swapper/108:51:591
2847699191,16cyclictest0-21swapper/110:48:531
2847699191,16cyclictest0-21swapper/108:30:091
2847699191,14cyclictest0-21swapper/111:50:551
28476991911,3cyclictest0-21swapper/107:41:311
2847699191,10cyclictest29871-21cat08:25:021
2847699191,10cyclictest2311-21cat07:20:161
2847699191,10cyclictest20195-21cat10:30:021
2847699191,10cyclictest0-21swapper/107:10:151
2847699190,14cyclictest0-21swapper/110:57:301
2847699190,14cyclictest0-21swapper/108:15:471
2847699190,14cyclictest0-21swapper/107:37:131
2846699198,3cyclictest12393-21/usr/sbin/munin07:45:220
2846699198,3cyclictest12393-21/usr/sbin/munin07:45:220
2846699196,2cyclictest16920-21kworker/0:109:22:230
2846699195,2cyclictest24149-21kworker/0:210:37:560
2846699195,2cyclictest15848-21kworker/0:011:35:160
2846699191,10cyclictest800-21snmpd11:43:250
2846699191,10cyclictest32177-21/usr/sbin/munin08:30:190
2848699188,2cyclictest0-21swapper/311:55:183
2848699186,2cyclictest87150irq/126-enp1s0-09:25:133
2848699181,6cyclictest41-21ksoftirqd/308:05:173
2848699181,6cyclictest0-21swapper/312:33:353
2848699181,5cyclictest21168-21cat08:05:023
28486991814,2cyclictest41-21ksoftirqd/310:00:143
2848699181,15cyclictest0-21swapper/308:10:163
2848699181,13cyclictest7073-21gltestperf08:45:153
2848699181,10cyclictest0-21swapper/312:27:463
28486991810,5cyclictest7103-21kworker/u8:108:20:123
2848299187,3cyclictest0-21swapper/207:40:242
2848299185,2cyclictest17192-21diskmemload09:15:222
2848299184,5cyclictest6073-21/usr/sbin/munin12:25:142
2848299181,6cyclictest0-21swapper/208:14:392
2848299181,5cyclictest0-21swapper/211:53:022
2848299181,3cyclictest87150irq/126-enp1s0-11:45:142
2848299181,2cyclictest0-21swapper/211:11:072
28482991812,3cyclictest28810-21netstat08:20:192
2848299181,11cyclictest0-21swapper/208:47:222
2848299181,10cyclictest0-21swapper/211:30:212
2848299181,10cyclictest0-21swapper/209:12:092
2847699181,3cyclictest0-21swapper/111:30:291
2847699181,15cyclictest0-21swapper/111:45:111
2847699180,14cyclictest0-21swapper/111:58:001
2847699180,14cyclictest0-21swapper/108:39:121
2846699187,3cyclictest7031-21/usr/sbin/munin10:00:180
2846699185,3cyclictest88350irq/132-enp2s0-08:58:360
2846699185,2cyclictest30354-21kworker/0:107:21:380
2846699185,2cyclictest15466-21kworker/0:010:25:480
2846699184,3cyclictest24873-21kworker/0:211:55:160
2846699184,2cyclictest87150irq/126-enp1s0-10:20:140
2846699181,8cyclictest18090-21kworker/0:112:00:180
2846699181,6cyclictest0-21swapper/012:20:170
2846699181,5cyclictest0-21swapper/008:00:220
2846699181,3cyclictest3867-21cat07:25:120
2846699181,3cyclictest17192-21diskmemload10:05:220
2846699181,3cyclictest17192-21diskmemload10:05:220
28466991813,3cyclictest14246-21/usr/sbin/munin11:30:160
2846699181,10cyclictest17192-21diskmemload11:08:190
2846699181,10cyclictest0-21swapper/012:33:350
305522170,2sleep00-21swapper/007:13:580
2848699177,2cyclictest0-21swapper/309:35:183
2848699175,5cyclictest87450irq/129-enp1s0-08:40:143
2848699173,3cyclictest39-21rcuc/309:10:113
2848699171,3cyclictest0-21swapper/309:19:433
28486991712,3cyclictest30707-21ntp_states12:05:213
28486991712,3cyclictest1396-21df_inode11:00:133
28486991711,4cyclictest17101-21swap10:20:223
2848699171,14cyclictest0-21swapper/307:47:083
2848699171,14cyclictest0-21swapper/307:47:083
28486991711,3cyclictest8003-21/usr/sbin/munin07:35:163
28486991711,3cyclictest28246-21kernelversion12:00:173
2848699171,10cyclictest0-21swapper/309:07:133
2848699170,4cyclictest0-21swapper/307:30:103
2848699170,15cyclictest0-21swapper/311:45:413
2848299179,5cyclictest30014-21systemd-journal09:35:322
2848299172,4cyclictest29740-21ntp_states07:10:162
2848299172,3cyclictest29988-21df_inode12:05:132
2848299172,3cyclictest13631-21/usr/sbin/munin10:15:182
2848299171,9cyclictest0-21swapper/208:35:222
2848299171,6cyclictest0-21swapper/207:55:092
2848299171,6cyclictest0-21swapper/207:15:252
28482991712,4cyclictest7103-21kworker/u8:107:20:332
28482991712,3cyclictest32-21ksoftirqd/209:53:012
28482991712,3cyclictest30600-21/usr/sbin/munin09:40:192
28482991711,4cyclictest2283-21head12:15:152
2848299171,14cyclictest0-21swapper/210:41:192
2848299171,13cyclictest30083-21ntp_states10:50:222
28482991711,3cyclictest20866-21gltestperf10:30:162
2846699179,2cyclictest87150irq/126-enp1s0-12:36:240
2846699177,8cyclictest14988-21dump-pmu-power09:05:010
2846699176,3cyclictest2510-21kworker/0:111:17:140
2846699174,8cyclictest27918-40dnf09:31:580
2846699174,2cyclictest1166-21kworker/0:012:14:000
2846699173,3cyclictest23709-21kworker/0:009:38:520
28466991712,3cyclictest6073-21/usr/sbin/munin12:25:110
2846699171,13cyclictest19385-21grep09:15:010
2846699171,10cyclictest0-21swapper/007:37:370
2848699168,4cyclictest28203-21cat09:35:023
2848699161,3cyclictest0-21swapper/310:30:143
28486991612,3cyclictest0-21swapper/310:07:383
28486991612,3cyclictest0-21swapper/310:07:383
2848699161,14cyclictest0-21swapper/311:16:063
2848699161,13cyclictest17083-21cstates07:55:113
2848699161,12cyclictest31451-21cstates10:55:133
2848699161,12cyclictest29020-21/usr/sbin/munin10:50:213
2848699161,12cyclictest22849-21grep09:20:193
28486991610,3cyclictest17787-21timerandwakeup11:35:243
2848699160,9cyclictest0-21swapper/308:15:113
2848299162,5cyclictest32-21ksoftirqd/212:39:392
2848299161,8cyclictest0-21swapper/210:10:222
2848299161,6cyclictest4216-21/usr/sbin/munin08:40:172
2848299161,5cyclictest21697-21fschecks_time08:05:152
2848299161,5cyclictest13009-21/usr/sbin/munin09:00:152
2848299161,4cyclictest0-21swapper/208:18:172
2848299161,2cyclictest0-21swapper/207:45:142
2848299161,2cyclictest0-21swapper/207:45:142
28482991612,3cyclictest32-21ksoftirqd/207:35:162
28482991612,3cyclictest0-21swapper/211:26:342
28482991612,3cyclictest0-21swapper/210:35:332
28482991611,3cyclictest645-21missed_timers08:30:172
28482991611,3cyclictest1046-21/usr/sbin/munin11:00:212
2848299160,14cyclictest0-21swapper/207:51:322
28466991611,3cyclictest31572-21/usr/sbin/munin07:15:120
2846699161,12cyclictest4216-21/usr/sbin/munin08:40:140
28486991510,3cyclictest4646-21ntpq07:25:203
2848299159,3cyclictest16015-21expr10:20:102
2848299158,4cyclictest32436-21grep12:10:152
2848299157,3cyclictest972-21cut09:45:142
28482991510,3cyclictest7049-21cpu10:00:112
2846699154,2cyclictest88350irq/132-enp2s0-08:15:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional