You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-25 - 19:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 25, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
174152240164,24sleep00-21swapper/007:08:570
174892237165,23sleep20-21swapper/207:09:562
172012236163,24sleep30-21swapper/307:06:123
173032212181,20sleep10-21swapper/107:07:291
17597991241,116cyclictest0-21swapper/210:19:102
1759799106103,2cyclictest0-21swapper/210:38:142
238332490,3sleep10-21swapper/109:49:061
253692480,4sleep011-21rcuc/007:25:140
280752470,1sleep00-21swapper/012:25:160
1759199451,27cyclictest21712-21latency_hist09:45:011
1759199443,26cyclictest808-21snmpd11:18:181
1759199421,32cyclictest10940-21missed_timers11:45:191
1759199421,29cyclictest29126-21grep10:00:171
1760299411,29cyclictest0-21swapper/310:26:093
1759199412,35cyclictest29889-21munin-run08:50:011
1759199411,36cyclictest4126-21sh09:05:011
1759199401,22cyclictest29075-21latency_hist11:15:011
1759199381,33cyclictest6628-21runrttasks10:21:021
1759199381,28cyclictest10279-21unixbench_multi09:15:261
1759199381,26cyclictest15959-21/usr/sbin/munin10:45:161
1759199381,23cyclictest18858-21latency_hist08:25:021
1759199381,21cyclictest12509-21unixbench_singl09:20:261
1759199378,17cyclictest0-21swapper/112:30:211
1759199371,33cyclictest26979-21mailstats08:40:251
1759199371,23cyclictest5233-21cat11:33:101
17591993710,19cyclictest0-21swapper/112:15:231
1759199361,27cyclictest14829-21ntp_states10:40:221
1759199361,23cyclictest26242-21grep07:25:231
1759199361,23cyclictest11868-21unixbench_singl08:05:301
1759199361,19cyclictest29626-21sh12:30:011
1759199351,22cyclictest24258-21chrt11:01:301
1759199351,20cyclictest23883-21kworker/u8:209:58:561
1759199351,20cyclictest17544-21/usr/sbin/munin09:35:151
1759199351,20cyclictest15253-21irqstats11:55:181
1760299341,22cyclictest0-21swapper/308:07:413
1759199342,24cyclictest4258-21sed10:15:261
1759199341,30cyclictest6781-21head11:35:241
1759199341,30cyclictest6781-21head11:35:241
1759199341,30cyclictest21422-21diskstats12:10:161
1759199341,30cyclictest14324-21/usr/sbin/munin09:25:231
1759199341,24cyclictest6908-21diskstats09:10:161
1759199341,24cyclictest6908-21diskstats09:10:161
1759199341,24cyclictest12228-21/usr/sbin/munin11:50:181
1759199341,22cyclictest0-21swapper/108:15:231
1759199341,21cyclictest0-21swapper/111:25:151
1759199341,16cyclictest2301-21kworker/u8:310:12:091
1760299331,30cyclictest0-21swapper/311:15:333
1760299331,22cyclictest0-21swapper/307:13:493
1760299331,15cyclictest0-21swapper/308:29:373
1759199333,20cyclictest25083-21cstates11:05:121
1759199331,29cyclictest18878-21/usr/sbin/munin12:05:231
1759199331,29cyclictest11570-21/usr/sbin/munin10:35:141
1759199331,21cyclictest5982-21ls07:55:141
1759199331,21cyclictest21618-21sensors_temp10:55:241
17591993310,20cyclictest9101-21swap11:40:261
1758799335,13cyclictest3471-21munin-run07:50:010
1760299321,20cyclictest0-21swapper/309:10:213
1760299321,20cyclictest0-21swapper/309:10:213
1760299321,15cyclictest0-21swapper/310:19:253
1760299320,21cyclictest0-21swapper/311:56:493
1760299320,13cyclictest0-21swapper/309:25:013
1759199327,16cyclictest87650irq/129-enp1s0-07:10:191
1759199322,26cyclictest3173-40dnf07:46:291
1759199322,26cyclictest22551-21smart_sda08:30:251
1759199321,28cyclictest9318-21cat10:30:011
1759199321,28cyclictest26074-21grep12:20:181
1759199321,27cyclictest18511-21/usr/sbin/munin10:50:171
1759199321,23cyclictest6335-21diskmemload09:33:151
1759199321,21cyclictest0-21swapper/108:35:221
1759199321,19cyclictest31505-21cat07:40:011
1759199321,18cyclictest32051-21kworker/u8:012:36:501
1759199320,29cyclictest4360-21/usr/sbin/munin09:05:291
1759199320,23cyclictest0-21swapper/112:00:201
1760299311,21cyclictest0-21swapper/308:34:413
1760299311,19cyclictest0-21swapper/307:55:213
1760299311,18cyclictest0-21swapper/312:08:533
1760299311,12cyclictest0-21swapper/311:34:263
1759199316,22cyclictest30119-21/usr/sbin/munin08:50:191
1759199314,25cyclictest19201-21kworker/u8:109:50:241
1759199311,18cyclictest28023-21kworker/u8:010:08:361
1759199311,15cyclictest5281-21uptime07:50:261
1759199310,22cyclictest0-21swapper/108:10:221
1758799311,9cyclictest26695-21kworker/0:211:11:090
1760299301,13cyclictest0-21swapper/311:43:493
1760299301,13cyclictest0-21swapper/307:34:293
1760299301,12cyclictest0-21swapper/310:02:573
1760299301,11cyclictest0-21swapper/310:35:533
17597993027,2cyclictest0-21swapper/207:33:562
1759799301,9cyclictest0-21swapper/207:43:112
1759199301,26cyclictest31656-21/usr/sbin/munin07:40:181
1759199301,26cyclictest31356-21/usr/sbin/munin11:20:251
1759199301,26cyclictest27263-21/usr/sbin/munin07:30:121
1759199301,26cyclictest21693-21netstat07:15:191
1759199300,27cyclictest8296-21cstates08:00:131
1760299291,2cyclictest0-21swapper/310:10:133
1760299291,14cyclictest0-21swapper/309:15:093
1760299291,13cyclictest0-21swapper/312:15:013
1760299291,13cyclictest0-21swapper/308:23:093
1760299291,12cyclictest0-21swapper/311:46:213
1760299291,12cyclictest0-21swapper/311:23:253
1760299291,12cyclictest0-21swapper/310:53:013
1760299291,12cyclictest0-21swapper/308:02:423
1760299291,11cyclictest0-21swapper/311:35:053
1760299291,11cyclictest0-21swapper/311:35:053
1760299291,11cyclictest0-21swapper/309:52:573
1760299290,20cyclictest0-21swapper/307:51:213
1760299290,19cyclictest0-21swapper/309:01:533
1760299290,19cyclictest0-21swapper/308:13:173
1759799290,11cyclictest0-21swapper/212:31:552
1759199291,25cyclictest32316-21/usr/sbin/munin08:55:171
1759199291,25cyclictest22870-21/usr/sbin/munin07:20:201
1759199291,16cyclictest9376-21/usr/sbin/munin10:30:161
1759199291,16cyclictest19271-21cstates08:25:141
1760299281,13cyclictest0-21swapper/312:29:133
1760299281,13cyclictest0-21swapper/311:53:133
1760299281,11cyclictest0-21swapper/312:16:013
1760299281,11cyclictest0-21swapper/312:04:013
1760299281,11cyclictest0-21swapper/307:21:453
1760299280,19cyclictest0-21swapper/308:17:493
1760299280,13cyclictest0-21swapper/309:28:373
1760299280,13cyclictest0-21swapper/309:06:373
1760299280,12cyclictest0-21swapper/307:46:173
1760299280,12cyclictest0-21swapper/307:15:413
1760299280,11cyclictest0-21swapper/312:30:053
1760299280,10cyclictest0-21swapper/308:54:133
1759799281,11cyclictest6335-21diskmemload11:51:132
1759799281,11cyclictest0-21swapper/210:40:192
1760299271,14cyclictest0-21swapper/308:39:293
1760299271,13cyclictest0-21swapper/311:10:293
1760299271,12cyclictest0-21swapper/312:24:333
1760299271,12cyclictest0-21swapper/309:30:293
1760299271,11cyclictest0-21swapper/310:21:533
1760299271,11cyclictest0-21swapper/308:41:213
1760299271,10cyclictest0-21swapper/312:39:373
1760299271,10cyclictest0-21swapper/311:09:293
1760299270,12cyclictest0-21swapper/311:29:133
1760299270,11cyclictest0-21swapper/310:55:573
1760299270,10cyclictest0-21swapper/310:33:013
1759799271,11cyclictest2301-21kworker/u8:310:20:052
1759799271,10cyclictest0-21swapper/211:29:492
1759799270,11cyclictest0-21swapper/212:08:192
1758799271,13cyclictest20850-21munin-run12:10:010
1760299266,3cyclictest0-21swapper/310:06:533
1760299261,10cyclictest0-21swapper/310:47:013
1760299260,11cyclictest0-21swapper/307:35:133
1759799263,10cyclictest17796-40dnf10:48:242
1759799260,12cyclictest0-21swapper/212:14:492
1758799266,17cyclictest22650-21munin-run07:20:010
1758799262,8cyclictest8194-21kworker/0:209:24:590
1758799261,8cyclictest26779-21kworker/0:007:30:030
1758799261,3cyclictest0-21swapper/011:57:370
221532250,11sleep232-21ksoftirqd/209:45:142
1760299259,8cyclictest0-21swapper/307:44:573
1760299251,12cyclictest0-21swapper/307:27:373
1760299251,11cyclictest0-21swapper/309:39:263
1760299251,10cyclictest0-21swapper/311:00:213
1760299250,12cyclictest0-21swapper/309:48:013
1760299250,10cyclictest0-21swapper/309:59:453
1760299250,10cyclictest0-21swapper/309:42:053
1759799250,11cyclictest0-21swapper/211:33:512
1758799251,9cyclictest8194-21kworker/0:209:32:170
1760299241,10cyclictest0-21swapper/308:56:093
1760299240,12cyclictest0-21swapper/308:49:293
1759799241,3cyclictest0-21swapper/209:57:532
1759799241,13cyclictest0-21swapper/208:49:452
1759799241,11cyclictest0-21swapper/208:26:332
1759799241,10cyclictest0-21swapper/208:30:372
1759799240,9cyclictest0-21swapper/211:21:112
1758799241,3cyclictest23504-21/usr/sbin/munin08:35:170
1758799240,9cyclictest746-21kworker/0:111:31:310
1759799238,6cyclictest0-21swapper/210:10:262
1759799231,2cyclictest89050irq/133-enp2s0-09:02:372
1759799231,11cyclictest0-21swapper/208:07:252
1758799239,3cyclictest25781-21kworker/0:209:54:450
1758799237,4cyclictest87672sed11:40:210
1758799232,9cyclictest27409-21kworker/0:108:54:350
1758799231,9cyclictest27409-21kworker/0:109:12:010
1758799231,9cyclictest27409-21kworker/0:109:12:010
1758799231,9cyclictest27409-21kworker/0:108:45:090
1758799231,11cyclictest26340-21/usr/sbin/munin09:55:170
1758799230,8cyclictest32488-21kworker/0:010:15:090
1758799230,11cyclictest1259-21perf07:45:010
1758799230,11cyclictest0-21swapper/007:15:010
1759799223,2cyclictest89150irq/134-enp2s0-10:30:332
1759799221,10cyclictest0-21swapper/207:55:452
1758799229,3cyclictest18776-21kworker/0:008:40:070
1758799229,2cyclictest25781-21kworker/0:210:11:160
1758799228,3cyclictest5346-21kworker/0:307:57:430
1758799225,2cyclictest9468-21kworker/0:211:46:270
1758799222,9cyclictest9164-21kworker/0:010:31:450
1758799222,8cyclictest3163-21kworker/0:207:50:390
1758799222,8cyclictest18776-21kworker/0:008:31:590
1758799222,11cyclictest18482-21kworker/0:212:13:100
1758799221,8cyclictest26324-21kworker/0:212:21:110
1758799221,11cyclictest4975-21/usr/sbin/munin10:20:170
143572220,4sleep341-21ksoftirqd/310:40:173
1759799215,9cyclictest3173-40dnf07:46:272
1759799215,10cyclictest4528-21kworker/2:208:23:072
1759799212,3cyclictest89250irq/135-enp2s0-10:00:292
1759799212,11cyclictest0-21swapper/211:40:262
1759799211,12cyclictest0-21swapper/211:11:072
1759799211,11cyclictest22563-21/usr/sbin/munin11:00:142
17597992110,3cyclictest20697-21df10:55:142
1759799210,11cyclictest0-21swapper/209:08:232
1758799219,2cyclictest29607-21kworker/0:112:32:530
1758799215,2cyclictest18776-21kworker/0:008:29:230
1758799212,9cyclictest9800-21kworker/0:108:23:050
1758799212,9cyclictest9800-21kworker/0:108:13:050
1758799212,9cyclictest9164-21kworker/0:010:37:050
1758799212,9cyclictest19834-21kworker/0:010:52:150
1758799212,8cyclictest9800-21kworker/0:108:04:130
1758799212,8cyclictest12028-21kworker/0:208:09:010
1758799211,9cyclictest14276-21kworker/0:008:17:210
1758799211,8cyclictest746-21kworker/0:111:36:170
1758799211,8cyclictest746-21kworker/0:111:36:170
1758799211,8cyclictest24559-21kworker/0:207:36:250
1758799211,8cyclictest13975-21kworker/0:112:02:190
1758799211,7cyclictest24559-21kworker/0:207:22:030
1758799210,7cyclictest26695-21kworker/0:211:15:010
1759799205,3cyclictest87650irq/129-enp1s0-07:25:152
1759799204,6cyclictest0-21swapper/209:51:472
1759799201,9cyclictest0-21swapper/210:52:052
1759799201,7cyclictest0-21swapper/208:35:252
1759799201,10cyclictest808-21snmpd09:22:562
1758799208,3cyclictest8194-21kworker/0:209:18:030
1758799205,2cyclictest1299-21kworker/0:012:38:210
1758799202,8cyclictest11694-21kworker/0:011:51:170
1758799202,10cyclictest21945-21/usr/sbin/munin09:45:150
1758799201,9cyclictest27409-21kworker/0:109:06:250
1758799201,9cyclictest22055-21kworker/0:111:03:010
1758799201,9cyclictest16126-21kworker/0:210:58:370
1758799201,9cyclictest13303-21kworker/0:110:41:450
1758799201,8cyclictest26695-21kworker/0:211:22:050
1758799201,10cyclictest29609-21sshd10:00:240
193802190,2sleep20-21swapper/207:10:272
1759799199,2cyclictest16449-21awk12:00:012
1759799195,3cyclictest87350irq/126-enp1s0-09:25:222
1759799194,7cyclictest0-21swapper/211:45:132
1759799194,4cyclictest32708-21sh10:10:012
1759799194,3cyclictest23268-21/usr/sbin/munin12:15:212
1759799192,5cyclictest808-21snmpd11:38:222
1759799192,5cyclictest808-21snmpd11:38:222
1759799191,6cyclictest808-21snmpd07:17:322
1759799191,5cyclictest17927-21sshd12:00:242
1759799190,8cyclictest0-21swapper/207:38:112
1758799197,2cyclictest27409-21kworker/0:108:58:510
1758799196,3cyclictest17269-21kworker/0:309:36:570
1758799196,2cyclictest8194-21kworker/0:209:29:170
1758799196,2cyclictest17269-21kworker/0:309:40:390
1758799195,2cyclictest13303-21kworker/0:110:48:290
1758799191,4cyclictest23268-21/usr/sbin/munin12:15:200
1758799191,10cyclictest7176-21/usr/sbin/munin10:25:190
1758799190,9cyclictest746-21kworker/0:111:28:330
1759799186,3cyclictest153832sed08:15:172
1759799184,7cyclictest0-21swapper/212:38:212
1759799184,7cyclictest0-21swapper/207:51:572
1759799184,6cyclictest0-21swapper/208:50:312
1759799181,10cyclictest0-21swapper/209:42:262
1759799181,10cyclictest0-21swapper/208:42:352
1758799186,3cyclictest30746-21/usr/sbin/munin10:05:250
17587991813,3cyclictest25109-21df_inode11:05:130
1759799173,7cyclictest0-21swapper/208:01:512
1759799171,9cyclictest87650irq/129-enp1s0-11:05:152
1759799171,9cyclictest0-21swapper/212:25:232
1759799171,6cyclictest0-21swapper/209:11:532
1759799171,6cyclictest0-21swapper/209:11:522
1759799171,4cyclictest18245-1kworker/u9:209:19:522
17597991712,3cyclictest17315-21ls09:35:012
1759799171,12cyclictest0-21swapper/208:10:312
1759799170,11cyclictest0-21swapper/212:20:512
1758799175,2cyclictest27409-21kworker/0:109:00:110
1759799162,6cyclictest7524-21diskstats10:25:152
1759799161,5cyclictest18104-21perl09:35:162
1759799161,4cyclictest32388-21tr08:55:122
17597991612,3cyclictest0-21swapper/211:15:022
1759799160,9cyclictest0-21swapper/207:22:512
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional