You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 20:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Mar 13, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
253352182101,24sleep30-21swapper/307:05:463
255092181107,25sleep00-21swapper/007:08:000
256522178106,58sleep10-21swapper/107:09:511
256222177101,63sleep20-21swapper/207:09:272
2272321700,4sleep22576099cyclictest11:55:122
2272321700,4sleep22576099cyclictest11:55:122
25767991080,1cyclictest0-21swapper/311:43:173
25767999991,3cyclictest0-21swapper/307:55:063
25767997067,2cyclictest0-21swapper/311:07:283
83352680,4sleep00-21swapper/012:35:160
221302590,2sleep20-21swapper/210:40:122
196772560,4sleep014441-21diskmemload09:20:140
2575599341,17cyclictest21-21rcuc/109:24:571
2574899337,12cyclictest11-21rcuc/012:15:000
2574899292,12cyclictest29847-21perf09:45:010
2574899281,12cyclictest27651-21basename09:40:010
2574899270,13cyclictest14457-21awk09:10:010
2574899261,13cyclictest7393-21kworker/0:107:45:010
2574899261,12cyclictest20432-21munin-run08:10:000
2576799256,3cyclictest12050irq/123-ahci[0009:50:013
2576099259,5cyclictest0-21swapper/211:34:272
25760992518,4cyclictest8167-21ntp_states11:20:202
89862241,19sleep09066-21cut08:55:200
2575599248,2cyclictest15780-21kworker/1:212:25:011
2575599241,15cyclictest0-21swapper/111:40:091
2574899246,15cyclictest12264-21perf09:05:020
2576799239,1cyclictest0-21swapper/311:55:183
2576799239,1cyclictest0-21swapper/311:55:183
2575599231,12cyclictest0-21swapper/109:37:481
2575599230,17cyclictest0-21swapper/109:53:071
2575599230,16cyclictest0-21swapper/110:54:211
2574899230,11cyclictest21665-21basename10:40:000
2576799221,5cyclictest0-21swapper/311:38:343
2576799221,16cyclictest0-21swapper/310:14:113
2576099221,6cyclictest0-21swapper/211:05:362
2575599229,7cyclictest0-21swapper/110:56:451
2575599222,11cyclictest0-21swapper/107:55:151
2575599221,13cyclictest15780-21kworker/1:210:30:221
2575599221,11cyclictest32653-21munin-run11:05:001
2575599221,11cyclictest11974-21grep07:50:111
25755992210,3cyclictest22122-21unixbench-2d08:10:241
2574899221,11cyclictest4700-21perf11:15:020
2576799213,10cyclictest19072-21/usr/sbin/munin09:20:203
2576799211,11cyclictest14441-21diskmemload10:05:243
2576799211,11cyclictest0-21swapper/310:25:013
25767992110,6cyclictest41-21ksoftirqd/309:11:073
2576799210,9cyclictest0-21swapper/312:15:233
2575599219,3cyclictest18845-21munin-run09:20:011
2575599219,3cyclictest1216-21cat08:39:561
2575599216,3cyclictest15780-21kworker/1:210:11:221
25755992115,4cyclictest10900-21find07:45:231
2575599211,18cyclictest0-21swapper/111:56:161
2575599211,18cyclictest0-21swapper/111:56:161
2575599211,18cyclictest0-21swapper/110:01:451
2575599211,11cyclictest27145-21cat12:05:111
25755992110,3cyclictest30003-21memory12:10:181
2574899215,2cyclictest88150irq/126-enp1s0-10:15:420
2574899211,14cyclictest0-21swapper/011:55:120
2574899211,14cyclictest0-21swapper/011:55:120
2576799209,2cyclictest30846-40dnf12:12:313
2576799204,13cyclictest3987-21cut12:25:173
2576799201,9cyclictest0-21swapper/311:20:113
2576799201,8cyclictest0-21swapper/310:45:293
2576799201,5cyclictest5256-21diskstats11:15:133
2576799201,5cyclictest0-21swapper/310:03:483
2576799201,11cyclictest6947-21ntp_states08:50:213
2576799201,11cyclictest6874-21sort12:30:243
2576799201,11cyclictest19683-21/usr/sbin/munin10:35:173
2576799201,10cyclictest11697-21irqstats10:15:153
2576099201,5cyclictest0-21swapper/209:45:512
25760992014,3cyclictest30105-21gltestperf08:30:132
2576099201,3cyclictest22069-21interrupts09:25:172
2575599207,4cyclictest27198-21cat08:25:011
2575599207,12cyclictest30944-21kworker/u8:012:19:201
2575599205,3cyclictest15780-21kworker/1:210:20:041
2575599201,5cyclictest6951-21cut10:05:141
2575599201,3cyclictest0-21swapper/108:48:451
2575599201,17cyclictest0-21swapper/112:27:151
2575599201,17cyclictest0-21swapper/111:25:461
2575599201,17cyclictest0-21swapper/111:15:151
2575599201,17cyclictest0-21swapper/111:05:151
2575599201,17cyclictest0-21swapper/110:45:461
2575599201,17cyclictest0-21swapper/110:40:151
2575599201,17cyclictest0-21swapper/108:51:151
2575599201,17cyclictest0-21swapper/107:30:151
2575599201,17cyclictest0-21swapper/107:25:151
2575599201,11cyclictest29186-21df_inode07:15:131
2575599201,11cyclictest15732-21ntp_states09:10:201
2575599201,10cyclictest27865-21gltestperf08:25:151
2575599201,10cyclictest22414-21cat11:55:021
2574899208,3cyclictest1434-21unixbench_singl09:50:250
2574899206,3cyclictest0-21swapper/010:33:210
2574899205,4cyclictest0-21swapper/012:21:040
2574899204,2cyclictest13793-21basename10:20:150
2574899201,6cyclictest0-21swapper/010:47:420
25748992013,4cyclictest7493-21ls12:35:020
2574899201,11cyclictest2784-21sshd07:29:210
2574899201,11cyclictest0-21swapper/012:25:330
2574899201,10cyclictest20284-21/usr/sbin/munin11:50:130
2574899200,10cyclictest8160-21kworker/u8:110:11:100
2576799192,15cyclictest0-21swapper/309:06:263
2576799191,7cyclictest3710-21needreboot11:10:203
2576799191,4cyclictest0-21swapper/309:58:243
2576799191,12cyclictest0-21swapper/309:00:263
2576799191,11cyclictest0-21swapper/308:45:253
2576799191,10cyclictest32301-21smart_sda07:20:203
2576799190,11cyclictest25126-21cpuspeed_turbos08:20:113
2576099198,3cyclictest3670-21df_inode12:25:152
2576099195,11cyclictest1-21systemd12:20:212
2576099193,9cyclictest3870-21runrttasks09:57:402
2576099192,7cyclictest26893-21/usr/sbin/munin12:05:172
2576099192,14cyclictest17391-21grep08:00:212
2576099191,8cyclictest20968-21ls10:35:242
2576099191,8cyclictest14609-21perl07:55:142
2576099191,7cyclictest0-21swapper/211:41:072
2576099191,6cyclictest0-21swapper/211:48:402
2576099191,6cyclictest0-21swapper/210:25:042
2576099191,5cyclictest0-21swapper/212:39:572
2576099191,5cyclictest0-21swapper/210:32:402
2576099191,12cyclictest0-21swapper/210:16:492
2576099191,12cyclictest0-21swapper/210:00:222
25755991915,3cyclictest0-21swapper/112:39:151
2575599191,3cyclictest88250irq/127-enp1s0-08:15:571
25755991913,4cyclictest20278-21grep10:35:151
2575599191,17cyclictest0-21swapper/108:57:451
2575599191,17cyclictest0-21swapper/107:21:451
2575599191,16cyclictest0-21swapper/112:00:161
2575599191,16cyclictest0-21swapper/111:45:161
2575599191,16cyclictest0-21swapper/111:35:161
2575599191,16cyclictest0-21swapper/111:20:151
2575599191,16cyclictest0-21swapper/109:05:151
2575599191,10cyclictest8276-21irqstats07:40:171
2575599191,10cyclictest2550-21ntp_states08:40:211
2575599191,10cyclictest2276-21cstates09:55:131
2575599191,10cyclictest16052-21/usr/sbin/munin10:25:161
2575599191,10cyclictest11610-21if_enp1s010:15:151
2575599191,10cyclictest10450-21cstates09:00:131
2575599190,9cyclictest0-21swapper/112:33:571
2574899198,3cyclictest7461-21cat10:05:180
2574899195,7cyclictest18555-21kworker/u8:109:49:030
2574899192,4cyclictest1237-21munin-run08:40:010
2574899192,11cyclictest0-21swapper/009:28:070
2574899191,6cyclictest0-21swapper/009:59:520
2574899191,6cyclictest0-21swapper/008:52:120
25748991913,3cyclictest4881-21grep10:00:130
2574899191,11cyclictest0-21swapper/012:17:350
2574899191,11cyclictest0-21swapper/010:53:260
2574899191,10cyclictest7393-21kworker/0:109:30:140
2574899191,10cyclictest7393-21kworker/0:109:30:140
2574899191,10cyclictest17068-21ps11:40:220
116132190,4sleep123-21ksoftirqd/111:30:111
31992181,2sleep30-21swapper/308:41:553
2576799187,3cyclictest31645-21/usr/sbin/munin08:35:103
2576799182,5cyclictest17126-1kworker/u9:111:30:363
2576799182,2cyclictest88450irq/129-enp1s0-11:26:153
2576799181,9cyclictest0-21swapper/310:34:143
25767991814,3cyclictest0-21swapper/309:54:093
25767991814,3cyclictest0-21swapper/307:42:553
2576799181,3cyclictest2466-21users07:25:253
25767991812,3cyclictest823-21snmpd10:20:203
25767991812,3cyclictest17164-21memory08:00:183
2576799181,14cyclictest9212-21grep08:55:223
2576799181,13cyclictest29608-21tr08:30:113
2576099185,6cyclictest4295-21uptime11:10:262
2576099183,4cyclictest6825-21kworker/u8:209:34:342
2576099183,4cyclictest6825-21kworker/u8:209:34:342
2576099181,7cyclictest31847-21perl12:15:152
2576099181,7cyclictest0-21swapper/210:49:552
2576099181,7cyclictest0-21swapper/209:37:372
2576099181,6cyclictest0-21swapper/210:55:162
2576099181,5cyclictest0-21swapper/210:24:212
2576099181,4cyclictest0-21swapper/210:07:052
2576099181,3cyclictest7807-21df_inode07:40:132
25760991812,3cyclictest3608-21cat08:45:012
2576099180,4cyclictest0-21swapper/212:30:152
2575599183,9cyclictest0-21swapper/109:31:061
2575599183,9cyclictest0-21swapper/109:31:051
2575599182,13cyclictest31133-21ntp_states09:45:221
2575599181,9cyclictest0-21swapper/111:10:111
2575599181,3cyclictest0-21swapper/109:40:141
2575599181,16cyclictest0-21swapper/108:30:151
2575599181,16cyclictest0-21swapper/108:06:151
2575599181,16cyclictest0-21swapper/107:35:151
25755991810,5cyclictest17513-21smart_sda08:00:231
2574899189,3cyclictest0-21swapper/007:35:240
2574899185,8cyclictest28063-21munin-node09:11:090
2574899185,3cyclictest16548-21sh10:25:230
2574899184,5cyclictest0-21swapper/011:28:530
2574899182,13cyclictest25823-21tr12:00:200
2574899181,4cyclictest27253-21/usr/sbin/munin08:25:270
25748991813,3cyclictest29839-21timerandwakeup10:55:250
2574899181,15cyclictest0-21swapper/011:00:240
2574899181,15cyclictest0-21swapper/010:41:320
2574899181,15cyclictest0-21swapper/007:11:060
2574899181,15cyclictest0-21swapper/007:11:050
2574899181,11cyclictest0-21swapper/011:47:510
2574899181,10cyclictest0-21swapper/011:18:330
2576799179,2cyclictest39-21rcuc/309:41:243
2576799178,5cyclictest21725-40dnf08:10:213
2576799176,3cyclictest0-21swapper/308:25:223
2576799172,5cyclictest28251-21munin-run10:55:013
2576799171,9cyclictest0-21swapper/311:50:243
2576799171,8cyclictest0-21swapper/307:10:123
2576799171,8cyclictest0-21swapper/307:10:123
2576799171,6cyclictest8066-21diskstats12:35:143
2576799171,6cyclictest0-21swapper/310:59:273
2576799171,6cyclictest0-21swapper/310:44:153
2576799171,3cyclictest18443-21diskstats11:45:163
25767991713,3cyclictest0-21swapper/312:05:173
25767991713,3cyclictest0-21swapper/307:36:043
25767991712,3cyclictest843-21runrttasks09:39:413
2576799171,13cyclictest23462-21/usr/sbin/munin09:30:153
2576799171,13cyclictest23462-21/usr/sbin/munin09:30:153
25767991711,3cyclictest21266-21/usr/sbin/munin09:25:253
2576799171,10cyclictest0-21swapper/312:00:083
2576799171,10cyclictest0-21swapper/307:50:153
2576099179,2cyclictest88450irq/129-enp1s0-09:10:092
2576099176,3cyclictest88450irq/129-enp1s0-12:00:252
2576099176,3cyclictest88150irq/126-enp1s0-11:15:242
2576099171,7cyclictest0-21swapper/210:50:212
2576099171,6cyclictest823-21snmpd09:55:012
2576099171,6cyclictest0-21swapper/211:00:402
2576099171,6cyclictest0-21swapper/209:17:472
2576099171,6cyclictest0-21swapper/207:50:242
2576099171,5cyclictest0-21swapper/211:39:102
25760991713,3cyclictest0-21swapper/209:41:212
25760991712,3cyclictest32712-21ntp_states08:35:192
25760991712,3cyclictest28319-21ntp_states08:25:222
25760991711,3cyclictest8424-21entropy08:55:142
25760991710,4cyclictest2107-21/usr/sbin/munin07:25:232
2576099170,6cyclictest0-21swapper/212:14:212
2575599171,15cyclictest0-21swapper/109:25:451
2575599171,15cyclictest0-21swapper/107:10:451
2575599171,15cyclictest0-21swapper/107:10:441
2574899175,4cyclictest0-21swapper/008:00:220
2574899175,3cyclictest0-21swapper/011:35:060
2574899171,9cyclictest0-21swapper/012:06:120
2574899171,3cyclictest0-21swapper/007:55:540
25748991712,3cyclictest11662-21sort07:50:010
2574899171,15cyclictest0-21swapper/011:06:590
2574899171,15cyclictest0-21swapper/007:32:060
2574899171,10cyclictest0-21swapper/011:30:150
236352171,3sleep30-21swapper/308:15:183
47302160,2sleep30-21swapper/307:31:073
2576799166,3cyclictest88450irq/129-enp1s0-07:15:143
2576799165,3cyclictest88450irq/129-enp1s0-07:45:153
2576799161,4cyclictest0-21swapper/311:00:143
2576799161,2cyclictest17860-21cut09:15:203
25767991611,3cyclictest19986-21awk08:05:243
2576799160,15cyclictest0-21swapper/312:20:473
2576099166,3cyclictest88450irq/129-enp1s0-08:10:182
2576099165,3cyclictest88150irq/126-enp1s0-09:05:152
2576099164,4cyclictest27207-21sendmail_mailtr07:10:222
2576099164,4cyclictest27207-21sendmail_mailtr07:10:222
2576099162,3cyclictest0-21swapper/211:50:342
2576099161,6cyclictest0-21swapper/207:30:232
2576099161,5cyclictest0-21swapper/210:12:332
25760991612,3cyclictest0-21swapper/209:00:052
25760991612,3cyclictest0-21swapper/208:18:142
2576099161,14cyclictest0-21swapper/208:54:452
2576099161,14cyclictest0-21swapper/208:22:052
25760991611,3cyclictest6417-21tr07:35:222
25760991611,3cyclictest3788-21expr08:45:122
25760991611,3cyclictest32286-21sendmail_mailtr07:20:212
2576099161,13cyclictest25341-21kworker/u8:007:15:192
25760991611,3cyclictest20538-21sed09:20:252
25760991610,4cyclictest0-21swapper/211:25:162
2574899162,11cyclictest4150-21fschecks_time08:45:160
2574899161,8cyclictest0-21swapper/009:16:510
2574899161,4cyclictest7103-21/usr/sbin/munin11:20:220
25748991612,3cyclictest0-21swapper/008:22:500
2574899161,14cyclictest0-21swapper/008:15:080
25748991611,3cyclictest29445-21/usr/sbin/munin08:30:100
25748991611,3cyclictest28843-21/usr/sbin/munin07:15:190
25748991611,3cyclictest2873-21sed08:40:240
25748991611,3cyclictest20788-21expr08:10:110
2574899161,13cyclictest0-21swapper/007:50:200
2576099154,3cyclictest19754-21sed08:05:232
25760991510,3cyclictest10020-21entropy07:45:142
25748991510,3cyclictest843-21runrttasks07:20:420
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional