You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 01:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Apr 06, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2107422660,3sleep3401ktimersoftd/307:05:193
220902222155,55sleep20-21swapper/207:08:252
218982202167,23sleep10-21swapper/107:05:541
220152199163,24sleep00-21swapper/007:07:260
228532630,4sleep121-21rcuc/109:35:151
57582500,2sleep121-21rcuc/110:10:161
2230099378,13cyclictest1229-21munin-run11:15:000
2231799318,4cyclictest0-21swapper/312:05:153
2230599302,18cyclictest22503-21/usr/sbin/munin07:10:241
22310992824,3cyclictest0-21swapper/210:11:252
2231099261,5cyclictest0-21swapper/209:24:162
2231099261,5cyclictest0-21swapper/209:24:162
2230599267,16cyclictest30965-21latency_hist09:55:021
2231799251,6cyclictest0-21swapper/312:13:503
2231099256,6cyclictest23718-21sort10:50:182
2231099251,6cyclictest0-21swapper/211:43:422
2231099251,6cyclictest0-21swapper/211:43:422
2231099251,5cyclictest16574-21kworker/u8:111:30:282
2231099251,2cyclictest0-21swapper/210:01:012
79562240,4sleep1221ktimersoftd/111:30:011
48262241,3sleep14928-21memory07:40:181
2231799244,16cyclictest18745-40dnf11:54:113
2231099247,4cyclictest27307-21switchtime08:30:242
2231099241,3cyclictest0-21swapper/210:19:502
22305992418,3cyclictest3969-21memory10:05:201
2230599241,4cyclictest0-21swapper/110:46:491
22305992413,2cyclictest13693-21kworker/u8:210:30:401
22305992413,2cyclictest13693-21kworker/u8:210:30:401
2230599241,10cyclictest0-21swapper/109:31:371
2230599240,17cyclictest0-21swapper/111:10:501
2230099249,3cyclictest29786-21/usr/sbin/munin07:25:220
2230099244,9cyclictest24245-21if_err_enp2s012:05:170
2230099241,7cyclictest0-21swapper/010:07:540
2230099241,11cyclictest6215-21/usr/sbin/munin07:45:170
2230099241,11cyclictest4015-21/usr/sbin/munin07:40:250
2230099241,11cyclictest28945-21kworker/0:011:19:370
2231799236,6cyclictest20533-21tune2fs09:30:143
2231799231,6cyclictest0-21swapper/310:23:333
2231799231,6cyclictest0-21swapper/309:41:423
2231799231,16cyclictest0-21swapper/310:00:223
2231799231,12cyclictest0-21swapper/311:38:323
2231099231,7cyclictest0-21swapper/212:15:332
2231099231,5cyclictest0-21swapper/212:27:262
2231099231,5cyclictest0-21swapper/209:55:332
22305992311,3cyclictest2506-21logger10:01:011
2230599231,12cyclictest3625-21ls11:20:011
2230099234,10cyclictest15502-21fschecks_time08:05:150
2230099234,10cyclictest15502-21fschecks_time08:05:150
2230099232,11cyclictest8638-21kworker/0:209:52:540
2230099231,11cyclictest27678-21ntp_states09:45:220
2230099231,10cyclictest0-21swapper/010:31:380
2230099231,10cyclictest0-21swapper/010:31:380
2231799227,6cyclictest0-21swapper/309:35:123
2231799226,3cyclictest87350irq/126-enp1s0-09:51:103
2231799223,11cyclictest23807-21/usr/sbin/munin08:25:153
2231799221,7cyclictest0-21swapper/312:20:133
2231799221,5cyclictest0-21swapper/310:46:013
2231799221,10cyclictest0-21swapper/311:30:063
2231799220,9cyclictest0-21swapper/309:27:543
2231099224,5cyclictest25561-21expr07:15:122
2231099224,4cyclictest4224-40dnf08:52:582
2231099223,5cyclictest0-21swapper/210:25:492
2231099223,3cyclictest12582-21munin-run08:00:012
2231099222,5cyclictest0-21swapper/211:54:112
2231099222,5cyclictest0-21swapper/209:30:022
2231099221,5cyclictest0-21swapper/210:38:392
2230599224,3cyclictest0-21swapper/109:10:451
2230599223,11cyclictest1676-1kworker/u9:210:58:131
2230599221,5cyclictest18745-40dnf11:54:101
2230599221,18cyclictest27542-21cat09:45:191
2230599221,18cyclictest2634-21irqstats07:35:171
2230599221,15cyclictest0-21swapper/109:40:381
2230099228,11cyclictest14440-21perf11:45:010
2230099228,11cyclictest14440-21perf11:45:010
2230099226,4cyclictest29634-21cat11:05:120
2230099225,4cyclictest11203-21sed10:20:250
2230099221,6cyclictest0-21swapper/010:27:060
2230099221,5cyclictest0-21swapper/010:38:430
2230099221,13cyclictest0-21swapper/011:47:460
2230099221,11cyclictest32250-21/usr/sbin/munin12:25:120
2230099221,11cyclictest20362-21diskstats09:30:130
2230099221,10cyclictest5252-21/usr/sbin/munin10:10:240
2230099221,10cyclictest3344-21kworker/0:111:25:220
2230099221,10cyclictest25661-21/usr/sbin/munin12:10:160
2230099221,10cyclictest22836-21/usr/sbin/munin10:50:140
198812220,2sleep339-21rcuc/311:55:193
223252211,3sleep322385-21pmu-power12:00:223
2231799217,6cyclictest39-21rcuc/311:05:253
2231799216,9cyclictest0-21swapper/310:44:193
2231799215,7cyclictest0-21swapper/311:11:403
2231799211,9cyclictest579-21systemd-journal07:30:013
2231799211,6cyclictest0-21swapper/310:50:243
2231799211,11cyclictest0-21swapper/309:58:333
2231799210,8cyclictest0-21swapper/309:13:083
2231099218,3cyclictest87550irq/128-enp1s0-11:25:302
2231099216,4cyclictest87350irq/126-enp1s0-08:20:012
2231099215,3cyclictest28218-21needreboot11:00:192
2231099213,5cyclictest0-21swapper/211:57:212
2231099213,3cyclictest0-21swapper/209:37:352
2231099212,2cyclictest0-21swapper/212:00:532
2231099211,8cyclictest0-21swapper/212:31:102
2231099211,7cyclictest0-21swapper/210:33:312
2231099211,7cyclictest0-21swapper/210:33:302
2231099211,3cyclictest0-21swapper/210:57:152
2231099211,3cyclictest0-21swapper/210:09:172
2230599218,7cyclictest0-21swapper/109:24:161
2230599218,7cyclictest0-21swapper/109:24:161
2230599217,9cyclictest0-21swapper/112:13:021
2230599217,3cyclictest27567-21df11:00:131
2230599211,7cyclictest5325-21taskset11:21:071
2230599211,7cyclictest0-21swapper/109:16:061
22305992116,3cyclictest32266-21sensors09:55:241
22305992113,7cyclictest0-21swapper/108:38:291
2230599211,17cyclictest2435-21acpi08:50:111
22305992111,3cyclictest0-21swapper/110:19:511
2230599211,12cyclictest20044-21strings11:55:221
2230099218,3cyclictest0-21swapper/008:20:210
2230099211,8cyclictest0-21swapper/008:40:160
2230099211,7cyclictest11001-21diskmemload12:35:170
2230099211,6cyclictest0-21swapper/012:22:430
2230099211,6cyclictest0-21swapper/012:03:040
2230099211,6cyclictest0-21swapper/009:40:350
2230099211,13cyclictest0-21swapper/012:16:080
2230099211,12cyclictest0-21swapper/010:58:090
2230099211,11cyclictest6330-21kworker/0:109:00:570
2230099211,11cyclictest27859-21sshd08:33:430
2230099211,11cyclictest25739-21sshd08:29:430
2230099211,10cyclictest19468-21sshd09:26:040
2230099210,6cyclictest0-21swapper/011:50:040
165232211,3sleep10-21swapper/111:48:521
2231799206,6cyclictest0-21swapper/311:22:413
2231799206,4cyclictest0-21swapper/310:35:323
2231799205,6cyclictest0-21swapper/309:47:253
2231799205,4cyclictest0-21swapper/312:28:323
2231799204,6cyclictest0-21swapper/312:17:083
2231799201,5cyclictest0-21swapper/312:35:473
22317992015,3cyclictest12315-21grep10:25:153
2231799201,3cyclictest0-21swapper/310:15:383
2231799201,16cyclictest25503-21grep07:15:123
2231799201,10cyclictest15623-21/usr/sbin/munin09:20:173
2231799201,10cyclictest15623-21/usr/sbin/munin09:20:163
2231799201,10cyclictest0-21swapper/311:42:503
2231799201,10cyclictest0-21swapper/311:42:503
2231799200,11cyclictest0-21swapper/311:15:133
2231099203,2cyclictest87350irq/126-enp1s0-12:13:072
2231099201,9cyclictest0-21swapper/208:57:092
2231099201,7cyclictest0-21swapper/211:10:162
2231099201,6cyclictest0-21swapper/211:37:112
2231099201,3cyclictest0-21swapper/211:16:482
22310992013,4cyclictest29532-21grep11:05:122
2230599209,7cyclictest0-21swapper/111:43:421
2230599209,7cyclictest0-21swapper/111:43:421
2230599208,3cyclictest443-21kworker/u8:011:08:201
2230599205,8cyclictest16236-21/usr/sbin/munin10:35:171
2230599203,2cyclictest87550irq/128-enp1s0-12:06:161
2230599201,3cyclictest0-21swapper/111:30:171
2230599201,3cyclictest0-21swapper/110:25:151
2230599201,17cyclictest0-21swapper/110:22:221
2230599200,15cyclictest0-21swapper/112:30:481
2230599200,13cyclictest0-21swapper/108:26:431
2230099204,3cyclictest87550irq/128-enp1s0-07:30:220
2230099201,9cyclictest8298-21expr11:30:140
22300992017,2cyclictest0-21swapper/009:05:260
2230099201,6cyclictest0-21swapper/010:40:440
2230099201,6cyclictest0-21swapper/009:17:490
2230099201,5cyclictest0-21swapper/010:46:240
2230099201,5cyclictest0-21swapper/009:38:380
2230099201,4cyclictest32061-21ntpq09:55:220
2230099201,12cyclictest0-21swapper/009:12:520
2230099201,12cyclictest0-21swapper/007:35:120
2230099200,6cyclictest0-21swapper/010:00:150
87612191,4sleep28845-21fschecks_count07:50:152
42352193,2sleep081ktimersoftd/008:53:010
2231799198,3cyclictest87350irq/126-enp1s0-07:40:023
2231799197,3cyclictest0-21swapper/308:24:503
2231799196,3cyclictest0-21swapper/311:47:113
2231799193,6cyclictest17208-21/usr/sbin/munin08:10:223
2231799192,14cyclictest25144-21tr10:55:143
2231799191,4cyclictest0-21swapper/310:07:593
2231799191,4cyclictest0-21swapper/309:15:383
22317991912,3cyclictest0-21swapper/311:01:493
2231799191,11cyclictest0-21swapper/310:33:173
2231799191,11cyclictest0-21swapper/310:33:173
2231799191,10cyclictest10612-21/usr/sbin/munin07:55:143
2231799191,10cyclictest0-21swapper/312:32:253
2231799191,10cyclictest0-21swapper/311:27:123
2231799191,10cyclictest0-21swapper/310:12:343
2231099198,2cyclictest87550irq/128-enp1s0-10:42:322
2231099196,3cyclictest87650irq/129-enp1s0-08:10:252
2231099195,3cyclictest87550irq/128-enp1s0-09:10:212
2231099195,3cyclictest87550irq/128-enp1s0-08:25:172
2231099195,3cyclictest5889-21taskset07:43:562
2231099193,4cyclictest15406-21awk09:20:012
2231099192,4cyclictest2207-21munin-run08:50:022
2231099191,9cyclictest0-21swapper/209:47:172
2231099191,6cyclictest20632-21/usr/sbin/munin10:45:152
2231099191,5cyclictest24522-21munin-node07:15:012
2231099191,5cyclictest10224-21perl10:20:152
2231099191,5cyclictest0-21swapper/211:21:452
2231099191,11cyclictest0-21swapper/209:51:212
2230599198,3cyclictest45672cat12:35:131
2230599196,4cyclictest21250-21/usr/sbin/munin12:00:261
2230599191,9cyclictest0-21swapper/110:51:481
2230599191,6cyclictest0-21swapper/112:25:211
22305991916,2cyclictest0-21swapper/110:44:571
22305991916,2cyclictest0-21swapper/108:40:141
22305991916,2cyclictest0-21swapper/107:27:481
2230599191,2cyclictest0-21swapper/107:15:201
2230599191,17cyclictest0-21swapper/107:32:231
2230599191,16cyclictest0-21swapper/108:45:541
2230599191,16cyclictest0-21swapper/108:06:381
2230599191,16cyclictest0-21swapper/108:06:381
2230599191,11cyclictest0-21swapper/112:21:101
2230599190,17cyclictest0-21swapper/109:00:141
2230599190,14cyclictest0-21swapper/111:36:181
2230099196,2cyclictest31884-21kworker/0:312:33:450
2230099194,3cyclictest0-21swapper/011:00:180
2230099191,6cyclictest0-21swapper/011:58:230
2230099191,6cyclictest0-21swapper/007:20:180
2230099191,3cyclictest11840-21/usr/sbin/munin11:35:250
22300991912,4cyclictest0-21swapper/010:15:140
2230099191,11cyclictest0-21swapper/009:20:480
2230099191,11cyclictest0-21swapper/009:20:480
2230099191,10cyclictest0-21swapper/007:13:420
2231799183,5cyclictest0-21swapper/308:52:563
2231799180,2cyclictest87550irq/128-enp1s0-09:01:193
2231099184,5cyclictest11006-21munin-run09:10:012
2231099184,3cyclictest87350irq/126-enp1s0-09:02:372
2231099184,3cyclictest87350irq/126-enp1s0-08:20:212
2231099181,6cyclictest0-21swapper/211:46:282
2231099181,5cyclictest4959-21hddtemp_smartct12:35:182
2231099181,5cyclictest30394-21/usr/sbin/munin08:40:182
2231099181,5cyclictest30052-21/usr/sbin/munin12:20:272
2231099181,5cyclictest29786-21/usr/sbin/munin07:25:212
2231099181,5cyclictest27928-21df_inode07:20:132
2231099181,5cyclictest24412-21/usr/sbin/munin09:40:142
2230599182,8cyclictest0-21swapper/108:20:121
2230599182,13cyclictest19068-21sendmail_mailtr09:25:241
2230599181,9cyclictest12807-21/usr/sbin/munin08:00:151
2230599181,8cyclictest29178-21users07:20:251
2230599181,7cyclictest27150-21pmu-power08:30:221
22305991815,2cyclictest0-21swapper/108:12:361
22305991813,3cyclictest9975-21grep07:50:251
2230599181,16cyclictest0-21swapper/109:06:311
2230599181,16cyclictest0-21swapper/108:57:521
2230599181,16cyclictest0-21swapper/107:59:421
2230599181,16cyclictest0-21swapper/107:46:501
2230599180,17cyclictest0-21swapper/112:15:591
2230099181,6cyclictest4629-21/usr/sbin/munin08:55:120
2230099181,6cyclictest0-21swapper/007:55:230
2230099181,4cyclictest87550irq/128-enp1s0-07:15:030
22300991814,3cyclictest0-21swapper/008:01:200
22300991813,3cyclictest3672-21/usr/sbin/munin11:20:210
2231799178,2cyclictest0-21swapper/308:45:163
2231799177,3cyclictest0-21swapper/308:40:203
2231799171,7cyclictest9727-21switchtime07:50:243
2231799171,6cyclictest29428-21grep08:35:223
2231799171,6cyclictest0-21swapper/308:55:203
2231799171,6cyclictest0-21swapper/308:00:263
2231799171,6cyclictest0-21swapper/307:45:233
2231799171,3cyclictest0-21swapper/307:11:533
2231799171,14cyclictest0-21swapper/307:31:513
22317991711,3cyclictest20550-21pmu-power08:15:223
2231099172,4cyclictest17818-21/usr/sbin/munin09:25:252
2231099171,7cyclictest1596-21perf07:39:562
2231099171,4cyclictest13163-21entropy08:00:142
22310991711,3cyclictest6634-21grep07:45:152
22310991711,3cyclictest28775-21perl08:35:142
22305991714,2cyclictest0-21swapper/108:16:521
22300991712,3cyclictest9085-21seq07:50:170
22300991712,3cyclictest28564-21grep08:35:130
22300991711,3cyclictest1278-21ntpq08:45:210
2230099170,6cyclictest0-21swapper/008:11:500
2231799166,3cyclictest0-21swapper/307:21:253
2231799161,7cyclictest0-21swapper/308:05:283
2231799161,7cyclictest0-21swapper/308:05:283
2231799161,5cyclictest0-21swapper/309:05:203
2231799161,3cyclictest0-21swapper/308:30:073
22317991610,3cyclictest4153-21grep07:40:123
2231099161,6cyclictest31982-21/usr/sbin/munin07:30:112
22310991611,3cyclictest15691-21hddtemp_smartct08:05:172
22310991611,3cyclictest15691-21hddtemp_smartct08:05:172
2231099160,10cyclictest0-21swapper/212:05:162
2230099152,3cyclictest0-21swapper/008:15:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional