You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 23:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 15, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
227562237161,25sleep00-21swapper/007:06:320
227972203168,23sleep10-21swapper/107:07:061
226762202165,24sleep30-21swapper/307:05:313
230182201164,24sleep20-21swapper/207:09:572
23115999690,3cyclictest32163-21df09:55:132
23115999591,3cyclictest0-21swapper/208:01:482
23115997167,3cyclictest0-21swapper/209:49:182
23115996461,2cyclictest0-21swapper/210:07:252
32002620,1sleep10-21swapper/112:30:101
23115995653,2cyclictest0-21swapper/211:59:392
23115995148,2cyclictest0-21swapper/211:29:582
23115994745,1cyclictest0-21swapper/209:04:592
305832460,4sleep30-21swapper/307:25:013
2310199412,29cyclictest7031-21/usr/sbin/munin07:45:180
2310199400,28cyclictest0-21swapper/007:40:130
2312299391,11cyclictest23035-21/usr/sbin/munin09:35:213
2310199392,32cyclictest30141-40dnf07:22:120
2310199391,30cyclictest17678-21/usr/sbin/munin11:50:230
2310199385,29cyclictest28804-21idleruntime-cro08:35:010
2310199381,29cyclictest20837-21/usr/sbin/munin09:30:170
2310199381,17cyclictest31249-21fschecks_count12:20:150
2312299371,11cyclictest30599-21/usr/sbin/munin07:25:223
2312299371,11cyclictest30599-21/usr/sbin/munin07:25:223
2310199373,26cyclictest87550irq/128-enp1s0-07:15:170
2312299368,3cyclictest23657-21/usr/sbin/munin10:50:183
2312299361,6cyclictest4842-21diskstats11:20:123
2312299361,3cyclictest31940-21ntp_states12:20:213
2312299361,10cyclictest7646-21/usr/sbin/munin09:00:133
2310199361,5cyclictest670-21grep11:10:160
2310199361,26cyclictest6070-21/usr/sbin/munin10:10:260
2312299358,4cyclictest27819-21idleruntime-cro11:00:013
2312299357,7cyclictest0-21swapper/310:35:153
2312299351,5cyclictest0-21swapper/309:26:533
2312299351,10cyclictest0-21swapper/309:22:103
2310199352,21cyclictest87350irq/126-enp1s0-12:25:150
2310199351,25cyclictest12052-21/usr/sbin/munin09:10:260
2312299347,4cyclictest0-21swapper/312:09:503
2312299347,3cyclictest11820-21diskmemload11:13:063
2312299346,3cyclictest87350irq/126-enp1s0-12:20:023
2312299345,3cyclictest12050irq/123-ahci[0012:35:023
23122993414,17cyclictest21615-21tr10:45:143
23122993412,19cyclictest22256-21cat08:20:013
23122993412,19cyclictest12667-21/usr/sbin/munin10:25:203
2312299341,11cyclictest0-21swapper/309:51:283
2312299341,10cyclictest0-21swapper/309:41:403
2311099341,12cyclictest0-21swapper/107:38:551
23101993416,14cyclictest20813-21grep08:15:160
2312299336,3cyclictest0-21swapper/309:58:173
2312299335,3cyclictest87350irq/126-enp1s0-11:25:133
2312299333,3cyclictest7074-21latency_hist12:40:023
2312299331,9cyclictest0-21swapper/310:08:263
23122993317,13cyclictest28447-21sh12:15:013
23122993317,13cyclictest28447-21sh12:15:013
2312299331,6cyclictest28587-21/usr/sbin/munin09:45:223
2312299331,29cyclictest808-21snmpd10:32:013
23122993311,19cyclictest13623-21/usr/sbin/munin08:00:183
2312299331,10cyclictest0-21swapper/311:58:203
2312299331,10cyclictest0-21swapper/311:36:493
2310199333,3cyclictest87350irq/126-enp1s0-09:40:110
2310199330,5cyclictest0-21swapper/009:50:140
2312299323,6cyclictest11820-21diskmemload10:23:263
2312299321,7cyclictest31118-21cat08:40:013
2312299321,6cyclictest0-21swapper/309:32:123
2312299321,12cyclictest0-21swapper/312:02:083
2312299321,10cyclictest0-21swapper/310:17:383
2312299320,9cyclictest0-21swapper/312:26:473
2310199329,20cyclictest31428-21proc_pri11:05:230
2310199323,3cyclictest87550irq/128-enp1s0-10:20:140
2310199321,5cyclictest5715-21runrttasks10:08:290
2310199321,5cyclictest25026-21awk09:40:000
2310199321,5cyclictest0-21swapper/009:45:110
2310199321,3cyclictest28876-21cstates12:15:120
2312299313,3cyclictest8099-21ntpq07:45:223
2312299311,2cyclictest15317-21sh09:15:213
23122993111,3cyclictest5450-21/usr/sbin/munin08:55:233
23122993111,3cyclictest20320-21ntpq10:40:213
23122993111,17cyclictest17269-21cat08:05:263
2312299311,10cyclictest0-21swapper/310:12:433
2310199318,15cyclictest808-21snmpd11:42:140
2310199317,21cyclictest5219-21turbostat08:55:010
2310199317,20cyclictest827-21runrttasks10:40:160
2310199313,3cyclictest88950irq/132-enp2s0-11:39:170
2310199313,25cyclictest19538-21grep08:10:250
2310199312,26cyclictest491-21cpuspeed_turbos07:30:100
2310199311,5cyclictest14363-21/usr/sbin/munin08:00:170
2310199311,3cyclictest2910-21/usr/sbin/munin12:30:180
2310199311,21cyclictest87350irq/126-enp1s0-11:00:190
2310199311,21cyclictest87350irq/126-enp1s0-11:00:190
23101993112,16cyclictest1666-21/usr/sbin/munin10:00:230
23101993111,17cyclictest4489-21/usr/sbin/munin11:20:220
2312299303,5cyclictest1379-21diskstats08:45:133
2312299301,4cyclictest0-21swapper/310:00:243
2312299301,27cyclictest0-21swapper/311:00:213
2312299301,27cyclictest0-21swapper/311:00:213
2312299301,26cyclictest30260-21/usr/sbin/munin11:05:163
23122993010,17cyclictest27927-21/usr/sbin/munin08:30:223
2311099306,3cyclictest0-21swapper/111:10:431
2310199308,19cyclictest827-21runrttasks08:38:570
2310199306,1cyclictest0-21swapper/009:55:150
2310199303,3cyclictest12667-21/usr/sbin/munin10:25:190
2310199303,3cyclictest12050irq/123-ahci[0007:30:020
2310199303,3cyclictest12050irq/123-ahci[0007:30:010
2310199303,24cyclictest24179-40dnf08:22:230
2310199302,5cyclictest19901-21smart_sda09:25:220
2310199301,6cyclictest11820-21diskmemload12:39:400
2310199301,4cyclictest9917-21/usr/sbin/munin11:30:190
2310199301,3cyclictest18064-21needreboot10:35:210
2310199301,3cyclictest14248-21/usr/sbin/munin09:15:130
2310199300,18cyclictest0-21swapper/008:45:170
2312299299,4cyclictest9895-21gltestperf07:50:153
2312299298,4cyclictest279972chrt07:17:553
2312299298,3cyclictest1063-21sed07:30:143
2312299298,3cyclictest106052chrt11:31:533
23122992913,13cyclictest25381-21cut07:15:013
23122992910,3cyclictest14403-21pmu-power11:40:223
2311099292,18cyclictest0-21swapper/107:40:271
2311099291,12cyclictest0-21swapper/109:45:191
2311099291,12cyclictest0-21swapper/108:41:171
2311099291,11cyclictest0-21swapper/111:25:351
2311099290,21cyclictest0-21swapper/110:50:291
2310199299,17cyclictest7646-21/usr/sbin/munin09:00:140
2310199299,17cyclictest24465-21interrupts10:50:170
2310199298,18cyclictest827-21runrttasks10:59:300
2310199298,18cyclictest2292-21/usr/sbin/munin11:15:220
2310199297,19cyclictest909-21sssd_nss07:55:200
2310199297,19cyclictest6817-21grep11:25:120
2310199292,4cyclictest15609-21/usr/sbin/munin10:30:170
2310199291,3cyclictest22898-21/usr/sbin/munin12:00:170
2310199291,3cyclictest15847-21/usr/sbin/munin11:45:130
23101992910,16cyclictest11588-21sshd09:07:110
2312299289,3cyclictest31399-21cstates08:40:133
2312299281,3cyclictest87350irq/126-enp1s0-07:40:013
23122992813,12cyclictest4834-21/usr/sbin/munin07:40:133
2312299281,11cyclictest11820-21diskmemload09:10:133
2312299280,4cyclictest0-21swapper/311:15:113
2311099281,13cyclictest0-21swapper/111:39:111
2311099281,11cyclictest0-21swapper/108:07:441
2311099280,12cyclictest0-21swapper/110:43:561
2310199288,17cyclictest21688-21expr10:45:140
2310199287,18cyclictest25038-21if_enp2s012:05:160
2310199284,21cyclictest9852-21gltestperf07:50:140
2310199282,4cyclictest20550-21hddtemp_smartct11:55:150
2310199281,5cyclictest27277-21/usr/sbin/munin12:10:150
2310199281,5cyclictest27277-21/usr/sbin/munin12:10:150
2310199281,4cyclictest31384-21/usr/sbin/munin08:40:130
2310199281,3cyclictest16606-21/usr/sbin/munin09:20:110
2310199281,19cyclictest0-21swapper/008:25:230
23101992810,15cyclictest16145-21expr08:05:130
2312299278,3cyclictest15635-21cpuspeed_turbos11:45:113
2312299278,2cyclictest827-21runrttasks09:06:293
2312299273,2cyclictest0-21swapper/308:20:113
2312299272,10cyclictest19574-21unixbench_singl08:10:253
2312299271,10cyclictest0-21swapper/308:50:183
2311099272,15cyclictest0-21swapper/109:00:471
2311099271,11cyclictest0-21swapper/111:18:431
2311099271,11cyclictest0-21swapper/108:10:441
2311099270,9cyclictest0-21swapper/109:41:571
2311099270,10cyclictest0-21swapper/110:28:421
2310199272,4cyclictest2635-21/usr/sbin/munin07:35:100
2310199271,4cyclictest24129-21irqstats07:10:180
2310199271,3cyclictest5450-21/usr/sbin/munin08:55:190
23101992710,14cyclictest8790-21cut10:15:140
2312299261,3cyclictest24625-21/usr/sbin/munin08:25:183
23115992623,2cyclictest0-21swapper/211:22:282
2311099268,6cyclictest0-21swapper/108:37:271
2311099268,3cyclictest31460-21fw_packets12:20:141
2311099267,9cyclictest0-21swapper/108:28:151
2311099262,17cyclictest0-21swapper/112:06:101
23110992611,3cyclictest27850-21timerandwakeup12:10:231
23110992611,3cyclictest27850-21timerandwakeup12:10:231
2311099261,11cyclictest0-21swapper/110:09:071
2311099261,11cyclictest0-21swapper/109:06:191
2311099261,10cyclictest0-21swapper/108:47:431
2311099261,10cyclictest0-21swapper/107:34:391
2311099260,11cyclictest0-21swapper/112:39:321
2311099260,11cyclictest0-21swapper/109:18:191
2312299256,4cyclictest12722-21switchtime07:55:263
2312299251,3cyclictest18945-21sh11:50:243
2311599259,6cyclictest14931-21ls10:30:122
2311599252,11cyclictest30141-40dnf07:22:142
2311099253,15cyclictest0-21swapper/107:57:141
2311099251,9cyclictest0-21swapper/112:15:091
2311099251,9cyclictest0-21swapper/108:02:301
2311099251,11cyclictest0-21swapper/108:58:191
2311099251,10cyclictest0-21swapper/109:14:051
2311099251,10cyclictest0-21swapper/108:16:101
2311099251,10cyclictest0-21swapper/107:13:341
23110992510,3cyclictest0-21swapper/110:37:421
2311099250,9cyclictest0-21swapper/109:23:291
2311099250,11cyclictest0-21swapper/110:19:541
2311099250,10cyclictest0-21swapper/111:01:511
2311099250,10cyclictest0-21swapper/111:01:511
2311099247,3cyclictest0-21swapper/107:27:071
2311099247,3cyclictest0-21swapper/107:27:071
2311099241,9cyclictest0-21swapper/109:27:351
2311099241,8cyclictest0-21swapper/109:33:351
2311099241,10cyclictest0-21swapper/112:03:111
2311099241,10cyclictest0-21swapper/108:50:031
2311099241,10cyclictest0-21swapper/107:22:311
2311099240,11cyclictest0-21swapper/111:33:291
2311099240,11cyclictest0-21swapper/110:14:491
2311099240,11cyclictest0-21swapper/109:58:171
2311599231,5cyclictest487-21fschecks_time11:10:162
23115992311,3cyclictest26206-21/usr/sbin/munin07:15:222
2311099238,3cyclictest0-21swapper/112:26:341
2311099238,3cyclictest0-21swapper/110:56:071
2311099231,9cyclictest0-21swapper/110:34:381
2311099231,4cyclictest30610-21diskstats11:05:141
2311099231,3cyclictest14341-21/usr/sbin/munin11:40:221
2311099231,11cyclictest0-21swapper/111:56:311
2311099230,16cyclictest0-21swapper/108:32:151
2311099230,11cyclictest0-21swapper/109:50:051
2311099228,3cyclictest0-21swapper/110:49:351
2311099228,3cyclictest0-21swapper/107:47:271
2311099227,3cyclictest0-21swapper/109:35:591
2311099225,3cyclictest0-21swapper/107:16:471
2311099220,9cyclictest0-21swapper/110:02:051
2311099220,12cyclictest0-21swapper/111:23:111
2311099211,9cyclictest0-21swapper/111:45:071
2311099211,14cyclictest0-21swapper/110:22:011
2311099211,10cyclictest0-21swapper/108:20:431
2311599209,3cyclictest9230-21sh11:30:132
2311099209,3cyclictest0-21swapper/111:50:071
2311099206,3cyclictest0-21swapper/107:54:431
2311599191,9cyclictest9257-21strings10:15:182
2311599191,6cyclictest0-21swapper/210:38:102
2311599191,5cyclictest11820-21diskmemload10:23:232
2311599191,4cyclictest0-21swapper/212:07:472
23115991914,3cyclictest2910-21/usr/sbin/munin12:30:102
23115991914,3cyclictest21987-21grep09:30:212
2311599185,5cyclictest12234-21cstates09:10:132
2311599183,3cyclictest0-21swapper/211:00:152
2311599183,3cyclictest0-21swapper/211:00:152
2311599182,13cyclictest5749-21cut12:35:162
2311599181,9cyclictest29361-21df_inode08:35:122
2311599181,7cyclictest2406-21sh07:35:012
2311599181,7cyclictest0-21swapper/210:42:352
2311599181,6cyclictest29399-21basename12:15:162
2311599181,5cyclictest0-21swapper/212:21:062
2311599181,3cyclictest20796-21sort09:30:012
23115991813,3cyclictest4635-21cat08:50:232
23115991813,3cyclictest16407-21tune2fs08:05:162
23115991812,3cyclictest27957-21grep12:10:242
23115991812,3cyclictest27957-21grep12:10:242
2311599181,15cyclictest11192-21turbostat.cron07:55:012
2311599181,15cyclictest0-21swapper/208:27:452
23115991811,4cyclictest31815-21grep11:05:262
23115991811,4cyclictest18304-21grep11:50:152
2311599178,3cyclictest87550irq/128-enp1s0-11:43:472
2311599177,3cyclictest87650irq/129-enp1s0-07:55:162
2311599174,3cyclictest24055-21latency_hist12:05:012
2311599173,2cyclictest87650irq/129-enp1s0-08:30:172
2311599172,6cyclictest12153-21ntp_states11:35:212
2311599172,12cyclictest3284-21strings11:15:182
2311599171,9cyclictest0-21swapper/210:55:082
2311599171,7cyclictest0-21swapper/210:29:192
2311599171,6cyclictest0-21swapper/210:46:292
2311599171,6cyclictest0-21swapper/209:15:182
2311599171,3cyclictest0-21swapper/210:53:432
2311599171,3cyclictest0-21swapper/208:10:152
23115991712,3cyclictest2711-21sshd08:46:112
23115991712,3cyclictest17048-21expr09:20:142
2311599171,15cyclictest0-21swapper/210:02:252
2311599171,15cyclictest0-21swapper/209:36:322
2311599171,13cyclictest32767-21unixbench_singl08:40:252
23115991711,3cyclictest23341-21memory08:20:182
2311599167,2cyclictest4586-21cat07:39:562
2311599161,6cyclictest6886-21irqstats10:10:172
2311599161,5cyclictest10766-21memory09:05:192
2311599161,3cyclictest0-21swapper/209:40:272
2311599161,3cyclictest0-21swapper/207:26:552
2311599161,3cyclictest0-21swapper/207:26:552
2311599161,2cyclictest0-21swapper/212:25:342
2311599161,2cyclictest0-21swapper/208:18:442
23115991612,3cyclictest0-21swapper/209:50:082
2311599161,14cyclictest0-21swapper/207:12:472
23115991611,3cyclictest5450-21/usr/sbin/munin08:55:242
2311599160,6cyclictest0-21swapper/207:40:242
2311599151,6cyclictest0-21swapper/211:47:012
2311599151,6cyclictest0-21swapper/207:45:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional