You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 20:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 10, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
42672224149,24sleep30-21swapper/307:05:033
62122202165,24sleep20-21swapper/207:09:332
59902202165,24sleep10-21swapper/107:06:431
61372201165,24sleep00-21swapper/007:08:340
851321580,3sleep2634599cyclictest08:25:172
6349999996,2cyclictest0-21swapper/311:52:453
164972570,1sleep00-21swapper/011:10:170
6349995450,3cyclictest0-21swapper/307:34:553
82892500,4sleep123-21ksoftirqd/108:25:161
27012500,2sleep10-21swapper/110:40:121
85252480,1sleep10-21swapper/107:15:001
188882450,2sleep00-21swapper/011:15:190
633899333,23cyclictest8590-21kworker/u8:211:42:491
288342320,2sleep30-21swapper/311:40:033
633899281,3cyclictest0-21swapper/109:19:081
633899280,26cyclictest0-21swapper/110:20:151
227482280,4sleep122751-21fschecks_time11:25:151
6338992715,6cyclictest20199-21cpuspeed_turbos11:20:141
633899270,4cyclictest5746-21kworker/u8:009:56:521
633899261,17cyclictest7233-21users08:20:251
633899261,16cyclictest0-21swapper/112:09:281
6338992611,3cyclictest20758-21diskstats07:40:141
633399268,15cyclictest28977-21munin-run08:00:010
633399261,13cyclictest25220-21awk09:05:010
6338992513,3cyclictest0-21swapper/108:15:131
6338992511,7cyclictest0-21swapper/109:20:231
633899251,16cyclictest0-21swapper/109:52:451
633899251,16cyclictest0-21swapper/108:47:591
633899251,16cyclictest0-21swapper/108:47:591
633899251,16cyclictest0-21swapper/108:37:471
633899251,16cyclictest0-21swapper/107:16:381
633899251,15cyclictest0-21swapper/108:31:101
633899251,15cyclictest0-21swapper/107:32:391
633899251,11cyclictest2249-21sh10:40:021
6338992510,3cyclictest11571-21memory09:45:171
634999241,20cyclictest24211-21strings09:00:193
634999241,16cyclictest0-21swapper/312:00:123
634999241,16cyclictest0-21swapper/311:10:223
633899242,9cyclictest17650-21munin-run11:15:011
6338992413,2cyclictest0-21swapper/110:01:001
6338992412,3cyclictest0-21swapper/112:35:591
6338992412,3cyclictest0-21swapper/107:38:331
6338992412,3cyclictest0-21swapper/107:38:331
633899241,20cyclictest31538-21expr08:05:131
633899241,20cyclictest23177-21fschecks_time07:45:161
633899241,17cyclictest0-21swapper/111:34:331
633899241,15cyclictest0-21swapper/112:32:211
633899241,15cyclictest0-21swapper/112:25:581
633899241,15cyclictest0-21swapper/109:25:281
633899241,15cyclictest0-21swapper/107:54:381
633899241,15cyclictest0-21swapper/107:24:311
633899241,11cyclictest27645-21cpu09:10:111
633899241,11cyclictest0-21swapper/109:05:151
633899240,15cyclictest0-21swapper/108:56:541
633399241,11cyclictest11694-21perf12:15:020
634999235,3cyclictest12050irq/123-ahci[0011:25:023
634999231,9cyclictest0-21swapper/310:35:473
633899231,4cyclictest7304-21ntp_kernel_pll_09:35:211
633899231,19cyclictest29115-21cat10:25:191
633899231,15cyclictest0-21swapper/112:14:281
633899231,14cyclictest0-21swapper/109:00:281
633899231,14cyclictest0-21swapper/108:12:201
633899231,14cyclictest0-21swapper/108:04:311
633899231,11cyclictest7823-21/usr/sbin/munin10:50:181
633899231,11cyclictest31168-21if_enp1s010:30:161
633899231,11cyclictest28630-21latency_hist11:40:021
633899231,10cyclictest9270-21irqstats09:40:181
633899231,10cyclictest21688-21cpu10:10:131
633899231,10cyclictest15320-21sort08:40:171
633899230,21cyclictest0-21swapper/111:05:281
633899230,21cyclictest0-21swapper/111:05:281
633899230,15cyclictest0-21swapper/112:00:041
634999229,2cyclictest13846-21head09:50:183
634999222,3cyclictest19851-21perf11:20:013
634999222,11cyclictest18214-21/usr/sbin/munin07:35:233
634999222,11cyclictest18214-21/usr/sbin/munin07:35:233
634999221,5cyclictest30645-21uname09:15:163
634999221,15cyclictest0-21swapper/312:07:523
634999220,5cyclictest0-21swapper/310:40:123
633899227,4cyclictest25816-21latency_hist10:20:011
633899221,3cyclictest27919-21memory07:55:181
633899221,3cyclictest11948-21hddtemp_smartct11:00:161
633899221,19cyclictest0-21swapper/111:45:091
633899221,19cyclictest0-21swapper/110:46:441
633899221,14cyclictest0-21swapper/112:16:051
633899221,14cyclictest0-21swapper/111:56:501
633899221,10cyclictest0-21swapper/107:25:251
633899220,14cyclictest0-21swapper/112:24:361
633899220,14cyclictest0-21swapper/108:50:111
6333992215,4cyclictest18357-21date12:30:010
4352220,1sleep20-21swapper/208:05:272
634999218,2cyclictest13100-21unixbench-2d07:20:243
634999216,9cyclictest0-21swapper/308:30:133
634999215,8cyclictest88450irq/129-enp1s0-11:30:183
634999212,11cyclictest6534-21/usr/sbin/munin07:10:193
634999212,11cyclictest23848-21/usr/sbin/munin10:15:143
634999212,10cyclictest27629-21/usr/sbin/munin09:10:183
634999211,7cyclictest7013-21if_enp2s009:35:173
634999211,7cyclictest17781-21pmu-power08:45:243
634999211,7cyclictest17781-21pmu-power08:45:243
634999211,18cyclictest0-21swapper/310:48:103
634999211,13cyclictest0-21swapper/308:15:183
634999211,12cyclictest28245-21/usr/sbin/munin10:25:183
634999211,12cyclictest0-21swapper/310:05:253
634999211,10cyclictest21684-21strings12:35:193
634599211,9cyclictest16348-21hddtemp_smartct11:10:152
634599211,8cyclictest0-21swapper/211:55:182
6345992116,3cyclictest24688-21grep09:00:242
634599211,12cyclictest0-21swapper/211:38:082
633899218,3cyclictest769-21avahi-daemon09:30:071
633899211,18cyclictest9994-21memory10:55:191
633899211,11cyclictest0-21swapper/111:16:041
633399216,10cyclictest0-21swapper/012:03:290
633399212,4cyclictest0-21swapper/012:17:430
633399211,6cyclictest32229-21kworker/0:010:32:570
633399211,12cyclictest26045-21/usr/sbin/munin10:20:230
225972210,4sleep011-21rcuc/007:45:010
634999208,3cyclictest0-21swapper/311:55:493
634999208,3cyclictest0-21swapper/308:25:453
634999204,3cyclictest22174-21grep08:55:243
634999202,9cyclictest4059-21/usr/sbin/munin09:30:223
6349992012,3cyclictest0-21swapper/312:19:543
634999201,11cyclictest27954-21sshd10:24:193
634999201,11cyclictest0-21swapper/311:28:023
634599205,5cyclictest9489-21perf12:10:012
634599204,6cyclictest19859-21awk11:20:012
634599201,8cyclictest0-21swapper/211:08:272
634599201,8cyclictest0-21swapper/211:08:262
634599201,5cyclictest0-21swapper/211:30:182
633899201,18cyclictest0-21swapper/110:05:471
633899201,17cyclictest0-21swapper/111:51:251
633399209,3cyclictest6260-21/usr/sbin/munin09:35:200
633399207,4cyclictest0-21swapper/012:36:160
633399207,3cyclictest0-21swapper/010:40:440
633399207,3cyclictest0-21swapper/010:05:010
633399201,7cyclictest27739-21ntp_states11:35:220
633399201,5cyclictest15052-21/usr/sbin/munin09:55:240
6333992014,3cyclictest32616-21tail09:20:160
633399201,11cyclictest862-21/usr/sbin/munin11:50:200
633399201,10cyclictest0-21swapper/010:50:310
6333992010,3cyclictest843-21runrttasks09:45:550
633399200,6cyclictest0-21swapper/009:41:150
634999195,3cyclictest5750irq/9-acpi11:00:263
634999195,3cyclictest12050irq/123-ahci[0011:00:013
634999194,3cyclictest88450irq/129-enp1s0-09:40:163
634999193,8cyclictest88450irq/129-enp1s0-09:45:113
634999192,4cyclictest23629-21turbostat10:15:013
634999191,9cyclictest29852-21head11:40:203
634999191,9cyclictest0-21swapper/310:00:333
634999191,6cyclictest24825-21cpu07:50:113
6349991914,4cyclictest0-21swapper/312:26:503
6349991914,3cyclictest7540-21tail10:50:143
6349991914,2cyclictest15039-21tail08:40:153
6349991913,4cyclictest31750-21sh08:05:143
6349991913,3cyclictest842-21lldpd12:22:423
634999191,10cyclictest31289-21kernelversion10:30:173
634999191,10cyclictest16531-21sed09:55:253
634999191,10cyclictest13479-21/usr/sbin/munin11:05:163
634999191,10cyclictest13479-21/usr/sbin/munin11:05:163
634999191,10cyclictest12237-21/usr/sbin/munin08:35:143
634599196,7cyclictest14423-40dnf08:40:072
634599196,4cyclictest88150irq/126-enp1s0-10:00:192
634599192,7cyclictest5639-21/usr/sbin/munin08:20:152
634599191,7cyclictest0-21swapper/212:37:042
6345991913,3cyclictest1526-21cstates08:10:132
634599191,14cyclictest20741-21sed11:20:172
634599191,10cyclictest0-21swapper/212:18:012
633399197,4cyclictest0-21swapper/010:28:550
633399196,4cyclictest6564-21missed_timers08:20:180
633399191,9cyclictest27398-21diskmemload10:40:010
633399191,7cyclictest0-21swapper/011:06:320
633399191,7cyclictest0-21swapper/011:06:320
633399191,6cyclictest0-21swapper/011:20:210
633399191,5cyclictest0-21swapper/011:30:170
6333991913,3cyclictest17273-21gltestperf08:45:170
6333991913,3cyclictest17273-21gltestperf08:45:170
6333991912,4cyclictest17252-21find07:30:240
633399191,10cyclictest29200-21/usr/sbin/munin08:00:150
633399191,10cyclictest18482-21kworker/u8:209:10:230
633399191,10cyclictest14099-21/usr/sbin/munin12:20:240
633399191,10cyclictest0-21swapper/010:57:350
633399190,6cyclictest0-21swapper/009:16:070
182542191,3sleep20-21swapper/212:29:582
134832190,4sleep230-21rcuc/208:35:242
65272181,2sleep00-21swapper/007:10:070
634999184,3cyclictest88450irq/129-enp1s0-07:55:183
634999183,8cyclictest39-21rcuc/312:30:123
634999183,3cyclictest88150irq/126-enp1s0-08:10:193
634999181,6cyclictest0-21swapper/309:25:163
634999181,6cyclictest0-21swapper/307:25:143
634999181,5cyclictest32372-21diskstats09:20:133
634999181,5cyclictest31827-21ethtool11:45:173
634999181,4cyclictest0-21swapper/307:15:193
6349991811,4cyclictest10953-21/usr/sbin/munin12:10:233
634599182,5cyclictest0-21swapper/210:21:522
634599182,14cyclictest0-21swapper/207:11:302
634599181,8cyclictest0-21swapper/208:55:162
634599181,7cyclictest31067-21/usr/sbin/munin11:45:222
634599181,6cyclictest29221-21sshd10:25:192
6345991812,3cyclictest13250-21idleruntime-cro11:05:022
634599181,15cyclictest0-21swapper/207:23:282
634599181,12cyclictest0-21swapper/209:29:202
634599181,11cyclictest0-21swapper/212:21:322
634599181,11cyclictest0-21swapper/211:42:252
634599181,11cyclictest0-21swapper/209:57:422
634599181,10cyclictest0-21swapper/207:57:192
633399188,8cyclictest88150irq/126-enp1s0-08:50:190
633399187,3cyclictest0-21swapper/011:55:080
633399187,3cyclictest0-21swapper/009:50:450
633399186,3cyclictest0-21swapper/010:11:020
633399186,3cyclictest0-21swapper/009:32:210
633399186,3cyclictest0-21swapper/009:28:090
633399184,4cyclictest0-21swapper/012:05:270
633399182,4cyclictest19234-21munin-run10:05:010
633399181,9cyclictest823-21snmpd08:40:180
633399181,9cyclictest27398-21diskmemload12:32:540
633399181,9cyclictest13820-21/usr/sbin/munin07:25:190
633399181,9cyclictest0-21swapper/007:23:200
633399181,7cyclictest0-21swapper/011:45:270
633399181,6cyclictest24524-21smartctl10:15:140
633399181,5cyclictest22808-21fschecks_time11:25:150
633399181,11cyclictest0-21swapper/011:05:000
633399181,10cyclictest0-21swapper/011:40:360
634999178,6cyclictest0-21swapper/307:45:193
634999175,2cyclictest0-21swapper/307:40:153
634999174,3cyclictest88450irq/129-enp1s0-08:20:243
634999172,9cyclictest0-21swapper/308:00:263
634999171,5cyclictest19591-21if_enp2s008:50:183
6349991712,3cyclictest26892-21unixbench-2d09:05:253
634599175,3cyclictest88150irq/126-enp1s0-09:30:122
634599172,6cyclictest27629-21/usr/sbin/munin09:10:182
634599172,13cyclictest20804-21grep10:05:252
634599172,12cyclictest20066-21head12:30:252
634599172,11cyclictest872-21gltestperf10:35:152
634599171,6cyclictest3440-21/usr/sbin/munin08:15:172
634599171,6cyclictest0-21swapper/211:54:312
634599171,6cyclictest0-21swapper/210:56:402
634599171,6cyclictest0-21swapper/209:50:082
6345991713,3cyclictest0-21swapper/209:36:522
6345991713,3cyclictest0-21swapper/207:30:522
6345991713,3cyclictest0-21swapper/207:16:052
634599171,2cyclictest0-21swapper/207:40:242
6345991712,3cyclictest31390-21sed08:05:032
6345991712,3cyclictest0-21swapper/210:14:082
6345991712,2cyclictest8459-21/usr/sbin/munin09:40:192
634599171,15cyclictest0-21swapper/207:35:182
634599171,15cyclictest0-21swapper/207:35:182
634599171,14cyclictest32-21ksoftirqd/210:50:002
634599171,13cyclictest32140-21tr09:20:122
6345991711,3cyclictest26938-21cat07:55:012
6345991711,3cyclictest17811-21processes08:45:242
6345991711,3cyclictest17811-21processes08:45:242
634599170,6cyclictest0-21swapper/210:50:192
634599170,5cyclictest3693-21processes10:40:222
633399173,8cyclictest0-21swapper/010:48:210
633399172,12cyclictest21026-21/usr/sbin/munin08:55:250
633399171,3cyclictest1220-21/usr/sbin/munin08:10:230
6333991711,3cyclictest23901-21cat07:45:240
633399171,11cyclictest0-21swapper/008:05:240
256462171,2sleep20-21swapper/210:17:592
241142171,2sleep20-21swapper/211:28:342
110772170,2sleep00-21swapper/007:16:050
634599162,7cyclictest0-21swapper/212:10:162
634599162,11cyclictest31405-21unixbench_singl09:15:242
634599161,9cyclictest0-21swapper/210:30:152
634599161,5cyclictest25572-21cpuspeed_turbos09:05:122
6345991612,3cyclictest0-21swapper/212:02:592
6345991612,3cyclictest0-21swapper/208:51:372
6345991612,3cyclictest0-21swapper/207:26:262
6345991611,3cyclictest10037-21/usr/sbin/munin08:30:172
6333991611,3cyclictest24809-21/usr/sbin/munin07:50:200
6333991611,3cyclictest18214-21/usr/sbin/munin07:35:170
6333991611,3cyclictest18214-21/usr/sbin/munin07:35:170
6333991611,3cyclictest12237-21/usr/sbin/munin08:35:170
6333991611,3cyclictest11738-21seq08:31:410
633399160,9cyclictest0-21swapper/008:25:140
122682161,2sleep20-21swapper/209:45:322
634599159,3cyclictest22899-21expr07:45:132
633399151,5cyclictest3788-21diskstats08:15:130
6333991511,3cyclictest0-21swapper/009:05:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional