You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 00:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Jan 09, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
50512233157,25sleep20-21swapper/207:09:302
48412203166,25sleep10-21swapper/107:06:481
48232201166,23sleep30-21swapper/307:06:343
49852199162,24sleep00-21swapper/007:08:390
795821790,4sleep3519099cyclictest10:55:123
1256321670,4sleep3519099cyclictest08:35:233
689721390,4sleep3519099cyclictest12:05:153
5183995956,2cyclictest0-21swapper/211:52:342
323172560,1sleep20-21swapper/209:20:242
516999388,14cyclictest18683-21munin-run11:20:010
519099331,13cyclictest0-21swapper/308:45:353
519099331,13cyclictest0-21swapper/307:33:313
516999338,12cyclictest23077-21munin-run11:30:020
519099321,14cyclictest0-21swapper/310:36:313
517799321,23cyclictest8886-21users09:40:261
517799321,18cyclictest0-21swapper/110:10:331
519099311,15cyclictest0-21swapper/309:34:183
519099311,12cyclictest0-21swapper/309:06:183
519099301,6cyclictest0-21swapper/309:24:103
519099301,18cyclictest0-21swapper/311:13:553
519099301,14cyclictest0-21swapper/312:31:153
519099301,13cyclictest0-21swapper/310:50:473
519099301,12cyclictest0-21swapper/312:00:253
517799301,18cyclictest0-21swapper/109:11:121
516999305,12cyclictest9898-21basename11:00:010
519099291,9cyclictest26242-21diskmemload09:19:023
519099291,6cyclictest0-21swapper/309:53:383
519099291,13cyclictest0-21swapper/312:25:273
519099291,13cyclictest0-21swapper/311:56:233
519099291,13cyclictest0-21swapper/310:42:253
519099291,13cyclictest0-21swapper/308:40:233
519099291,12cyclictest12315-21uptime12:15:253
519099291,12cyclictest0-21swapper/309:44:023
519099291,12cyclictest0-21swapper/308:18:553
519099291,11cyclictest0-21swapper/309:45:343
519099291,10cyclictest0-21swapper/309:59:193
517799292,4cyclictest11075-21df12:15:131
517799291,14cyclictest2163-21cat11:55:131
519099281,17cyclictest0-21swapper/312:10:193
519099281,12cyclictest0-21swapper/311:07:593
519099281,12cyclictest0-21swapper/308:07:313
519099281,11cyclictest26242-21diskmemload10:23:113
519099281,11cyclictest0-21swapper/308:10:113
519099281,11cyclictest0-21swapper/308:03:373
519099280,14cyclictest18661-21kworker/u8:208:20:593
519099280,13cyclictest0-21swapper/310:05:473
517799281,25cyclictest0-21swapper/110:20:121
5177992810,9cyclictest0-21swapper/111:27:461
516999289,16cyclictest8654-21munin-run08:30:010
519099271,12cyclictest0-21swapper/308:55:433
519099271,11cyclictest0-21swapper/310:04:473
519099271,11cyclictest0-21swapper/307:48:393
519099271,10cyclictest0-21swapper/310:49:513
518399271,6cyclictest0-21swapper/210:19:142
518399271,12cyclictest16145-21hddtemp_smartct08:45:142
517799271,16cyclictest0-21swapper/109:27:271
517799271,12cyclictest13047-21munin-run08:40:001
517799270,18cyclictest0-21swapper/110:29:141
516999273,11cyclictest7352-21munin-run07:15:000
519099261,9cyclictest26242-21diskmemload09:37:223
519099261,12cyclictest0-21swapper/311:15:073
519099261,12cyclictest0-21swapper/307:39:593
519099261,11cyclictest0-21swapper/311:44:303
519099261,11cyclictest0-21swapper/307:21:543
519099261,10cyclictest26242-21diskmemload11:39:063
518399268,5cyclictest0-21swapper/210:30:522
517799262,7cyclictest3243-21diskstats09:30:141
517799262,6cyclictest778-21ntpd09:55:201
517799261,17cyclictest0-21swapper/111:19:161
5177992611,7cyclictest0-21swapper/109:35:391
517799261,15cyclictest87350irq/128-enp1s0-10:53:411
517799261,15cyclictest0-21swapper/110:30:181
517799261,11cyclictest7731-21fschecks_count07:15:021
5177992610,9cyclictest0-21swapper/110:08:221
519099251,13cyclictest0-21swapper/311:27:433
519099251,13cyclictest0-21swapper/309:04:463
519099251,12cyclictest0-21swapper/310:11:073
519099251,11cyclictest32763-21smartctl11:50:153
519099251,11cyclictest0-21swapper/312:38:433
519099251,11cyclictest0-21swapper/311:45:463
519099251,11cyclictest0-21swapper/310:28:073
519099251,11cyclictest0-21swapper/310:16:433
519099251,11cyclictest0-21swapper/309:26:503
519099251,11cyclictest0-21swapper/308:52:193
519099250,11cyclictest496-21jbd2/dm-0-807:56:053
518399252,7cyclictest23377-40dnf09:00:242
518399252,16cyclictest0-21swapper/212:39:382
5177992518,4cyclictest9827-21df09:45:111
517799251,7cyclictest0-21swapper/112:20:231
517799251,6cyclictest15685-21grep12:25:151
517799251,16cyclictest0-21swapper/110:01:131
517799251,15cyclictest0-21swapper/112:11:011
517799250,16cyclictest0-21swapper/112:01:381
516999251,4cyclictest25217-21kworker/u8:011:42:180
516999251,12cyclictest19314-21awk12:35:020
516999250,12cyclictest21515-21basename12:40:020
519099241,15cyclictest0-21swapper/307:43:153
519099241,11cyclictest0-21swapper/308:27:353
518399242,6cyclictest0-21swapper/212:01:442
517799245,5cyclictest0-21swapper/109:51:251
5177992420,3cyclictest0-21swapper/109:15:241
5177992420,3cyclictest0-21swapper/107:40:221
5177992419,3cyclictest20382-21sort12:35:161
517799241,19cyclictest0-21swapper/111:35:191
517799240,6cyclictest0-21swapper/110:38:271
517799240,16cyclictest0-21swapper/111:31:011
517799240,15cyclictest0-21swapper/111:54:331
517799240,14cyclictest0-21swapper/111:12:411
130822240,4sleep3519099cyclictest12:20:123
519099231,9cyclictest0-21swapper/310:32:273
519099231,15cyclictest0-21swapper/308:33:593
519099231,10cyclictest0-21swapper/311:00:033
519099231,10cyclictest0-21swapper/311:00:033
519099230,15cyclictest0-21swapper/309:10:423
518399236,9cyclictest15990-21sed11:10:252
518399234,6cyclictest30-21rcuc/210:55:222
518399231,5cyclictest0-21swapper/211:05:352
5177992319,3cyclictest0-21swapper/110:15:231
5177992319,3cyclictest0-21swapper/108:58:431
5177992319,3cyclictest0-21swapper/107:47:261
517799231,8cyclictest0-21swapper/112:05:311
5177992318,2cyclictest18396-21ntpq12:30:211
517799231,7cyclictest30881-21cpu09:20:111
517799231,7cyclictest0-21swapper/111:20:201
517799231,7cyclictest0-21swapper/108:15:181
517799231,7cyclictest0-21swapper/107:20:431
5177992317,3cyclictest17115-21egrep07:35:111
517799231,4cyclictest12663-21diskstats11:05:151
517799231,16cyclictest0-21swapper/108:45:171
517799231,15cyclictest0-21swapper/111:01:581
517799231,15cyclictest0-21swapper/111:01:571
517799231,11cyclictest5536-21ntp_offset08:20:191
517799230,13cyclictest0-21swapper/111:46:261
516999231,3cyclictest0-21swapper/009:53:230
519099227,3cyclictest0-21swapper/307:12:073
519099221,13cyclictest0-21swapper/311:31:263
519099221,13cyclictest0-21swapper/307:26:303
519099221,13cyclictest0-21swapper/307:26:303
519099221,11cyclictest0-21swapper/311:21:233
518399225,4cyclictest4546-21chrt10:45:202
518399223,6cyclictest19251-21/usr/sbin/munin07:40:112
518399223,5cyclictest0-21swapper/210:23:542
518399221,5cyclictest30914-21gltestperf08:05:142
518399221,4cyclictest6671-21switchtime07:10:242
518399221,3cyclictest0-21swapper/209:18:122
518399220,5cyclictest9749-21kworker/u8:111:20:572
5177992218,3cyclictest0-21swapper/110:56:091
5177992218,3cyclictest0-21swapper/108:04:581
517799221,6cyclictest23533-21sed09:00:241
517799221,5cyclictest0-21swapper/110:40:111
517799220,7cyclictest0-21swapper/110:45:541
516999224,9cyclictest3430-21latency_hist10:45:010
516999221,6cyclictest1644-21cat09:25:210
519099214,3cyclictest0-21swapper/307:15:353
518399214,5cyclictest10218-21timerandwakeup08:30:232
518399214,4cyclictest27391-21awk07:55:272
518399213,6cyclictest32-21ksoftirqd/208:55:152
518399212,10cyclictest0-21swapper/207:38:452
518399211,7cyclictest9768-21sadc12:10:232
518399211,7cyclictest29029-21kworker/u8:011:00:102
518399211,7cyclictest29029-21kworker/u8:011:00:102
518399211,7cyclictest27427-21diskstats10:25:142
518399211,7cyclictest1068-21munin-run10:40:012
518399211,6cyclictest27723-21sa109:10:222
518399211,6cyclictest0-21swapper/211:33:202
518399211,6cyclictest0-21swapper/209:45:112
518399211,5cyclictest0-21swapper/212:25:192
517799219,7cyclictest0-21swapper/108:10:171
517799217,11cyclictest30254-21acpi08:05:091
5177992117,3cyclictest0-21swapper/107:31:381
5177992117,3cyclictest0-21swapper/107:12:171
517799211,6cyclictest28046-21/usr/sbin/munin11:40:141
517799211,4cyclictest24406-21if_enp2s007:50:151
517799211,18cyclictest0-21swapper/108:40:221
517799211,10cyclictest24443-21cstates09:05:121
517799210,19cyclictest0-21swapper/108:25:191
516999219,3cyclictest87350irq/128-enp1s0-12:15:390
516999217,4cyclictest13184-21latency11:05:190
516999211,3cyclictest0-21swapper/010:15:030
5169992112,3cyclictest3482-21grep09:30:160
516999211,11cyclictest5684-21grep12:00:240
519099201,13cyclictest0-21swapper/307:50:273
518399202,7cyclictest0-21swapper/210:07:092
518399202,6cyclictest32-21ksoftirqd/208:25:142
518399202,6cyclictest30-21rcuc/211:15:152
518399202,5cyclictest16195-21threads07:30:242
518399201,7cyclictest9622-21cat07:15:232
518399201,7cyclictest3965-21ntp_states09:30:232
518399201,7cyclictest0-21swapper/211:47:222
518399201,6cyclictest9996-21kworker/2:108:10:252
518399201,6cyclictest25878-21fschecks_count11:35:132
518399201,6cyclictest1399-21idleruntime09:25:182
518399201,6cyclictest13009-21diskstats07:25:142
518399201,6cyclictest13009-21diskstats07:25:142
518399201,5cyclictest2706-21fschecks_count08:15:152
518399201,3cyclictest660-21kworker/u8:210:40:272
518399201,12cyclictest0-21swapper/212:09:482
5177992016,3cyclictest0-21swapper/108:32:091
517799201,5cyclictest12661-21/usr/sbin/munin07:25:161
517799201,5cyclictest12661-21/usr/sbin/munin07:25:161
517799201,3cyclictest0-21swapper/107:55:331
516999209,4cyclictest407-1kworker/u9:011:59:010
516999207,3cyclictest32119-21kworker/0:108:15:150
516999207,3cyclictest18832-21ps08:50:210
516999202,5cyclictest407-1kworker/u9:011:12:380
516999202,4cyclictest13403-21/usr/sbin/munin07:25:160
516999202,4cyclictest13403-21/usr/sbin/munin07:25:160
516999201,7cyclictest6048-21df10:50:140
516999201,6cyclictest0-21swapper/009:19:500
516999201,11cyclictest26411-21memory11:35:170
516999201,10cyclictest557-21latency_hist09:25:010
516999201,10cyclictest22456-21turbostat.cron10:15:010
321712200,3sleep00-21swapper/011:50:100
518399197,2cyclictest30-21rcuc/207:45:212
518399195,3cyclictest12050irq/123-ahci[0008:55:012
518399194,3cyclictest0-21swapper/212:15:182
518399193,4cyclictest6012-21memory09:35:182
518399193,4cyclictest15022-21sed12:25:022
518399191,7cyclictest30-21rcuc/211:25:222
518399191,6cyclictest0-21swapper/209:50:062
518399191,6cyclictest0-21swapper/209:40:402
518399191,5cyclictest17721-40dnf10:00:522
518399191,5cyclictest14631-21timerandwakeup08:40:232
518399191,15cyclictest10588-21cpuspeed_turbos07:20:112
517799190,8cyclictest0-21swapper/108:50:161
516999196,3cyclictest10462-21/usr/sbin/munin07:20:110
516999195,3cyclictest20271-21munin-run10:10:010
516999194,5cyclictest20890-21ntp_kernel_err08:55:200
516999191,5cyclictest0-21swapper/011:03:270
516999191,5cyclictest0-21swapper/011:03:270
516999191,5cyclictest0-21swapper/010:38:020
516999191,4cyclictest22055-21/usr/sbin/munin09:00:170
5169991914,3cyclictest19894-21needreboot11:20:190
516999191,12cyclictest0-21swapper/009:35:180
516999191,11cyclictest0-21swapper/012:25:220
516999191,10cyclictest27642-21sshd07:57:590
516999191,10cyclictest0-21swapper/010:28:470
204262191,2sleep00-21swapper/007:40:220
150722190,4sleep011-21rcuc/009:55:210
518399183,5cyclictest30-21rcuc/211:55:172
518399183,3cyclictest23945-21cstates07:50:122
518399181,6cyclictest26242-21diskmemload12:34:212
518399181,6cyclictest0-21swapper/210:50:102
518399181,5cyclictest28718-21hddtemp_smartct08:00:152
518399181,5cyclictest26242-21diskmemload10:10:242
518399181,5cyclictest13887-21/usr/sbin/munin09:55:192
518399181,4cyclictest27695-21/usr/sbin/munin11:40:152
5183991813,3cyclictest4478-21/usr/sbin/munin08:20:202
5183991812,3cyclictest11072-21/usr/sbin/munin08:35:092
516999188,7cyclictest27106-21gltestperf09:10:150
516999186,3cyclictest9064-21cut12:10:150
516999185,3cyclictest31475-21users11:45:250
516999182,9cyclictest30531-21sensors_temp10:30:240
516999182,9cyclictest16426-21df_inode10:00:110
516999181,9cyclictest16977-21grep08:45:240
516999181,4cyclictest8148-21kworker/u8:012:08:040
516999181,4cyclictest22052-21grep07:45:140
5169991813,3cyclictest14813-21unixbench_singl08:40:250
5169991812,3cyclictest9586-21sort09:45:100
516999181,10cyclictest266322sleep010:22:170
232952181,2sleep00-21swapper/011:30:100
518399171,4cyclictest25181-21memory09:05:172
516999174,4cyclictest24364-21basename07:50:150
516999174,3cyclictest8309-21netstat09:40:200
516999171,6cyclictest9854-21users07:15:260
516999171,6cyclictest5082-21gltestperf08:20:160
5169991713,2cyclictest31746-21unixbench_multi08:05:230
5169991711,3cyclictest17401-21/usr/sbin/munin07:35:120
5169991711,3cyclictest12307-21grep08:35:200
516999171,10cyclictest0-21swapper/012:23:420
516999161,6cyclictest0-21swapper/007:30:550
516999161,3cyclictest0-21swapper/009:05:180
5169991611,3cyclictest4445-21sed10:45:190
5169991611,3cyclictest32721-21expr08:10:120
5169991611,3cyclictest28345-21expr08:00:110
516999151,4cyclictest9262-21cut08:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional