You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 14:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon May 18, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
196652204168,24sleep30-21swapper/307:09:363
193562203167,24sleep10-21swapper/107:05:361
195702201166,23sleep20-21swapper/207:08:232
194172201164,25sleep00-21swapper/007:06:230
81221760,3sleep11978699cyclictest07:35:261
2375621510,3sleep341-21ksoftirqd/309:43:383
117521470,4sleep01978099cyclictest08:50:210
2265821370,5sleep31979799cyclictest08:25:243
296302580,4sleep11978699cyclictest07:30:121
316522530,2sleep30-21swapper/310:00:193
242922530,1sleep30-21swapper/307:15:253
19780994214,14cyclictest5668-21munin-run07:50:000
1979799401,28cyclictest23668-21tune2fs12:10:163
1979799395,24cyclictest25423-21cat09:45:253
1979799382,20cyclictest31413-40dnf07:34:553
1979799381,19cyclictest1078-21unixbench_multi12:30:253
1979299383,9cyclictest0-21swapper/207:34:572
1979799372,27cyclictest17449-21chrt09:30:013
1979799372,23cyclictest27152-21cat09:50:173
1979299373,5cyclictest0-21swapper/210:05:012
19792993711,4cyclictest21950-21latency_hist07:15:012
1978099378,14cyclictest20737-21idleruntime-cro12:05:010
1979799364,23cyclictest87550irq/128-enp1s0-08:00:163
1979799364,15cyclictest23580-21ntp_kernel_pll_10:55:253
1979799364,14cyclictest18516-21perf12:00:023
1979799363,22cyclictest16464-21fschecks_time10:40:153
1979799361,2cyclictest87350irq/126-enp1s0-12:35:173
1979799361,22cyclictest29051-21perl09:55:143
19792993610,3cyclictest7263-21tr10:20:132
1979799352,24cyclictest3827-21sshd08:56:543
1979799351,24cyclictest19704-21/usr/sbin/munin09:35:173
1979799351,18cyclictest0-21swapper/310:35:263
1979799351,18cyclictest0-21swapper/310:35:263
19792993511,3cyclictest31608-21cat11:15:142
19792993511,3cyclictest13491-21idleruntime-cro10:35:012
1979799343,22cyclictest5316-21latency_hist11:30:013
1979799342,23cyclictest3975-21sendmail_mailqu10:10:243
1979799342,21cyclictest31320-21/usr/sbin/munin11:15:273
1979799341,21cyclictest26413-21runrttasks12:15:223
1979799332,21cyclictest12050irq/123-ahci[0011:45:023
1979799331,3cyclictest808-21snmpd08:37:453
1979799331,29cyclictest8092-21/usr/sbin/munin07:55:113
1979799331,29cyclictest31091-21latency_hist11:15:023
1979799331,28cyclictest27891-21diskstats12:20:183
1979799331,24cyclictest1-21systemd10:50:203
1979799331,23cyclictest19071-21missed_timers10:45:203
19797993310,8cyclictest14098-1kworker/u9:010:19:263
1979299337,8cyclictest15326-21grep08:10:172
1979299333,3cyclictest4839-21pmu-power07:45:222
1979799324,12cyclictest0-21swapper/311:03:353
1979799323,16cyclictest24266-21kworker/u8:211:05:153
1979799322,22cyclictest6519-21/usr/sbin/munin09:05:263
1979799321,3cyclictest15189-21grep11:50:183
1979799321,12cyclictest0-21swapper/312:01:373
19797993210,19cyclictest12276-21grep11:45:133
19797993210,19cyclictest12276-21grep11:45:133
1979299321,7cyclictest28877-21chrt08:40:202
1979299321,7cyclictest25451-21latency_hist08:35:022
1979299321,6cyclictest5892-21/usr/sbin/munin07:50:232
1979299321,6cyclictest18123-21/usr/sbin/munin10:45:122
1979299321,6cyclictest11262-21diskstats09:15:152
1979299321,4cyclictest4157-21interrupts11:25:162
19792993211,4cyclictest13720-21head11:45:252
19792993211,4cyclictest13720-21head11:45:252
19792993211,3cyclictest32162-21cat08:50:022
1979799315,18cyclictest87350irq/126-enp1s0-07:10:233
1979799313,19cyclictest87650irq/129-enp1s0-07:40:243
1979799311,4cyclictest440-21/usr/sbin/munin10:05:283
1979799311,27cyclictest17799-21expr09:30:123
1979799311,19cyclictest12889-21awk09:20:013
1979799311,18cyclictest12591-21ntp_states10:30:213
1979799311,17cyclictest30126-21fschecks_count12:25:153
1979799310,18cyclictest21566-21gltestperf12:05:183
1979299319,4cyclictest25986-21/usr/sbin/munin07:20:182
1979299312,11cyclictest22300-21munin-run10:55:012
1979299311,8cyclictest0-21swapper/212:04:362
19792993112,3cyclictest4934-21/usr/sbin/munin10:15:242
19792993112,3cyclictest458-21head08:50:132
19792993112,3cyclictest29122-21/usr/sbin/munin11:10:182
19792993112,3cyclictest10007-21hddtemp_smartct10:25:162
19792993111,4cyclictest1919-21grep07:40:152
19792993111,3cyclictest17676-21if_err_enp2s008:15:162
1978099319,11cyclictest0-21swapper/011:10:010
1979799305,14cyclictest87650irq/129-enp1s0-09:10:213
1979799304,14cyclictest87650irq/129-enp1s0-07:25:123
1979799302,25cyclictest1155-21smart_sda08:50:213
1979799302,24cyclictest5896-21diskstats11:30:143
1979799301,4cyclictest7154-21/usr/sbin/munin07:50:243
1979799301,3cyclictest14695-21/usr/sbin/munin08:10:193
1979799300,20cyclictest0-21swapper/311:20:163
1979299306,6cyclictest0-21swapper/212:26:282
1979299306,3cyclictest87650irq/129-enp1s0-11:53:542
1979299302,11cyclictest23249-21crond08:30:012
1979299301,5cyclictest10621-21df08:00:132
1979299301,4cyclictest9946-21/usr/sbin/munin11:40:102
19792993012,3cyclictest17515-21users10:40:262
19792993011,4cyclictest8439-21/usr/sbin/munin07:55:132
19792993011,3cyclictest27542-21/usr/sbin/munin12:20:262
19792993011,3cyclictest15856-21cat10:40:012
19792993011,3cyclictest15856-21cat10:40:012
1979799299,17cyclictest26565-21unixbench_multi07:20:253
1979799297,19cyclictest27992-21cpuspeed_turbos08:40:123
1979799296,20cyclictest0-21swapper/311:35:273
1979799291,3cyclictest19878-21/usr/sbin/munin08:20:173
1979799291,3cyclictest10247-21/usr/sbin/munin10:25:193
1979799291,25cyclictest4760-21ntp_states07:45:213
1979799291,24cyclictest5380-21ntp_states09:00:213
1979799290,21cyclictest0-21swapper/309:25:013
1979299297,2cyclictest7534-21awk11:35:012
19792992910,3cyclictest20246-21head08:20:222
19780992918,2cyclictest19690-21kworker/0:009:41:110
1979799286,19cyclictest8313-21cut10:20:233
1979799281,24cyclictest910-21systemd-logind09:20:203
1979799281,23cyclictest31670-21/usr/sbin/munin07:35:123
1979799281,23cyclictest30662-21/usr/sbin/munin08:45:163
19797992810,15cyclictest23487-21/usr/sbin/munin08:30:203
1979299288,8cyclictest0-21swapper/208:05:532
1979299283,5cyclictest0-21swapper/209:42:562
1979299281,11cyclictest0-21swapper/208:39:572
1978099281,12cyclictest920-21awk11:20:010
1979799271,3cyclictest13550-21/usr/sbin/munin08:05:213
1979799271,23cyclictest17573-21/usr/sbin/munin08:15:153
1979299276,3cyclictest0-21swapper/207:17:252
1979299271,5cyclictest0-21swapper/212:07:152
1978099272,17cyclictest10348-21kworker/0:109:16:230
1978099271,11cyclictest22555-21kworker/0:212:34:430
1978099271,10cyclictest19956-21kworker/0:111:02:230
1979299261,4cyclictest4309-21/usr/sbin/munin09:00:252
1979299261,3cyclictest0-21swapper/210:10:392
1979299261,3cyclictest0-21swapper/209:46:312
1979299261,12cyclictest0-21swapper/209:35:412
1979299261,10cyclictest0-21swapper/209:10:172
19792992610,3cyclictest0-21swapper/211:07:452
1978699262,4cyclictest27319-21munin-run12:20:011
1978099262,9cyclictest19690-21kworker/0:009:55:350
1978099261,12cyclictest16315-21munin-run11:55:010
1978099261,10cyclictest4963-21kworker/0:111:27:110
1979299258,3cyclictest0-21swapper/208:57:452
1979299257,3cyclictest0-21swapper/207:38:292
1979299251,3cyclictest0-21swapper/212:30:472
1979299251,11cyclictest0-21swapper/209:34:012
1978699254,16cyclictest15400-40dnf10:36:241
1978699254,16cyclictest15400-40dnf10:36:241
1978699251,3cyclictest26708-21perf11:05:011
1978699251,3cyclictest20104-21munin-run10:50:011
1978699251,16cyclictest0-21swapper/110:20:111
1978699251,15cyclictest0-21swapper/112:10:541
1978099258,15cyclictest12287-21perf08:05:020
1978099252,10cyclictest30222-21kworker/0:210:03:550
19780992513,3cyclictest17253-21kworker/0:209:31:390
1978099251,12cyclictest29858-21munin-run08:45:010
1978099251,11cyclictest29252-21awk07:30:010
1978099251,10cyclictest15472-21kworker/0:210:52:150
1979299248,3cyclictest7990-21expr11:35:132
1979299248,3cyclictest0-21swapper/212:16:132
1979299246,3cyclictest0-21swapper/212:35:252
1979299241,12cyclictest0-21swapper/211:55:452
19792992410,3cyclictest0-21swapper/209:28:292
1978699241,15cyclictest24161-21cpu09:45:121
1978699241,11cyclictest0-21swapper/109:16:141
1978699240,17cyclictest0-21swapper/112:35:161
1978099248,9cyclictest556-21kworker/0:007:16:390
1978099242,10cyclictest32494-21kworker/0:110:09:110
1978099241,10cyclictest22555-21kworker/0:212:36:590
1979299238,3cyclictest0-21swapper/210:07:332
1979299237,3cyclictest0-21swapper/209:55:292
1979299235,3cyclictest0-21swapper/207:28:332
1978699232,12cyclictest17355-21irqstats11:55:181
1978699232,11cyclictest19477-21munin-run09:35:011
19786992317,3cyclictest31506-21irqstats10:00:181
19786992313,2cyclictest2092-21cat08:55:021
1978699231,16cyclictest0-21swapper/109:59:441
1978699231,14cyclictest0-21swapper/109:52:111
1978699231,11cyclictest0-21swapper/110:06:291
1978699230,9cyclictest0-21swapper/108:08:311
1978099239,3cyclictest28368-21kworker/0:012:23:470
1978099238,2cyclictest22858-21kworker/0:312:16:030
1978099236,3cyclictest556-21kworker/0:007:12:550
1978099235,2cyclictest16376-21kworker/0:008:15:310
1978099232,8cyclictest15472-21kworker/0:210:57:030
1978099232,11cyclictest13078-21kworker/0:108:10:390
1978099231,12cyclictest11031-21kworker/0:210:28:470
1978099231,11cyclictest4083-21munin-run09:00:000
1978099231,11cyclictest17253-21kworker/0:209:38:070
19780992310,3cyclictest13264-21kworker/0:010:36:110
19780992310,3cyclictest13264-21kworker/0:010:36:110
1979299226,3cyclictest0-21swapper/211:02:372
1979299226,3cyclictest0-21swapper/210:56:452
1979299221,13cyclictest0-21swapper/209:50:132
1978699228,4cyclictest0-21swapper/112:24:341
1978699222,9cyclictest0-21swapper/112:00:241
1978699221,6cyclictest0-21swapper/112:30:501
1978699221,6cyclictest0-21swapper/109:35:161
1978699221,3cyclictest24146-21seq10:55:431
1978699221,15cyclictest0-21swapper/107:29:581
1978699221,12cyclictest17239-21diskstats08:15:131
19786992210,3cyclictest1472-21df11:20:121
1978699220,16cyclictest0-21swapper/111:40:101
1978699220,16cyclictest0-21swapper/110:14:081
1978099229,2cyclictest3315-21kworker/0:007:58:190
1978099227,3cyclictest18657-21kworker/0:208:22:480
1978099226,2cyclictest32494-21kworker/0:110:10:110
1978099222,8cyclictest18423-21kworker/0:012:07:270
1978099222,11cyclictest9444-21kworker/0:011:38:190
19780992211,2cyclictest15013-21kworker/0:009:28:430
257832210,1sleep10-21swapper/108:35:131
1979299217,3cyclictest0-21swapper/212:12:052
1979299214,3cyclictest0-21swapper/209:08:132
1978699219,4cyclictest27047-21awk07:25:011
1978699212,4cyclictest25461-21munin-run08:35:011
1978699211,18cyclictest17225-21sed10:40:241
1978699211,15cyclictest0-21swapper/112:28:481
1978699211,15cyclictest0-21swapper/110:17:591
1978699211,15cyclictest0-21swapper/108:59:091
1978699211,15cyclictest0-21swapper/108:49:001
1978699211,14cyclictest0-21swapper/110:50:361
19786992110,3cyclictest2764-21smart_sda07:40:241
1978699210,16cyclictest0-21swapper/108:22:331
1978699210,16cyclictest0-21swapper/108:13:581
1978099218,2cyclictest27354-21kworker/0:208:38:110
1978099217,3cyclictest9839-21kworker/0:208:05:120
1978099217,2cyclictest28674-21kworker/0:211:14:590
1978099217,2cyclictest15472-21kworker/0:210:47:590
1978099215,5cyclictest0-21swapper/010:30:180
1978099214,2cyclictest11669-21kworker/0:211:45:590
1978099214,2cyclictest11669-21kworker/0:211:45:590
1978099213,3cyclictest87550irq/128-enp1s0-07:22:070
1978099213,2cyclictest32494-21kworker/0:110:22:550
1978099212,9cyclictest10348-21kworker/0:109:20:070
1978099212,9cyclictest10348-21kworker/0:109:20:070
1978099212,11cyclictest29463-21/usr/sbin/munin07:30:240
1978099212,10cyclictest15924-21/usr/sbin/munin10:40:250
1978099211,9cyclictest29617-21kworker/0:008:47:390
1978099210,9cyclictest4963-21kworker/0:111:40:470
216892201,2sleep10-21swapper/107:11:471
1979299206,3cyclictest0-21swapper/209:23:332
1979299206,3cyclictest0-21swapper/209:23:332
1979299203,3cyclictest0-21swapper/211:20:452
19786992012,3cyclictest0-21swapper/111:17:271
19786992012,2cyclictest0-21swapper/109:00:531
1978699201,16cyclictest7204-21runrttasks11:31:051
1978699201,15cyclictest0-21swapper/110:29:031
1978699201,14cyclictest0-21swapper/111:36:131
1978699201,14cyclictest0-21swapper/108:41:081
1978699201,14cyclictest0-21swapper/108:28:331
1978699201,14cyclictest0-21swapper/107:52:421
1978699201,12cyclictest24266-21kworker/u8:212:05:411
1978699201,11cyclictest28893-21latency_hist11:10:021
1978699201,11cyclictest22909-21netstat09:40:191
1978699201,11cyclictest15410-21ntp_states11:50:211
1978699201,11cyclictest13311-21wc11:45:221
1978699201,11cyclictest13311-21wc11:45:221
1978699200,9cyclictest0-21swapper/111:13:571
1978699200,19cyclictest0-21swapper/110:33:431
1978699200,15cyclictest0-21swapper/108:02:311
1978099209,2cyclictest31386-21kworker/0:112:26:230
1978099208,2cyclictest19690-21kworker/0:009:51:510
1978099206,3cyclictest25762-21kworker/0:209:49:270
1978099206,2cyclictest4613-21kworker/0:210:17:150
1978099202,10cyclictest20967-21kworker/0:108:30:550
1978699197,3cyclictest15653-21diskstats09:25:131
1978699192,14cyclictest9789-21ntp_states09:10:231
19786991912,3cyclictest0-21swapper/111:26:481
1978699191,10cyclictest5745-21cat07:50:011
1978699191,10cyclictest24851-21munin-run07:20:011
1978699190,14cyclictest0-21swapper/107:57:291
1978099197,3cyclictest740-21kworker/0:011:24:550
1978099196,2cyclictest4963-21kworker/0:111:30:510
1978099195,3cyclictest31085-21kworker/0:307:44:350
1978099191,9cyclictest31085-21kworker/0:307:35:070
1978099191,10cyclictest16541-21/usr/sbin/munin11:55:220
1978699181,16cyclictest0-21swapper/109:05:551
1978699180,10cyclictest13306-21cat09:20:111
1978699180,10cyclictest13306-21cat09:20:111
1978099185,8cyclictest22858-21kworker/0:312:12:270
1978099182,10cyclictest6686-21kworker/0:009:10:230
19780991813,3cyclictest6555-21tail07:50:160
1978099181,10cyclictest4309-21/usr/sbin/munin09:00:200
1978099175,3cyclictest20967-21kworker/0:108:25:070
1978099161,3cyclictest6519-21/usr/sbin/munin09:05:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional