You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 23:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri May 22, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
258542235167,56sleep20-21swapper/207:06:162
259802233157,24sleep00-21swapper/007:07:540
260452202166,23sleep30-21swapper/307:08:463
258312202166,24sleep10-21swapper/107:05:581
2623699104101,2cyclictest0-21swapper/211:14:472
2623699104101,2cyclictest0-21swapper/211:14:472
26236999188,2cyclictest0-21swapper/210:31:252
26236998164,16cyclictest0-21swapper/212:33:482
26236996259,2cyclictest0-21swapper/209:22:022
26236995653,2cyclictest0-21swapper/210:55:512
174552500,2sleep20-21swapper/211:40:212
147582470,1sleep20-21swapper/207:55:122
289172430,4sleep30-21swapper/308:25:223
13662430,2sleep20-21swapper/207:25:102
2622199379,13cyclictest19953-21munin-run10:35:010
2622999341,25cyclictest0-21swapper/109:32:541
2622999340,8cyclictest0-21swapper/109:55:531
159792340,5sleep32624199cyclictest09:10:173
2622999321,13cyclictest0-21swapper/110:15:051
2622199326,12cyclictest2979-21idleruntime-cro11:10:020
205152320,4sleep30-21swapper/310:35:143
2622999311,11cyclictest0-21swapper/110:12:051
2622199317,2cyclictest10978-21kworker/0:110:18:190
2622999301,6cyclictest0-21swapper/111:27:211
2622999301,6cyclictest0-21swapper/107:19:091
2622999301,13cyclictest28362-21users10:50:251
2622999301,13cyclictest0-21swapper/109:28:261
2622999301,12cyclictest4963-21cat10:00:111
2622999301,12cyclictest4963-21cat10:00:111
2622999301,11cyclictest0-21swapper/112:02:101
2622999300,14cyclictest0-21swapper/110:29:511
265702290,11sleep07-21ksoftirqd/012:00:260
2622999291,12cyclictest0-21swapper/110:57:151
2622999291,11cyclictest0-21swapper/111:03:331
2622999291,11cyclictest0-21swapper/109:51:331
2622999291,10cyclictest0-21swapper/110:41:371
2622999281,9cyclictest0-21swapper/108:27:031
2622999281,9cyclictest0-21swapper/107:21:291
2622999281,18cyclictest0-21swapper/111:33:381
2622999281,12cyclictest0-21swapper/112:08:111
2622999281,11cyclictest1525-21fschecks_time11:05:151
2622999281,11cyclictest0-21swapper/111:46:221
2622999281,10cyclictest0-21swapper/111:56:381
2622999271,9cyclictest0-21swapper/107:33:501
2622999271,8cyclictest0-21swapper/108:42:191
2622999271,17cyclictest0-21swapper/112:33:131
2622999271,11cyclictest0-21swapper/111:40:081
2622999271,10cyclictest0-21swapper/111:15:291
2622999270,12cyclictest0-21swapper/110:48:111
2622999270,11cyclictest0-21swapper/109:16:541
2622199271,7cyclictest0-21swapper/012:23:580
2622199271,7cyclictest0-21swapper/012:23:580
2622199271,7cyclictest0-21swapper/011:19:200
26221992710,15cyclictest12132-21perf07:50:010
2622999261,9cyclictest0-21swapper/112:10:331
2622999261,9cyclictest0-21swapper/111:54:221
2622999261,9cyclictest0-21swapper/108:14:191
2622999261,16cyclictest0-21swapper/110:39:331
2622999261,16cyclictest0-21swapper/109:42:291
2622999261,14cyclictest32199-21cpuspeed_turbos08:35:131
2622999261,10cyclictest0-21swapper/112:27:171
2622999261,10cyclictest0-21swapper/110:07:551
2624199251,16cyclictest0-21swapper/311:26:343
2622999251,9cyclictest0-21swapper/109:35:581
2622999251,8cyclictest0-21swapper/108:19:131
2622999251,16cyclictest0-21swapper/109:00:221
2622999251,13cyclictest0-21swapper/109:14:461
2622999251,12cyclictest0-21swapper/108:56:091
2622999251,10cyclictest0-21swapper/112:21:131
2622999251,10cyclictest0-21swapper/112:21:131
2622999251,10cyclictest0-21swapper/110:34:251
2622999251,10cyclictest0-21swapper/109:24:461
2622999250,17cyclictest0-21swapper/107:55:081
2622999250,13cyclictest0-21swapper/111:20:211
2622199251,5cyclictest0-21swapper/012:27:580
26221992513,4cyclictest18599-21/usr/sbin/munin11:45:220
26241992417,2cyclictest0-21swapper/310:10:213
2624199241,14cyclictest0-21swapper/310:20:053
2623699249,11cyclictest22760-21latency_hist11:55:012
2622999241,9cyclictest0-21swapper/107:28:241
2622999241,10cyclictest0-21swapper/112:38:411
2622999241,10cyclictest0-21swapper/108:45:091
2622199241,5cyclictest0-21swapper/010:09:090
287952230,4sleep230-21rcuc/209:40:152
279512231,2sleep30-21swapper/310:50:243
2624199231,6cyclictest0-21swapper/310:55:323
2624199231,16cyclictest0-21swapper/309:30:163
2624199230,5cyclictest0-21swapper/311:00:543
2623699231,3cyclictest0-21swapper/209:58:142
2622999232,18cyclictest7467-21open_inodes08:50:211
2622999232,18cyclictest7467-21open_inodes08:50:211
26229992318,3cyclictest14935-21diskmemload11:12:241
26229992318,3cyclictest14935-21diskmemload11:12:241
2622999231,7cyclictest0-21swapper/107:52:331
2622999231,6cyclictest0-21swapper/107:36:211
2622999231,16cyclictest0-21swapper/108:32:581
2622999231,12cyclictest0-21swapper/107:12:131
2622999231,11cyclictest0-21swapper/109:09:131
2622999231,10cyclictest0-21swapper/111:36:071
2622999230,7cyclictest0-21swapper/110:21:311
2622199235,16cyclictest32524-21munin-run09:50:000
2622199233,10cyclictest9347-21if_err_enp2s008:55:180
2622199233,10cyclictest10226-21sort12:40:020
2622199231,5cyclictest0-21swapper/012:11:360
2622199231,5cyclictest0-21swapper/009:43:070
2624199226,7cyclictest87650irq/129-enp1s0-08:45:113
2624199226,4cyclictest2754-21/usr/sbin/munin09:55:123
2624199224,3cyclictest0-21swapper/312:20:143
2624199224,3cyclictest0-21swapper/312:20:143
2624199223,6cyclictest7239-40dnf11:18:253
2624199223,3cyclictest0-21swapper/311:36:343
2624199221,6cyclictest0-21swapper/310:25:453
2624199221,6cyclictest0-21swapper/309:19:003
2624199221,15cyclictest0-21swapper/312:34:203
2624199221,13cyclictest0-21swapper/311:52:013
2624199221,13cyclictest0-21swapper/311:40:173
2622999228,3cyclictest0-21swapper/112:17:491
2622999221,9cyclictest0-21swapper/108:23:491
2622999221,16cyclictest0-21swapper/108:00:031
2622199227,6cyclictest24655-1kworker/u9:009:52:380
2622199226,3cyclictest14935-21diskmemload10:40:580
2622199224,2cyclictest87550irq/128-enp1s0-09:00:190
2622199223,5cyclictest24655-1kworker/u9:012:33:500
2622199222,2cyclictest5152-21kworker/u8:011:25:250
26221992211,3cyclictest827-21runrttasks10:38:330
2622199220,11cyclictest0-21swapper/007:15:010
2622199220,10cyclictest81ktimersoftd/010:50:110
2624199217,3cyclictest25542-21diskstats12:00:143
2624199214,8cyclictest0-21swapper/311:49:463
2624199211,9cyclictest0-21swapper/309:36:123
26241992118,2cyclictest0-21swapper/307:15:043
2624199211,6cyclictest0-21swapper/310:46:253
2624199211,14cyclictest0-21swapper/311:31:143
2624199211,14cyclictest0-21swapper/310:30:293
2624199211,14cyclictest0-21swapper/308:19:453
2624199211,13cyclictest0-21swapper/311:05:093
26241992110,2cyclictest14697-21sshd09:07:083
2623699214,3cyclictest87650irq/129-enp1s0-09:11:072
2623699212,8cyclictest7719-21munin-run07:40:022
2623699212,16cyclictest1634-21head11:05:152
2623699211,7cyclictest0-21swapper/211:59:372
2623699211,6cyclictest9268-21sshd11:21:082
2623699211,6cyclictest23955-21/usr/sbin/munin09:30:142
2623699211,5cyclictest16695-21memory10:25:172
2623699211,5cyclictest0-21swapper/212:01:232
2622999216,3cyclictest0-21swapper/109:46:171
2622999210,8cyclictest0-21swapper/107:43:551
2622999210,8cyclictest0-21swapper/107:43:551
2622199217,2cyclictest27406-21/usr/sbin/munin12:05:130
2622199212,9cyclictest0-21swapper/009:15:260
2622199211,5cyclictest0-21swapper/009:34:550
2622199211,2cyclictest0-21swapper/011:10:130
2622199211,2cyclictest0-21swapper/011:10:130
2622199211,13cyclictest0-21swapper/010:55:140
2622199211,10cyclictest0-21swapper/010:49:010
2622199210,4cyclictest20617-21kworker/u8:109:10:240
2624199209,3cyclictest11384-21/usr/sbin/munin10:15:223
2624199201,6cyclictest7339-21diskstats10:05:133
2624199201,5cyclictest0-21swapper/309:41:173
2624199201,5cyclictest0-21swapper/309:20:363
2624199201,14cyclictest0-21swapper/312:28:413
2624199201,11cyclictest8528-21memory11:20:183
2624199201,10cyclictest761-21ntp_states08:35:223
2623699206,3cyclictest87650irq/129-enp1s0-10:37:072
2623699204,3cyclictest11379-21sshd10:15:082
2623699203,5cyclictest5445-21systemd-journal07:34:112
2623699203,3cyclictest19550-21head08:05:172
2623699201,7cyclictest8235-21/usr/sbin/munin12:35:212
2623699201,7cyclictest31519-21/usr/sbin/munin07:20:232
2623699201,7cyclictest0-21swapper/210:07:432
2623699201,6cyclictest6365-21/usr/sbin/munin08:50:152
2623699201,6cyclictest6365-21/usr/sbin/munin08:50:152
2623699201,6cyclictest4166-21/usr/sbin/munin08:45:152
2623699201,5cyclictest5757-21diskstats11:15:132
2623699201,4cyclictest0-21swapper/210:40:052
2623699200,6cyclictest0-21swapper/207:10:192
26229992011,2cyclictest0-21swapper/108:05:251
2622999200,8cyclictest0-21swapper/107:47:191
2622199209,3cyclictest808-21snmpd09:37:330
2622199208,4cyclictest87550irq/128-enp1s0-11:30:130
2622199207,3cyclictest808-21snmpd07:57:150
2622199206,3cyclictest87350irq/126-enp1s0-07:25:100
2622199202,10cyclictest4166-21/usr/sbin/munin08:45:260
2622199201,6cyclictest0-21swapper/011:20:230
2622199201,3cyclictest22995-21/usr/sbin/munin11:55:220
26221992011,2cyclictest87550irq/128-enp1s0-07:30:130
2622199201,11cyclictest0-21swapper/010:10:020
2622199201,10cyclictest0-21swapper/010:23:560
2624199196,3cyclictest87350irq/126-enp1s0-07:35:193
2624199195,3cyclictest87650irq/129-enp1s0-12:10:173
2624199191,7cyclictest14935-21diskmemload12:38:093
2624199191,6cyclictest18946-21/usr/sbin/munin08:05:183
2624199191,6cyclictest0-21swapper/310:00:343
2624199191,6cyclictest0-21swapper/310:00:343
26241991915,3cyclictest0-21swapper/309:50:373
26241991913,3cyclictest10762-21/usr/sbin/munin09:00:183
2624199191,10cyclictest0-21swapper/312:15:193
2623699194,4cyclictest29838-21cstates12:10:132
2623699193,4cyclictest1568-21cat12:20:022
2623699193,3cyclictest87650irq/129-enp1s0-08:15:222
2623699192,15cyclictest2364-21runrttasks09:55:022
2623699191,7cyclictest0-21swapper/210:12:102
2623699191,6cyclictest5145-21grep12:25:242
2623699191,6cyclictest0-21swapper/208:20:172
2623699191,5cyclictest0-21swapper/211:45:332
2623699191,4cyclictest0-21swapper/209:19:202
26236991913,3cyclictest10791-21gltestperf07:45:162
2622199198,3cyclictest6365-21/usr/sbin/munin08:50:150
2622199198,3cyclictest6365-21/usr/sbin/munin08:50:150
2622199198,10cyclictest0-21swapper/008:20:010
2622199195,3cyclictest6077-21cat10:00:230
2622199195,3cyclictest6077-21cat10:00:230
2622199195,11cyclictest21764-21head08:10:160
2622199194,7cyclictest0-21swapper/007:20:010
2622199192,2cyclictest87550irq/128-enp1s0-10:25:130
2622199191,5cyclictest0-21swapper/011:54:440
2622199191,12cyclictest0-21swapper/009:55:210
2622199191,11cyclictest0-21swapper/009:25:140
2622199191,10cyclictest1972-21/usr/sbin/munin08:40:240
292512181,2sleep20-21swapper/212:08:362
2624199185,3cyclictest0-21swapper/309:49:473
2624199181,6cyclictest0-21swapper/311:55:473
26241991813,3cyclictest14540-21/usr/sbin/munin07:55:103
2624199181,15cyclictest0-21swapper/312:06:063
2624199181,15cyclictest0-21swapper/310:40:193
2624199180,6cyclictest0-21swapper/311:13:043
2624199180,6cyclictest0-21swapper/311:13:043
2623699183,3cyclictest87650irq/129-enp1s0-09:35:212
2623699181,7cyclictest0-21swapper/209:45:352
2623699181,5cyclictest22429-21hddtemp_smartct09:25:152
2623699181,5cyclictest14942-21munin-run09:10:012
2623699181,5cyclictest0-21swapper/211:37:452
2623699181,5cyclictest0-21swapper/208:35:262
2623699181,4cyclictest2527-21perl08:40:152
2623699181,4cyclictest0-21swapper/210:04:182
2623699181,4cyclictest0-21swapper/210:04:182
2623699181,3cyclictest0-21swapper/211:32:282
26236991812,4cyclictest27861-21grep08:25:122
26236991812,3cyclictest2320-21smartctl12:20:152
26236991812,3cyclictest2320-21smartctl12:20:152
2623699181,10cyclictest0-21swapper/207:50:202
2622199189,2cyclictest87550irq/128-enp1s0-11:43:380
2622199187,3cyclictest29938-21/usr/sbin/munin08:30:210
2622199183,6cyclictest87350irq/126-enp1s0-12:15:120
2622199183,12cyclictest31519-21/usr/sbin/munin07:20:170
2622199182,6cyclictest839-21lldpd07:50:130
2622199181,9cyclictest808-21snmpd11:35:080
2622199181,9cyclictest16735-21/usr/sbin/munin08:00:230
2622199181,5cyclictest11-21rcuc/008:05:270
2622199180,5cyclictest27509-21latency_hist08:25:020
2624199172,12cyclictest3553-21/usr/sbin/munin07:30:113
26241991713,3cyclictest0-21swapper/307:50:193
2624199171,2cyclictest0-21swapper/308:42:063
2624199171,13cyclictest22023-21expr09:25:123
2624199171,13cyclictest2025-21hddtemp_smartct07:25:143
26241991710,4cyclictest32746-21/usr/sbin/munin07:20:223
26241991710,4cyclictest27047-21gltestperf07:10:163
2624199170,15cyclictest0-21swapper/307:41:193
2624199170,15cyclictest0-21swapper/307:41:193
2623699171,6cyclictest0-21swapper/210:47:282
2623699171,4cyclictest13922-21df10:20:132
2623699171,15cyclictest0-21swapper/208:31:492
26236991710,4cyclictest8622-21sh07:40:162
26236991710,4cyclictest8622-21sh07:40:162
26236991710,4cyclictest11031-21gunzip11:25:222
2622199177,3cyclictest27740-21/usr/sbin/munin08:25:180
2622199172,14cyclictest0-21swapper/008:36:440
2622199171,8cyclictest0-21swapper/011:00:220
26221991712,3cyclictest12968-21/usr/sbin/munin09:05:240
2624199169,4cyclictest9597-21ntpq08:55:193
2624199169,4cyclictest25966-21fschecks_count08:20:153
26241991611,3cyclictest17901-21proc_pri08:00:233
26241991610,3cyclictest6714-21diskstats08:50:133
26241991610,3cyclictest6714-21diskstats08:50:133
2623699161,3cyclictest0-21swapper/207:16:322
2623699161,14cyclictest0-21swapper/208:03:252
26236991611,3cyclictest8791-21expr08:55:122
26236991611,3cyclictest827-21runrttasks10:50:072
26236991611,3cyclictest348-21run-parts11:01:012
2623699160,15cyclictest0-21swapper/209:02:532
2622199161,14cyclictest0-21swapper/007:41:500
2622199161,14cyclictest0-21swapper/007:41:500
2622199160,2cyclictest0-21swapper/009:20:100
26241991511,3cyclictest0-21swapper/307:49:093
26241991510,3cyclictest22171-21ntp_kernel_err08:10:223
2623699151,3cyclictest21490-21diskstats08:10:142
2622199152,4cyclictest5770-21grep07:35:130
2624199148,4cyclictest31011-21ntpq08:30:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional