You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 01:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 30, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323422238162,24sleep10-21swapper/107:06:061
325992203166,25sleep30-21swapper/307:09:263
324032201165,24sleep00-21swapper/007:06:530
322982200164,24sleep20-21swapper/207:05:302
593821790,4sleep03273399cyclictest09:45:250
2851521480,5sleep13273899cyclictest08:10:191
2591821330,4sleep33274999cyclictest08:05:153
3274999107104,2cyclictest0-21swapper/308:45:353
327499910092,3cyclictest0-21swapper/310:53:033
32749999793,3cyclictest0-21swapper/308:53:143
32749999591,3cyclictest0-21swapper/309:28:203
216002590,6sleep011-21rcuc/010:20:250
58672550,4sleep081ktimersoftd/007:20:130
236642520,1sleep20-21swapper/211:40:162
3273899421,12cyclictest13508-21missed_timers12:30:201
3273899394,6cyclictest26320-40dnf10:33:241
3273899391,3cyclictest15871-21cstates10:10:131
32749993734,2cyclictest0-21swapper/307:57:323
3273899362,3cyclictest31818-21munin-run08:20:011
32738993612,3cyclictest29637-21tr11:55:141
3273899353,7cyclictest21440-21diskmemload11:45:271
3273899353,13cyclictest0-21swapper/111:15:241
3273899351,13cyclictest28669-21awk10:40:011
3273899351,11cyclictest5046-21/usr/sbin/munin07:15:261
3273399352,18cyclictest18844-21missed_timers10:15:210
3273399352,18cyclictest18844-21missed_timers10:15:200
3273899345,6cyclictest0-21swapper/107:30:151
3273899342,13cyclictest22690-21munin-run11:40:011
3273899341,5cyclictest2580-21munin-node07:15:011
32738993411,4cyclictest8635-21/usr/sbin/munin11:05:231
32738993410,3cyclictest2215-21df08:25:111
3273399344,21cyclictest27017-21missed_timers09:20:190
3273899339,4cyclictest14604-21/usr/sbin/munin10:05:221
3273899339,4cyclictest0-21swapper/112:10:271
3273899339,3cyclictest6880-21processes07:20:241
3273899336,7cyclictest2705-21pmu-power12:05:241
3273899331,11cyclictest8763-21head12:20:151
3273899331,11cyclictest5105-21sleep11:00:021
3273899331,10cyclictest20811-21turbostat.cron07:55:011
3273899331,10cyclictest20811-21turbostat.cron07:55:011
32738993310,3cyclictest15518-21ntp_states07:40:211
197772330,2sleep33274999cyclictest11:30:223
3273899329,3cyclictest10441-21latency_hist08:45:011
3273899323,7cyclictest5838-40dnf08:32:231
3273899322,13cyclictest3035-21fschecks_time09:40:141
3273899321,6cyclictest1092-21irqstats09:35:171
3273899321,3cyclictest14197-21cpuspeed_turbos11:20:111
32738993212,2cyclictest16709-21sh12:40:011
32738993212,2cyclictest16709-21sh12:40:011
32738993210,3cyclictest15123-21cpu08:55:131
3273399325,12cyclictest24893-21awk11:45:010
3273899319,3cyclictest27417-21latency_hist08:10:011
3273899318,2cyclictest0-21swapper/107:55:131
3273899316,3cyclictest11636-21diskstats10:00:121
3273899316,3cyclictest0-21swapper/107:48:081
3273899311,2cyclictest11923-21uptime12:25:271
32738993112,3cyclictest23864-21/usr/sbin/munin09:15:231
32738993111,3cyclictest12333-21cat07:35:121
3273899311,12cyclictest23432-21tune2fs11:40:151
3273899311,12cyclictest17254-21taskset09:00:041
3273899311,12cyclictest10846-21sshd09:57:211
32738993110,3cyclictest30125-21expr10:40:221
32738993110,3cyclictest1581-21sed10:50:171
32738993012,3cyclictest8205-21diskstats07:25:121
3273899301,11cyclictest0-21swapper/108:35:121
3273899301,10cyclictest0-21swapper/109:25:161
32738993010,3cyclictest27554-21ls11:50:161
3273899300,5cyclictest0-21swapper/109:55:011
3273399305,13cyclictest23352-21ntp_states10:25:230
3273399304,5cyclictest26734-21wc08:05:240
3273399303,18cyclictest86550irq/126-enp1s0-12:35:160
3273399303,18cyclictest86550irq/126-enp1s0-12:35:160
3273899291,3cyclictest584-21cat10:49:571
3273899291,2cyclictest21440-21diskmemload11:00:211
3273899291,16cyclictest0-21swapper/112:03:531
3273899291,16cyclictest0-21swapper/109:49:291
3273899291,10cyclictest0-21swapper/110:25:191
32738992910,3cyclictest10601-21latency11:10:191
3273399297,10cyclictest25000-21pmu-power09:15:220
3273399293,15cyclictest29843-21/usr/sbin/munin08:15:200
3273399292,4cyclictest0-21swapper/011:28:520
3273399292,4cyclictest0-21swapper/011:28:520
3274999280,18cyclictest29874-21strings10:40:183
3273899281,6cyclictest12872-21/usr/sbin/munin08:50:251
3273899281,6cyclictest0-21swapper/110:22:591
3273899281,3cyclictest30808-21diskstats09:30:131
3273899281,16cyclictest0-21swapper/109:06:361
32738992810,3cyclictest20102-21users11:30:241
3273899280,4cyclictest27089-21netstat09:20:201
3273399288,12cyclictest11699-21munin-run11:15:010
3273399288,10cyclictest11922-21/usr/sbin/munin11:15:240
3273399286,14cyclictest32041-21/usr/sbin/munin08:20:170
3273399285,13cyclictest9836-21munin-run07:30:000
32749992714,5cyclictest13887-21latency_hist11:20:013
3274999271,10cyclictest11690-21latency_hist11:15:013
3273899271,9cyclictest0-21swapper/108:20:221
3273899271,15cyclictest0-21swapper/108:49:011
3273899271,10cyclictest0-21swapper/109:11:011
3273899270,6cyclictest0-21swapper/110:16:231
3273899270,6cyclictest0-21swapper/110:16:231
3273399279,4cyclictest2852-21diskstats09:40:130
3273399278,16cyclictest31080-21/usr/sbin/munin10:45:120
3273399277,6cyclictest20292-21kworker/0:011:34:390
3273399275,7cyclictest771-21rngd11:54:390
3273399275,10cyclictest12395-21cat12:30:020
3273399273,12cyclictest24670-21cstates10:30:130
32733992711,5cyclictest28650-21fschecks_count09:25:150
32733992711,12cyclictest9410-21smart_sda12:20:250
3273399270,23cyclictest21440-21diskmemload11:35:210
3274999267,8cyclictest7631-21latency_hist07:25:023
3274999263,10cyclictest24708-21chrt09:15:183
3274999261,13cyclictest0-21swapper/310:04:463
32738992612,2cyclictest0-21swapper/111:29:431
32738992612,2cyclictest0-21swapper/111:29:431
3273399266,16cyclictest3466-21/usr/sbin/munin07:15:180
3273399266,11cyclictest8478-21/usr/sbin/munin08:40:190
3273399265,10cyclictest1514-21gunzip09:35:220
3273399264,19cyclictest0-21swapper/010:59:100
3273399263,10cyclictest21440-21diskmemload12:09:260
3273399262,16cyclictest86550irq/126-enp1s0-12:13:140
3273399261,3cyclictest8457-21sort11:05:190
3273399261,11cyclictest6470-21sshd11:00:220
3274399251,7cyclictest0-21swapper/212:08:422
3273899250,9cyclictest0-21swapper/112:16:101
3273399258,5cyclictest2539-21latency_hist07:15:010
3273399258,5cyclictest10065-21acpi07:30:110
3273399255,11cyclictest16352-21sshd07:44:150
3273399254,9cyclictest25378-21cstates11:45:150
3273399253,11cyclictest833-21runrttasks09:10:570
3273399252,9cyclictest17264-21/usr/sbin/munin09:00:160
32733992511,3cyclictest15807-21expr10:10:120
3273399250,7cyclictest0-21swapper/009:55:250
3274999247,8cyclictest86550irq/126-enp1s0-09:30:143
3274999247,8cyclictest0-21swapper/309:35:143
3274999246,9cyclictest7722-21kernelversion09:50:193
32749992415,3cyclictest86550irq/126-enp1s0-07:10:173
3274399241,5cyclictest0-21swapper/211:10:162
3273399247,4cyclictest86750irq/128-enp1s0-08:50:150
3273399246,9cyclictest2157-21sendmail_mailqu10:50:230
3273399246,8cyclictest20812-21latency_hist07:55:010
3273399246,8cyclictest20812-21latency_hist07:55:010
3273399245,4cyclictest6282-21/usr/sbin/munin08:35:170
3273399244,17cyclictest30259-21timerandwakeup10:40:240
3273399243,13cyclictest31185-21kworker/u8:212:15:230
3273399242,16cyclictest423-21kworker/u8:111:22:180
3273399242,12cyclictest86750irq/128-enp1s0-10:35:210
3273399242,12cyclictest21059-21bc07:55:130
32733992415,5cyclictest5838-40dnf08:32:240
32733992414,6cyclictest32247-40dnf09:32:480
3274999236,3cyclictest31080-21/usr/sbin/munin10:45:183
3274999232,9cyclictest5435-21latency_hist07:20:023
32749992313,3cyclictest86550irq/126-enp1s0-12:25:263
32749992313,3cyclictest86550irq/126-enp1s0-12:20:123
32749992313,2cyclictest12050irq/123-ahci[0009:15:013
32738992315,7cyclictest0-21swapper/108:00:251
3273399238,3cyclictest31471-21latency_hist12:00:010
3273399233,9cyclictest21440-21diskmemload10:08:100
3273399232,18cyclictest19471-21/usr/sbin/munin09:05:220
32733992311,4cyclictest11966-21/usr/sbin/munin10:00:150
3274999225,9cyclictest0-21swapper/310:35:173
3274999224,15cyclictest851-21sssd_nss09:04:213
3274999223,6cyclictest22563-21kworker/3:210:09:153
3274999222,12cyclictest26550-21cat10:35:013
3274999221,7cyclictest0-21swapper/311:25:143
3274999221,7cyclictest0-21swapper/311:25:143
3274999221,13cyclictest28375-21ntp_states11:50:253
32749992211,3cyclictest14224-21latency_hist07:40:013
3274999221,13cyclictest0-21swapper/310:18:083
3274999221,13cyclictest0-21swapper/310:18:083
3274999221,11cyclictest3852-21idleruntime-cro08:30:013
3274999220,9cyclictest32727-21/usr/sbin/munin12:00:213
3274999220,15cyclictest14451-40dnf12:34:243
3274399221,5cyclictest0-21swapper/210:40:202
3273399226,14cyclictest86550irq/126-enp1s0-09:50:140
3273399221,9cyclictest812-21snmpd08:58:010
32733992215,4cyclictest2796-21memory08:25:180
32733992215,4cyclictest17717-21ntp_states07:45:210
32733992213,6cyclictest12931-21/usr/sbin/munin07:35:160
264312220,4sleep339-21rcuc/311:45:283
3274999216,3cyclictest6294-21users09:45:293
3274999215,8cyclictest86550irq/126-enp1s0-11:20:233
3274999215,8cyclictest6415-21open_inodes11:00:213
3274999214,9cyclictest2683-21munin-node11:35:293
3274999212,16cyclictest7687-21tr11:05:133
3274999211,6cyclictest21650-21head10:20:243
3274999211,6cyclictest0-21swapper/312:10:123
3274999211,6cyclictest0-21swapper/310:55:273
3274999211,6cyclictest0-21swapper/310:26:343
3274999211,6cyclictest0-21swapper/309:58:283
3274999211,6cyclictest0-21swapper/309:23:073
3274999211,5cyclictest3118550kworker/u8:212:07:223
3274999211,3cyclictest8026-21chrt08:37:173
3274999211,15cyclictest0-21swapper/307:31:493
3274999211,13cyclictest0-21swapper/307:52:403
3274999211,13cyclictest0-21swapper/307:52:403
32749992111,2cyclictest22563-21kworker/3:210:14:163
3274399215,8cyclictest86850irq/129-enp1s0-10:35:272
3274399211,8cyclictest0-21swapper/209:47:082
3274399211,8cyclictest0-21swapper/209:37:532
3274399211,6cyclictest0-21swapper/212:03:282
3274399211,6cyclictest0-21swapper/211:46:402
3274399211,6cyclictest0-21swapper/210:51:122
32743992112,3cyclictest86850irq/129-enp1s0-09:20:272
32743992110,3cyclictest9732-21sendmail_mailtr08:40:232
32733992113,5cyclictest24386-21pmu-power08:00:210
3273399211,13cyclictest0-21swapper/012:04:440
3274999206,2cyclictest21447-21perf09:10:013
3274999202,8cyclictest86550irq/126-enp1s0-07:40:213
3274999201,7cyclictest0-21swapper/312:36:123
3274999201,7cyclictest0-21swapper/312:36:123
3274999201,4cyclictest0-21swapper/312:18:373
3274999201,14cyclictest86550irq/126-enp1s0-08:20:213
3274999201,12cyclictest0-21swapper/309:40:133
3274999201,12cyclictest0-21swapper/308:58:223
3274999201,12cyclictest0-21swapper/307:29:173
3274999201,11cyclictest0-21swapper/311:40:183
3274399209,3cyclictest25796-21diskstats08:05:142
3274399201,6cyclictest24054-21irqstats08:00:172
3274399201,5cyclictest0-21swapper/209:17:482
3274399201,5cyclictest0-21swapper/209:13:372
3274399201,4cyclictest15363-21kworker/u8:210:26:312
3274399201,4cyclictest0-21swapper/209:59:262
3274399201,3cyclictest3124-21/usr/sbin/munin10:55:212
32733992013,4cyclictest13457-21memory12:30:190
32733992011,6cyclictest27649-21/usr/sbin/munin08:10:150
32733992011,5cyclictest11020-21diskstats08:45:140
97792191,3sleep20-21swapper/212:20:512
79172190,2sleep20-21swapper/208:35:532
3274999193,8cyclictest86550irq/126-enp1s0-08:15:223
3274999191,12cyclictest0-21swapper/308:32:003
3274999191,11cyclictest836-21lldpd07:47:443
3274999191,11cyclictest0-21swapper/311:55:083
3274999191,10cyclictest0-21swapper/308:12:523
3274399193,6cyclictest0-21swapper/211:30:132
3274399193,3cyclictest21440-21diskmemload11:50:242
3274399191,8cyclictest0-21swapper/211:23:292
3274399191,8cyclictest0-21swapper/209:33:222
3274399191,7cyclictest32073-21needreboot10:45:192
3274399191,5cyclictest0-21swapper/210:18:392
3274399191,5cyclictest0-21swapper/210:18:392
3274399191,4cyclictest0-21swapper/210:10:292
32743991913,3cyclictest1204-21gltestperf07:10:142
3274399191,10cyclictest0-21swapper/207:45:132
3274399190,11cyclictest15555-21kworker/2:111:01:152
125332191,3sleep20-21swapper/212:30:062
3274999182,8cyclictest0-21swapper/308:00:143
3274399184,4cyclictest25155-21gltestperf10:30:152
3274399181,9cyclictest31093-21users11:55:282
3274399181,8cyclictest20090-21/usr/sbin/munin10:20:112
3274399181,7cyclictest16003-21smart_sda12:35:242
3274399181,7cyclictest16003-21smart_sda12:35:242
3274399181,7cyclictest0-21swapper/210:07:332
3274399181,7cyclictest0-21swapper/210:01:072
3274399181,5cyclictest21440-21diskmemload09:40:132
3274399181,5cyclictest0-21swapper/211:27:192
3274399181,5cyclictest0-21swapper/211:27:192
3274399181,3cyclictest8534-21hddtemp_smartct07:25:142
32743991811,4cyclictest8413-21sort11:05:192
3274399181,11cyclictest0-21swapper/208:26:152
3274399181,10cyclictest0-21swapper/211:39:382
3274399181,10cyclictest0-21swapper/207:20:132
296512180,3sleep23274399cyclictest08:15:012
190202180,2sleep20-21swapper/209:02:182
3274999173,3cyclictest12050irq/123-ahci[0008:45:003
3274399178,7cyclictest17041-21turbostat09:00:012
3274399171,8cyclictest23027-21awk08:00:012
3274399171,8cyclictest0-21swapper/209:50:222
3274399171,7cyclictest0-21swapper/212:27:122
3274399171,7cyclictest0-21swapper/212:11:022
3274399171,6cyclictest580-21systemd-journal09:10:012
3274399171,6cyclictest18952-21grep07:50:132
3274399171,6cyclictest18952-21grep07:50:132
3274399171,6cyclictest0-21swapper/211:17:022
3274399171,2cyclictest0-21swapper/212:15:062
3274399171,15cyclictest0-21swapper/208:32:322
3274399171,10cyclictest0-21swapper/208:52:102
3274399169,4cyclictest16420-21latency_hist07:45:022
3274399161,7cyclictest456-21sed08:20:182
3274399161,6cyclictest29851-21acpi08:15:112
3274399161,6cyclictest29525-21smart_sda09:25:252
32743991611,3cyclictest10058-21/usr/sbin/munin07:30:192
3274399151,5cyclictest3165-21ntp_states07:15:032
32743991510,3cyclictest10757-21cat08:45:122
3274399150,10cyclictest0-21swapper/207:35:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional