You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 15:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Dec 04, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278382240165,24sleep30-21swapper/307:07:323
277012237162,24sleep10-21swapper/107:05:441
278812236163,24sleep00-21swapper/007:08:040
278832235161,24sleep20-21swapper/207:08:072
2813499108105,2cyclictest0-21swapper/209:36:342
300772620,3sleep30-21swapper/308:25:143
265542560,4sleep081ktimersoftd/009:30:180
28134994633,2cyclictest0-21swapper/209:06:192
2813499451,14cyclictest2746-21kworker/2:107:40:182
28122994211,14cyclictest3026-21munin-run07:25:010
28129994011,26cyclictest20842-21/usr/sbin/munin08:05:261
2812999394,8cyclictest9634-40dnf12:31:141
2812999393,2cyclictest86850irq/129-enp1s0-09:10:021
2812999383,2cyclictest4508-21/usr/sbin/munin09:55:171
2812999373,16cyclictest25866-40dnf09:30:161
28138993620,5cyclictest0-21swapper/310:27:213
2812999362,4cyclictest27438-21/usr/sbin/munin08:20:181
2812999361,10cyclictest1571-21cat08:35:021
2812999351,9cyclictest0-21swapper/112:11:571
2812999351,8cyclictest0-21swapper/109:35:431
2812999351,10cyclictest15115-21sed07:50:191
28134993414,4cyclictest25290-21kworker/2:110:09:532
2812999348,3cyclictest11238-21ntp_states12:35:221
2812999341,10cyclictest22733-21smart_sda09:20:251
28138993222,3cyclictest0-21swapper/311:46:153
2813499321,10cyclictest0-21swapper/212:31:092
28129993213,3cyclictest0-21swapper/107:40:401
2812999321,18cyclictest21047-21awk10:30:231
2812999321,14cyclictest0-21swapper/109:45:261
2812999321,11cyclictest1362-21/usr/sbin/munin12:15:271
2813499318,3cyclictest0-21swapper/208:55:262
2812999312,2cyclictest86850irq/129-enp1s0-11:39:361
2812999311,9cyclictest0-21swapper/111:49:321
2812999311,16cyclictest0-21swapper/111:08:241
2812999311,12cyclictest0-21swapper/110:29:541
2812999310,16cyclictest0-21swapper/107:47:471
135002310,4sleep339-21rcuc/307:45:243
28129993012,4cyclictest0-21swapper/111:55:351
2812999300,16cyclictest0-21swapper/109:52:291
2812999300,15cyclictest780-21rngd08:03:161
2812999300,15cyclictest0-21swapper/110:42:391
2812999300,15cyclictest0-21swapper/108:18:441
2812299304,13cyclictest25012-21munin-run08:15:010
2812299301,13cyclictest0-21swapper/007:35:130
2813899295,3cyclictest86850irq/129-enp1s0-07:40:153
2813499297,3cyclictest0-21swapper/209:43:262
2813499297,3cyclictest0-21swapper/209:43:252
2812999298,3cyclictest86750irq/128-enp1s0-08:42:161
28129992913,2cyclictest0-21swapper/107:26:031
2813499286,3cyclictest2746-21kworker/2:107:46:162
2812999281,14cyclictest0-21swapper/110:05:181
2813899277,4cyclictest780-21rngd12:30:193
2813899275,10cyclictest780-21rngd12:20:093
2813899271,6cyclictest0-21swapper/308:14:233
2813499275,3cyclictest7957-21kworker/2:208:49:532
2813499275,2cyclictest25290-21kworker/2:110:58:152
2812999278,2cyclictest86750irq/128-enp1s0-07:10:571
2812999276,2cyclictest0-21swapper/107:55:201
28129992716,3cyclictest30993-21sort08:25:251
28129992710,3cyclictest16847-21diskmemload11:34:461
2812299272,20cyclictest7-21ksoftirqd/010:05:520
2813499266,3cyclictest0-21swapper/207:55:262
2813499262,11cyclictest7957-21kworker/2:209:00:562
2812999265,5cyclictest780-21rngd11:24:581
2812999261,11cyclictest0-21swapper/107:35:101
2812999261,10cyclictest0-21swapper/109:00:301
2812999260,13cyclictest0-21swapper/108:36:581
2812999260,13cyclictest0-21swapper/108:36:581
2812299261,13cyclictest22506-21munin-run11:50:010
2813899257,4cyclictest780-21rngd07:10:083
2813899256,3cyclictest780-21rngd11:19:153
2813899254,10cyclictest780-21rngd11:41:353
2813499254,2cyclictest86750irq/128-enp1s0-09:55:372
2813499252,4cyclictest25290-21kworker/2:110:24:112
2812999252,3cyclictest1401-21hddtemp_smartct11:00:181
2812999252,12cyclictest0-21swapper/110:24:361
28129992513,2cyclictest0-21swapper/112:00:541
2813899242,4cyclictest28993-21memory09:35:213
28138992419,2cyclictest148752sleep309:05:133
28138992412,3cyclictest4423-21ps07:25:233
28138992410,3cyclictest0-21swapper/309:21:533
2813899240,9cyclictest0-21swapper/308:20:263
2813499248,3cyclictest2746-21kworker/2:107:40:022
2813499245,4cyclictest25290-21kworker/2:111:10:172
2813499245,3cyclictest2746-21kworker/2:107:51:142
2813499245,11cyclictest30-21rcuc/207:14:202
2813499244,3cyclictest86550irq/126-enp1s0-12:10:252
28134992414,2cyclictest25290-21kworker/2:112:07:352
2813499241,11cyclictest0-21swapper/209:45:572
2813499241,10cyclictest0-21swapper/211:46:372
2813499241,10cyclictest0-21swapper/210:50:292
2813499241,10cyclictest0-21swapper/210:30:132
2812999241,8cyclictest780-21rngd12:26:161
2812999241,6cyclictest0-21swapper/112:20:261
2812999241,11cyclictest16847-21diskmemload09:25:101
2812999240,12cyclictest0-21swapper/107:17:551
2812299247,10cyclictest0-21swapper/010:02:060
2812299240,9cyclictest1115-21kworker/0:308:49:050
2813899237,5cyclictest1362-21/usr/sbin/munin12:15:253
2813899236,3cyclictest0-21swapper/310:20:133
28138992313,2cyclictest0-21swapper/307:30:173
28138992313,2cyclictest0-21swapper/307:30:173
2813499239,2cyclictest0-21swapper/209:50:092
2813499238,3cyclictest0-21swapper/211:30:132
2813499231,9cyclictest0-21swapper/210:15:242
2813499231,9cyclictest0-21swapper/208:38:132
2813499231,9cyclictest0-21swapper/208:38:132
28134992316,2cyclictest25290-21kworker/2:111:02:312
2813499231,10cyclictest0-21swapper/208:00:062
2812999231,9cyclictest0-21swapper/111:40:041
2812999231,13cyclictest0-21swapper/111:25:261
2812999231,10cyclictest0-21swapper/110:39:101
2812999231,10cyclictest0-21swapper/108:57:261
2812999230,16cyclictest0-21swapper/109:10:271
2812299236,5cyclictest0-21swapper/011:01:250
28122992316,4cyclictest581-21systemd-journal07:50:010
2812299231,5cyclictest0-21swapper/012:14:450
2812299231,5cyclictest0-21swapper/011:38:130
2812299231,11cyclictest14634-21snmpd11:13:250
2812299230,11cyclictest17496-21perf10:25:020
2813899228,3cyclictest0-21swapper/310:44:263
2813899227,3cyclictest0-21swapper/312:12:263
2813899226,3cyclictest0-21swapper/311:14:263
2813899221,10cyclictest0-21swapper/310:05:543
2813899220,9cyclictest0-21swapper/308:15:173
2813499229,7cyclictest0-21swapper/209:19:002
2813499228,2cyclictest2746-21kworker/2:107:30:162
2813499228,2cyclictest2746-21kworker/2:107:30:162
2813499227,3cyclictest0-21swapper/208:05:182
2813499221,5cyclictest17589-21fschecks_time09:10:172
2813499221,10cyclictest0-21swapper/211:44:052
2813499221,10cyclictest0-21swapper/210:25:252
2812999228,2cyclictest86850irq/129-enp1s0-11:52:581
2812999221,9cyclictest0-21swapper/107:21:321
28129992214,2cyclictest16847-21diskmemload11:15:581
2812999221,15cyclictest0-21swapper/108:10:231
2812999221,10cyclictest0-21swapper/112:09:421
2812999221,10cyclictest0-21swapper/110:55:341
2812999221,10cyclictest0-21swapper/110:53:341
2812999221,10cyclictest0-21swapper/110:47:501
2812999221,10cyclictest0-21swapper/110:47:501
2812999220,16cyclictest0-21swapper/109:15:261
2812299221,5cyclictest0-21swapper/010:19:110
2812299221,4cyclictest0-21swapper/012:17:000
2812299220,6cyclictest0-21swapper/010:54:010
2813899218,3cyclictest0-21swapper/309:00:553
2813899214,3cyclictest0-21swapper/309:18:593
28138992110,3cyclictest0-21swapper/310:15:253
2813899210,15cyclictest0-21swapper/307:22:283
2813499214,8cyclictest30-21rcuc/211:20:152
2813499211,12cyclictest25866-40dnf09:30:102
2813499210,7cyclictest32-21ksoftirqd/211:58:112
2813499210,4cyclictest2746-21kworker/2:108:27:292
2812999218,3cyclictest0-21swapper/111:13:301
2812999211,4cyclictest10779-21kworker/u8:210:12:231
2812999211,12cyclictest0-21swapper/110:16:141
2812999211,11cyclictest0-21swapper/107:33:181
2812999211,11cyclictest0-21swapper/107:33:181
2812299217,7cyclictest0-21swapper/010:55:280
2812299217,2cyclictest27049-21kworker/0:108:22:530
2812299214,3cyclictest86750irq/128-enp1s0-08:30:230
2812299211,4cyclictest0-21swapper/010:47:520
2812299211,4cyclictest0-21swapper/010:47:520
2812299211,11cyclictest28322-21/usr/sbin/munin07:10:160
52592201,3sleep20-21swapper/212:21:252
2813899207,3cyclictest784-21NetworkManager09:10:083
2813899207,3cyclictest0-21swapper/310:45:213
2813899207,3cyclictest0-21swapper/310:45:213
2813899206,3cyclictest0-21swapper/308:05:183
2813899206,2cyclictest86850irq/129-enp1s0-08:35:133
2813899206,2cyclictest86850irq/129-enp1s0-08:35:133
2813899205,3cyclictest0-21swapper/312:05:253
2813899202,2cyclictest86850irq/129-enp1s0-12:36:193
2813899201,7cyclictest22732-21/usr/sbin/munin11:50:163
2813899201,7cyclictest0-21swapper/311:33:003
2813899201,4cyclictest18114-21munin-run11:40:003
28138992011,2cyclictest7645-21/usr/sbin/munin07:35:173
2813899200,11cyclictest0-21swapper/311:00:253
2813499207,3cyclictest2746-21kworker/2:107:19:352
2813499205,3cyclictest86550irq/126-enp1s0-11:25:172
2813499203,3cyclictest86850irq/129-enp1s0-10:00:202
2813499203,3cyclictest2746-21kworker/2:107:24:332
2813499201,7cyclictest28107-21sed12:00:202
2813499201,7cyclictest0-21swapper/211:35:422
2813499201,6cyclictest0-21swapper/211:50:342
2813499201,5cyclictest16847-21diskmemload10:40:232
2813499201,4cyclictest0-21swapper/212:38:392
2813499201,4cyclictest0-21swapper/210:11:302
2813499201,16cyclictest5881-21grep12:25:142
2813499200,7cyclictest0-21swapper/208:41:232
2812999209,3cyclictest30830-21perl09:40:151
2812999209,3cyclictest30830-21perl09:40:151
2812999202,10cyclictest8271-21/usr/sbin/munin08:50:301
2812999201,6cyclictest0-21swapper/110:00:391
2812299209,3cyclictest16847-21diskmemload11:53:060
2812299207,10cyclictest11-21rcuc/009:30:000
2812299204,9cyclictest14634-21snmpd10:41:430
2812299202,9cyclictest1115-21kworker/0:308:44:550
2812299202,2cyclictest86550irq/126-enp1s0-11:15:240
2812299201,9cyclictest0-21swapper/010:25:150
28122992015,3cyclictest12170-21cat12:40:010
2812299201,11cyclictest27132-21/usr/sbin/munin12:00:160
2812299201,11cyclictest0-21swapper/012:22:300
2812299201,11cyclictest0-21swapper/011:23:290
266372201,2sleep00-21swapper/011:56:440
217302200,7sleep230-21rcuc/209:20:132
2813899199,3cyclictest16437-21/usr/sbin/munin07:55:153
2813899197,3cyclictest0-21swapper/312:01:533
2813899193,13cyclictest30909-21/usr/sbin/munin10:55:283
2813899191,5cyclictest3382-21ntp_states09:50:223
2813899191,5cyclictest0-21swapper/310:39:203
2813899191,5cyclictest0-21swapper/309:56:453
28138991915,3cyclictest0-21swapper/311:21:053
28138991913,3cyclictest8278-21unixbench_singl10:00:243
28138991913,3cyclictest11536-21ntp_states08:55:243
2813899191,2cyclictest86750irq/128-enp1s0-12:28:113
2813499196,3cyclictest0-21swapper/210:35:012
2813499193,4cyclictest25290-21kworker/2:110:45:032
2813499193,4cyclictest25290-21kworker/2:110:45:032
2813499192,3cyclictest23670-21/usr/sbin/munin09:25:172
2813499191,6cyclictest7950-21gltestperf11:15:182
2813499191,6cyclictest0-21swapper/212:19:202
2813499191,6cyclictest0-21swapper/211:09:302
28134991912,4cyclictest27778-21df_inode08:20:152
2812299198,3cyclictest2952-21/usr/sbin/munin11:05:190
2812299198,3cyclictest13942-21/usr/sbin/munin11:30:140
2812299197,4cyclictest0-21swapper/011:40:020
2812299196,3cyclictest0-21swapper/009:50:100
2812299192,8cyclictest11447-21df10:10:140
2812299192,2cyclictest86550irq/126-enp1s0-09:40:430
2812299192,2cyclictest86550irq/126-enp1s0-09:40:430
28122991913,3cyclictest5117-21tail09:55:170
2812299191,2cyclictest86550irq/126-enp1s0-10:35:290
2812299191,2cyclictest86550irq/126-enp1s0-09:16:490
2812299191,15cyclictest0-21swapper/009:10:160
2812299191,12cyclictest86550irq/126-enp1s0-12:30:290
2812299191,11cyclictest21471-21/usr/sbin/munin09:20:240
2812299191,10cyclictest14634-21snmpd09:48:080
2812299191,10cyclictest0-21swapper/012:05:060
2812299191,10cyclictest0-21swapper/011:27:580
2812299191,10cyclictest0-21swapper/010:34:320
2812299191,10cyclictest0-21swapper/009:36:170
2813899183,3cyclictest0-21swapper/309:25:053
2813899181,7cyclictest0-21swapper/309:33:513
2813899181,5cyclictest0-21swapper/309:46:343
2813899181,4cyclictest26692-21sshd11:57:253
28138991812,3cyclictest19658-21nfs4_client08:00:183
28138991812,3cyclictest12272-21proc_pri10:10:233
2813899181,15cyclictest0-21swapper/311:25:123
2813899181,15cyclictest0-21swapper/308:42:233
2813899181,14cyclictest848-21sssd_nss08:46:253
2813899181,14cyclictest19907-21/usr/sbin/munin10:30:153
28138991811,4cyclictest14844-21grep07:50:163
2813499182,13cyclictest2746-21kworker/2:108:30:152
28134991811,4cyclictest3779-21grep07:25:162
2813499180,5cyclictest2746-21kworker/2:108:10:152
2812999181,9cyclictest0-21swapper/108:45:181
2812299187,4cyclictest11-21rcuc/009:08:180
2812299184,7cyclictest9602-21kworker/0:207:42:550
2812299184,5cyclictest780-21rngd08:35:430
2812299184,5cyclictest780-21rngd08:35:430
2812299182,14cyclictest9415-21grep08:50:260
2812299181,8cyclictest20616-21grep08:05:010
2812299181,3cyclictest16218-21perf07:55:010
2812299180,8cyclictest7343-21sed12:25:290
2813899176,3cyclictest0-21swapper/310:50:293
28138991713,2cyclictest30611-21df09:40:123
28138991713,2cyclictest30611-21df09:40:123
28138991712,3cyclictest0-21swapper/307:15:443
2813499172,2cyclictest86750irq/128-enp1s0-08:51:572
2812299176,3cyclictest86550irq/126-enp1s0-08:55:280
2812299172,3cyclictest0-21swapper/009:03:500
28122991713,3cyclictest0-21swapper/008:08:350
28122991711,3cyclictest5218-21latency_hist07:30:010
28122991711,3cyclictest5218-21latency_hist07:30:010
28138991611,3cyclictest3976-21nfs4_client11:05:203
28138991610,3cyclictest8453-21cstates08:50:133
2813899160,3cyclictest0-21swapper/308:30:053
28134991611,3cyclictest25914-21sh08:15:162
2812299164,4cyclictest25239-21/usr/sbin/munin08:15:260
2812299161,7cyclictest0-21swapper/008:29:300
28122991612,3cyclictest0-21swapper/007:56:370
28122991611,3cyclictest3834-21gltestperf07:25:170
2812299160,5cyclictest0-21swapper/007:15:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional