You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 07:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Mar 28, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287452240163,65sleep20-21swapper/219:05:112
303632230154,25sleep10-21swapper/119:05:391
304972201168,22sleep00-21swapper/019:07:190
304002201162,26sleep30-21swapper/319:06:093
374321550,3sleep00-21swapper/020:30:300
2567321470,4sleep23080499cyclictest20:10:012
30810995854,3cyclictest0-21swapper/323:29:543
77762510,1sleep00-21swapper/000:20:250
56632470,2sleep00-21swapper/023:05:130
74282430,1sleep00-21swapper/019:25:240
253902410,1sleep00-21swapper/021:20:220
217112390,1sleep20-21swapper/220:00:142
3079399381,14cyclictest0-21swapper/000:34:060
30798993621,3cyclictest9794-21ntp_states20:45:231
3079399350,23cyclictest0-21swapper/019:42:410
30798993415,4cyclictest0-21swapper/120:00:141
3079899341,22cyclictest0-21swapper/122:45:191
3079399341,19cyclictest0-21swapper/021:35:140
3079399331,23cyclictest0-21swapper/000:28:360
3079399321,13cyclictest0-21swapper/000:03:460
3079399320,25cyclictest0-21swapper/020:58:360
3079899311,12cyclictest985-21ntp_states20:25:231
3079399311,13cyclictest0-21swapper/020:01:390
3079399311,12cyclictest15214-21latency23:25:160
3079399311,12cyclictest0-21swapper/022:05:540
3079399311,11cyclictest0-21swapper/019:13:110
3079399310,17cyclictest0-21swapper/000:14:000
3079399310,12cyclictest0-21swapper/023:21:320
3079899301,11cyclictest28611-21ntpq23:55:201
3079399301,27cyclictest0-21swapper/020:25:260
3079399301,14cyclictest0-21swapper/021:54:300
3079399301,13cyclictest0-21swapper/022:51:190
3079399301,13cyclictest0-21swapper/022:33:390
3079399301,13cyclictest0-21swapper/022:33:390
3079399301,12cyclictest0-21swapper/022:41:270
3079399301,12cyclictest0-21swapper/021:05:440
3079399301,12cyclictest0-21swapper/000:18:220
3079899298,2cyclictest0-21swapper/120:56:231
3079399291,23cyclictest0-21swapper/022:15:250
30793992912,15cyclictest11335-21awk22:05:000
3079399291,13cyclictest0-21swapper/021:28:580
3079399291,13cyclictest0-21swapper/021:04:200
3079399291,13cyclictest0-21swapper/020:44:460
3079399291,12cyclictest0-21swapper/023:41:100
3079399291,12cyclictest0-21swapper/023:41:100
3079399291,12cyclictest0-21swapper/019:37:050
3079399290,23cyclictest0-21swapper/021:19:280
3079899286,13cyclictest0-21swapper/123:09:391
3079899282,3cyclictest0-21swapper/120:53:231
3079899281,14cyclictest0-21swapper/123:30:511
3079899281,13cyclictest0-21swapper/100:04:501
3079399288,16cyclictest1586-21munin-run00:10:010
3079399281,13cyclictest0-21swapper/023:02:350
3079399281,13cyclictest0-21swapper/022:13:590
3079399281,13cyclictest0-21swapper/020:36:560
3079399281,11cyclictest0-21swapper/022:57:390
3079399281,11cyclictest0-21swapper/022:20:330
3079399281,11cyclictest0-21swapper/021:46:100
3079399281,10cyclictest0-21swapper/021:43:560
3079399281,10cyclictest0-21swapper/020:15:200
3079399280,17cyclictest0-21swapper/020:12:050
3079399280,12cyclictest0-21swapper/021:12:560
3079399280,12cyclictest0-21swapper/019:59:370
185602281,2sleep118585-21ntp_states21:05:201
44382270,3sleep23080499cyclictest23:00:212
3081099272,8cyclictest741-21munin-node19:15:013
3079899279,5cyclictest31672-21df_inode22:50:151
3079899276,3cyclictest0-21swapper/100:05:221
3079899271,8cyclictest19494-21diskmemload22:43:271
30798992716,3cyclictest0-21swapper/122:17:501
3079399271,11cyclictest0-21swapper/023:12:280
3079399271,11cyclictest0-21swapper/022:45:230
3079399270,9cyclictest0-21swapper/023:49:080
3079399270,10cyclictest0-21swapper/000:36:330
271412270,3sleep127145-21kernelversion21:25:171
3081099261,4cyclictest0-21swapper/322:02:143
30798992618,5cyclictest1139-21munin-node19:15:041
30798992612,3cyclictest0-21swapper/100:35:461
3079899260,11cyclictest0-21swapper/122:36:321
3079399261,9cyclictest0-21swapper/019:54:290
3079399261,18cyclictest0-21swapper/023:30:260
3079399261,10cyclictest0-21swapper/021:31:380
3079399261,10cyclictest0-21swapper/020:53:380
3079399261,10cyclictest0-21swapper/020:45:140
3079399261,10cyclictest0-21swapper/020:21:480
30798992520,2cyclictest19909-21irqstats19:55:191
3079899251,14cyclictest1978-21cpuspeed_turbos00:10:101
3079899251,11cyclictest0-21swapper/122:28:221
3079899250,11cyclictest0-21swapper/121:55:141
3079399251,8cyclictest0-21swapper/019:19:260
3079399251,10cyclictest0-21swapper/023:35:380
3079399251,10cyclictest0-21swapper/022:37:230
3079399251,10cyclictest0-21swapper/020:05:230
3079399251,10cyclictest0-21swapper/020:05:230
3079399251,10cyclictest0-21swapper/019:32:320
3079899247,4cyclictest22911-21grep22:30:141
3079899247,4cyclictest22911-21grep22:30:141
3079899246,3cyclictest26179-21sed23:50:181
3079899246,12cyclictest86650irq/127-enp1s0-21:02:551
3079899243,8cyclictest0-21swapper/119:45:441
3079899241,9cyclictest0-21swapper/119:11:051
3079399241,20cyclictest0-21swapper/019:45:130
3079399241,15cyclictest0-21swapper/023:17:540
3079399241,10cyclictest0-21swapper/023:59:060
3079399241,10cyclictest0-21swapper/022:28:470
3079399241,10cyclictest0-21swapper/019:22:220
3081099234,7cyclictest12050irq/123-ahci[0023:05:013
3081099234,2cyclictest12050irq/123-ahci[0020:30:163
3081099231,6cyclictest0-21swapper/300:39:043
3079899239,12cyclictest5911-21kworker/u8:120:05:221
3079899239,12cyclictest5911-21kworker/u8:120:05:221
3079899235,3cyclictest0-21swapper/123:38:351
3079899234,4cyclictest18877-21munin-run19:55:001
3079899233,9cyclictest10700-21timerandwakeup22:00:241
3079899233,7cyclictest0-21swapper/123:43:551
3079899233,7cyclictest0-21swapper/123:43:551
3079899232,18cyclictest11033-21ntp_states23:15:221
30798992315,3cyclictest0-21swapper/121:11:071
3079899231,16cyclictest0-21swapper/123:47:421
3079899231,11cyclictest9198-21ntp_states19:30:221
3079399231,15cyclictest0-21swapper/023:54:100
3079399231,14cyclictest0-21swapper/021:56:580
28162230,3sleep12820-21cpuspeed_turbos21:45:111
12972230,7sleep230-21rcuc/221:40:162
3081099225,5cyclictest12050irq/123-ahci[0020:55:013
3081099225,2cyclictest86550irq/126-enp1s0-00:14:273
3081099225,10cyclictest19349-21chrt21:08:173
3081099222,15cyclictest0-21swapper/300:32:583
3081099221,6cyclictest41-21ksoftirqd/300:24:073
3081099221,11cyclictest0-21swapper/323:06:363
3081099220,11cyclictest0-21swapper/321:26:203
3080499221,6cyclictest0-21swapper/222:34:022
3080499221,6cyclictest0-21swapper/222:34:022
30804992215,4cyclictest29110-21grep21:30:172
3079899228,4cyclictest0-21swapper/123:14:231
3079899228,4cyclictest0-21swapper/121:18:311
3079899227,2cyclictest6727-21interrupts19:25:161
3079899222,12cyclictest0-21swapper/100:25:281
3079899221,3cyclictest20119-21perf22:25:001
3079899221,12cyclictest30506-21/usr/sbin/munin20:20:151
3079899220,8cyclictest0-21swapper/123:24:271
3079899220,17cyclictest0-21swapper/119:44:271
3079899220,15cyclictest0-21swapper/121:30:171
3081099216,4cyclictest0-21swapper/323:38:443
3081099215,7cyclictest0-21swapper/321:19:463
3081099212,8cyclictest31174-21sed00:00:263
3081099212,3cyclictest29123-21/usr/sbin/munin22:45:203
3081099211,11cyclictest445-21/usr/sbin/munin21:40:103
3081099211,10cyclictest0-21swapper/321:34:553
3081099210,8cyclictest0-21swapper/322:58:253
3081099210,10cyclictest0-21swapper/321:38:503
3080499219,4cyclictest2361-21processes22:55:242
3080499215,7cyclictest771-21rngd20:48:172
3080499211,9cyclictest0-21swapper/221:25:192
3080499211,5cyclictest23493-21df_inode23:45:142
3080499211,11cyclictest9076-21switchtime23:10:252
3079899215,9cyclictest1329-21cstates22:55:151
3079899214,2cyclictest0-21swapper/122:05:241
3079899213,6cyclictest0-21swapper/123:26:511
3079899213,3cyclictest0-21swapper/121:53:561
3079899213,3cyclictest0-21swapper/120:13:271
3079899212,13cyclictest86750irq/128-enp1s0-21:36:381
3079899211,9cyclictest0-21swapper/120:44:051
30798992117,3cyclictest0-21swapper/119:20:211
3079899211,10cyclictest0-21swapper/123:03:151
3079899211,10cyclictest0-21swapper/121:22:001
3079899210,15cyclictest0-21swapper/119:35:061
3079899210,12cyclictest0-21swapper/120:15:391
3081099209,3cyclictest86550irq/126-enp1s0-00:25:143
3081099208,4cyclictest39-21rcuc/319:44:023
3081099207,7cyclictest86550irq/126-enp1s0-22:42:333
3081099207,4cyclictest0-21swapper/322:21:143
3081099206,6cyclictest0-21swapper/300:17:003
3081099206,4cyclictest782-21in:imjournal23:50:163
3081099204,7cyclictest86850irq/129-enp1s0-22:15:253
3081099203,9cyclictest86550irq/126-enp1s0-22:25:123
3081099202,7cyclictest86850irq/129-enp1s0-19:25:133
3081099201,6cyclictest0-21swapper/322:05:033
30810992016,3cyclictest41-21ksoftirqd/321:00:003
30810992013,4cyclictest25325-21grep22:35:143
3081099201,11cyclictest2747-21/usr/sbin/munin21:45:153
3081099201,11cyclictest0-21swapper/321:54:323
3081099201,10cyclictest0-21swapper/323:30:023
30810992010,2cyclictest86850irq/129-enp1s0-21:55:163
3081099200,5cyclictest41-21ksoftirqd/323:20:353
3081099200,10cyclictest0-21swapper/322:52:223
3080499205,4cyclictest1549-40dnf00:09:512
30804992013,4cyclictest10375-21dump-pmu-power00:30:002
3079899204,7cyclictest14905-21proc_pri22:10:221
3079899204,5cyclictest0-21swapper/100:15:511
3079899201,3cyclictest0-21swapper/120:36:231
30798992012,3cyclictest0-21swapper/100:22:401
3079899200,7cyclictest0-21swapper/120:30:031
3081099198,3cyclictest32292-21/usr/sbin/munin20:25:273
3081099196,8cyclictest0-21swapper/321:10:483
3081099196,3cyclictest86550irq/126-enp1s0-20:00:143
3081099194,4cyclictest86550irq/126-enp1s0-23:15:253
3081099193,6cyclictest0-21swapper/323:42:253
3081099193,6cyclictest0-21swapper/323:42:253
3081099192,7cyclictest0-21swapper/323:13:493
3081099192,11cyclictest0-21swapper/323:58:453
3081099192,10cyclictest0-21swapper/323:47:343
3081099191,7cyclictest0-21swapper/320:20:223
3081099191,15cyclictest6692sleep300:05:243
3081099191,10cyclictest812-21snmpd19:59:133
3081099191,10cyclictest0-21swapper/322:11:103
3081099190,8cyclictest0-21swapper/321:22:433
3080499191,9cyclictest14141-21munin-run23:25:012
3080499191,6cyclictest0-21swapper/200:19:092
3080499191,5cyclictest18414-21unixbench_multi19:50:252
3080499191,2cyclictest86750irq/128-enp1s0-21:23:142
30798991915,3cyclictest0-21swapper/100:31:311
3079899191,16cyclictest0-21swapper/121:41:331
185902190,4sleep230-21rcuc/221:05:202
32942180,2sleep30-21swapper/319:17:383
3081099188,2cyclictest5696-21awk19:25:023
3081099188,2cyclictest0-21swapper/321:02:153
3081099186,3cyclictest12455-21cat19:40:023
3081099184,6cyclictest12050irq/123-ahci[0020:40:013
3081099182,12cyclictest14703-21/usr/sbin/munin19:45:223
3081099181,7cyclictest17823-21cat19:50:183
30810991815,2cyclictest0-21swapper/322:33:033
30810991815,2cyclictest0-21swapper/322:33:033
3081099181,15cyclictest0-21swapper/320:46:213
3081099181,15cyclictest0-21swapper/320:41:463
3080499189,3cyclictest0-21swapper/223:25:122
3080499189,2cyclictest4736-21awk21:50:022
3080499186,4cyclictest28324-21sshd23:55:172
3080499185,4cyclictest3337-21unixbench_multi00:10:252
3080499184,5cyclictest0-21swapper/221:50:252
3080499182,7cyclictest16906-21df_inode23:30:152
3080499181,6cyclictest0-21swapper/222:08:132
3080499181,5cyclictest0-21swapper/200:02:112
3080499181,3cyclictest0-21swapper/222:13:182
30804991813,3cyclictest31518-21grep19:10:142
3080499181,2cyclictest0-21swapper/221:37:482
30804991812,3cyclictest8179-21munin-plugin-st00:25:012
30804991812,3cyclictest18140-21/usr/sbin/munin22:20:172
3080499181,10cyclictest0-21swapper/223:44:312
3080499181,10cyclictest0-21swapper/223:44:312
3080499181,10cyclictest0-21swapper/221:59:432
3080499180,11cyclictest0-21swapper/219:30:062
53392171,2sleep20-21swapper/219:20:392
3081099171,7cyclictest25715-21grep20:10:113
30810991712,3cyclictest9161-21ntp_kernel_pll_19:30:223
3081099171,15cyclictest0-21swapper/320:15:533
3081099171,10cyclictest0-21swapper/320:05:233
3081099171,10cyclictest0-21swapper/320:05:233
3080499174,5cyclictest771-21rngd20:33:462
3080499174,5cyclictest13182-21hddtemp_smartct19:40:162
3080499174,3cyclictest86550irq/126-enp1s0-22:45:212
3080499172,5cyclictest25771-1kworker/u9:020:20:202
3080499172,5cyclictest13872-21ntp_states00:35:212
3080499171,7cyclictest32212-21latency22:50:192
3080499171,6cyclictest6285-21cat20:39:572
3080499171,6cyclictest0-21swapper/221:15:182
3080499171,5cyclictest0-21swapper/222:43:162
3080499171,4cyclictest0-21swapper/222:36:482
30804991712,3cyclictest7667-21pmu-power20:40:212
30804991712,3cyclictest11026-21grep20:50:122
30804991712,3cyclictest10753-21cut00:30:122
3080499171,15cyclictest0-21swapper/221:04:272
3080499171,15cyclictest0-21swapper/220:29:322
3080499171,13cyclictest9997-21sed22:00:152
3080499171,10cyclictest0-21swapper/223:19:102
253342171,2sleep20-21swapper/223:50:032
220162171,2sleep20-21swapper/222:26:272
207022170,3sleep20-21swapper/223:39:522
103012170,7sleep230-21rcuc/219:35:022
3080499165,3cyclictest86550irq/126-enp1s0-20:10:162
30804991612,3cyclictest0-21swapper/220:55:172
30804991611,3cyclictest6699-21cut23:05:222
30804991611,3cyclictest6622-21idleruntime19:25:152
30804991611,3cyclictest20164-21sh19:55:222
3080499161,10cyclictest0-21swapper/222:18:442
30804991610,3cyclictest19982-21cat21:10:152
3080499151,5cyclictest27893-21/usr/sbin/munin20:15:242
3080499151,13cyclictest0-21swapper/219:46:152
30804991510,3cyclictest25012-21unixbench_multi20:05:252
3080499149,3cyclictest1393-21processes19:15:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional