You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 09:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jun 23, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
236592238163,24sleep20-21swapper/219:07:542
236342237166,58sleep30-21swapper/319:07:353
235722204165,26sleep00-21swapper/019:06:460
237722203166,24sleep10-21swapper/119:09:211
2082721480,5sleep22391899cyclictest23:55:122
1319421230,3sleep32392499cyclictest21:10:143
2391899104101,2cyclictest0-21swapper/223:02:312
169932600,4sleep230-21rcuc/223:45:172
184682560,4sleep011-21rcuc/021:20:230
23918994844,3cyclictest0-21swapper/219:51:242
23913993814,9cyclictest32432-21cat23:05:251
124022370,5sleep02390699cyclictest23:35:150
2392499340,20cyclictest0-21swapper/320:29:253
2392499331,22cyclictest0-21swapper/322:30:113
2392499331,21cyclictest0-21swapper/323:52:033
2390699338,12cyclictest5397-21munin-run19:40:000
2392499321,23cyclictest0-21swapper/323:31:143
2392499321,21cyclictest0-21swapper/322:39:473
2392499320,21cyclictest0-21swapper/321:23:223
2392499320,21cyclictest0-21swapper/320:09:113
2392499320,20cyclictest0-21swapper/321:42:593
23918993229,2cyclictest0-21swapper/220:48:512
2390699316,12cyclictest11994-21munin-run19:55:000
2392499301,20cyclictest0-21swapper/323:40:153
2392499301,13cyclictest0-21swapper/322:08:553
2392499301,13cyclictest0-21swapper/300:10:473
2392499301,12cyclictest0-21swapper/321:28:573
2392499300,20cyclictest0-21swapper/322:14:233
2390699303,12cyclictest8217-21awk21:00:010
2392499291,19cyclictest14097-21gltestperf22:25:133
2392499291,19cyclictest0-21swapper/323:45:313
2392499291,14cyclictest0-21swapper/321:30:193
2392499291,10cyclictest0-21swapper/320:42:493
2392499290,12cyclictest0-21swapper/322:18:033
2392499290,10cyclictest0-21swapper/319:24:493
2391399291,22cyclictest0-21swapper/122:09:281
23924992816,3cyclictest0-21swapper/322:03:273
23924992816,3cyclictest0-21swapper/322:03:273
2392499281,14cyclictest0-21swapper/321:58:153
2392499281,12cyclictest0-21swapper/321:50:473
2392499281,12cyclictest0-21swapper/321:37:273
2392499280,9cyclictest0-21swapper/300:17:133
2392499280,19cyclictest0-21swapper/323:20:103
2391899281,12cyclictest0-21swapper/221:35:172
2392499271,12cyclictest0-21swapper/319:41:053
2392499271,12cyclictest0-21swapper/319:41:053
2392499270,16cyclictest0-21swapper/323:15:363
2390699271,12cyclictest11654-21munin-run23:35:010
23906992710,14cyclictest26084-21munin-run19:15:010
2392499261,16cyclictest0-21swapper/319:19:223
2392499261,10cyclictest0-21swapper/323:39:213
2392499261,10cyclictest0-21swapper/323:04:473
2392499260,11cyclictest0-21swapper/300:31:413
2392499260,10cyclictest0-21swapper/319:12:493
2391399261,18cyclictest0-21swapper/121:59:011
2390699261,6cyclictest24860-21munin-run00:05:000
2390699261,12cyclictest32403-21munin-run21:55:010
2392499251,9cyclictest0-21swapper/319:46:233
2392499251,7cyclictest0-21swapper/320:52:213
2392499251,12cyclictest0-21swapper/322:40:273
2392499251,11cyclictest0-21swapper/322:46:473
2392499251,11cyclictest0-21swapper/319:29:293
2392499250,7cyclictest0-21swapper/319:36:373
2392499250,10cyclictest0-21swapper/319:32:253
23913992514,1cyclictest0-21swapper/123:40:021
2390699256,16cyclictest980-21munin-run19:30:000
2390699251,12cyclictest20444-21munin-run23:55:010
2392499249,3cyclictest16156-21seq20:02:053
2392499247,11cyclictest0-21swapper/320:46:373
2392499241,11cyclictest0-21swapper/300:00:393
2392499240,11cyclictest0-21swapper/323:56:253
2391399242,15cyclictest0-21swapper/122:20:211
2390699241,8cyclictest20660-21kworker/0:120:18:250
2390699241,15cyclictest23435-21kworker/0:021:42:160
2392499231,2cyclictest0-21swapper/322:57:353
2392499231,12cyclictest0-21swapper/321:45:033
23924992310,2cyclictest0-21swapper/323:13:503
2392499230,11cyclictest0-21swapper/320:37:013
2392499230,10cyclictest0-21swapper/320:22:533
2391899231,7cyclictest311ktimersoftd/200:20:012
23913992314,2cyclictest89150irq/134-enp2s0-23:13:131
2391399231,18cyclictest0-21swapper/122:19:161
2391399231,12cyclictest8852-21cut21:00:151
2391399230,17cyclictest0-21swapper/100:24:501
2391399230,17cyclictest0-21swapper/100:24:501
2391399230,16cyclictest87650irq/129-enp1s0-00:14:301
2390699233,5cyclictest32178-40dnf23:05:240
2390699231,7cyclictest22883-21kworker/0:220:20:570
2392499227,3cyclictest32035-21sh00:20:133
2392499227,3cyclictest32035-21sh00:20:133
2392499222,3cyclictest87550irq/128-enp1s0-20:15:213
2392499221,11cyclictest0-21swapper/323:26:183
2392499221,11cyclictest0-21swapper/300:25:033
2392499221,10cyclictest0-21swapper/320:13:293
2392499220,7cyclictest0-21swapper/319:50:173
2392499220,20cyclictest0-21swapper/322:23:433
2391899224,9cyclictest2121-21grep00:25:162
23913992217,3cyclictest27473-21grep19:15:141
2391399221,15cyclictest0-21swapper/122:54:211
2391399221,15cyclictest0-21swapper/122:01:221
2391399221,15cyclictest0-21swapper/122:01:221
2390699225,2cyclictest87350irq/126-enp1s0-00:39:330
2390699224,3cyclictest21323-21runrttasks21:29:100
23906992217,3cyclictest27395-21awk20:30:010
171882220,4sleep339-21rcuc/321:20:013
55852213,4sleep25661-21needreboot22:05:212
2392499213,3cyclictest0-21swapper/319:58:533
2392499211,9cyclictest0-21swapper/323:05:543
2392499210,12cyclictest0-21swapper/320:30:413
2392499210,10cyclictest0-21swapper/321:06:073
2391899216,5cyclictest87350irq/126-enp1s0-00:00:122
2391899216,5cyclictest0-21swapper/200:37:432
2391399219,4cyclictest18997-21cat20:10:121
2391399218,4cyclictest21730-21ntp_states23:55:211
2391399212,10cyclictest23156-21cat20:20:011
2391399211,9cyclictest0-21swapper/123:29:371
23913992117,3cyclictest0-21swapper/119:10:261
2391399211,3cyclictest32328-21cstates20:40:121
2391399211,3cyclictest30870-21ntp_states20:35:201
23913992113,3cyclictest0-21swapper/121:45:291
2391399211,17cyclictest8462-21/usr/sbin/munin22:10:271
2391399211,15cyclictest0-21swapper/122:41:341
2391399211,15cyclictest0-21swapper/121:44:101
2391399211,15cyclictest0-21swapper/121:37:021
2391399211,15cyclictest0-21swapper/121:10:541
2391399211,11cyclictest29200-21diskstats23:00:161
2391399211,11cyclictest18556-21ls23:50:121
2391399211,11cyclictest17787-21cat22:35:021
2391399211,11cyclictest15060-21grep21:15:121
2391399211,11cyclictest13804-21users19:55:261
2391399211,10cyclictest3604-21cat00:30:011
2391399210,19cyclictest0-21swapper/120:50:171
2391399210,19cyclictest0-21swapper/100:05:401
2390699215,14cyclictest0-21swapper/023:05:010
2390699212,11cyclictest21632-21/usr/sbin/munin21:30:190
2390699211,8cyclictest17819-21kworker/0:022:49:410
2390699211,11cyclictest853-21/usr/sbin/munin23:10:240
2390699211,11cyclictest0-21swapper/022:35:030
2392499207,3cyclictest0-21swapper/320:57:353
2392499206,9cyclictest0-21swapper/300:07:213
2392499201,10cyclictest5926-21/usr/sbin/munin00:35:133
2392499201,10cyclictest0-21swapper/321:03:153
2391899201,7cyclictest32-21ksoftirqd/200:25:012
2391899201,7cyclictest32-21ksoftirqd/200:25:012
2391899201,16cyclictest14111-21/usr/sbin/munin21:10:262
2391899200,6cyclictest32-21ksoftirqd/221:24:102
2391399206,5cyclictest0-21swapper/123:47:391
2391399205,4cyclictest0-21swapper/123:34:041
2391399204,13cyclictest28246-21grep20:30:151
2391399201,9cyclictest0-21swapper/121:25:431
2391399201,2cyclictest827-21runrttasks19:20:131
2391399201,14cyclictest0-21swapper/121:24:561
2391399201,11cyclictest25809-21grep20:25:151
2391399201,11cyclictest19920-21cat22:40:021
2391399201,11cyclictest0-21swapper/122:45:221
2391399201,10cyclictest5390-21latency_hist19:40:011
2391399201,10cyclictest15569-21unixbench_multi23:40:261
2391399201,10cyclictest15438-21munin-run22:30:021
23913992010,3cyclictest7789-21cat19:45:021
2391399200,18cyclictest0-21swapper/119:30:131
2390699207,4cyclictest0-21swapper/000:34:520
2390699206,5cyclictest8462-21cpu21:00:120
2390699206,5cyclictest0-21swapper/000:21:430
2390699206,5cyclictest0-21swapper/000:21:430
2390699201,8cyclictest8540-21kworker/0:022:13:130
2390699201,8cyclictest12525-21kworker/0:221:10:390
2390699201,4cyclictest0-21swapper/022:18:450
2390699201,11cyclictest20667-21/usr/sbin/munin23:55:200
2390699201,11cyclictest0-21swapper/022:07:050
2390699201,10cyclictest0-21swapper/023:15:210
2390699200,8cyclictest2567-21kworker/0:123:26:310
132922200,6sleep2311ktimersoftd/223:35:252
2391899199,3cyclictest87350irq/126-enp1s0-21:45:102
2391899196,4cyclictest32-21ksoftirqd/222:50:112
2391899196,4cyclictest19895-21fschecks_count21:25:142
2391899192,4cyclictest26830-21gltestperf19:15:052
2391899191,7cyclictest0-21swapper/223:14:142
2391899191,4cyclictest0-21swapper/223:53:552
23918991914,3cyclictest22251-21/usr/sbin/munin22:45:112
2391899191,15cyclictest13720-21taskset19:55:252
2391399197,5cyclictest0-21swapper/123:20:451
23913991916,2cyclictest0-21swapper/123:15:221
23913991915,3cyclictest0-21swapper/120:59:151
23913991915,3cyclictest0-21swapper/120:21:101
23913991915,3cyclictest0-21swapper/119:45:151
23913991914,3cyclictest24377-21grep00:00:271
2391399191,17cyclictest0-21swapper/100:37:271
2391399191,11cyclictest0-21swapper/100:15:151
2391399191,10cyclictest6701-21open_files19:40:211
2391399191,10cyclictest26997-21df_inode22:55:141
2391399191,10cyclictest21677-21cpu21:30:121
2391399191,10cyclictest15480-21ntp_states20:00:221
2391399191,10cyclictest0-21swapper/120:45:131
2390699195,6cyclictest0-21swapper/022:22:210
2390699195,5cyclictest0-21swapper/000:05:080
2390699194,6cyclictest0-21swapper/021:36:280
2390699194,3cyclictest87350irq/126-enp1s0-22:30:090
2390699192,2cyclictest87350irq/126-enp1s0-00:10:290
2390699191,5cyclictest0-21swapper/022:28:360
2390699191,5cyclictest0-21swapper/021:16:320
2390699191,15cyclictest3054-21gltestperf22:00:160
2390699191,15cyclictest3054-21gltestperf22:00:160
2390699191,11cyclictest0-21swapper/022:40:490
2390699191,10cyclictest808-21snmpd19:34:450
2390699191,10cyclictest12623-21diskmemload21:55:160
2392499181,12cyclictest0-21swapper/322:50:513
2391899183,7cyclictest27069-21latency_hist00:10:012
2391899183,2cyclictest311ktimersoftd/223:20:162
2391899182,9cyclictest0-21swapper/200:30:132
2391899182,7cyclictest8435-21/usr/sbin/munin21:00:182
2391899181,7cyclictest26513-21fschecks_time21:40:142
2391899181,7cyclictest0-21swapper/222:30:192
2391899181,7cyclictest0-21swapper/220:20:152
2391899181,6cyclictest0-21swapper/223:40:182
2391899181,6cyclictest0-21swapper/222:42:392
2391899181,6cyclictest0-21swapper/222:28:052
2391899181,6cyclictest0-21swapper/220:25:152
2391899181,3cyclictest87350irq/126-enp1s0-22:00:252
2391899181,3cyclictest87350irq/126-enp1s0-22:00:252
23918991812,3cyclictest6152-21fschecks_time19:40:152
23918991812,3cyclictest6152-21fschecks_time19:40:152
2391899181,11cyclictest0-21swapper/219:25:242
2391899181,10cyclictest0-21swapper/223:25:202
2391899181,10cyclictest0-21swapper/221:19:132
23913991815,2cyclictest0-21swapper/100:34:291
2391399181,3cyclictest10725-21cat21:05:141
2391399181,16cyclictest0-21swapper/120:05:061
23913991811,4cyclictest0-21swapper/121:50:271
2391399181,11cyclictest0-21swapper/119:25:441
2391399181,10cyclictest0-21swapper/119:50:141
2390699186,3cyclictest0-21swapper/023:21:280
2390699185,3cyclictest168692cat20:05:120
2390699184,11cyclictest27285-21sed22:55:160
2390699181,9cyclictest17333-21ntpq23:45:210
2390699181,9cyclictest0-21swapper/000:15:160
2390699181,11cyclictest0-21swapper/000:26:300
2390699181,10cyclictest18823-21/usr/sbin/munin20:10:260
2390699181,10cyclictest10649-21/usr/sbin/munin21:05:200
2390699181,10cyclictest0-21swapper/022:50:130
2390699180,8cyclictest0-21swapper/023:41:570
289942171,2sleep20-21swapper/200:11:382
2391899174,4cyclictest9630-21sshd19:48:032
2391899174,4cyclictest32-21ksoftirqd/222:15:262
2391899173,7cyclictest0-21swapper/222:56:062
2391899172,3cyclictest13424-21cat22:25:012
2391899171,6cyclictest31344-21unixbench_singl20:35:242
2391899171,6cyclictest0-21swapper/223:05:182
23918991713,3cyclictest0-21swapper/221:55:062
23918991713,3cyclictest0-21swapper/220:55:292
2391899171,15cyclictest0-21swapper/221:31:002
2391899171,14cyclictest0-21swapper/222:10:112
23918991711,3cyclictest31466-21ntp_kernel_pll_21:50:212
2391899171,10cyclictest0-21swapper/219:37:212
23906991710,4cyclictest32119-21cut20:40:120
2391899163,3cyclictest30-21rcuc/223:15:162
2391899162,8cyclictest32-21ksoftirqd/220:10:172
2391899161,9cyclictest0-21swapper/220:02:492
2391899161,9cyclictest0-21swapper/219:30:132
2391899161,6cyclictest32003-21/usr/sbin/munin20:40:152
23918991612,3cyclictest0-21swapper/222:37:582
23918991612,3cyclictest0-21swapper/220:15:342
23918991611,3cyclictest30740-21grep19:20:252
23918991611,3cyclictest29330-21seq20:31:552
2391899160,5cyclictest0-21swapper/220:55:002
2391899160,10cyclictest30-21rcuc/223:30:252
2390699169,4cyclictest31311-21wc19:25:010
2390699163,3cyclictest15563-21pmu-power20:00:230
2390699161,7cyclictest27003-21/usr/sbin/munin19:15:110
2390699161,14cyclictest0-21swapper/020:49:050
23906991611,3cyclictest8159-21sh19:45:120
23906991611,3cyclictest5626-21/usr/sbin/munin19:40:210
23906991611,3cyclictest5626-21/usr/sbin/munin19:40:210
2390699161,13cyclictest0-21swapper/019:55:400
2390699161,12cyclictest0-21swapper/021:45:050
23906991610,4cyclictest30403-21grep20:35:150
2390699160,10cyclictest0-21swapper/020:30:250
2391899152,4cyclictest18241-21chrt20:05:412
23918991511,3cyclictest0-21swapper/219:10:022
23918991510,3cyclictest10981-21expr21:05:152
23906991510,3cyclictest4039-21/usr/sbin/munin20:50:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional