You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 13:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 29, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
151192237160,26sleep00-21swapper/019:08:580
149242231164,54sleep20-21swapper/219:06:262
143082216159,47sleep10-21swapper/119:05:211
148562203165,25sleep30-21swapper/319:05:353
1721721540,5sleep31530199cyclictest21:40:023
1494521440,5sleep11529099cyclictest20:20:131
660821370,4sleep11529099cyclictest21:15:151
1332221290,4sleep21529799cyclictest20:15:212
218932650,4sleep11529099cyclictest00:15:221
186742560,5sleep318678-21gltestperf22:55:183
301322550,1sleep30-21swapper/300:35:153
91342500,1sleep10-21swapper/122:35:011
287022500,1sleep10-21swapper/100:30:231
15284993810,13cyclictest12172-21munin-run20:15:010
27942370,4sleep11529099cyclictest22:20:131
27942370,4sleep11529099cyclictest22:20:131
261092370,5sleep21529799cyclictest23:10:262
1529099342,3cyclictest10193-21chrt20:10:101
1529099331,15cyclictest0-21swapper/122:42:441
1529099331,15cyclictest0-21swapper/121:25:421
1529099301,6cyclictest0-21swapper/123:19:021
15290992910,3cyclictest18381-21users20:25:261
1529099281,14cyclictest23-21ksoftirqd/120:00:421
1528499281,11cyclictest31835-21kworker/0:023:34:300
1528499281,11cyclictest31835-21kworker/0:023:34:300
1529099271,14cyclictest0-21swapper/121:49:161
1529099271,14cyclictest0-21swapper/120:40:261
1528499278,16cyclictest19396-21awk21:45:010
1528499271,14cyclictest5567-21munin-run20:00:010
1528499271,14cyclictest5567-21munin-run20:00:010
163232260,2sleep3401ktimersoftd/319:10:213
1530199263,5cyclictest4832-21/usr/sbin/munin22:25:153
1529099261,12cyclictest3992-21diskmemload00:09:281
1528499269,15cyclictest7438-21perf23:45:010
1528499268,15cyclictest3373-21munin-run19:55:010
1528499268,15cyclictest11821-21munin-run23:55:010
1528499263,13cyclictest481-40dnf23:27:010
1528499260,12cyclictest0-21swapper/019:30:010
1529099253,7cyclictest87350irq/126-enp1s0-19:35:061
1529099251,21cyclictest23778-21munin-run21:55:011
1529099250,10cyclictest0-21swapper/123:05:261
1528499257,15cyclictest22223-21perf23:05:020
1528499251,9cyclictest24745-21kworker/0:100:22:040
1528499251,12cyclictest20362-21idleruntime-cro19:20:010
1530199241,11cyclictest839-21lldpd22:50:023
1529799246,5cyclictest311ktimersoftd/220:50:252
1529799241,12cyclictest311ktimersoftd/221:10:012
1529799241,12cyclictest311ktimersoftd/221:10:012
1529099241,9cyclictest0-21swapper/120:30:201
1529099241,15cyclictest0-21swapper/119:54:281
1529099241,12cyclictest0-21swapper/122:29:321
1529099241,11cyclictest0-21swapper/120:07:441
1529099240,13cyclictest0-21swapper/123:59:261
15301992314,4cyclictest0-21swapper/321:41:173
15301992314,4cyclictest0-21swapper/320:57:543
1530199231,12cyclictest0-21swapper/321:45:403
1529799238,4cyclictest87350irq/126-enp1s0-22:55:152
1529099231,8cyclictest0-21swapper/100:22:521
1529099231,7cyclictest0-21swapper/122:48:301
15290992314,3cyclictest0-21swapper/121:04:261
1529099231,3cyclictest3296-21switchtime21:05:241
1529099231,3cyclictest3296-21switchtime21:05:241
1529099231,13cyclictest3992-21diskmemload22:17:041
1529099231,10cyclictest0-21swapper/123:00:221
1529099231,10cyclictest0-21swapper/121:42:011
1529099231,10cyclictest0-21swapper/100:39:171
1529099230,16cyclictest0-21swapper/120:46:541
1528499232,9cyclictest4547-21kworker/0:122:36:020
1528499232,8cyclictest14941-21kworker/0:221:35:260
1528499232,6cyclictest32292-21kworker/0:322:24:120
1528499232,6cyclictest32292-21kworker/0:322:24:120
1528499232,6cyclictest21302-21kworker/0:222:04:040
305762220,3sleep011-21rcuc/023:20:270
1529099223,4cyclictest0-21swapper/119:20:141
1529099221,8cyclictest0-21swapper/119:57:441
1529099221,8cyclictest0-21swapper/119:57:441
1529099221,5cyclictest23-21ksoftirqd/123:36:561
1529099221,15cyclictest3992-21diskmemload00:13:041
1529099221,10cyclictest0-21swapper/123:42:531
1529099221,10cyclictest0-21swapper/122:10:411
1529099221,10cyclictest0-21swapper/121:22:381
15290992210,4cyclictest10145-21memory22:35:171
15290992210,3cyclictest14074-21ntp_states21:30:221
1529099220,9cyclictest0-21swapper/122:05:161
1529099220,16cyclictest0-21swapper/121:11:091
1529099220,15cyclictest0-21swapper/119:35:011
1528499222,7cyclictest3161-21kworker/0:120:08:280
1528499221,8cyclictest21302-21kworker/0:222:05:360
1528499221,5cyclictest0-21swapper/023:35:110
1528499221,5cyclictest0-21swapper/022:51:180
1528499221,11cyclictest17474-21perf19:15:010
1528499220,9cyclictest7351-21kworker/0:000:15:320
281252210,4sleep011-21rcuc/019:35:180
1530199211,11cyclictest19452-21ntp_states19:15:243
1530199211,11cyclictest18677-21/usr/sbin/munin00:10:123
1530199210,15cyclictest0-21swapper/321:18:593
1529799214,5cyclictest0-21swapper/222:19:562
1529799214,3cyclictest30-21rcuc/220:10:192
1529799213,6cyclictest808-21snmpd23:27:222
1529799211,8cyclictest7503-21open_files21:15:252
1529799211,5cyclictest0-21swapper/219:38:072
1529099219,4cyclictest25542-21memory23:10:191
1529099217,3cyclictest31627-21cpuspeed_turbos19:45:141
1529099217,3cyclictest0-21swapper/120:18:301
1529099217,3cyclictest0-21swapper/119:18:341
1529099214,3cyclictest0-21swapper/119:12:581
1529099214,2cyclictest87650irq/129-enp1s0-23:47:581
15290992114,2cyclictest0-21swapper/100:28:231
1529099211,2cyclictest0-21swapper/122:50:251
15290992112,2cyclictest3992-21diskmemload00:02:111
1529099211,15cyclictest0-21swapper/122:59:271
1529099211,14cyclictest0-21swapper/121:55:191
1529099211,11cyclictest744-21latency_hist23:30:011
1529099210,15cyclictest0-21swapper/123:30:231
1529099210,15cyclictest0-21swapper/123:30:231
1528499219,3cyclictest13973-21diskstats22:45:160
1528499218,3cyclictest9587-21kworker/0:223:55:300
1528499212,10cyclictest7351-21kworker/0:000:12:240
1528499211,9cyclictest4547-21kworker/0:122:44:460
1528499211,9cyclictest12632-21kworker/0:021:34:500
1528499211,8cyclictest32292-21kworker/0:322:26:180
1528499211,8cyclictest10279-21kworker/0:121:24:240
1528499211,6cyclictest0-21swapper/022:16:210
1528499211,5cyclictest18770-21munin-run20:30:010
1528499211,11cyclictest0-21swapper/000:05:070
1530199206,8cyclictest0-21swapper/321:14:463
1530199201,9cyclictest0-21swapper/323:10:223
1530199201,6cyclictest31179-21NetworkManager19:43:103
1530199201,5cyclictest0-21swapper/323:08:543
15301992012,5cyclictest495-21jbd2/dm-0-819:47:593
1530199201,16cyclictest30947-21diskstats22:10:133
1530199201,11cyclictest3939-21df_inode19:55:133
1530199201,11cyclictest3939-21df_inode19:55:133
1530199201,10cyclictest0-21swapper/321:58:463
1530199201,10cyclictest0-21swapper/300:33:463
15301992010,3cyclictest0-21swapper/322:54:043
1529799205,3cyclictest0-21swapper/220:49:462
1529799203,7cyclictest0-21swapper/223:15:082
1529799203,3cyclictest87650irq/129-enp1s0-22:45:182
1529799201,8cyclictest87650irq/129-enp1s0-22:10:112
1529799201,8cyclictest0-21swapper/223:09:422
1529799201,7cyclictest0-21swapper/222:24:392
1529799201,7cyclictest0-21swapper/222:24:392
1529799201,7cyclictest0-21swapper/221:10:352
1529799201,7cyclictest0-21swapper/200:34:162
1529799201,6cyclictest0-21swapper/222:53:382
1529799201,6cyclictest0-21swapper/221:51:542
1529799201,6cyclictest0-21swapper/200:11:102
1529799201,5cyclictest0-21swapper/200:29:472
15297992012,7cyclictest32-21ksoftirqd/221:20:202
1529099201,4cyclictest6119-21kworker/u8:221:38:321
15290992013,2cyclictest0-21swapper/120:59:161
1529099201,14cyclictest0-21swapper/120:36:451
1529099201,11cyclictest22999-21cstates19:25:131
1529099200,4cyclictest0-21swapper/123:20:031
1528499209,3cyclictest32170-21/usr/sbin/munin21:00:130
1528499209,3cyclictest23740-21seq20:40:150
1528499208,3cyclictest24523-21kworker/0:019:32:380
1528499204,7cyclictest755-21rngd20:00:230
1528499203,9cyclictest0-21swapper/000:27:030
1528499202,10cyclictest15180-21kworker/0:119:21:300
1528499201,9cyclictest4547-21kworker/0:122:33:380
1528499201,5cyclictest14685-21kworker/u8:022:57:000
1528499201,4cyclictest0-21swapper/021:48:170
1528499201,11cyclictest0-21swapper/023:10:440
1528499201,11cyclictest0-21swapper/021:50:380
1528499201,11cyclictest0-21swapper/000:38:020
1528499201,11cyclictest0-21swapper/000:33:120
1528499201,10cyclictest3992-21diskmemload23:06:320
1528499200,8cyclictest1626-21kworker/0:021:15:060
170012191,3sleep20-21swapper/221:37:532
1530199198,3cyclictest23385-21/usr/sbin/munin20:40:173
1530199195,6cyclictest0-21swapper/321:25:243
1530199194,3cyclictest87350irq/126-enp1s0-22:20:133
1530199194,3cyclictest87350irq/126-enp1s0-22:20:133
1530199191,6cyclictest6127-21hddtemp_smartct23:40:153
1530199191,4cyclictest0-21swapper/322:18:313
15301991911,2cyclictest8829-21sshd22:32:543
1530199191,11cyclictest0-21swapper/300:08:033
1530199191,10cyclictest0-21swapper/323:55:273
1530199190,10cyclictest0-21swapper/322:09:463
1529799198,2cyclictest0-21swapper/222:43:352
1529799195,3cyclictest87550irq/128-enp1s0-23:31:022
1529799195,3cyclictest87550irq/128-enp1s0-23:31:022
1529799193,2cyclictest87650irq/129-enp1s0-21:43:122
1529799192,7cyclictest0-21swapper/200:06:282
1529799191,9cyclictest0-21swapper/222:30:212
1529799191,7cyclictest0-21swapper/223:01:162
1529799191,7cyclictest0-21swapper/200:02:222
1529799191,6cyclictest0-21swapper/223:56:122
1529799191,6cyclictest0-21swapper/222:29:132
1529799191,6cyclictest0-21swapper/221:25:212
1529799191,6cyclictest0-21swapper/220:35:152
1529799191,5cyclictest9856-21/usr/sbin/munin23:50:142
1529799191,5cyclictest0-21swapper/223:47:532
15297991914,3cyclictest29974-21gltestperf19:40:152
15297991913,3cyclictest20915-21df_inode19:20:142
15297991913,3cyclictest20877-21/usr/sbin/munin00:15:292
1529799191,11cyclictest0-21swapper/223:35:062
1529099197,3cyclictest9879-21wc23:50:141
1529099195,2cyclictest30053-21sed19:40:161
1529099191,11cyclictest28451-21hddtemp_smartct20:50:181
1529099191,10cyclictest27110-21memory22:00:191
1528499198,2cyclictest5123-21kworker/0:123:47:540
1528499192,4cyclictest7351-21kworker/0:000:03:160
1528499192,14cyclictest26689-21ls20:45:220
1528499191,9cyclictest3992-21diskmemload21:14:110
1528499191,5cyclictest0-21swapper/022:11:360
1528499191,4cyclictest20970-21perf20:35:010
1528499191,3cyclictest0-21swapper/021:57:460
1528499191,10cyclictest808-21snmpd20:50:190
1528499191,10cyclictest29364-21/usr/sbin/munin19:40:150
1528499191,10cyclictest27416-21grep23:15:170
1528499191,10cyclictest2022-21/usr/sbin/munin21:05:190
1528499191,10cyclictest2022-21/usr/sbin/munin21:05:190
281542181,2sleep30-21swapper/322:04:553
1530199188,2cyclictest1431-21sshd21:00:543
1530199186,6cyclictest0-21swapper/300:26:463
1530199184,3cyclictest9247-21cpu22:35:113
1530199183,7cyclictest0-21swapper/300:03:333
1530199183,12cyclictest21808-21/usr/sbin/munin21:50:143
1530199181,5cyclictest30081-21taskset23:20:223
1530199181,5cyclictest0-21swapper/323:47:033
1530199181,5cyclictest0-21swapper/322:44:483
1530199181,4cyclictest22225-21/usr/sbin/munin20:35:203
1530199181,3cyclictest0-21swapper/320:08:593
15301991812,3cyclictest26109-21fschecks_time20:45:173
1530199180,12cyclictest0-21swapper/323:32:023
1530199180,12cyclictest0-21swapper/323:32:023
1530199180,12cyclictest0-21swapper/323:25:223
1530199180,10cyclictest0-21swapper/321:33:593
1529799183,3cyclictest87650irq/129-enp1s0-21:30:192
1529799182,3cyclictest0-21swapper/220:30:412
1529799181,7cyclictest16931-21sort19:10:282
1529799181,6cyclictest30084-21ntpq23:20:222
1529799181,6cyclictest0-21swapper/220:20:132
1529799181,6cyclictest0-21swapper/200:36:192
1529799181,5cyclictest3270-21sshd19:53:542
1529799181,5cyclictest0-21swapper/221:48:142
1529799181,4cyclictest30-21rcuc/221:03:242
1529799181,3cyclictest30-21rcuc/222:00:132
15297991812,3cyclictest9397-21expr22:35:112
1529799181,13cyclictest0-21swapper/219:18:022
1528499181,9cyclictest808-21snmpd19:45:060
1528499181,8cyclictest29391-21kworker/0:120:56:120
1528499181,3cyclictest0-21swapper/021:25:200
1530199179,5cyclictest23891-21cut19:25:233
1530199178,6cyclictest6017-21expr20:00:113
1530199171,4cyclictest20572-21diskstats23:00:163
15301991712,3cyclictest22125-21sendmail_mailtr00:15:263
15301991712,3cyclictest15657-21ntp_states20:20:193
15301991711,3cyclictest27350-21cstates19:35:113
1530199171,10cyclictest0-21swapper/320:50:263
15301991710,2cyclictest0-21swapper/323:52:023
15301991710,1cyclictest0-21swapper/319:50:183
1530199170,6cyclictest0-21swapper/323:35:153
1529799175,3cyclictest12050irq/123-ahci[0019:30:012
1529799171,9cyclictest30723-21kworker/2:119:49:462
1529799171,7cyclictest87650irq/129-enp1s0-20:55:202
1529799171,4cyclictest5455-21/usr/sbin/munin23:40:222
15297991713,3cyclictest0-21swapper/219:30:472
1529799171,15cyclictest0-21swapper/219:55:292
1529799171,15cyclictest0-21swapper/219:55:292
15297991710,4cyclictest25063-21ntpq21:55:222
15297991710,4cyclictest16793-21munin-node20:25:122
15297991710,4cyclictest0-21swapper/222:05:262
1528499172,6cyclictest808-21snmpd20:15:120
1530199169,4cyclictest12914-21tune2fs20:15:173
1530199162,3cyclictest14712-21kworker/3:119:22:003
15301991612,3cyclictest0-21swapper/319:30:483
1530199161,14cyclictest0-21swapper/320:11:373
15301991611,3cyclictest27732-21load23:15:203
15301991611,3cyclictest18994-21/usr/sbin/munin20:30:143
1530199160,10cyclictest0-21swapper/320:25:273
1529799161,6cyclictest0-21swapper/200:21:512
1529799161,3cyclictest0-21swapper/220:00:162
15297991611,3cyclictest24424-21ntp_kernel_pll_20:40:212
1528499163,5cyclictest755-21rngd20:20:460
15284991611,3cyclictest21189-21/usr/sbin/munin20:35:180
1530199159,3cyclictest23136-21cpuspeed_turbos00:20:133
15301991510,3cyclictest8626-21acpi21:20:123
15297991510,3cyclictest30-21rcuc/220:05:232
1530199141,3cyclictest87650irq/129-enp1s0-21:05:113
1530199141,3cyclictest87650irq/129-enp1s0-21:05:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional