You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 00:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun May 31, 2026 12:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19122119910596,7cyclictest23-21ksoftirqd/110:50:021
191220599939,41cyclictest0-21swapper/011:45:020
191220599899,46cyclictest1958173-21latency_hist10:40:010
1912205998920,29cyclictest1922345-21ssh09:18:490
191220599851,62cyclictest0-21swapper/009:05:020
191220599809,41cyclictest0-21swapper/009:35:010
1912205997619,21cyclictest1915014-21munin-plugin-st08:05:000
191220599744,32cyclictest0-21swapper/011:34:590
1912205997420,28cyclictest0-21swapper/010:40:320
1912205997420,28cyclictest0-21swapper/010:40:320
1912205997413,22cyclictest1968685-21ssh11:04:050
1912205997413,22cyclictest1968685-21ssh11:04:040
1912205997323,13cyclictest1944812-21latency_hist10:10:010
191220599714,31cyclictest0-21swapper/011:30:020
191220599713,34cyclictest0-21swapper/010:28:380
1912205997119,19cyclictest1923484-21ssh09:21:150
1912205997033,34cyclictest1938134-21ssh09:54:470
1912205997019,18cyclictest1987523-21ssh11:46:230
1912205997019,18cyclictest1987523-21ssh11:46:230
1912205996850,13cyclictest3467602-21systemd-journal07:15:000
1912205996727,11cyclictest0-21swapper/010:59:540
191220599672,48cyclictest0-21swapper/011:38:140
191220599666,32cyclictest2009880-21rm12:37:160
1912205996531,30cyclictest1977274-21ssh11:23:200
1912205996524,23cyclictest1918392-21ssh09:10:010
1912205996524,23cyclictest1918392-21ssh09:10:010
1912205996518,15cyclictest0-21swapper/011:10:550
1912205996518,15cyclictest0-21swapper/011:10:550
1912205996516,14cyclictest2001808-21ssh12:19:120
1912205996510,19cyclictest1950460-21sh10:22:270
1912205996431,30cyclictest1933702-21ssh09:44:370
1912205996415,18cyclictest1996368-21ssh12:06:330
191220599632,52cyclictest0-21swapper/007:45:010
1912205996319,16cyclictest1954138-21sh10:30:470
1912205996319,16cyclictest1954138-21sh10:30:470
1912205996319,14cyclictest2003199-21ssh12:21:590
1912205996316,13cyclictest1936067-21sh09:50:000
1912205996316,13cyclictest1936067-21sh09:50:000
1912205996315,17cyclictest1964888-21sed10:55:020
1912219996138,18cyclictest1-21systemd10:05:003
1912219996111,6cyclictest131rcu_preempt10:25:053
1912205996116,14cyclictest1940726-21date10:00:030
1912205996110,12cyclictest1990128-21ssh11:52:270
191221999607,12cyclictest37-21ksoftirqd/309:20:013
191220599602,32cyclictest0-21swapper/012:04:250
191221999597,13cyclictest37-21ksoftirqd/309:55:023
1912215995948,7cyclictest1962698-21kworker/2:211:34:432
1912205995919,11cyclictest1946224-21ssh10:12:510
1912205995919,11cyclictest1946224-21ssh10:12:510
1912205995912,16cyclictest1916557-21latency_hist08:35:020
1912219995848,7cyclictest361rcuc/311:17:513
191221999583,11cyclictest37-21ksoftirqd/311:45:033
191220599584,22cyclictest0-21swapper/010:18:270
1912205995835,16cyclictest1-21systemd09:00:010
1912205995821,33cyclictest0-21swapper/011:08:580
1912205995815,39cyclictest1999885-21ssh12:14:480
1912205995815,39cyclictest1999885-21ssh12:14:480
191221999575,10cyclictest37-21ksoftirqd/311:30:013
1912219995716,6cyclictest131rcu_preempt11:55:003
1912215995748,5cyclictest1902691-21kworker/2:011:04:252
1912215995748,5cyclictest1902691-21kworker/2:011:04:252
1912205995714,13cyclictest2005340-21ssh12:26:570
1912219995617,6cyclictest131rcu_preempt12:20:003
1912219995610,4cyclictest131rcu_preempt11:56:463
1912215995646,6cyclictest1901656-21kworker/2:309:10:012
1912215995646,6cyclictest1901656-21kworker/2:309:10:002
1912215995638,14cyclictest275-21dbus-daemon08:50:012
1912205995620,32cyclictest1919873-21rm09:13:230
1912205995617,7cyclictest1940122-21ssh09:59:080
1912205995612,39cyclictest0-21swapper/009:30:030
191221999559,6cyclictest131rcu_preempt12:26:023
1912219995545,7cyclictest361rcuc/310:24:003
191221999553,13cyclictest37-21ksoftirqd/311:25:023
1912219995511,6cyclictest131rcu_preempt09:43:553
1912215995546,5cyclictest1902691-21kworker/2:010:28:222
1912215995546,5cyclictest1902691-21kworker/2:009:41:232
1912215995540,7cyclictest1901656-21kworker/2:309:05:012
1912205995535,17cyclictest48-21kcompactd008:45:060
191221999544,9cyclictest37-21ksoftirqd/310:38:343
1912219995413,5cyclictest131rcu_preempt09:22:353
1912219995411,10cyclictest37-21ksoftirqd/309:40:003
1912215995446,5cyclictest1902691-21kworker/2:011:41:012
1912215995444,6cyclictest1931666-21kworker/2:110:08:092
1912215995443,7cyclictest1902691-21kworker/2:012:13:312
1912215995443,7cyclictest1902691-21kworker/2:012:13:312
1912215995442,5cyclictest1902691-21kworker/2:009:48:162
1912215995442,5cyclictest1902691-21kworker/2:009:48:162
1912211995437,13cyclictest1-21systemd08:00:001
1912205995436,13cyclictest1913673-21sshd07:38:340
1912205995436,13cyclictest1913673-21sshd07:38:330
1912205995419,31cyclictest1960732-21sh10:45:320
1912205995414,13cyclictest1992156-21ssh11:57:020
191221999538,10cyclictest37-21ksoftirqd/311:35:343
191221999531,7cyclictest1918124-21grep09:05:003
1912219995314,5cyclictest131rcu_preempt08:10:013
1912215995345,5cyclictest1902691-21kworker/2:011:56:492
1912215995345,5cyclictest1902691-21kworker/2:009:35:012
1912215995345,5cyclictest1902691-21kworker/2:009:20:012
1912215995344,6cyclictest1902691-21kworker/2:012:08:332
1912215995344,5cyclictest1931666-21kworker/2:110:15:082
1912215995343,6cyclictest1902691-21kworker/2:010:54:132
1912215995343,6cyclictest1902691-21kworker/2:008:05:012
1912215995341,7cyclictest1902691-21kworker/2:010:35:002
1912215995341,7cyclictest1902691-21kworker/2:010:35:002
1912215995336,13cyclictest1-21systemd11:15:012
1912215995336,13cyclictest1-21systemd11:15:012
1912205995343,5cyclictest16950irq/224-800000011:20:000
1912205995336,14cyclictest1917063-21sshd08:42:330
1912205995319,30cyclictest0-21swapper/008:50:550
191221999526,13cyclictest37-21ksoftirqd/309:30:003
1912219995240,9cyclictest37-21ksoftirqd/312:20:293
1912219995225,22cyclictest361rcuc/310:15:023
1912219995225,22cyclictest361rcuc/310:15:023
1912215995244,5cyclictest1902691-21kworker/2:012:21:522
1912215995244,5cyclictest1902691-21kworker/2:011:51:202
1912215995244,5cyclictest1902691-21kworker/2:011:07:332
1912215995244,5cyclictest1902691-21kworker/2:010:35:022
1912215995244,5cyclictest1901656-21kworker/2:308:10:002
1912215995243,5cyclictest1902691-21kworker/2:012:25:372
1912215995241,7cyclictest1902691-21kworker/2:011:49:402
1912215995241,7cyclictest1902691-21kworker/2:011:49:402
1912215995237,7cyclictest1902691-21kworker/2:009:27:242
191220599524,42cyclictest0-21swapper/012:35:010
191220599523,43cyclictest1916807-21modprobe08:40:010
191221999518,8cyclictest2008843-21rm12:35:023
191221999516,6cyclictest131rcu_preempt11:31:163
191221999514,11cyclictest37-21ksoftirqd/311:15:003
191221999514,11cyclictest37-21ksoftirqd/311:15:003
1912219995136,11cyclictest1-21systemd07:55:003
1912219995113,10cyclictest131rcu_preempt10:20:013
1912219995110,12cyclictest37-21ksoftirqd/312:36:393
1912215995143,5cyclictest1986766-21kworker/2:312:34:582
1912215995143,5cyclictest1931666-21kworker/2:110:20:312
1912215995143,5cyclictest1902691-21kworker/2:011:17:032
1912215995142,6cyclictest1902691-21kworker/2:012:04:032
1912215995142,5cyclictest1902691-21kworker/2:009:22:202
1912215995141,6cyclictest1902691-21kworker/2:007:40:022
1912215995141,6cyclictest1902691-21kworker/2:007:40:012
1912205995135,13cyclictest1915730-21sshd08:15:330
191221999505,8cyclictest37-21ksoftirqd/310:56:473
191221999504,8cyclictest37-21ksoftirqd/311:45:463
191221999504,8cyclictest37-21ksoftirqd/311:45:463
191221999503,11cyclictest37-21ksoftirqd/311:00:303
191221999503,11cyclictest37-21ksoftirqd/311:00:303
1912219995010,8cyclictest37-21ksoftirqd/310:10:023
1912219995010,6cyclictest131rcu_preempt10:52:543
1912215995043,5cyclictest1902691-21kworker/2:010:58:502
1912215995042,5cyclictest1986766-21kworker/2:312:37:242
1912215995042,5cyclictest1986766-21kworker/2:312:17:462
1912215995042,5cyclictest1962698-21kworker/2:211:27:502
1912215995042,5cyclictest1931666-21kworker/2:110:12:482
1912215995042,5cyclictest1931666-21kworker/2:110:12:482
1912215995042,5cyclictest1902691-21kworker/2:009:53:592
1912215995042,5cyclictest1901656-21kworker/2:308:55:002
1912215995042,5cyclictest1901656-21kworker/2:308:25:032
1912215995042,5cyclictest1901656-21kworker/2:308:18:412
1912215995041,6cyclictest1902691-21kworker/2:009:57:152
1912215995041,5cyclictest1902691-21kworker/2:010:01:392
1912215995041,5cyclictest1902691-21kworker/2:009:38:102
1912215995039,6cyclictest1902691-21kworker/2:010:45:002
1912215995039,6cyclictest1902691-21kworker/2:010:45:002
1912215995031,14cyclictest275-21dbus-daemon07:35:012
191221199509,5cyclictest131rcu_preempt08:20:021
191220599508,39cyclictest0-21swapper/007:20:010
1912205995043,4cyclictest1913447-21munin-run07:35:010
1912205995034,13cyclictest1914696-21sshd07:55:340
1912205995011,35cyclictest1931614-21modprobe09:40:000
191221999496,9cyclictest37-21ksoftirqd/308:50:013
1912219994926,7cyclictest37-21ksoftirqd/312:00:523
1912219994912,6cyclictest131rcu_preempt12:10:023
1912219994912,6cyclictest131rcu_preempt12:10:023
1912215994941,5cyclictest1901656-21kworker/2:309:12:572
1912215994940,6cyclictest1902691-21kworker/2:011:20:052
1912215994939,7cyclictest1902691-21kworker/2:010:47:252
1912211994935,10cyclictest1-21systemd10:25:001
1912211994934,11cyclictest3467602-21systemd-journal08:15:001
1912211994930,15cyclictest1-21systemd08:40:011
1912211994910,6cyclictest131rcu_preempt12:05:021
1912205994933,11cyclictest1916036-21sshd08:24:350
1912205994933,11cyclictest1916036-21sshd08:24:350
1912205994931,14cyclictest364-21snmpd07:23:530
1912205994921,19cyclictest0-21swapper/007:10:010
191221999486,10cyclictest37-21ksoftirqd/311:09:133
1912219994836,8cyclictest37-21ksoftirqd/312:10:003
191221999483,5cyclictest131rcu_preempt10:44:233
191221999483,5cyclictest131rcu_preempt10:44:223
191221999483,11cyclictest37-21ksoftirqd/307:30:003
1912219994829,14cyclictest309-21in:imuxsock07:40:003
1912219994829,14cyclictest309-21in:imuxsock07:39:593
1912219994819,9cyclictest37-21ksoftirqd/308:55:013
1912219994812,5cyclictest131rcu_preempt09:58:573
191221999481,11cyclictest1918372-21modprobe09:10:013
191221999481,11cyclictest1918372-21modprobe09:10:003
1912219994810,5cyclictest131rcu_preempt10:49:313
191221199485,5cyclictest131rcu_preempt11:20:001
191221199483,6cyclictest131rcu_preempt11:45:041
191221199481,6cyclictest150-21pr/ttyS211:26:021
1912211994811,6cyclictest131rcu_preempt09:25:011
1912205994833,11cyclictest1913174-21sshd07:29:320
191221999477,9cyclictest37-21ksoftirqd/309:33:233
191221999471,5cyclictest0-21swapper/310:33:173
191221999471,5cyclictest0-21swapper/310:33:173
191221999471,5cyclictest0-21swapper/307:15:023
1912215994738,5cyclictest1901656-21kworker/2:308:45:012
1912215994732,6cyclictest1902691-21kworker/2:011:35:082
191221599473,18cyclictest0-21swapper/208:00:032
191221199477,5cyclictest131rcu_preempt12:40:001
1912211994728,14cyclictest275-21dbus-daemon10:30:011
1912211994710,5cyclictest131rcu_preempt11:15:021
1912211994710,5cyclictest131rcu_preempt11:15:021
1912205994734,10cyclictest1914196-21sshd07:46:330
1912205994733,10cyclictest1914460-21sshd07:54:340
191221999467,11cyclictest37-21ksoftirqd/309:10:453
191221999463,11cyclictest37-21ksoftirqd/308:25:013
191221999463,11cyclictest37-21ksoftirqd/308:25:013
1912219994614,7cyclictest131rcu_preempt07:10:023
1912215994637,6cyclictest1901656-21kworker/2:307:15:032
1912211994623,4cyclictest23-21ksoftirqd/111:21:561
1912211994619,6cyclictest23-21ksoftirqd/110:06:541
1912205994633,10cyclictest1915495-21sshd08:14:340
1912205994622,19cyclictest1915267-21systemd-run08:10:010
191221999457,12cyclictest37-21ksoftirqd/308:30:013
191221999455,8cyclictest37-21ksoftirqd/307:45:013
1912219994534,9cyclictest37-21ksoftirqd/308:35:003
1912219994533,10cyclictest37-21ksoftirqd/308:45:023
1912215994537,5cyclictest1902691-21kworker/2:007:45:002
191221199457,6cyclictest131rcu_preempt10:35:021
191221199457,6cyclictest131rcu_preempt10:35:021
191221199457,6cyclictest131rcu_preempt09:55:001
191221199452,7cyclictest0-21swapper/112:30:021
1912211994510,6cyclictest131rcu_preempt09:30:001
191221999448,5cyclictest131rcu_preempt08:20:003
191221999446,5cyclictest131rcu_preempt09:47:353
191221999446,5cyclictest131rcu_preempt09:47:343
1912219994427,12cyclictest3467602-21systemd-journal08:58:353
191221999441,16cyclictest0-21swapper/308:00:023
1912215994434,5cyclictest1901656-21kworker/2:308:15:002
191221599441,6cyclictest131rcu_preempt08:25:032
191221199443,5cyclictest131rcu_preempt11:50:001
191221199443,5cyclictest131rcu_preempt11:50:001
1912211994429,10cyclictest3467602-21systemd-journal09:45:001
191221199442,8cyclictest131rcu_preempt11:00:001
191221199442,6cyclictest131rcu_preempt10:40:001
1912219994333,5cyclictest1912669-21sendmail07:20:013
1912219994329,10cyclictest37-21ksoftirqd/307:45:343
1912219994314,5cyclictest131rcu_preempt08:40:013
191221599434,8cyclictest1917897-21latency_hist09:00:022
1912211994333,7cyclictest48-21kcompactd012:22:131
1912211994330,7cyclictest309-21in:imuxsock07:25:011
1912211994329,11cyclictest23-21ksoftirqd/109:11:441
1912211994324,14cyclictest275-21dbus-daemon07:20:001
1912211994319,19cyclictest221rcuc/111:55:331
1912211994318,7cyclictest131rcu_preempt09:55:461
191221199431,6cyclictest131rcu_preempt07:55:011
1912211994316,23cyclictest221rcuc/111:30:081
1912205994335,4cyclictest1916272-21sshd08:25:330
1912219994210,12cyclictest37-21ksoftirqd/308:15:003
1912215994234,5cyclictest1902691-21kworker/2:007:52:092
1912215994234,5cyclictest1901656-21kworker/2:307:29:342
1912211994231,8cyclictest1931073-21sshd09:38:341
1912211994229,10cyclictest1915022-21systemctl08:05:011
191221199422,5cyclictest131rcu_preempt10:12:331
191221199422,5cyclictest131rcu_preempt10:12:321
1912211994221,5cyclictest23-21ksoftirqd/111:55:001
191221199421,9cyclictest364-21snmpd07:35:011
1912211994218,5cyclictest23-21ksoftirqd/110:51:531
191221999411,6cyclictest150-21pr/ttyS207:22:483
1912215994134,5cyclictest1901656-21kworker/2:308:31:312
1912215994129,9cyclictest1914421-21sed07:50:012
191221199419,5cyclictest131rcu_preempt12:12:591
191221199419,5cyclictest131rcu_preempt12:12:591
191221199414,3cyclictest131rcu_preempt11:09:581
191221199413,5cyclictest131rcu_preempt08:50:001
1912211994131,5cyclictest309-21in:imuxsock07:30:011
1912211994118,5cyclictest23-21ksoftirqd/108:45:001
191221199411,6cyclictest131rcu_preempt09:00:021
1912211994116,7cyclictest131rcu_preempt11:01:521
1912211994116,7cyclictest131rcu_preempt11:01:521
191221199411,5cyclictest131rcu_preempt08:10:011
1912211994111,6cyclictest131rcu_preempt08:34:341
1912219994028,9cyclictest37-21ksoftirqd/307:35:013
191221999401,4cyclictest1912199-21cyclictest08:05:003
1912215994033,4cyclictest1901656-21kworker/2:308:35:192
1912211994031,5cyclictest309-21in:imuxsock09:00:021
191221199402,5cyclictest131rcu_preempt12:34:341
1912211994025,12cyclictest1936038-21sh09:49:581
1912211994025,12cyclictest1936038-21sh09:49:581
1912211994019,15cyclictest275-21dbus-daemon08:30:011
191221199401,6cyclictest131rcu_preempt09:33:321
191221199401,4cyclictest131rcu_preempt10:45:011
191221199401,4cyclictest131rcu_preempt10:45:011
1912215993931,5cyclictest1901656-21kworker/2:307:23:282
1912215993927,9cyclictest1912519-21sed07:15:022
191221199394,5cyclictest131rcu_preempt12:15:431
191221199393,6cyclictest23-21ksoftirqd/111:35:121
1912211993927,8cyclictest3467602-21systemd-journal07:45:001
1912211993922,12cyclictest309-21in:imuxsock07:15:011
191221199391,5cyclictest364-21snmpd10:03:251
1912211993915,6cyclictest23-21ksoftirqd/112:07:041
1912211993915,15cyclictest23-21ksoftirqd/110:20:021
1912211993913,7cyclictest131rcu_preempt09:10:011
1912211993913,7cyclictest131rcu_preempt09:10:011
191221199388,6cyclictest131rcu_preempt08:54:341
191221199383,3cyclictest131rcu_preempt09:17:411
1912215993729,5cyclictest1901656-21kworker/2:308:20:532
191221199376,7cyclictest131rcu_preempt07:46:341
1912211993725,8cyclictest275-21dbus-daemon08:25:011
1912211993725,8cyclictest275-21dbus-daemon08:25:011
191221199371,5cyclictest131rcu_preempt07:40:011
191221199371,5cyclictest131rcu_preempt07:40:011
1912215993324,5cyclictest1912297-21latency_hist07:10:022
191221199322,3cyclictest131rcu_preempt07:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional