You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 14:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackaslot8s.osadl.org (updated Fri Aug 28, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
116222171118,45sleep20-21swapper/219:05:052
2664221600,1sleep30-21swapper/320:55:123
133572143107,25sleep30-21swapper/319:07:463
116212140103,26sleep10-21swapper/119:05:031
132232138103,25sleep00-21swapper/019:06:020
1353127558,10sleep10-21swapper/119:10:011
34242640,1chrt3420-21sh00:00:170
40662580,2sleep10-21swapper/121:11:551
120142580,1sleep00-21swapper/023:42:110
135822570,1sleep00-21swapper/000:31:180
26722510,2sleep10-21swapper/122:47:181
91982480,4sleep0131rcuc/023:15:490
183732480,1sleep20-21swapper/200:10:342
131402480,1sleep20-21swapper/200:30:522
86882470,1sleep20-21swapper/220:10:252
286682420,1sleep30-21swapper/323:30:213
182842380,2chrt0-21swapper/322:35:153
193082350,2sleep10-21swapper/121:47:221
188072350,2sleep20-21swapper/221:23:112
289582330,3chrt12340-1kworker/u9:023:55:150
163352310,2sleep20-21swapper/223:45:212
1380599315,2cyclictest141rcu_preempt23:14:593
13802993118,8cyclictest636-21runrttasks21:05:140
41602300,2sleep30-21swapper/322:00:163
110532300,3sleep012-21ksoftirqd/021:17:110
254512290,1sleep00-21swapper/022:15:310
246542290,4chrt24652-21ssh22:40:020
1380599297,10cyclictest34-21ksoftirqd/322:33:543
1380599296,10cyclictest34-21ksoftirqd/300:00:553
13805992925,3cyclictest331rcuc/321:54:593
13805992914,13cyclictest34-21ksoftirqd/300:20:013
1380599291,20cyclictest34-21ksoftirqd/321:36:113
1380599284,15cyclictest34-21ksoftirqd/321:21:373
13805992823,4cyclictest331rcuc/323:42:403
233852270,4sleep2281ksoftirqd/223:02:562
1380599277,10cyclictest34-21ksoftirqd/322:49:483
13805992723,3cyclictest331rcuc/321:56:363
13805992722,4cyclictest331rcuc/321:26:553
1380599271,6cyclictest34-21ksoftirqd/323:45:253
13805992711,4cyclictest34-21ksoftirqd/323:26:033
1380599270,6cyclictest17653-21/usr/sbin/munin00:35:233
89422260,2sleep10-21swapper/121:40:121
1380599266,10cyclictest34-21ksoftirqd/300:11:393
1380599263,6cyclictest20699-21ls23:00:253
1380599262,5cyclictest34-21ksoftirqd/321:30:153
13805992623,2cyclictest331rcuc/321:40:473
13805992622,3cyclictest331rcuc/321:45:103
13805992619,5cyclictest34-21ksoftirqd/323:59:593
13805992611,3cyclictest34-21ksoftirqd/322:08:393
1380599260,6cyclictest34-21ksoftirqd/300:20:013
13802992615,7cyclictest16074-21sh21:20:280
1380299261,1cyclictest0-21swapper/019:13:080
56982250,2sleep30-21swapper/322:50:083
324272250,2sleep00-21swapper/023:10:030
1380599258,9cyclictest34-21ksoftirqd/322:16:363
1380599258,8cyclictest34-21ksoftirqd/323:21:183
1380599256,9cyclictest34-21ksoftirqd/323:07:053
1380599255,10cyclictest34-21ksoftirqd/300:05:413
1380599252,8cyclictest34-21ksoftirqd/322:20:133
13805992519,4cyclictest34-21ksoftirqd/323:20:003
13805992518,6cyclictest331rcuc/322:28:473
13805992510,8cyclictest34-21ksoftirqd/323:38:373
13804992514,8cyclictest278-21systemd-journal19:31:492
13803992515,8cyclictest8005-21cut21:15:161
13803992510,6cyclictest29794-21grep21:55:161
1380299252,17cyclictest15235-21ssh23:45:120
168322240,1sleep10-21swapper/122:58:061
1380599247,8cyclictest34-21ksoftirqd/320:34:123
1380599246,10cyclictest34-21ksoftirqd/322:11:013
1380599245,8cyclictest0-21swapper/300:26:003
13805992421,2cyclictest331rcuc/323:53:253
13805992417,5cyclictest34-21ksoftirqd/320:20:003
1380599240,23cyclictest0-21swapper/319:12:253
13804992422,1cyclictest0-21swapper/219:12:172
13804992412,8cyclictest18994-21ntpq22:35:232
278692230,2sleep00-21swapper/021:54:030
1380599236,10cyclictest34-21ksoftirqd/300:30:273
1380599234,9cyclictest34-21ksoftirqd/322:40:253
1380599234,8cyclictest34-21ksoftirqd/321:12:263
1380599233,7cyclictest34-21ksoftirqd/321:00:153
1380599232,7cyclictest34-21ksoftirqd/320:40:213
1380599231,3cyclictest34-21ksoftirqd/320:25:223
13805992310,11cyclictest34-21ksoftirqd/321:20:003
1380499233,14cyclictest390-21diskmemload22:50:132
1380499233,10cyclictest390-21diskmemload23:43:562
1380499232,7cyclictest28-21ksoftirqd/222:46:342
1380499231,8cyclictest0-21swapper/221:55:422
13804992316,5cyclictest11536-21grep22:05:232
1380499231,11cyclictest1209-21ssh21:10:132
1380399231,20cyclictest23973-21cat00:15:171
1380299232,15cyclictest13842-21taskset23:20:040
13802992316,5cyclictest24547-21ssh21:27:120
13802992311,7cyclictest28844-21sh21:30:220
31812220,1sleep20-21swapper/222:23:582
13805992217,3cyclictest331rcuc/322:55:463
13805992216,4cyclictest331rcuc/320:38:313
13805992215,5cyclictest34-21ksoftirqd/319:50:003
13805992212,2cyclictest141rcu_preempt19:29:423
1380599220,7cyclictest331rcuc/320:10:003
1380499228,3cyclictest141rcu_preempt20:20:002
13804992216,4cyclictest18720-21sh00:35:222
1380399224,4cyclictest390-21diskmemload21:35:161
1380399221,6cyclictest24202-21munin-run22:40:001
13803992215,5cyclictest24274-21ssh23:51:041
13803992212,6cyclictest4497-21grep22:00:191
13803992211,7cyclictest53050irq/142-enp3s0-23:20:031
1380399221,12cyclictest59862ssh00:01:441
1380299223,12cyclictest15959-21taskset20:30:130
13802992215,5cyclictest29013-21ssh00:19:340
13802992213,6cyclictest278-21systemd-journal19:19:590
132072221,2sleep00-21swapper/022:06:400
1380599217,2cyclictest141rcu_preempt19:37:253
1380599214,10cyclictest0-21swapper/320:01:433
13805992111,2cyclictest141rcu_preempt21:05:053
13805992111,2cyclictest141rcu_preempt19:31:563
13805992110,3cyclictest141rcu_preempt19:23:493
1380599210,9cyclictest34-21ksoftirqd/319:50:143
1380499219,3cyclictest28-21ksoftirqd/221:35:002
1380499211,6cyclictest3809-21ls00:00:192
13804992115,3cyclictest390-21diskmemload00:17:492
13804992113,6cyclictest21727-21ntp_states23:25:202
13804992112,6cyclictest10771-21cron22:30:002
1380499211,14cyclictest21064-21ssh21:25:112
1380499211,13cyclictest24702-21ssh22:40:042
1380499211,11cyclictest21945-21apt-get23:50:132
1380399218,11cyclictest32095-21vmstat22:20:281
1380399214,3cyclictest390-21diskmemload23:19:171
1380399211,6cyclictest16952-21apt-get22:10:101
13803992112,7cyclictest30765-21cut21:05:011
13803992112,7cyclictest2515-21tr23:59:591
13803992112,6cyclictest6081-21users00:25:251
1380399211,18cyclictest28599-21cron19:45:011
1380399210,19cyclictest32002-21seq23:33:071
1380299213,12cyclictest75-21kswapd000:35:120
1380299213,12cyclictest12321-21kworker/u8:022:10:110
13802992115,4cyclictest12-21ksoftirqd/022:00:000
13802992112,6cyclictest278-21systemd-journal21:44:590
13802992110,6cyclictest31698-21sh22:20:240
13805992014,4cyclictest34-21ksoftirqd/320:25:003
13805992010,3cyclictest141rcu_preempt20:51:163
1380499209,3cyclictest254522sh22:15:302
1380499203,4cyclictest28-21ksoftirqd/221:47:492
1380499202,4cyclictest0-21swapper/221:45:002
1380499202,16cyclictest1354-21cron21:54:592
1380499201,6cyclictest0-21swapper/223:11:152
13804992014,3cyclictest390-21diskmemload23:33:532
13804992013,5cyclictest31179-21cat19:50:122
13804992012,6cyclictest560-21dbus-daemon23:55:242
13804992012,6cyclictest28620-21sed21:00:002
13804992012,6cyclictest24921-21cut20:50:152
13804992011,6cyclictest16418-21tune2fs19:15:152
13804992011,6cyclictest10749-21cat00:30:022
1380499201,14cyclictest2065-21taskset21:35:042
1380499201,10cyclictest0-21swapper/222:13:502
1380399206,3cyclictest90262fschecks_time23:40:121
1380399203,4cyclictest390-21diskmemload22:43:171
1380399203,3cyclictest390-21diskmemload23:25:341
1380399203,3cyclictest390-21diskmemload21:54:041
1380399202,13cyclictest27738-21runrttasks21:30:111
1380399201,5cyclictest25378-21users22:15:251
13803992014,4cyclictest53050irq/142-enp3s0-00:30:131
13803992013,5cyclictest16648-21sed22:10:011
13803992013,4cyclictest1221-21gltestperf19:55:141
13803992012,6cyclictest4659-21awk22:25:161
13803992011,7cyclictest15343-21ls23:45:131
13803992011,6cyclictest24507-21cat19:35:001
1380399200,18cyclictest24517-21ssh23:04:031
1380399200,11cyclictest266412/usr/sbin/munin20:55:111
1380299204,9cyclictest856-21ssh21:10:110
13802992013,4cyclictest636-21runrttasks22:31:320
13802992012,6cyclictest4342-21sed22:00:180
13802992012,5cyclictest636-21runrttasks23:53:060
1380299201,16cyclictest1488-21awk23:35:010
104342201,4sleep10-21swapper/120:15:211
1380599191,7cyclictest34-21ksoftirqd/320:10:153
1380599191,6cyclictest34-21ksoftirqd/319:40:223
1380599191,6cyclictest34-21ksoftirqd/319:15:203
13805991911,6cyclictest22492-21expr20:45:143
1380599190,7cyclictest1477-21/usr/sbin/munin19:55:173
1380499197,10cyclictest52750irq/139-enp3s0-22:55:152
1380499193,3cyclictest390-21diskmemload23:39:182
1380499192,3cyclictest390-21diskmemload23:22:362
1380499191,5cyclictest0-21swapper/221:19:422
13804991914,3cyclictest3937-21idleruntime-cro00:25:002
13804991912,5cyclictest9741-21grep00:05:122
13804991912,5cyclictest15717-21basename20:30:012
13804991912,4cyclictest390-21diskmemload22:01:382
13804991911,6cyclictest22183-21date20:45:012
13804991911,6cyclictest18268-21expr19:20:122
13804991911,5cyclictest2226-21who19:55:242
13804991910,7cyclictest5032-21apt-key20:05:002
13804991910,6cyclictest30326-21seq21:00:252
13804991910,6cyclictest12272-21cat20:20:172
1380399191,8cyclictest26944-21ntpq23:05:211
1380399191,7cyclictest18788-21/usr/sbin/munin00:35:221
1380399191,4cyclictest390-21diskmemload21:26:201
13803991913,4cyclictest5208-21sh23:37:031
13803991912,5cyclictest21969-21seq19:27:331
13803991912,5cyclictest18847-21ssh21:23:131
13803991912,5cyclictest12946-21seq22:30:281
13803991911,6cyclictest4450-21seq20:01:071
13803991911,6cyclictest29092-21expr19:45:121
13803991911,6cyclictest25899-21cat19:35:241
13803991911,6cyclictest17082-21which19:15:221
13803991911,5cyclictest6191-21cut20:05:251
13803991910,7cyclictest19880-21seq20:38:561
13803991910,7cyclictest10115-21tune2fs00:05:161
1380299199,7cyclictest1-21systemd00:20:010
1380299192,12cyclictest75-21kswapd022:35:130
1380299192,10cyclictest636-21runrttasks23:25:110
1380299191,6cyclictest0-21swapper/000:25:160
13802991913,3cyclictest390-21diskmemload23:00:140
13802991912,5cyclictest390-21diskmemload22:53:230
13802991912,5cyclictest15866-21sed20:30:000
13802991912,4cyclictest636-21runrttasks00:08:560
13802991912,4cyclictest390-21diskmemload21:36:060
13802991911,6cyclictest6495-21sed20:05:260
13802991911,6cyclictest19917-21seq19:23:400
13802991911,5cyclictest3421-21timerwakeupswit23:35:250
1380299191,14cyclictest4606-21ssh22:25:170
13802991910,7cyclictest28892-21cat21:00:120
13802991910,6cyclictest4353-21grep20:00:260
13802991910,5cyclictest15673-21apt-get21:45:120
1380499181,8cyclictest0-21swapper/223:08:022
1380499181,4cyclictest0-21swapper/222:33:192
13804991812,4cyclictest52750irq/139-enp3s0-23:15:142
13804991811,5cyclictest32637-21seq21:07:292
13804991811,5cyclictest30115-21expr19:45:242
13804991810,6cyclictest23163-21cat20:45:192
1380399189,6cyclictest18616-21gltestperf19:20:141
1380399187,7cyclictest7368-21apt-get20:10:111
1380399187,3cyclictest53050irq/142-enp3s0-23:10:491
1380399181,8cyclictest32281-21rm00:20:541
13803991813,3cyclictest8411-21sh22:51:211
13803991811,5cyclictest20833-21idleruntime20:40:171
13803991811,5cyclictest16880-21ntpq20:30:211
13803991811,4cyclictest0-21swapper/100:10:041
13803991810,6cyclictest32565-21seq19:51:041
13803991810,6cyclictest12106-21perl20:20:161
1380299182,10cyclictest515-21irqbalance19:45:110
1380299181,3cyclictest0-21swapper/022:55:180
13802991812,3cyclictest2615-21sendmail20:00:000
13802991810,6cyclictest31884-21df22:45:140
13802991810,6cyclictest16490-21basename00:10:110
13802991810,5cyclictest22782-21cut19:30:140
13802991810,4cyclictest25948-21cpu23:05:140
1380499179,6cyclictest20592-21cat19:25:122
13804991710,5cyclictest18289-21expr20:35:132
13803991712,3cyclictest30947-21acpi21:05:121
1380399171,12cyclictest22391-21apt-get20:45:131
1380299179,6cyclictest560-21dbus-daemon20:54:590
1380299179,6cyclictest560-21dbus-daemon19:40:010
1380299179,6cyclictest13134-21awk20:20:250
13802991712,3cyclictest7342-21/usr/sbin/munin20:10:140
13802991712,3cyclictest20262-21/usr/sbin/munin19:25:200
13802991710,5cyclictest27810-21grep20:55:230
13802991710,5cyclictest27453-21sed19:40:180
13802991710,5cyclictest19172-21gzip20:35:230
1380499169,5cyclictest24707-21grep19:35:132
1380499168,6cyclictest18039-21cut20:35:002
1380499165,2cyclictest28-21ksoftirqd/219:44:552
1380399169,5cyclictest14074-21expr20:25:121
13803991611,3cyclictest24501-21/usr/sbin/munin20:50:141
1380299169,5cyclictest23098-21sed20:45:180
1380299168,5cyclictest31840-21grep19:50:180
13802991611,3cyclictest636-21runrttasks20:19:000
13802991611,3cyclictest21191-21open_files20:40:220
1380499151,5cyclictest3500-21if_enp3s020:00:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional