You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-06 - 15:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Fri Mar 06, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3390199105101,2cyclictest0-21swapper/012:00:030
3390199832,5cyclictest44375-21sshd09:19:170
33901998316,6cyclictest16850irq/134-800000010:56:390
33901998215,7cyclictest42541-21sed09:15:020
33901998014,61cyclictest16850irq/134-800000011:52:550
3390199773,7cyclictest51314-21munin-run09:35:010
3390199771,58cyclictest125781-21ssh12:21:250
33906997634,37cyclictest275-21dbus-daemon07:20:001
33906997632,40cyclictest78172-21sed10:35:021
33901997619,7cyclictest16850irq/134-800000009:39:440
33901997524,47cyclictest16950irq/224-800000011:56:440
33901997516,54cyclictest16850irq/134-800000010:18:470
3390199741,68cyclictest0-21swapper/011:11:270
3390199741,3cyclictest0-21swapper/009:46:530
33901997327,30cyclictest0-21swapper/012:34:040
33901997317,46cyclictest16850irq/134-800000010:42:230
33901997317,46cyclictest16850irq/134-800000010:42:220
3390199731,68cyclictest86433-21sh10:53:370
3390199731,67cyclictest0-21swapper/010:24:400
33901997226,29cyclictest0-21swapper/011:33:260
33901997221,17cyclictest106667-21ssh11:39:030
33901997221,12cyclictest66436-21ssh10:08:500
33901997221,12cyclictest66436-21ssh10:08:490
33901997217,12cyclictest78947-21ssh10:36:410
3390199713,63cyclictest275-21dbus-daemon08:20:010
33901997120,47cyclictest16950irq/224-800000012:10:010
33901997120,47cyclictest16950irq/224-800000012:10:010
33906997023,42cyclictest3467602-21systemd-journal11:20:001
33901997016,49cyclictest16850irq/134-800000011:25:270
33901997016,49cyclictest16850irq/134-800000011:25:260
33906996915,48cyclictest275-21dbus-daemon12:20:011
33901996924,42cyclictest0-21swapper/010:28:200
3390199691,64cyclictest0-21swapper/009:29:180
3390199691,64cyclictest0-21swapper/009:29:170
33901996827,37cyclictest83428-21sh10:46:340
3390199682,6cyclictest64746-21munin-run10:05:000
3390199681,62cyclictest0-21swapper/012:38:140
3390699671,12cyclictest125901-21sshd12:21:441
33901996720,13cyclictest127884-21ssh12:26:050
33901996720,13cyclictest127884-21ssh12:26:050
3390199671,61cyclictest0-21swapper/012:10:120
3390199671,3cyclictest0-21swapper/011:03:080
33901996713,7cyclictest16850irq/134-800000009:24:310
33906996631,31cyclictest82553-21latency_hist10:45:021
33906996631,31cyclictest82553-21latency_hist10:45:011
33906996623,39cyclictest73713-21wc10:25:011
33901996622,40cyclictest92611-21ssh11:07:160
33906996526,34cyclictest85185-21sh10:50:281
33901996522,39cyclictest97107-21rm11:17:170
33901996520,8cyclictest49090-21kworker/u8:009:51:470
33906996425,35cyclictest94891-21sh11:12:261
3390699641,15cyclictest131702-21munin-run12:35:001
3390699641,12cyclictest89916-21ssh11:01:081
33901996418,14cyclictest62092-21ssh09:59:110
3390699632,12cyclictest121322-21ssh12:11:251
3390699631,13cyclictest47237-21ssh09:25:181
3390699631,13cyclictest47237-21ssh09:25:181
3390699631,11cyclictest63571-21ssh10:02:151
3390199636,16cyclictest69136-21grep10:15:000
33901996330,29cyclictest3467602-21systemd-journal08:00:010
33901996330,29cyclictest3467602-21systemd-journal08:00:010
3390199631,58cyclictest0-21swapper/010:32:440
3390199631,57cyclictest0-21swapper/011:47:500
33901996313,6cyclictest16850irq/134-800000011:40:540
33912996214,12cyclictest37-21ksoftirqd/312:32:053
33906996227,31cyclictest23-21ksoftirqd/109:57:181
3390699622,16cyclictest39850-21modprobe09:05:001
33906996221,37cyclictest117743-21sh12:03:411
3390699621,14cyclictest364-21snmpd10:59:201
3390699621,12cyclictest44963-21ssh09:20:161
3390199623,54cyclictest0-21swapper/007:30:440
3390199622,21cyclictest35432-21munin-plugin-st07:39:590
3391299619,15cyclictest44261-21ssh09:19:013
33912996117,9cyclictest37-21ksoftirqd/312:25:013
33906996123,34cyclictest92200-21sh11:06:171
3390699612,16cyclictest39329-21munin-run08:55:011
3390699612,16cyclictest114555-21ssh11:56:181
3390699612,14cyclictest37009-21/usr/sbin/munin08:10:011
33906996119,38cyclictest40075-21kworker/u8:009:12:581
33901996120,37cyclictest99369-21rm11:22:230
3390699602,18cyclictest110848-21ssh11:48:151
3390699602,14cyclictest364-21snmpd10:27:551
3390699602,13cyclictest48-21kcompactd011:35:051
3390699602,13cyclictest38307-21munin-run08:35:011
33906996020,36cyclictest50459-21sh09:32:541
33906996019,37cyclictest44421-1kworker/u9:009:39:251
3390699601,14cyclictest60133-21sshd09:54:441
33901996017,37cyclictest0-21swapper/007:20:010
3390199601,25cyclictest124293-21ssh12:18:220
33912995911,10cyclictest37-21ksoftirqd/310:26:293
33906995925,31cyclictest23-21ksoftirqd/111:43:031
3390699592,13cyclictest99137-21sh11:21:501
3390699591,16cyclictest78417-21ssh10:35:251
33912995835,17cyclictest3467602-21systemd-journal10:49:443
33910995849,5cyclictest38597-21kworker/2:209:27:272
33910995849,5cyclictest38597-21kworker/2:209:27:272
33910995849,5cyclictest107140-21kworker/2:112:21:522
33906995833,22cyclictest23-21ksoftirqd/111:30:201
3390699582,16cyclictest119143-21ssh12:06:371
3390699582,16cyclictest119143-21ssh12:06:371
3390699581,14cyclictest364-21snmpd07:29:171
3390699581,13cyclictest364-21snmpd07:56:481
3390699581,13cyclictest364-21snmpd07:56:471
33901995836,18cyclictest16850irq/134-800000007:30:000
33912995714,11cyclictest37-21ksoftirqd/310:00:003
33910995748,5cyclictest38597-21kworker/2:210:08:332
33910995748,5cyclictest38597-21kworker/2:210:08:332
33906995733,20cyclictest43681-21sh09:17:391
3390699572,15cyclictest66505-21ssh10:09:001
3390699572,15cyclictest66505-21ssh10:08:591
3390699572,15cyclictest101750-21sh11:27:471
3390699572,15cyclictest101750-21sh11:27:471
3390699572,14cyclictest83313-21ssh10:46:171
3390699572,13cyclictest113455-21ssh11:54:121
33906995716,36cyclictest40114-21diskmemload10:17:291
3390699571,15cyclictest364-21snmpd09:48:081
33901995713,10cyclictest275-21dbus-daemon09:10:000
3391299567,17cyclictest110251-21ssh11:46:423
33910995647,6cyclictest11831-21kworker/2:109:47:382
33910995647,5cyclictest38597-21kworker/2:209:34:552
33910995646,6cyclictest107140-21kworker/2:112:39:312
33910995646,5cyclictest38597-21kworker/2:210:38:572
33910995645,6cyclictest11831-21kworker/2:109:54:372
33910995637,13cyclictest1-21systemd08:10:012
3390699562,15cyclictest132229-21ssh12:36:061
3390699562,14cyclictest38839-21munin-run08:45:001
3390699562,12cyclictest364-21snmpd07:54:271
33906995614,10cyclictest63133-1kworker/u9:010:12:091
3390699561,15cyclictest35186-21munin-run07:35:011
3390699561,13cyclictest127908-21ssh12:26:081
3390699561,13cyclictest127908-21ssh12:26:071
3390699561,12cyclictest39045-21sshd08:46:441
3390199563,14cyclictest38545-21sh08:40:010
3390199563,14cyclictest38545-21sh08:40:000
3390199563,11cyclictest39671-21grep09:00:020
33901995625,25cyclictest275-21dbus-daemon08:45:010
3391299551,33cyclictest0-21swapper/307:55:013
33910995547,5cyclictest11831-21kworker/2:108:30:012
33910995546,6cyclictest38597-21kworker/2:209:17:192
33910995546,5cyclictest38597-21kworker/2:211:40:012
33910995545,7cyclictest38597-21kworker/2:211:29:042
33910995545,7cyclictest38597-21kworker/2:211:29:032
33910995545,6cyclictest72106-21kworker/2:011:08:172
33910995545,6cyclictest38597-21kworker/2:211:01:102
33910995545,5cyclictest38597-21kworker/2:211:30:522
3390699559,4cyclictest0-21swapper/108:21:321
3390699559,4cyclictest0-21swapper/108:21:311
3390699553,8cyclictest364-21snmpd08:13:041
3390699553,8cyclictest364-21snmpd08:13:041
3390699551,16cyclictest54600-21ssh09:42:091
3390199552,16cyclictest39089-21unin-run08:50:010
33901995516,8cyclictest12-21ksoftirqd/008:55:000
33901995516,4cyclictest40114-21diskmemload09:43:360
3391299547,11cyclictest37-21ksoftirqd/311:39:263
33910995445,5cyclictest38597-21kworker/2:210:34:372
33910995444,6cyclictest38597-21kworker/2:211:49:222
33910995444,5cyclictest38597-21kworker/2:209:10:032
33910995435,14cyclictest1-21systemd07:34:592
33906995418,31cyclictest23-21ksoftirqd/108:20:021
3390699541,11cyclictest364-21snmpd07:49:521
3391299537,7cyclictest91929-21ssh11:05:383
3391299536,19cyclictest64513-21ssh10:04:293
33912995331,17cyclictest275-21dbus-daemon09:35:013
33912995310,8cyclictest37-21ksoftirqd/307:15:013
33910995345,5cyclictest38597-21kworker/2:210:55:582
33910995345,4cyclictest38597-21kworker/2:211:55:152
33910995345,4cyclictest107140-21kworker/2:112:12:352
33910995344,6cyclictest38597-21kworker/2:211:43:582
33910995344,5cyclictest11831-21kworker/2:109:42:532
33910995343,7cyclictest38597-21kworker/2:210:13:482
33910995334,11cyclictest34378-21cron07:20:012
3390699532,13cyclictest1-21systemd08:40:011
3390699532,13cyclictest1-21systemd08:40:001
3390699532,12cyclictest35743-21sed07:45:011
3390199533,20cyclictest36198-21cron07:54:590
33912995212,9cyclictest37-21ksoftirqd/311:55:573
33910995244,5cyclictest38597-21kworker/2:211:15:472
33910995244,5cyclictest107140-21kworker/2:112:07:072
33910995244,5cyclictest107140-21kworker/2:112:07:072
33910995243,6cyclictest72106-21kworker/2:010:54:102
33910995243,6cyclictest38597-21kworker/2:210:20:272
33910995243,5cyclictest72106-21kworker/2:010:49:402
33910995243,5cyclictest11831-21kworker/2:108:00:022
33910995243,5cyclictest11831-21kworker/2:108:00:012
33910995242,6cyclictest107140-21kworker/2:112:30:012
33910995242,6cyclictest107140-21kworker/2:112:30:002
3390699521,13cyclictest34682-21unin-run07:25:011
3390199522,17cyclictest37174-21latency_hist08:10:020
3390199522,13cyclictest36759-21unin-run08:05:010
3390199521,5cyclictest39901-21cat09:05:020
33901995215,8cyclictest12-21ksoftirqd/007:25:000
3391299514,15cyclictest42004-21sh09:14:063
33912995116,7cyclictest37-21ksoftirqd/309:42:053
33912995113,8cyclictest0-21swapper/309:23:183
33910995143,5cyclictest107140-21kworker/2:112:15:142
33910995142,5cyclictest38597-21kworker/2:210:43:272
33910995142,5cyclictest38597-21kworker/2:210:43:272
33910995142,5cyclictest38597-21kworker/2:209:21:282
33910995142,5cyclictest11831-21kworker/2:107:50:012
33910995142,5cyclictest11831-21kworker/2:107:30:032
3390699512,13cyclictest35425-21sh07:40:001
3390699511,11cyclictest364-21snmpd08:29:161
33901995147,2cyclictest0-21swapper/008:10:020
33901995147,2cyclictest0-21swapper/008:10:010
3390199513,43cyclictest0-21swapper/008:30:020
3390199513,14cyclictest37783-21modprobe08:25:000
3390199513,14cyclictest37783-21modprobe08:24:590
3390199512,10cyclictest0-21swapper/007:15:020
33912995015,6cyclictest361rcuc/311:23:153
33910995042,5cyclictest38597-21kworker/2:211:13:072
33910995042,5cyclictest107140-21kworker/2:112:33:292
33910995041,6cyclictest38597-21kworker/2:210:26:592
33910995041,5cyclictest38597-21kworker/2:210:01:022
33910995041,5cyclictest107140-21kworker/2:112:05:022
33910995040,7cyclictest38597-21kworker/2:211:51:412
3390699501,13cyclictest364-21snmpd09:07:381
3390699501,11cyclictest34131-21munin-run07:15:001
3390699501,11cyclictest34011-21date07:10:021
33901995012,9cyclictest12-21ksoftirqd/007:50:010
3391299495,7cyclictest37-21ksoftirqd/312:26:103
3391299495,7cyclictest37-21ksoftirqd/312:26:103
33912994910,9cyclictest0-21swapper/311:54:053
33910994941,5cyclictest11831-21kworker/2:108:15:012
33910994941,5cyclictest11831-21kworker/2:108:15:012
33910994940,5cyclictest38597-21kworker/2:211:21:092
33910994940,5cyclictest38597-21kworker/2:210:18:022
33910994940,5cyclictest38597-21kworker/2:209:36:022
33910994932,12cyclictest1-21systemd08:45:012
3390699492,5cyclictest39573-21modprobe09:00:001
33901994919,9cyclictest12-21ksoftirqd/008:35:010
3391299481,13cyclictest68806-21sh10:14:053
33910994840,5cyclictest38597-21kworker/2:209:11:592
33910994839,5cyclictest11831-21kworker/2:109:57:292
3390199481,24cyclictest1-21systemd07:40:440
3391299479,7cyclictest0-21swapper/310:43:203
3391299479,7cyclictest0-21swapper/310:43:203
3391299479,5cyclictest131rcu_preempt10:06:493
3391299479,5cyclictest131rcu_preempt10:06:493
3391299476,10cyclictest37-21ksoftirqd/310:37:193
33912994739,5cyclictest361rcuc/310:59:003
33912994732,11cyclictest34378-21cron07:20:013
33912994714,8cyclictest0-21swapper/311:12:123
3391299464,5cyclictest37-21ksoftirqd/310:20:263
33912994633,10cyclictest53546-21systemctl09:40:013
33912994612,7cyclictest0-21swapper/311:34:343
33912994612,5cyclictest0-21swapper/309:25:173
33912994612,5cyclictest0-21swapper/309:25:173
33912994611,7cyclictest0-21swapper/310:53:253
33910994639,4cyclictest38597-21kworker/2:208:56:452
33910994638,5cyclictest11831-21kworker/2:107:43:572
3390699462,11cyclictest36767-21grep08:05:011
3391299458,11cyclictest37-21ksoftirqd/311:16:193
3391299455,5cyclictest131rcu_preempt11:26:373
3391299455,5cyclictest131rcu_preempt11:26:373
33912994528,13cyclictest77420-21ssh10:33:273
33912994512,6cyclictest0-21swapper/311:03:113
33912994510,9cyclictest0-21swapper/310:18:213
33910994538,4cyclictest38597-21kworker/2:208:51:402
33910994537,5cyclictest11831-21kworker/2:107:38:152
3391299448,8cyclictest37-21ksoftirqd/311:40:273
3391299448,7cyclictest37-21ksoftirqd/312:35:553
3391299447,6cyclictest37-21ksoftirqd/309:45:503
3391299445,8cyclictest37-21ksoftirqd/312:06:103
3391299445,8cyclictest37-21ksoftirqd/312:06:103
3391299436,10cyclictest37-21ksoftirqd/312:12:403
33912994311,7cyclictest37-21ksoftirqd/312:01:313
33910994321,16cyclictest275-21dbus-daemon08:40:012
33910994321,16cyclictest275-21dbus-daemon08:40:012
33912994231,8cyclictest60383-21latency_hist09:55:023
33912994230,7cyclictest309-21in:imuxsock08:00:013
33912994230,7cyclictest309-21in:imuxsock08:00:013
33912994227,10cyclictest35172-21munin-plugin-st07:35:003
33910994234,5cyclictest38597-21kworker/2:209:05:022
33910994234,5cyclictest11831-21kworker/2:108:01:142
33910994234,5cyclictest11831-21kworker/2:107:25:002
33910994234,5cyclictest11831-21kworker/2:107:15:022
33910994233,6cyclictest11831-21kworker/2:108:50:002
33910994226,11cyclictest275-21dbus-daemon08:35:012
33912994129,9cyclictest275-21dbus-daemon09:00:013
33912994126,11cyclictest309-21in:imuxsock08:20:013
33910994132,5cyclictest11831-21kworker/2:108:25:002
33910994132,5cyclictest11831-21kworker/2:108:25:002
33912994022,12cyclictest35421-21modprobe07:40:003
33910994035,3cyclictest11831-21kworker/2:107:53:292
3391299396,8cyclictest37-21ksoftirqd/307:20:453
3391299393,7cyclictest37-21ksoftirqd/312:16:153
3391299386,7cyclictest0-21swapper/307:30:003
33910993829,6cyclictest11831-21kworker/2:108:20:012
3391299375,9cyclictest37-21ksoftirqd/308:27:443
33912993724,9cyclictest38509-21sshd08:35:443
33912993724,9cyclictest38509-21sshd08:35:443
33912993721,12cyclictest1-21systemd09:10:013
33912993620,13cyclictest38273-21sshd08:34:043
33912993525,6cyclictest39315-21cron08:55:003
33912993523,9cyclictest35678-21sshd07:42:443
33912993520,12cyclictest37753-21sshd08:20:453
33912993520,12cyclictest37753-21sshd08:20:443
33912993520,11cyclictest40039-21wc09:05:023
33912993520,10cyclictest37214-21sshd08:10:443
33912993520,10cyclictest37214-21sshd08:10:433
33912993422,9cyclictest37-21ksoftirqd/308:50:013
33912993317,12cyclictest3467602-21systemd-journal08:01:433
33912993314,15cyclictest364-21snmpd08:07:253
33912993123,5cyclictest38822-21latency_hist08:45:013
3391299291,8cyclictest0-21swapper/307:49:103
33901992917,9cyclictest33981-21ls07:10:020
33912992816,9cyclictest33997-21cat07:10:023
3391099289,15cyclictest33940-21wc07:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional