You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 16:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri May 08, 2026 12:45:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276197999921,88cyclictest0-21swapper/008:40:020
276197999921,88cyclictest0-21swapper/008:40:010
2761992998429,25cyclictest2846527-21sh11:52:143
276199299831,50cyclictest2863520-21ssh12:27:293
276198899822,66cyclictest2789999-21rm09:55:022
276199299812,57cyclictest285254350ssh12:05:013
276197999791,4cyclictest0-21swapper/009:20:030
276199299782,43cyclictest2816897-21ssh10:50:383
2761984997815,12cyclictest0-21swapper/112:40:021
276199299772,42cyclictest2837999-21ssh11:34:543
2761992997226,40cyclictest275-21dbus-daemon12:00:013
2761992997226,40cyclictest275-21dbus-daemon12:00:003
276199299711,45cyclictest2785453-21ssh09:45:223
276199299711,45cyclictest2785453-21ssh09:45:213
276197999717,7cyclictest12-21ksoftirqd/010:12:140
276199299691,42cyclictest2860107-21ssh12:20:113
276199299691,42cyclictest2860107-21ssh12:20:103
2761992996827,36cyclictest2808393-21sh10:33:203
2761992996822,42cyclictest37-21ksoftirqd/312:30:323
2761992996822,42cyclictest37-21ksoftirqd/312:30:313
276199299671,62cyclictest2795132-21ssh10:05:323
2761979996746,16cyclictest2814055-21latency_hist10:45:020
276199299662,35cyclictest2856430-21ssh12:12:493
2761992996622,40cyclictest2869427-21run-parts12:40:003
276199299661,60cyclictest2805529-21sshd10:27:153
276199299661,60cyclictest2805529-21sshd10:27:143
276198899662,10cyclictest30-21ksoftirqd/212:40:012
2761992996522,40cyclictest37-21ksoftirqd/311:39:573
276199299651,60cyclictest2810951-21ssh10:38:373
276199299651,60cyclictest2791990-21ssh09:59:253
276199299651,60cyclictest2791990-21ssh09:59:243
2761979996533,28cyclictest12-21ksoftirqd/010:30:390
2761992996427,34cyclictest2843579-21sh11:46:073
2761992996427,34cyclictest2843579-21sh11:46:063
276199299642,58cyclictest2827549-21ssh11:12:593
276199299642,57cyclictest2832309-21ssh11:22:553
2761992996421,38cyclictest2789152-21sh09:53:243
276199299641,5cyclictest2803891-21ssh10:23:563
276199299641,59cyclictest2773522-21ssh09:20:463
276199299641,58cyclictest2841504-21ssh11:41:473
276199299641,58cyclictest2776697-21ssh09:27:283
2761988996423,10cyclictest30-21ksoftirqd/209:30:012
2761992996319,40cyclictest37-21ksoftirqd/310:56:033
276199299631,58cyclictest2852973-21ssh12:05:293
276199299631,58cyclictest2821764-21ssh11:00:383
276199299631,58cyclictest2800716-21ssh10:17:243
276199299631,58cyclictest2793553-21ssh10:02:243
276199299631,58cyclictest2780943-21ssh09:36:133
276199299631,58cyclictest2779693-21ssh09:33:553
276199299631,58cyclictest2768820-21ssh09:11:093
276199299631,58cyclictest2768820-21ssh09:11:083
276199299631,4cyclictest2813115-21ssh10:42:543
276198499638,8cyclictest23-21ksoftirqd/110:15:021
2761984996342,10cyclictest23-21ksoftirqd/110:05:001
2761984996315,12cyclictest23-21ksoftirqd/110:46:021
2761984996315,12cyclictest23-21ksoftirqd/110:46:011
276199299622,55cyclictest2765304-21munin-run08:15:013
276199299622,55cyclictest2763232-21munin-run07:35:003
276199299621,8cyclictest2783752-21ssh09:42:023
276199299621,57cyclictest2858464-21ssh12:17:073
276199299621,57cyclictest2828722-21ssh11:15:093
276199299621,57cyclictest2824335-21ssh11:06:053
276199299621,57cyclictest2824335-21ssh11:06:043
276199299621,57cyclictest2814715-21ssh10:46:043
276199299621,57cyclictest2814715-21ssh10:46:033
276199299621,57cyclictest2771077-21ssh09:15:563
276199299621,56cyclictest2833892-21ssh11:25:533
276199299621,55cyclictest2766339-21munin-run08:35:013
276198499629,9cyclictest23-21ksoftirqd/110:21:251
2761979996222,9cyclictest2785097-21ls09:45:020
2761979996213,44cyclictest16850irq/134-800000012:30:010
276199299614,50cyclictest1-21systemd08:05:013
276199299612,55cyclictest2798986-21ssh10:13:543
276199299612,54cyclictest2766599-21munin-run08:40:003
276199299612,54cyclictest2766599-21munin-run08:40:003
276199299612,54cyclictest2763715-21sshd07:40:143
276199299612,54cyclictest2762980-21munin-run07:30:023
276199299612,54cyclictest2762980-21munin-run07:30:013
2761979996148,10cyclictest3467602-21systemd-journal11:25:020
276199299601,55cyclictest2767597-21sshd08:56:143
276199299591,53cyclictest2768124-21sshd09:05:143
276199299591,53cyclictest2764169-21cat07:50:023
276198899598,8cyclictest30-21ksoftirqd/210:12:442
2761988995915,10cyclictest2833422-21ssh11:25:012
2761988995912,18cyclictest2804347-21cron10:24:592
2761979995920,34cyclictest16850irq/134-800000009:35:020
276199299582,51cyclictest2764473-21sshd07:55:153
276199299582,51cyclictest2764473-21sshd07:55:143
276199299581,4cyclictest2762405-21sshd07:17:143
2761988995815,8cyclictest30-21ksoftirqd/207:40:012
2761988995814,8cyclictest30-21ksoftirqd/211:13:242
276197999588,5cyclictest131rcu_preempt11:48:480
276197999588,5cyclictest131rcu_preempt11:48:470
2761979995818,31cyclictest16850irq/134-800000012:10:280
276199299572,50cyclictest2765012-21sshd08:07:143
276199299571,51cyclictest48-21kcompactd008:20:453
276199299571,51cyclictest2767126-21munin-run08:50:003
276199299571,51cyclictest2767126-21munin-run08:50:003
276199299571,51cyclictest2765506-21sshd08:15:063
276198899577,11cyclictest2854336-21ssh12:08:372
276198499578,7cyclictest23-21ksoftirqd/109:19:591
276198499577,10cyclictest23-21ksoftirqd/111:55:021
276198499577,10cyclictest23-21ksoftirqd/111:55:021
2761979995719,30cyclictest16850irq/134-800000010:19:180
276199299563,49cyclictest364-21snmpd07:37:013
276199299563,44cyclictest275-21dbus-daemon09:05:023
276199299563,44cyclictest275-21dbus-daemon09:05:013
2761988995614,8cyclictest30-21ksoftirqd/211:19:292
276198499566,9cyclictest23-21ksoftirqd/110:41:341
276198499563,15cyclictest23-21ksoftirqd/109:52:141
2761984995610,6cyclictest131rcu_preempt12:26:541
2761984995610,5cyclictest131rcu_preempt09:24:591
2761979995628,12cyclictest2790439-21ssh09:55:510
2761979995628,12cyclictest2790439-21ssh09:55:500
2761979995627,14cyclictest2861212-21ssh12:22:460
2761979995627,14cyclictest2861212-21ssh12:22:450
2761979995627,14cyclictest2777104-21ssh09:28:280
276199299552,49cyclictest2767376-21munin-run08:55:013
276199299551,49cyclictest48-21kcompactd008:41:183
2761988995512,13cyclictest30-21ksoftirqd/212:15:022
276198499559,5cyclictest131rcu_preempt12:19:291
276198499557,8cyclictest23-21ksoftirqd/111:53:281
276198499556,6cyclictest131rcu_preempt11:05:301
276198499556,6cyclictest131rcu_preempt11:05:291
2761984995512,6cyclictest131rcu_preempt12:12:081
2761979995546,6cyclictest12-21ksoftirqd/012:05:390
2761979995535,15cyclictest3467602-21systemd-journal08:45:000
2761979995528,12cyclictest2866755-21ssh12:34:240
2761979995528,12cyclictest2866755-21ssh12:34:230
2761979995528,11cyclictest2803441-21ssh10:22:570
2761979995527,13cyclictest2832946-21ssh11:24:140
2761979995527,13cyclictest2789580-21ssh09:54:270
2761979995522,30cyclictest12-21ksoftirqd/010:30:010
2761979995522,30cyclictest12-21ksoftirqd/010:30:010
2761979995511,5cyclictest131rcu_preempt11:18:190
276199299542,49cyclictest364-21snmpd07:51:133
276198899546,10cyclictest30-21ksoftirqd/211:01:152
2761988995418,8cyclictest30-21ksoftirqd/209:34:072
276198499548,9cyclictest23-21ksoftirqd/112:23:091
276198499548,9cyclictest23-21ksoftirqd/112:23:081
276198499547,9cyclictest23-21ksoftirqd/110:39:301
276198499545,9cyclictest23-21ksoftirqd/111:39:141
276198499545,12cyclictest23-21ksoftirqd/111:15:131
276198499544,10cyclictest23-21ksoftirqd/109:57:331
276198499544,10cyclictest23-21ksoftirqd/109:57:321
2761984995411,5cyclictest131rcu_preempt09:38:391
2761979995444,7cyclictest12-21ksoftirqd/010:36:430
2761979995424,14cyclictest2815170-21ssh10:47:020
2761979995424,14cyclictest2815170-21ssh10:47:010
2761979995419,31cyclictest16850irq/134-800000010:04:340
2761979995418,32cyclictest16850irq/134-800000011:36:040
2761979995418,32cyclictest16850irq/134-800000011:32:320
2761979995418,31cyclictest16850irq/134-800000012:04:580
2761979995417,32cyclictest16850irq/134-800000011:10:270
2761979995417,32cyclictest16850irq/134-800000009:47:040
2761979995417,32cyclictest16850irq/134-800000009:47:030
276197999541,47cyclictest2768353-21ssh09:10:020
276199299531,47cyclictest2766093-21systemd-run08:30:003
276198899534,8cyclictest30-21ksoftirqd/212:27:022
276198899534,10cyclictest30-21ksoftirqd/209:50:012
276198899534,10cyclictest30-21ksoftirqd/209:50:012
2761988995314,8cyclictest30-21ksoftirqd/211:46:382
2761988995314,8cyclictest30-21ksoftirqd/211:46:382
276198499537,6cyclictest131rcu_preempt11:22:341
276198499536,8cyclictest23-21ksoftirqd/110:08:001
276198499534,9cyclictest23-21ksoftirqd/109:47:381
276198499534,9cyclictest23-21ksoftirqd/109:47:371
2761984995312,6cyclictest131rcu_preempt09:27:401
2761984995312,5cyclictest131rcu_preempt12:33:171
2761984995312,5cyclictest131rcu_preempt12:33:171
2761979995325,13cyclictest2817341-21ssh10:51:350
2761979995318,31cyclictest16850irq/134-800000009:12:220
2761979995318,31cyclictest16850irq/134-800000009:12:220
2761979995317,31cyclictest16850irq/134-800000009:21:240
276199299521,46cyclictest364-21snmpd07:13:583
276199299521,46cyclictest364-21snmpd07:13:573
276198899529,8cyclictest30-21ksoftirqd/209:38:252
276198899525,10cyclictest30-21ksoftirqd/212:02:252
2761988995214,5cyclictest131rcu_preempt11:40:422
276198499527,9cyclictest23-21ksoftirqd/110:28:121
276198499527,9cyclictest23-21ksoftirqd/110:28:111
276198499525,8cyclictest23-21ksoftirqd/109:42:401
276198499523,9cyclictest23-21ksoftirqd/111:34:241
2761984995216,6cyclictest131rcu_preempt07:50:001
2761984995211,5cyclictest131rcu_preempt09:11:131
2761984995211,5cyclictest131rcu_preempt09:11:131
2761984995210,5cyclictest131rcu_preempt11:02:371
276197999529,5cyclictest131rcu_preempt10:57:270
2761979995216,32cyclictest16850irq/134-800000012:18:440
2761979995216,32cyclictest16850irq/134-800000010:07:470
2761979995215,33cyclictest16850irq/134-800000009:35:030
2761979995215,32cyclictest16850irq/134-800000012:37:020
276199299514,37cyclictest3467602-21systemd-journal07:20:033
276198899516,5cyclictest30-21ksoftirqd/210:56:402
276198899515,5cyclictest30-21ksoftirqd/212:16:322
276198899514,7cyclictest30-21ksoftirqd/210:31:002
276198899514,10cyclictest30-21ksoftirqd/211:09:182
276198899514,10cyclictest30-21ksoftirqd/211:09:172
276198899511,18cyclictest2847715-21munin-run11:55:012
276198499518,5cyclictest131rcu_preempt11:44:061
276198499518,5cyclictest131rcu_preempt10:58:271
276198499514,7cyclictest23-21ksoftirqd/111:10:291
2761984995142,6cyclictest221rcuc/108:08:151
276198499513,11cyclictest23-21ksoftirqd/110:17:431
2761984995113,5cyclictest131rcu_preempt09:32:581
2761984995110,6cyclictest131rcu_preempt10:33:591
2761979995117,31cyclictest16850irq/134-800000011:44:190
2761979995117,19cyclictest16850irq/134-800000011:08:180
2761979995117,19cyclictest16850irq/134-800000011:08:180
2761979995110,37cyclictest309-21in:imuxsock08:16:590
276198899508,10cyclictest30-21ksoftirqd/210:08:042
276198899505,11cyclictest2767049-21grep08:45:022
2761988995042,5cyclictest2848268-21kworker/2:011:57:042
2761988995042,5cyclictest2848268-21kworker/2:011:57:032
2761988995042,5cyclictest2809171-21kworker/2:110:52:562
276198499509,6cyclictest131rcu_preempt12:05:161
276198499509,5cyclictest131rcu_preempt12:01:121
276198499508,5cyclictest131rcu_preempt08:45:031
276198499507,10cyclictest23-21ksoftirqd/111:27:061
276198499505,8cyclictest23-21ksoftirqd/111:46:461
276198499505,8cyclictest23-21ksoftirqd/111:46:461
276198499501,12cyclictest23-21ksoftirqd/110:52:041
2761979995016,30cyclictest16850irq/134-800000011:52:100
2761979995013,6cyclictest131rcu_preempt07:50:040
2761979995013,32cyclictest16850irq/134-800000011:55:050
2761979995013,32cyclictest16850irq/134-800000011:55:040
276198899498,9cyclictest30-21ksoftirqd/210:43:592
276198899496,9cyclictest30-21ksoftirqd/209:23:032
276198899495,6cyclictest30-21ksoftirqd/209:15:522
276198899495,5cyclictest30-21ksoftirqd/211:38:002
276198899494,8cyclictest30-21ksoftirqd/210:27:512
276198899494,8cyclictest30-21ksoftirqd/210:27:512
2761988994939,6cyclictest2766354-21kworker/2:310:00:002
2761988994939,6cyclictest2766354-21kworker/2:310:00:002
276198899492,12cyclictest30-21ksoftirqd/211:30:022
276198499493,7cyclictest23-21ksoftirqd/108:20:021
2761984994932,13cyclictest3467602-21systemd-journal08:25:011
2761984994911,5cyclictest131rcu_preempt08:46:141
2761984994911,5cyclictest131rcu_preempt08:46:141
2761984994910,6cyclictest131rcu_preempt08:00:141
2761979994911,33cyclictest16850irq/134-800000011:00:160
276198899484,8cyclictest30-21ksoftirqd/208:30:012
276198899484,11cyclictest30-21ksoftirqd/210:01:172
276198899483,7cyclictest30-21ksoftirqd/210:48:492
276198899483,7cyclictest30-21ksoftirqd/210:48:492
276198899482,13cyclictest30-21ksoftirqd/209:14:512
276198899482,13cyclictest30-21ksoftirqd/209:14:512
2761988994810,8cyclictest30-21ksoftirqd/210:35:252
276198499486,8cyclictest23-21ksoftirqd/108:35:011
276198499484,8cyclictest23-21ksoftirqd/109:00:001
2761979994813,6cyclictest131rcu_preempt08:24:140
276198899478,9cyclictest30-21ksoftirqd/209:41:182
276198899476,6cyclictest30-21ksoftirqd/211:30:522
276198899475,9cyclictest30-21ksoftirqd/212:33:292
276198899475,9cyclictest30-21ksoftirqd/212:33:292
276198899475,7cyclictest30-21ksoftirqd/207:23:592
276198899474,11cyclictest30-21ksoftirqd/209:10:012
276198499476,10cyclictest23-21ksoftirqd/109:09:331
2761984994730,13cyclictest2763457-21cron07:40:001
276198499472,13cyclictest23-21ksoftirqd/108:00:011
276198499472,13cyclictest23-21ksoftirqd/108:00:001
276197999478,7cyclictest12-21ksoftirqd/007:25:020
2761979994733,11cyclictest12-21ksoftirqd/009:05:030
2761979994733,11cyclictest12-21ksoftirqd/009:05:020
276198899465,9cyclictest30-21ksoftirqd/208:10:012
2761988994636,6cyclictest2765821-21cron08:25:002
2761988994610,5cyclictest131rcu_preempt10:16:052
276198499465,8cyclictest23-21ksoftirqd/107:15:001
276198499465,8cyclictest23-21ksoftirqd/107:15:001
2761984994610,5cyclictest131rcu_preempt07:45:021
2761979994613,6cyclictest131rcu_preempt08:30:000
276199299452,36cyclictest2762109-21latency_hist07:10:023
276198899457,9cyclictest0-21swapper/209:00:012
2761988994531,11cyclictest30-21ksoftirqd/208:39:042
2761988994531,11cyclictest30-21ksoftirqd/208:39:032
276198899451,7cyclictest2860227-21ssh12:20:342
276198899451,7cyclictest2860227-21ssh12:20:342
276198499455,9cyclictest23-21ksoftirqd/107:33:291
276198899447,8cyclictest30-21ksoftirqd/208:35:002
276198899446,8cyclictest30-21ksoftirqd/207:29:142
276198899446,8cyclictest30-21ksoftirqd/207:29:132
276198899441,13cyclictest30-21ksoftirqd/207:35:012
276198499449,5cyclictest131rcu_preempt08:55:021
2761984994429,12cyclictest23-21ksoftirqd/107:19:591
276197999445,11cyclictest12-21ksoftirqd/008:00:000
276197999445,11cyclictest12-21ksoftirqd/008:00:000
2761979994437,4cyclictest16950irq/224-800000009:00:000
2761979994429,11cyclictest275-21dbus-daemon08:50:020
2761979994429,11cyclictest275-21dbus-daemon08:50:010
276198899435,7cyclictest30-21ksoftirqd/207:50:002
2761988994312,7cyclictest30-21ksoftirqd/207:18:222
276198499438,5cyclictest131rcu_preempt07:25:031
2761984994327,12cyclictest3467602-21systemd-journal07:55:001
276198499430,8cyclictest150-21pr/ttyS207:27:371
276198499430,8cyclictest150-21pr/ttyS207:27:371
2761979994336,4cyclictest16950irq/224-800000007:40:000
276197999431,38cyclictest12-21ksoftirqd/008:55:010
276198899421,34cyclictest2764476-21kworker/u8:108:05:012
276198499427,6cyclictest131rcu_preempt08:12:141
276198499421,5cyclictest150-21pr/ttyS208:26:491
2761979994234,6cyclictest12-21ksoftirqd/007:35:000
276198899412,14cyclictest30-21ksoftirqd/207:55:542
276198899412,14cyclictest30-21ksoftirqd/207:55:532
276198499418,5cyclictest131rcu_preempt08:35:081
276198499418,5cyclictest131rcu_preempt08:35:071
276197999416,5cyclictest131rcu_preempt08:15:030
276197999416,5cyclictest12-21ksoftirqd/007:30:020
276197999416,5cyclictest12-21ksoftirqd/007:30:010
276198899409,7cyclictest0-21swapper/207:54:012
2761988994028,9cyclictest1-21systemd07:15:002
2761988994028,9cyclictest1-21systemd07:15:002
276198899401,9cyclictest30-21ksoftirqd/208:55:012
276198899401,11cyclictest2768078-21date09:05:022
276198899401,11cyclictest2768078-21date09:05:022
276198499407,7cyclictest23-21ksoftirqd/109:02:471
276198499407,7cyclictest23-21ksoftirqd/109:02:461
276197999399,6cyclictest131rcu_preempt07:50:020
2761979993933,4cyclictest16950irq/224-800000008:35:020
2761979993929,7cyclictest12-21ksoftirqd/007:20:000
2761979993929,7cyclictest12-21ksoftirqd/007:15:000
2761979993929,7cyclictest12-21ksoftirqd/007:15:000
2761979993927,9cyclictest12-21ksoftirqd/007:45:020
276197999391,8cyclictest0-21swapper/008:05:200
2761979993829,7cyclictest151rcuc/007:10:010
2761988993726,8cyclictest30-21ksoftirqd/207:45:012
2761979993725,9cyclictest12-21ksoftirqd/008:05:020
276198899364,6cyclictest30-21ksoftirqd/208:15:022
2761988993423,7cyclictest30-21ksoftirqd/208:45:542
2761988993423,7cyclictest30-21ksoftirqd/208:45:542
276198899339,5cyclictest131rcu_preempt07:10:022
276198899336,8cyclictest30-21ksoftirqd/208:16:352
2761984992821,4cyclictest2761917-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional