You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 00:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Jun 29, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2736051991021,85cyclictest0-21swapper/011:25:020
2736059999372,16cyclictest30-21ksoftirqd/210:40:022
2736059999372,16cyclictest30-21ksoftirqd/210:40:012
2736051998071,3cyclictest150-21pr/ttyS209:05:000
2736059997918,45cyclictest2739598-21kworker/u8:108:20:032
2736055997726,25cyclictest23-21ksoftirqd/107:25:021
273605199711,49cyclictest2807729-21kworker/u8:311:30:000
2736055997011,54cyclictest1-21systemd09:10:001
2736051997011,37cyclictest28151481ssh11:41:410
2736055996837,27cyclictest1-21systemd09:35:011
273606199673,35cyclictest2831085-21ssh12:15:013
273605199671,5cyclictest131rcu_preempt09:40:010
273605199671,5cyclictest131rcu_preempt09:40:000
2736055996626,36cyclictest2760071-21sh09:46:491
2736055996626,36cyclictest2760071-21sh09:46:481
2736055996527,34cyclictest2800741-21sh11:11:401
273605599652,18cyclictest2780642-21munin-run10:30:011
273605599651,14cyclictest2804817-21ssh11:20:061
273605599642,7cyclictest2830921-21munin-run12:15:011
273605599631,9cyclictest2746793-21ssh09:19:261
2736055996318,41cyclictest23-21ksoftirqd/111:49:021
273605599631,10cyclictest2789708-21ssh10:48:541
273605599631,10cyclictest2789708-21ssh10:48:541
273605599622,18cyclictest2808562-21ssh11:28:121
273605599622,13cyclictest2838914-21ssh12:31:241
273605599622,13cyclictest2838914-21ssh12:31:231
2736055996221,37cyclictest2810376-21rm11:31:501
273605599621,9cyclictest2786243-21ssh10:41:211
273605599621,9cyclictest2750344-21ssh09:26:311
273605599621,9cyclictest2750344-21ssh09:26:311
273605599621,8cyclictest2825673-21ssh12:03:591
273605599621,7cyclictest2842571-21ssh12:39:241
273605599621,7cyclictest2828279-21ssh12:09:271
273605599621,7cyclictest2757712-21ssh09:41:521
2736051996229,4cyclictest131rcu_preempt09:58:320
273605199621,29cyclictest27970681ssh11:04:220
2736059996150,7cyclictest30-21ksoftirqd/210:40:032
273605599612,7cyclictest2795696-21ssh11:01:001
273605599612,7cyclictest2769084-21ssh10:05:271
273605599612,7cyclictest2737819-21munin-run07:45:011
2736055996121,35cyclictest2742223-21diskmemload12:16:241
273605599611,9cyclictest2772613-21ssh10:13:031
273605599611,8cyclictest2837887-21ssh12:29:351
273605599611,8cyclictest2833796-21ssh12:20:311
273605599611,8cyclictest2833796-21ssh12:20:311
273605599611,8cyclictest2812340-21ssh11:35:441
273605599611,8cyclictest2778097-21ssh10:24:441
273605599611,7cyclictest2804088-21ssh11:18:561
273605599611,7cyclictest2797839-21ssh11:05:301
273605599611,7cyclictest2791971-21ssh10:53:381
273605599611,7cyclictest2785218-21ssh10:39:361
273605599611,7cyclictest2785218-21ssh10:39:361
273605599611,7cyclictest2766693-21ssh10:00:291
273605599611,14cyclictest2755835-21ssh09:38:111
273605599611,14cyclictest2755835-21ssh09:38:101
2736051996127,29cyclictest2788159-21sh10:45:220
2736051996127,29cyclictest2788159-21sh10:45:220
2736061996014,9cyclictest37-21ksoftirqd/310:44:073
2736061996013,10cyclictest37-21ksoftirqd/309:55:493
273605599602,7cyclictest2792867-21ssh10:55:091
273605599602,16cyclictest3467602-21systemd-journal07:10:211
273605599601,9cyclictest2773866-21ssh10:15:331
273605599601,9cyclictest2764288-21ssh09:55:331
273605599601,8cyclictest2743135-21ssh09:11:331
273605599601,7cyclictest2822805-21ssh11:57:561
273605599601,7cyclictest2780963-21ssh10:30:171
273605599601,6cyclictest2820563-21ssh11:53:161
273605599601,6cyclictest2820563-21ssh11:53:151
273605599601,6cyclictest2814590-21ssh11:40:311
273605599601,6cyclictest2764007-21ssh09:55:031
2736051996041,4cyclictest16850irq/134-800000010:16:590
2736061995922,9cyclictest131rcu_preempt09:43:513
273605599591,8cyclictest2747813-21ssh09:21:101
273605199599,12cyclictest12-21ksoftirqd/010:04:570
2736051995942,3cyclictest150-21pr/ttyS208:15:010
273605199593,32cyclictest131rcu_preempt07:40:010
2736051995929,13cyclictest2786436-21ssh10:41:460
2736061995812,7cyclictest0-21swapper/310:05:453
2736059995847,7cyclictest2741992-21kworker/2:209:46:052
2736059995847,7cyclictest2741992-21kworker/2:209:46:052
2736055995818,36cyclictest23-21ksoftirqd/107:29:261
2736055995818,36cyclictest23-21ksoftirqd/107:29:251
2736051995823,31cyclictest12-21ksoftirqd/012:13:060
2736051995820,11cyclictest12-21ksoftirqd/011:35:210
273605199581,41cyclictest2779509-21ssh10:27:240
273605199581,40cyclictest2790892-21ssh10:51:060
2736055995727,27cyclictest23-21ksoftirqd/107:55:011
273605599572,14cyclictest2737287-21munin-run07:35:001
273605599571,13cyclictest2739660-21munin-run08:20:021
273605199578,37cyclictest19350irq/34-mmc107:35:010
2736051995753,2cyclictest0-21swapper/007:25:010
273605199572,10cyclictest12-21ksoftirqd/009:20:000
2736059995646,7cyclictest2741992-21kworker/2:211:01:192
2736059995646,6cyclictest2809351-21kworker/2:012:26:392
2736059995646,6cyclictest2809351-21kworker/2:012:15:482
273605599562,6cyclictest2739900-21munin-run08:25:001
273605599562,16cyclictest2738070-21munin-run07:50:001
2736055995618,33cyclictest23-21ksoftirqd/107:20:001
2736051995628,14cyclictest2758535-21ssh09:43:460
2736051995628,13cyclictest2776831-21ssh10:21:470
273605199562,38cyclictest2781003-21ssh10:30:210
2736051995621,31cyclictest0-21swapper/011:49:350
2736061995538,12cyclictest275-21dbus-daemon09:40:003
2736061995538,12cyclictest275-21dbus-daemon09:40:003
2736061995521,10cyclictest37-21ksoftirqd/310:16:023
2736061995510,9cyclictest37-21ksoftirqd/311:10:003
273605599552,5cyclictest2738865-21munin-run08:05:011
273605599552,16cyclictest2737579-21munin-run07:40:001
273605599551,11cyclictest2739117-21munin-run08:10:001
273605199556,33cyclictest12-21ksoftirqd/009:54:300
2736051995528,12cyclictest2823020-21ssh11:58:230
2736061995415,9cyclictest37-21ksoftirqd/309:45:073
2736061995415,9cyclictest37-21ksoftirqd/309:45:073
2736061995412,8cyclictest0-21swapper/311:18:033
2736059995444,6cyclictest2741992-21kworker/2:210:10:412
273605599544,42cyclictest2740211-21grep08:30:021
273605599542,47cyclictest2741698-21munin-run09:00:011
273605599541,6cyclictest2741934-21sshd09:00:401
2736051995444,7cyclictest12-21ksoftirqd/012:04:220
2736051995438,12cyclictest1-21systemd10:40:010
2736051995438,12cyclictest1-21systemd10:40:010
2736051995428,22cyclictest19350irq/34-mmc112:22:420
2736051995428,22cyclictest19350irq/34-mmc112:22:420
273605199542,37cyclictest2747787-21ssh09:21:070
2736059995344,6cyclictest2741992-21kworker/2:211:07:132
2736059995343,6cyclictest2741992-21kworker/2:210:22:262
2736059995342,8cyclictest2741992-21kworker/2:209:30:012
2736059995342,8cyclictest2741992-21kworker/2:209:30:002
273605599532,4cyclictest364-21snmpd08:52:561
273605599531,9cyclictest364-21snmpd08:37:531
2736051995342,6cyclictest16950irq/224-800000007:20:000
2736051995339,8cyclictest12-21ksoftirqd/011:51:450
2736051995339,8cyclictest12-21ksoftirqd/011:51:450
2736051995325,12cyclictest2832146-21ssh12:17:200
2736051995322,27cyclictest2742226-21latency_hist09:10:020
273606199529,9cyclictest37-21ksoftirqd/310:30:023
273606199528,9cyclictest37-21ksoftirqd/309:34:033
273606199527,9cyclictest37-21ksoftirqd/310:13:063
273606199525,7cyclictest37-21ksoftirqd/312:25:293
2736061995214,8cyclictest37-21ksoftirqd/310:03:373
2736061995212,7cyclictest37-21ksoftirqd/310:58:113
2736059995243,6cyclictest2741992-21kworker/2:210:00:412
2736059995242,6cyclictest2741992-21kworker/2:211:43:592
2736059995242,5cyclictest2740943-21kworker/2:109:23:452
2736051995231,18cyclictest12-21ksoftirqd/012:31:180
2736051995231,18cyclictest12-21ksoftirqd/012:31:180
2736051995226,11cyclictest2827651-21ssh12:07:580
2736051995226,11cyclictest2744043-21ssh09:13:350
2736061995115,8cyclictest37-21ksoftirqd/309:25:263
2736061995115,8cyclictest37-21ksoftirqd/309:25:253
2736061995113,9cyclictest0-21swapper/311:54:133
2736061995113,9cyclictest0-21swapper/311:54:133
2736061995110,13cyclictest2747143-21latency_hist09:20:023
2736059995143,5cyclictest2741992-21kworker/2:209:53:362
2736059995143,5cyclictest2741992-21kworker/2:209:34:052
2736059995143,5cyclictest2738081-21kworker/2:308:25:012
2736059995142,6cyclictest2741992-21kworker/2:210:45:242
2736059995142,6cyclictest2741992-21kworker/2:210:45:232
2736059995141,6cyclictest2741992-21kworker/2:211:21:092
273605599513,8cyclictest2738582-21munin-run08:00:011
273605599512,5cyclictest2740415-21unin-run08:35:001
273605199516,10cyclictest12-21ksoftirqd/012:40:000
2736051995121,15cyclictest2750971-21ssh09:27:560
2736051995121,15cyclictest2750971-21ssh09:27:550
273606199506,8cyclictest37-21ksoftirqd/312:09:573
273606199506,8cyclictest37-21ksoftirqd/311:04:283
273606199505,9cyclictest37-21ksoftirqd/308:05:013
2736061995014,8cyclictest37-21ksoftirqd/311:59:153
2736061995011,6cyclictest0-21swapper/310:35:013
2736059995043,4cyclictest2741992-21kworker/2:209:55:332
2736059995042,5cyclictest2809351-21kworker/2:012:39:202
2736059995042,5cyclictest2741992-21kworker/2:210:17:332
2736059995042,5cyclictest2741992-21kworker/2:210:08:272
2736059995042,5cyclictest2740943-21kworker/2:109:14:112
2736059995042,4cyclictest2741992-21kworker/2:210:55:012
2736059995041,5cyclictest2773429-21kworker/2:110:33:452
2736059995039,7cyclictest2741992-21kworker/2:211:37:512
273605599501,5cyclictest364-21snmpd08:42:541
273605599501,5cyclictest364-21snmpd08:42:541
2736051995021,13cyclictest2753387-21ssh09:33:000
2736051995015,30cyclictest12-21ksoftirqd/010:13:540
273605199501,4cyclictest131rcu_preempt11:30:100
273606199499,9cyclictest2840524-21latency_hist12:35:023
273606199499,9cyclictest2840524-21latency_hist12:35:023
273606199497,10cyclictest37-21ksoftirqd/310:24:553
273606199491,10cyclictest2788540-21ssh10:46:103
273606199491,10cyclictest2788540-21ssh10:46:093
2736059994941,5cyclictest2809351-21kworker/2:011:55:232
2736059994941,5cyclictest2741992-21kworker/2:211:46:342
2736059994941,5cyclictest2741992-21kworker/2:211:17:252
2736059994941,5cyclictest2741992-21kworker/2:210:58:202
2736059994941,4cyclictest2809351-21kworker/2:012:14:152
2736059994933,10cyclictest309-21in:imuxsock09:35:022
2736059994933,10cyclictest309-21in:imuxsock09:35:022
2736055994916,29cyclictest3467602-21systemd-journal08:14:401
273605199494,33cyclictest2740377-21kworker/u8:108:55:020
2736051994938,6cyclictest12-21ksoftirqd/008:20:010
273605199492,7cyclictest1-21systemd12:30:000
2736051994912,33cyclictest16850irq/134-800000010:59:420
2736051994910,5cyclictest131rcu_preempt11:09:140
2736059994840,5cyclictest2740943-21kworker/2:109:15:452
2736059994839,6cyclictest2809351-21kworker/2:012:06:162
2736059994839,5cyclictest2751874-21kworker/2:309:40:042
273605599487,35cyclictest0-21swapper/108:47:271
2736051994836,10cyclictest12-21ksoftirqd/011:10:150
2736051994822,11cyclictest2760345-21ssh09:47:330
2736051994822,11cyclictest2760345-21ssh09:47:320
273606199479,6cyclictest37-21ksoftirqd/311:47:193
273606199474,8cyclictest37-21ksoftirqd/311:28:473
2736061994735,7cyclictest2763805-21cron09:55:003
2736061994711,9cyclictest0-21swapper/309:24:553
2736059994740,4cyclictest2809351-21kworker/2:012:20:092
2736059994740,4cyclictest2809351-21kworker/2:012:20:092
2736059994739,5cyclictest2809351-21kworker/2:012:00:412
2736059994739,5cyclictest2741992-21kworker/2:211:33:162
2736059994739,5cyclictest2738081-21kworker/2:308:31:402
273605599472,12cyclictest2736076-21modprobe07:10:041
2736051994734,10cyclictest12-21ksoftirqd/008:05:020
273606199465,8cyclictest37-21ksoftirqd/311:35:453
2736061994630,11cyclictest3467602-21systemd-journal09:09:593
2736061994613,7cyclictest0-21swapper/312:16:113
2736059994638,5cyclictest2809351-21kworker/2:012:33:012
2736059994638,5cyclictest2809351-21kworker/2:012:33:012
2736059994638,5cyclictest2741992-21kworker/2:211:12:222
2736059994638,5cyclictest2735948-21kworker/2:107:35:012
2736059994638,5cyclictest2726516-21kworker/2:007:21:492
2736059994636,6cyclictest2735948-21kworker/2:107:55:012
2736059994635,6cyclictest2773429-21kworker/2:110:30:012
2736051994632,10cyclictest275-21dbus-daemon08:35:020
2736051994632,10cyclictest2740664-21systemctl08:40:000
2736051994627,13cyclictest12-21ksoftirqd/009:00:010
273605199462,40cyclictest2804590-21munin-run11:20:010
2736051994618,7cyclictest12-21ksoftirqd/008:07:470
2736061994517,9cyclictest37-21ksoftirqd/312:37:073
273606199451,12cyclictest2824779-21ssh12:01:493
2736059994537,5cyclictest2738081-21kworker/2:308:45:022
2736059994537,5cyclictest2738081-21kworker/2:308:45:022
2736059994536,6cyclictest2735948-21kworker/2:107:55:142
2736059994526,15cyclictest30-21ksoftirqd/208:10:032
2736059994525,14cyclictest3467602-21systemd-journal07:15:002
2736051994529,12cyclictest1-21systemd07:55:010
2736051994527,14cyclictest2769189-21sshd10:05:400
2736061994410,8cyclictest0-21swapper/311:10:183
2736059994436,5cyclictest2809351-21kworker/2:011:53:472
2736059994436,5cyclictest2809351-21kworker/2:011:53:472
273606199439,8cyclictest2792396-21ssh10:54:313
273606199435,8cyclictest1-21systemd08:00:013
273606199435,5cyclictest37-21ksoftirqd/310:38:353
273606199435,5cyclictest37-21ksoftirqd/310:38:353
2736061994329,10cyclictest3467602-21systemd-journal08:40:003
2736061994329,10cyclictest3467602-21systemd-journal08:23:023
2736061994312,6cyclictest0-21swapper/311:23:233
2736061994312,5cyclictest0-21swapper/309:14:283
2736061994311,7cyclictest0-21swapper/312:24:563
2736061994311,7cyclictest0-21swapper/312:24:563
2736059994335,5cyclictest2741992-21kworker/2:211:28:582
2736059994326,12cyclictest30-21ksoftirqd/208:40:002
2736051994331,9cyclictest12-21ksoftirqd/008:25:010
2736051994328,10cyclictest3467602-21systemd-journal07:30:010
2736051994328,10cyclictest3467602-21systemd-journal07:30:010
273605199432,37cyclictest0-21swapper/007:50:020
273606199427,22cyclictest37-21ksoftirqd/308:30:023
273606199425,7cyclictest37-21ksoftirqd/311:30:283
2736061994232,5cyclictest309-21in:imuxsock08:10:013
2736061994228,9cyclictest309-21in:imuxsock07:35:003
273606199421,11cyclictest37-21ksoftirqd/311:44:253
2736059994234,5cyclictest2740943-21kworker/2:109:08:042
2736059994234,5cyclictest2735948-21kworker/2:107:47:492
273605199421,6cyclictest0-21swapper/007:45:010
2736061994128,8cyclictest3467602-21systemd-journal08:20:003
2736059994133,5cyclictest2738081-21kworker/2:308:50:002
2736051994127,11cyclictest2741160-21sshd08:47:400
2736059994025,11cyclictest30-21ksoftirqd/207:35:412
273605199401,6cyclictest2738614-21munin-run08:00:010
2736061993925,10cyclictest3467602-21systemd-journal09:00:013
2736061993921,13cyclictest3467602-21systemd-journal07:30:013
2736061993921,13cyclictest3467602-21systemd-journal07:30:003
2736061993920,16cyclictest2736463-21sshd07:16:413
2736059993927,9cyclictest30-21ksoftirqd/207:17:002
273605199395,23cyclictest2736054-21latency_hist07:10:010
2736051993933,4cyclictest12-21ksoftirqd/007:15:020
2736051993920,13cyclictest1-21systemd08:30:000
2736061993828,8cyclictest37-21ksoftirqd/309:05:023
2736061993826,9cyclictest2741171-21cron08:50:003
2736059993826,8cyclictest30-21ksoftirqd/208:05:002
2736059993825,8cyclictest30-21ksoftirqd/207:10:072
2736051993831,4cyclictest16950irq/224-800000008:45:010
2736051993831,4cyclictest16950irq/224-800000008:45:010
2736061993725,8cyclictest2739322-21sshd08:10:293
2736059993727,7cyclictest30-21ksoftirqd/208:55:022
2736059993727,7cyclictest2740145-21cron08:30:002
2736059993721,12cyclictest30-21ksoftirqd/209:04:402
2736059993711,11cyclictest309-21in:imuxsock08:57:402
273606199361,9cyclictest150-21pr/ttyS207:55:013
2736059993627,6cyclictest2737160-21latency_hist07:30:022
2736059993627,6cyclictest2737160-21latency_hist07:30:022
2736059993524,7cyclictest2737785-21sshd07:40:402
273606199344,9cyclictest37-21ksoftirqd/308:52:553
2736061993427,4cyclictest2736237-21latency_hist07:15:003
2736061993423,7cyclictest37-21ksoftirqd/307:10:023
2736061993418,11cyclictest1-21systemd07:50:003
273605999343,3cyclictest131rcu_preempt08:15:002
273606199333,24cyclictest2737547-21munin-run07:40:003
273606199332,23cyclictest0-21swapper/308:45:033
273606199332,23cyclictest0-21swapper/308:45:023
2736061993221,8cyclictest37-21ksoftirqd/307:25:023
2736061993020,6cyclictest37-21ksoftirqd/308:35:023
273606199301,4cyclictest0-21swapper/307:42:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional