You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 20:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 16, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27516849911147,57cyclictest2757869-21diskmemload12:35:020
2751684998417,35cyclictest16850irq/134-800000009:28:190
275168999832,58cyclictest0-21swapper/109:00:011
275168999811,45cyclictest2820622-21ssh11:29:381
2751684997819,37cyclictest16850irq/134-800000011:01:160
2751684997819,37cyclictest16850irq/134-800000011:01:150
275168999772,43cyclictest2756051-21systemd-run08:35:011
2751684997716,6cyclictest16850irq/134-800000012:27:390
275168999761,42cyclictest2814693-21ssh11:16:071
275168999761,42cyclictest2753584-21cut07:45:021
2751684997619,31cyclictest16850irq/134-800000009:32:100
2751684997619,31cyclictest16850irq/134-800000009:32:090
2751689997317,25cyclictest2757869-21diskmemload09:40:211
2751689997317,25cyclictest2757869-21diskmemload09:40:211
275168999731,41cyclictest2793395-21ssh10:29:231
275168999731,41cyclictest2793395-21ssh10:29:221
275168999721,46cyclictest2757869-21diskmemload10:40:201
275168999721,43cyclictest2764326-21ssh09:24:131
275168999721,43cyclictest2764326-21ssh09:24:121
2751684997217,34cyclictest16850irq/134-800000010:28:470
2751684997217,34cyclictest16850irq/134-800000010:28:470
275168999701,41cyclictest364-21snmpd10:39:481
275168999701,41cyclictest364-21snmpd10:39:471
275168999701,36cyclictest2759909-21ssh09:14:331
2751684997014,6cyclictest16850irq/134-800000010:53:560
2751684997014,6cyclictest16850irq/134-800000010:53:550
275168999691,39cyclictest2766169-21ssh09:28:211
2751684996915,7cyclictest16850irq/134-800000009:41:360
2751684996915,7cyclictest16850irq/134-800000009:41:360
275168999682,61cyclictest2753695-21sshd07:49:161
275168999682,61cyclictest2753695-21sshd07:49:151
2751689996822,42cyclictest23-21ksoftirqd/111:42:151
2751684996819,31cyclictest16850irq/134-800000010:18:130
2751684996819,31cyclictest16850irq/134-800000010:18:130
275169399676,35cyclictest2758047-21ssh09:10:032
275168999672,61cyclictest3467602-21systemd-journal12:29:281
275168999672,61cyclictest2803684-21ssh10:51:471
275168999672,61cyclictest2803684-21ssh10:51:461
275168999672,60cyclictest2821170-21ssh11:30:211
2751689996725,37cyclictest2831183-21sh11:52:571
275168999671,62cyclictest2829364-21ssh11:48:541
275168999671,44cyclictest2838586-21rm12:09:341
2751684996717,31cyclictest16850irq/134-800000012:00:150
275168999668,54cyclictest2756799-21sshd08:47:201
275168999662,60cyclictest2775454-21ssh09:49:101
275168999661,61cyclictest2844398-21ssh12:22:101
275168999661,61cyclictest2823422-21ssh11:35:251
275168999661,61cyclictest2812800-21ssh11:12:011
275168999661,61cyclictest2802673-21ssh10:49:531
275168999661,60cyclictest2835440-21ssh12:02:121
275168999661,40cyclictest2849542-21ssh12:33:451
2751684996614,34cyclictest16850irq/134-800000010:12:420
2751693996523,36cyclictest2816840-21sh11:20:392
2751693996521,39cyclictest2764759-21latency_hist09:25:022
2751693996521,39cyclictest2764759-21latency_hist09:25:022
275168999651,60cyclictest2839150-21ssh12:10:211
275168999651,60cyclictest2833788-21ssh11:58:471
275168999651,60cyclictest2811443-21ssh11:09:101
275168999651,60cyclictest2788538-21ssh10:18:181
275168999651,60cyclictest2788538-21ssh10:18:181
275168999651,60cyclictest2785958-21ssh10:12:201
275168999651,60cyclictest2761893-21ssh09:18:521
275168999651,59cyclictest2805363-21ssh10:55:241
275168999651,59cyclictest2791084-21ssh10:23:591
275168999651,59cyclictest2777353-21ssh09:53:231
275168999651,58cyclictest2778384-21ssh09:55:211
275168999641,59cyclictest2850368-21ssh12:35:161
275168999641,59cyclictest2841264-21ssh12:15:031
275168999641,59cyclictest2818285-21ssh11:24:191
275168999641,59cyclictest2794921-21ssh10:32:361
275168999641,59cyclictest2782913-21ssh10:05:201
275168999641,59cyclictest2769533-21ssh09:35:351
275168999641,59cyclictest2767525-21ssh09:31:141
275168999641,59cyclictest2767525-21ssh09:31:131
275168999641,58cyclictest2781266-21ssh10:01:531
2751693996322,37cyclictest2766832-21cron09:29:592
275168999632,57cyclictest2755214-21sshd08:15:151
275168999631,58cyclictest2807826-21ssh11:00:461
275168999631,58cyclictest2807826-21ssh11:00:451
275168999631,57cyclictest2752644-21sshd07:26:161
2751684996323,20cyclictest0-21swapper/011:47:440
275168999622,56cyclictest2754509-21munin-run08:05:011
275168999621,56cyclictest48-21kcompactd009:00:391
2751693996123,34cyclictest30-21ksoftirqd/212:16:392
2751693996117,5cyclictest2761820-21sh09:18:402
275168999611,56cyclictest2757045-21sshd08:50:191
2751684996120,21cyclictest16850irq/134-800000009:20:000
2751684996119,21cyclictest16850irq/134-800000011:30:510
275169899609,7cyclictest131rcu_preempt09:00:003
275169399602,19cyclictest2752171-21sendmail-msp07:19:592
2751693996019,4cyclictest2778018-21latency_hist09:55:022
275168999601,54cyclictest48-21kcompactd007:35:471
2751693995925,30cyclictest30-21ksoftirqd/212:05:012
275168999592,53cyclictest2756338-21munin-run08:40:021
275168999592,52cyclictest364-21snmpd07:17:381
275168999591,54cyclictest2756540-21sshd08:40:181
275168999591,53cyclictest48-21kcompactd009:08:251
275168999591,53cyclictest48-21kcompactd008:08:291
275168999591,53cyclictest48-21kcompactd008:08:291
2751684995955,2cyclictest0-21swapper/008:10:020
2751684995955,2cyclictest0-21swapper/008:10:020
275168499592,3cyclictest131rcu_preempt12:24:200
2751684995919,36cyclictest16850irq/134-800000010:40:200
2751684995919,21cyclictest16850irq/134-800000010:02:030
2751693995827,28cyclictest30-21ksoftirqd/210:57:302
2751693995823,31cyclictest2773356-21sh09:44:372
2751693995823,31cyclictest2773356-21sh09:44:362
2751693995820,7cyclictest30-21ksoftirqd/210:41:172
2751693995815,5cyclictest0-21swapper/210:20:002
2751693995815,5cyclictest0-21swapper/210:20:002
275169399581,13cyclictest2757914-21latency_hist09:10:022
275168999581,53cyclictest2755505-21sshd08:20:171
275168999581,53cyclictest2755505-21sshd08:20:171
275168999581,53cyclictest2755000-21munin-run08:15:001
2751684995819,35cyclictest16850irq/134-800000010:22:220
2751684995818,36cyclictest16850irq/134-800000011:09:160
2751684995818,35cyclictest16850irq/134-800000010:09:200
2751684995818,20cyclictest16850irq/134-800000011:27:120
2751684995818,20cyclictest16850irq/134-800000009:48:170
2751684995817,36cyclictest16850irq/134-800000009:51:240
2751693995725,29cyclictest30-21ksoftirqd/212:32:122
2751693995717,5cyclictest2757869-21diskmemload11:13:032
2751693995716,4cyclictest2810643-21sh11:07:102
2751693995714,7cyclictest2778045-21kworker/u8:112:15:012
275168999572,51cyclictest2753977-21munin-run07:55:011
275168999571,52cyclictest3467602-21systemd-journal07:10:111
275168999571,51cyclictest48-21kcompactd007:33:331
2751684995729,11cyclictest2780229-21ssh10:00:000
2751684995727,15cyclictest2840777-21ssh12:14:280
2751684995718,34cyclictest16850irq/134-800000011:50:560
2751684995718,20cyclictest16850irq/134-800000012:09:560
2751684995717,36cyclictest16850irq/134-800000012:17:440
2751684995717,20cyclictest16850irq/134-800000011:38:430
2751684995717,20cyclictest16850irq/134-800000010:39:300
2751684995717,20cyclictest16850irq/134-800000010:39:290
275169399562,17cyclictest2755271-21grep08:20:012
2751693995618,4cyclictest2757869-21diskmemload11:35:092
2751693995617,4cyclictest2757869-21diskmemload10:13:122
2751693995616,5cyclictest0-21swapper/210:25:012
2751693995615,10cyclictest30-21ksoftirqd/210:00:162
2751693995612,9cyclictest309-21in:imuxsock07:45:012
275168499566,12cyclictest2807248-21sendmail11:00:000
2751684995618,19cyclictest16850irq/134-800000011:18:370
2751693995545,6cyclictest2744705-21kworker/2:208:45:002
2751693995526,25cyclictest2793699-21sh10:30:012
2751693995526,25cyclictest2793699-21sh10:30:002
2751693995522,28cyclictest2807565-21sh11:00:112
2751693995522,28cyclictest2807565-21sh11:00:112
2751693995520,9cyclictest30-21ksoftirqd/212:35:192
2751693995520,5cyclictest2757869-21diskmemload11:51:222
275169399551,18cyclictest2827692-21wc11:45:022
275168999553,47cyclictest364-21snmpd07:20:401
275168999553,47cyclictest364-21snmpd07:20:401
275168999552,47cyclictest364-21snmpd07:55:141
275168999551,50cyclictest2755777-21cron08:29:591
2751684995525,15cyclictest2795701-21ssh10:34:300
2751684995517,34cyclictest16850irq/134-800000010:45:570
2751684995517,33cyclictest16850irq/134-800000009:35:130
2751684995516,34cyclictest16850irq/134-800000011:59:380
275169399542,15cyclictest3467718-21lldpd10:33:312
2751693995419,4cyclictest2819100-21sh11:25:442
2751693995417,4cyclictest2757869-21diskmemload11:57:202
275168499544,12cyclictest2755249-21cron08:20:000
2751684995424,15cyclictest2763466-21ssh09:22:080
2751684995424,15cyclictest2763466-21ssh09:22:070
2751684995417,33cyclictest16850irq/134-800000012:39:460
2751684995417,33cyclictest16850irq/134-800000011:14:020
2751684995415,5cyclictest16850irq/134-800000011:44:510
2751693995344,6cyclictest2762435-21kworker/2:309:57:092
2751693995328,22cyclictest30-21ksoftirqd/209:39:142
2751693995325,24cyclictest2845108-21sh12:23:562
2751693995324,26cyclictest2751622-21latency_hist07:10:022
275169399532,16cyclictest2805217-21latency_hist10:55:032
275169399532,16cyclictest2805217-21latency_hist10:55:022
2751693995320,4cyclictest2828863-21sh11:47:482
2751693995315,5cyclictest2797815-21sh10:39:142
2751693995315,5cyclictest2797815-21sh10:39:132
2751693995315,10cyclictest30-21ksoftirqd/212:26:572
2751693995312,11cyclictest275-21dbus-daemon09:05:012
275169899525,40cyclictest2773615-21grep09:45:023
275169899525,40cyclictest2773615-21grep09:45:013
275169399522,15cyclictest2756929-21cat08:50:012
2751693995216,5cyclictest30-21ksoftirqd/208:40:012
2751684995213,34cyclictest16850irq/134-800000011:22:540
2751693995123,25cyclictest30-21ksoftirqd/211:15:252
275169399512,18cyclictest309-21in:imuxsock08:30:002
275169399511,17cyclictest2752677-21munin-plugin-st07:29:592
2751684995127,8cyclictest12-21ksoftirqd/009:10:000
275169899502,15cyclictest2840293-21ssh12:13:143
275169399503,18cyclictest1-21systemd08:35:002
2751693995021,5cyclictest2838798-21latency_hist12:10:012
2751684995012,30cyclictest16850irq/134-800000009:13:530
2751693994916,6cyclictest2774010-21ssh09:45:352
2751693994915,8cyclictest30-21ksoftirqd/209:32:022
2751693994915,8cyclictest30-21ksoftirqd/209:32:022
2751693994915,7cyclictest30-21ksoftirqd/210:46:462
2751684994934,11cyclictest1-21systemd09:05:020
275169899487,5cyclictest131rcu_preempt08:19:163
2751698994819,20cyclictest361rcuc/310:35:013
2751693994821,7cyclictest30-21ksoftirqd/211:34:302
2751693994811,5cyclictest2755599-21cat08:25:012
2751693994811,5cyclictest2755599-21cat08:25:012
2751698994731,11cyclictest3467602-21systemd-journal08:35:013
2751693994738,5cyclictest2751289-21kworker/2:107:15:012
2751693994715,9cyclictest309-21in:imuxsock10:10:012
2751693994712,11cyclictest2754780-21date08:10:022
2751693994712,11cyclictest2754780-21date08:10:022
275169899469,6cyclictest131rcu_preempt10:13:103
275169899464,5cyclictest131rcu_preempt09:20:003
2751698994634,9cyclictest37-21ksoftirqd/307:30:003
275169899463,38cyclictest2778045-21kworker/u8:111:10:023
2751698994631,12cyclictest37-21ksoftirqd/311:53:073
2751698994631,11cyclictest309-21in:imuxsock12:00:003
2751693994619,4cyclictest3467562-21sshd07:51:162
275168999461,40cyclictest2751772-21latency_hist07:10:011
275168499463,6cyclictest2756306-21grep08:40:000
2751684994629,13cyclictest275-21dbus-daemon07:35:010
2751684994617,20cyclictest16850irq/134-800000007:11:170
2751698994535,8cyclictest37-21ksoftirqd/310:40:013
2751698994535,8cyclictest37-21ksoftirqd/310:40:013
275169899453,3cyclictest131rcu_preempt09:24:283
275169899453,3cyclictest131rcu_preempt09:24:273
2751698994532,9cyclictest3467602-21systemd-journal07:48:163
2751698994532,9cyclictest3467602-21systemd-journal07:48:163
2751698994530,11cyclictest37-21ksoftirqd/308:15:003
2751698994530,11cyclictest1-21systemd08:30:003
2751698994528,12cyclictest1-21systemd11:25:023
2751684994530,11cyclictest3467602-21systemd-journal09:00:000
2751684994526,15cyclictest2753147-21sshd07:36:160
275168499452,10cyclictest2756038-21latency_hist08:35:020
275169899444,5cyclictest131rcu_preempt09:55:413
275169899441,5cyclictest0-21swapper/310:55:113
275169899441,14cyclictest2790266-21ssh10:21:583
275169399442,17cyclictest2757218-21cat08:55:022
2751693994417,5cyclictest30-21ksoftirqd/207:40:002
2751684994429,11cyclictest12-21ksoftirqd/007:50:010
2751684994429,11cyclictest12-21ksoftirqd/007:50:000
275169899438,5cyclictest131rcu_preempt09:45:033
2751698994331,8cyclictest37-21ksoftirqd/310:05:013
275169899432,9cyclictest2765749-21ssh09:27:123
2751698994329,10cyclictest0-21swapper/309:35:013
2751698994329,10cyclictest0-21swapper/309:35:003
2751698994322,15cyclictest275-21dbus-daemon11:15:013
2751693994335,5cyclictest2751289-21kworker/2:108:12:502
275168499435,30cyclictest2751687-21grep07:10:020
2751684994332,6cyclictest2752145-21sendmail07:19:590
275168499432,35cyclictest311-21rs:main0
275169899427,4cyclictest131rcu_preempt09:10:133
275169899424,4cyclictest131rcu_preempt11:30:213
275169899423,5cyclictest2836021-21ssh12:03:413
2751698994231,8cyclictest37-21ksoftirqd/311:19:593
2751698994231,8cyclictest37-21ksoftirqd/309:53:403
2751698994231,7cyclictest3467602-21systemd-journal12:05:033
2751698994229,10cyclictest37-21ksoftirqd/310:28:073
2751698994229,10cyclictest37-21ksoftirqd/310:28:063
275169899422,5cyclictest0-21swapper/312:33:143
275168499422,6cyclictest1-21systemd08:45:010
2751684994226,12cyclictest3467602-21systemd-journal07:45:000
2751684994225,12cyclictest275-21dbus-daemon07:55:000
275169899418,5cyclictest131rcu_preempt12:19:223
2751698994130,7cyclictest37-21ksoftirqd/310:20:023
2751698994130,7cyclictest37-21ksoftirqd/310:20:013
2751698994128,10cyclictest37-21ksoftirqd/312:21:193
2751698994128,10cyclictest37-21ksoftirqd/311:35:273
2751698994127,10cyclictest2751761-21date07:10:013
2751693994125,11cyclictest30-21ksoftirqd/209:00:012
275169399411,19cyclictest2754424-21wc08:00:032
2751684994116,6cyclictest12-21ksoftirqd/008:04:590
275169899407,4cyclictest131rcu_preempt11:47:153
275169899405,6cyclictest131rcu_preempt11:25:203
275169899404,4cyclictest131rcu_preempt09:05:033
2751698994027,10cyclictest37-21ksoftirqd/309:36:363
275169899402,4cyclictest131rcu_preempt12:40:003
2751698994023,11cyclictest3467602-21systemd-journal07:17:013
275169899401,5cyclictest0-21swapper/310:05:233
275169899401,4cyclictest2825659-21ssh11:40:193
275169899401,10cyclictest2801604-21sh10:47:243
275168499404,4cyclictest131rcu_preempt07:27:150
275168499401,10cyclictest2757262-21sort08:55:010
275168499394,30cyclictest2752421-21modprobe07:25:010
275168499394,30cyclictest2752421-21modprobe07:25:010
2751684993929,7cyclictest309-21in:imuxsock08:25:040
275169899384,4cyclictest131rcu_preempt10:40:323
275169899384,4cyclictest131rcu_preempt07:45:013
275169899383,4cyclictest131rcu_preempt11:01:223
275169899383,4cyclictest131rcu_preempt11:01:223
275169899382,4cyclictest131rcu_preempt10:51:193
275169899382,4cyclictest131rcu_preempt10:51:183
2751698993822,11cyclictest0-21swapper/307:15:013
275169899381,4cyclictest131rcu_preempt08:40:023
2751693993811,5cyclictest131rcu_preempt07:22:402
2751693993811,5cyclictest131rcu_preempt07:22:392
275169899377,5cyclictest131rcu_preempt08:48:263
275169899373,4cyclictest131rcu_preempt08:25:033
275169899373,4cyclictest131rcu_preempt08:25:023
2751698993727,8cyclictest37-21ksoftirqd/308:45:023
2751698993725,9cyclictest37-21ksoftirqd/307:53:153
2751698993724,9cyclictest3467602-21systemd-journal07:30:163
275169899371,6cyclictest0-21swapper/312:27:173
2751684993721,12cyclictest3467602-21systemd-journal08:25:010
2751684993721,12cyclictest3467602-21systemd-journal08:25:000
275169899366,15cyclictest2753174-21modprobe07:40:003
275169899364,3cyclictest131rcu_preempt08:10:013
275169899364,3cyclictest131rcu_preempt08:10:003
275169899363,3cyclictest131rcu_preempt08:05:003
2751684993619,12cyclictest275-21dbus-daemon08:50:010
2751698993522,9cyclictest3467602-21systemd-journal07:20:023
2751698993522,9cyclictest3467602-21systemd-journal07:20:013
275168499351,30cyclictest2754965-21sshd08:10:160
275169899342,4cyclictest131rcu_preempt09:10:013
275169899342,3cyclictest131rcu_preempt08:00:013
2751693993426,5cyclictest2754493-21latency_hist08:05:012
2751698993322,6cyclictest309-21in:imuxsock08:54:193
2751693993323,7cyclictest30-21ksoftirqd/207:31:162
2751693993224,6cyclictest30-21ksoftirqd/207:45:032
2751693993224,6cyclictest30-21ksoftirqd/207:45:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional