You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-11 - 20:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 11, 2026 12:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2785390999721,38cyclictest2831893-21sh10:43:053
278538599812,75cyclictest2832771-21ssh10:45:031
278539099741,52cyclictest0-21swapper/309:10:023
278538199718,45cyclictest12-21ksoftirqd/011:05:340
278539099691,12cyclictest2802345-21cron09:35:003
2785381996826,17cyclictest28607101ssh11:50:000
278539099672,16cyclictest2834659-21ssh10:49:423
278539099672,16cyclictest2834659-21ssh10:49:413
278539099671,35cyclictest2791548-21diskmemload11:06:403
278539099662,60cyclictest2807708-21ssh09:47:133
278539099652,13cyclictest2858578-21munin-run11:45:003
278539099651,17cyclictest2847095-21sh11:18:163
2785381996516,42cyclictest131rcu_preempt10:37:400
278539099642,12cyclictest2786904-21munin-run07:40:023
278539099642,12cyclictest2786904-21munin-run07:40:013
278539099641,4cyclictest2799762-21ssh09:28:563
278539099641,4cyclictest2799762-21ssh09:28:553
278539099634,55cyclictest2878319-21ssh12:30:363
2785390996324,34cyclictest2868479-21sh12:07:433
278539099631,9cyclictest2812878-21ssh09:59:273
278539099631,9cyclictest2812878-21ssh09:59:263
278539099631,8cyclictest2797432-21ssh09:23:273
278539099631,5cyclictest2794790-21ssh09:17:183
278539099631,5cyclictest2794790-21ssh09:17:183
278539099631,58cyclictest2837264-21ssh10:55:083
278539099631,58cyclictest2837264-21ssh10:55:073
278539099631,18cyclictest2804167-21ssh09:39:163
278539099631,18cyclictest2804167-21ssh09:39:163
278538199631,39cyclictest28791721ssh12:32:440
278539099623,55cyclictest2828249-21sshd10:34:493
278539099622,20cyclictest2817506-21ssh10:10:003
278539099621,7cyclictest2866768-21ssh12:03:513
278539099621,5cyclictest2821297-21ssh10:18:423
278539099621,5cyclictest2821297-21ssh10:18:423
278539099621,19cyclictest2880291-21ssh12:35:043
278539099621,13cyclictest2845310-21ssh11:14:183
278539099612,5cyclictest3467602-21systemd-journal12:23:173
278539099612,53cyclictest2829646-21ssh10:37:543
278539099611,5cyclictest2849420-21ssh11:23:413
278539099611,5cyclictest2805092-21ssh09:41:023
278539099611,56cyclictest2823731-21ssh10:24:213
278539099611,13cyclictest2836387-21ssh10:53:353
2785387996139,13cyclictest2808848-21kworker/2:110:41:332
278538199616,36cyclictest19350irq/34-mmc109:10:020
278539099602,6cyclictest2819229-21ssh10:13:513
278539099602,51cyclictest2814010-21ssh10:01:423
278539099602,4cyclictest2873438-21ssh12:19:433
278539099601,8cyclictest2854118-21ssh11:34:413
2785381996049,7cyclictest12-21ksoftirqd/009:30:300
278538199601,40cyclictest27985011ssh09:25:380
278538199601,40cyclictest27985011ssh09:25:370
278539099592,11cyclictest2787107-21sshd07:40:163
278539099591,7cyclictest2869982-21ssh12:11:013
278539099591,7cyclictest2859498-21ssh11:46:483
278539099591,6cyclictest2877145-21ssh12:28:113
278539099591,5cyclictest2810647-21ssh09:54:173
2785390995915,38cyclictest275-21dbus-daemon08:35:013
278539099591,4cyclictest2791269-21sshd09:02:503
278539099591,17cyclictest2824703-21ssh10:26:203
278539099583,50cyclictest2789223-21munin-run08:25:013
278539099582,4cyclictest2789952-21sshd08:37:513
278539099582,4cyclictest2789952-21sshd08:37:503
278539099581,8cyclictest2850213-21ssh11:25:093
278539099581,4cyclictest2839505-21ssh11:00:193
278539099581,11cyclictest2790486-21sshd08:47:513
2785381995820,33cyclictest16850irq/134-800000011:33:350
2785381995811,6cyclictest131rcu_preempt09:53:190
278539099573,49cyclictest2787865-21sshd07:55:493
278539099572,50cyclictest2863198-21ssh11:55:153
2785390995724,28cyclictest2788426-21latency_hist08:10:023
278539099572,13cyclictest2790763-21munin-run08:55:013
278539099572,13cyclictest2790763-21munin-run08:55:013
278539099571,5cyclictest2854577-21ssh11:35:213
2785381995729,13cyclictest2861525-21ssh11:51:320
2785381995729,13cyclictest2861525-21ssh11:51:310
2785381995727,14cyclictest2872618-21ssh12:17:350
2785381995726,15cyclictest2849955-21ssh11:25:010
2785381995716,36cyclictest12-21ksoftirqd/010:30:090
278539099562,50cyclictest364-21snmpd07:15:093
278539099562,4cyclictest2861799-21ssh11:52:113
278539099562,4cyclictest2861799-21ssh11:52:113
2785387995646,7cyclictest2808848-21kworker/2:110:51:592
2785381995616,32cyclictest16850irq/134-800000009:20:400
2785381995615,37cyclictest12-21ksoftirqd/010:42:440
278538199561,45cyclictest28235611ssh10:23:570
2785381995610,21cyclictest131rcu_preempt11:44:300
278539099552,44cyclictest2787660-21unin-run07:55:013
278539099551,12cyclictest2788665-21sshd08:14:503
2785387995546,6cyclictest2808848-21kworker/2:109:59:402
2785387995546,6cyclictest2808848-21kworker/2:109:59:392
2785387995546,5cyclictest2775083-21kworker/2:008:55:022
2785387995546,5cyclictest2775083-21kworker/2:008:55:012
2785381995537,15cyclictest16850irq/134-800000010:30:010
2785381995528,12cyclictest2866954-21ssh12:04:170
2785381995527,24cyclictest2877151-21ssh12:28:120
2785381995526,23cyclictest2819681-21rm10:15:020
2785381995519,32cyclictest16850irq/134-800000012:08:180
2785381995519,32cyclictest16850irq/134-800000010:56:070
2785381995519,32cyclictest16850irq/134-800000010:56:070
2785381995518,18cyclictest16850irq/134-800000011:40:000
2785381995511,5cyclictest131rcu_preempt12:37:200
278539099542,5cyclictest2785570-21munin-run07:15:003
278539099542,45cyclictest2787394-21modprobe07:50:003
278539099541,6cyclictest2788179-21munin-run08:05:013
278539099541,4cyclictest2792484-21ssh09:11:533
2785387995437,13cyclictest1-21systemd10:30:002
2785387995436,13cyclictest275-21dbus-daemon11:15:012
278538199549,5cyclictest131rcu_preempt11:57:510
2785381995424,14cyclictest2792619-21ssh09:12:240
2785381995418,6cyclictest16850irq/134-800000012:21:320
2785381995418,31cyclictest16850irq/134-800000010:05:160
2785381995417,32cyclictest16850irq/134-800000011:17:080
2785381995416,34cyclictest16850irq/134-800000011:01:210
278539099531,3cyclictest0-21swapper/308:43:503
2785387995345,5cyclictest2830735-21kworker/2:012:09:172
2785387995334,7cyclictest2808848-21kworker/2:111:20:402
2785381995318,31cyclictest16850irq/134-800000010:47:290
2785381995318,31cyclictest16850irq/134-800000010:47:290
2785381995318,31cyclictest16850irq/134-800000010:02:470
2785381995318,31cyclictest16850irq/134-800000009:40:370
2785381995317,6cyclictest16850irq/134-800000009:49:570
2785381995317,32cyclictest16850irq/134-800000011:29:050
2785381995317,32cyclictest16850irq/134-800000010:15:550
2785381995317,32cyclictest16850irq/134-800000010:15:540
2785381995316,32cyclictest16850irq/134-800000009:35:560
2785381995316,32cyclictest16850irq/134-800000009:35:550
278539099522,43cyclictest1-21systemd07:35:013
2785387995244,5cyclictest2808848-21kworker/2:111:16:452
2785387995244,5cyclictest2790522-21kworker/2:309:48:072
2785387995244,5cyclictest2775083-21kworker/2:009:22:352
2785387995244,5cyclictest2775083-21kworker/2:009:19:452
2785387995244,5cyclictest2775083-21kworker/2:009:19:442
2785387995244,4cyclictest2808848-21kworker/2:112:35:012
2785387995243,5cyclictest2808848-21kworker/2:109:54:472
2785387995242,6cyclictest2808848-21kworker/2:111:30:012
2785387995242,6cyclictest2790522-21kworker/2:309:29:142
2785387995242,6cyclictest2790522-21kworker/2:309:29:132
278538599527,27cyclictest0-21swapper/109:05:041
2785385995232,15cyclictest275-21dbus-daemon10:00:011
2785385995232,15cyclictest275-21dbus-daemon10:00:011
278538199529,5cyclictest131rcu_preempt08:35:020
2785381995217,31cyclictest16850irq/134-800000009:18:050
2785381995217,31cyclictest16850irq/134-800000009:18:040
2785381995216,32cyclictest16850irq/134-800000009:55:490
2785381995216,32cyclictest16850irq/134-800000009:55:490
2785381995216,18cyclictest16850irq/134-800000010:54:210
278539099512,9cyclictest2788896-21sshd08:15:293
278539099512,5cyclictest2791020-21munin-run08:59:593
2785387995143,5cyclictest2808848-21kworker/2:112:16:172
2785387995143,5cyclictest2808848-21kworker/2:112:14:022
2785387995143,5cyclictest2808848-21kworker/2:111:37:212
2785387995143,5cyclictest2808848-21kworker/2:110:47:582
2785387995143,5cyclictest2808848-21kworker/2:110:47:582
2785387995143,5cyclictest2808848-21kworker/2:110:37:182
2785387995143,5cyclictest2808848-21kworker/2:110:08:202
2785387995143,5cyclictest2775083-21kworker/2:009:15:022
2785387995142,6cyclictest2808848-21kworker/2:112:39:562
2785387995142,6cyclictest2808848-21kworker/2:112:02:242
2785387995142,5cyclictest2775083-21kworker/2:008:20:012
2785387995134,12cyclictest3467602-21systemd-journal08:00:012
2785385995135,12cyclictest1-21systemd09:55:011
2785381995143,4cyclictest16850irq/134-800000011:15:000
2785381995118,29cyclictest12-21ksoftirqd/008:10:020
2785381995115,18cyclictest16850irq/134-800000012:13:410
278539099502,44cyclictest0-21swapper/307:20:493
278539099502,44cyclictest0-21swapper/307:20:483
278539099501,11cyclictest364-21snmpd07:27:393
2785387995042,5cyclictest2830735-21kworker/2:011:34:232
2785387995042,5cyclictest2808848-21kworker/2:110:21:102
2785387995041,6cyclictest2830735-21kworker/2:011:44:212
2785387995041,5cyclictest2830735-21kworker/2:011:50:362
2785387995041,5cyclictest2830735-21kworker/2:011:50:362
2785387995040,6cyclictest2775083-21kworker/2:009:32:402
2785381995032,14cyclictest1-21systemd08:50:000
278538199502,43cyclictest2785842-21systemctl07:20:010
278539099492,5cyclictest2789442-21modprobe08:25:503
2785387994942,4cyclictest2808848-21kworker/2:111:55:522
2785387994942,4cyclictest2808848-21kworker/2:110:10:102
2785387994941,5cyclictest2830735-21kworker/2:011:46:502
2785387994941,5cyclictest2830735-21kworker/2:011:09:512
2785387994941,5cyclictest2830735-21kworker/2:011:01:362
2785387994941,5cyclictest2808848-21kworker/2:112:25:362
2785387994941,5cyclictest2808848-21kworker/2:112:21:262
2785387994941,5cyclictest2808848-21kworker/2:110:56:432
2785387994941,5cyclictest2808848-21kworker/2:110:56:432
2785387994941,5cyclictest2790522-21kworker/2:310:19:002
2785387994941,5cyclictest2790522-21kworker/2:310:19:002
2785387994941,5cyclictest2775083-21kworker/2:009:36:072
2785387994941,5cyclictest2775083-21kworker/2:009:36:072
2785387994941,5cyclictest2775083-21kworker/2:007:55:002
2785385994931,15cyclictest23-21ksoftirqd/108:40:011
2785385994931,15cyclictest23-21ksoftirqd/108:40:001
278538599492,6cyclictest131rcu_preempt12:35:001
2785381994910,5cyclictest131rcu_preempt07:25:020
2785381994910,5cyclictest131rcu_preempt07:25:010
2785387994840,5cyclictest2808848-21kworker/2:110:32:582
2785387994840,5cyclictest2808848-21kworker/2:110:00:152
2785387994840,5cyclictest2775083-21kworker/2:009:41:152
2785387994833,10cyclictest1-21systemd08:15:012
2785385994834,10cyclictest1-21systemd10:30:021
2785385994831,13cyclictest1-21systemd07:15:021
2785385994827,15cyclictest275-21dbus-daemon11:40:011
2785385994821,22cyclictest23-21ksoftirqd/111:03:061
2785387994740,4cyclictest2775083-21kworker/2:008:21:022
2785387994739,5cyclictest2775083-21kworker/2:007:35:082
2785387994739,5cyclictest2775083-21kworker/2:007:35:082
2785385994731,12cyclictest2786873-21systemctl07:40:011
2785385994731,12cyclictest2786873-21systemctl07:40:001
2785385994721,21cyclictest23-21ksoftirqd/110:45:361
2785385994721,21cyclictest23-21ksoftirqd/110:45:361
2785385994718,15cyclictest23-21ksoftirqd/110:07:021
278539099461,40cyclictest2785452-21latency_hist07:10:023
2785387994638,5cyclictest2775083-21kworker/2:007:43:512
2785387994638,5cyclictest2775083-21kworker/2:007:28:542
2785385994630,12cyclictest275-21dbus-daemon12:10:001
2785385994622,15cyclictest23-21ksoftirqd/108:45:011
2785385994621,8cyclictest23-21ksoftirqd/109:35:511
2785385994621,8cyclictest23-21ksoftirqd/109:35:501
2785385994621,20cyclictest23-21ksoftirqd/111:11:341
2785385994621,17cyclictest23-21ksoftirqd/107:33:501
278538199468,8cyclictest12-21ksoftirqd/008:40:030
278538199468,8cyclictest12-21ksoftirqd/008:40:020
2785381994630,12cyclictest12-21ksoftirqd/008:55:010
2785381994630,12cyclictest12-21ksoftirqd/008:55:000
2785387994537,5cyclictest2775083-21kworker/2:009:05:222
2785387994537,5cyclictest2775083-21kworker/2:007:30:562
2785387994537,5cyclictest2775083-21kworker/2:007:21:192
2785387994537,5cyclictest2775083-21kworker/2:007:21:182
2785387994537,5cyclictest2775083-21kworker/2:007:17:342
2785387994532,9cyclictest30-21ksoftirqd/209:00:002
278538599457,5cyclictest131rcu_preempt11:20:001
2785385994529,11cyclictest1-21systemd09:25:001
2785385994521,20cyclictest23-21ksoftirqd/112:21:451
2785385994521,19cyclictest23-21ksoftirqd/110:32:331
278538599451,5cyclictest131rcu_preempt11:45:011
278538799449,28cyclictest30-21ksoftirqd/208:40:022
278538799449,28cyclictest30-21ksoftirqd/208:40:012
2785387994436,5cyclictest2775083-21kworker/2:008:30:002
2785387994436,5cyclictest2775083-21kworker/2:007:47:422
2785385994429,4cyclictest23-21ksoftirqd/109:50:011
2785385994429,11cyclictest1-21systemd07:50:001
2785385994420,20cyclictest23-21ksoftirqd/111:47:291
278538599441,4cyclictest0-21swapper/108:25:021
2785381994437,4cyclictest16950irq/224-800000009:00:010
2785381994427,14cyclictest2788146-21sshd08:00:510
2785387994336,4cyclictest2775083-21kworker/2:008:34:572
2785387994335,5cyclictest2775083-21kworker/2:008:45:562
2785387994335,5cyclictest2775083-21kworker/2:008:06:162
278538599437,31cyclictest1-21systemd10:40:001
2785385994331,9cyclictest2791520-21sshd09:07:501
2785385994326,4cyclictest23-21ksoftirqd/112:25:211
2785385994320,19cyclictest23-21ksoftirqd/112:35:571
2785385994320,19cyclictest23-21ksoftirqd/110:15:121
2785385994320,19cyclictest23-21ksoftirqd/110:15:121
2785385994319,20cyclictest23-21ksoftirqd/109:15:081
2785385994319,20cyclictest23-21ksoftirqd/109:15:081
278538599431,29cyclictest2842906-21kworker/u8:211:10:011
278538199432,37cyclictest2787155-21munin-run07:45:000
2785385994229,8cyclictest309-21in:imuxsock12:15:011
278538599421,6cyclictest131rcu_preempt07:20:001
2785385994213,21cyclictest0-21swapper/110:13:301
2785381994235,4cyclictest16950irq/224-800000008:45:010
2785381994228,10cyclictest1-21systemd07:55:000
2785381994227,10cyclictest275-21dbus-daemon07:15:000
2785387994133,5cyclictest2775083-21kworker/2:008:40:172
2785387994133,5cyclictest2775083-21kworker/2:007:10:592
2785385994133,5cyclictest2787905-21latency_hist08:00:001
2785385994114,19cyclictest131rcu_preempt09:30:281
2785385994112,15cyclictest23-21ksoftirqd/110:01:431
2785381994129,7cyclictest12-21ksoftirqd/008:25:010
2785381994128,10cyclictest3467602-21systemd-journal07:36:320
2785381994128,10cyclictest3467602-21systemd-journal07:36:320
2785381994124,8cyclictest12-21ksoftirqd/008:00:000
2785387994032,5cyclictest2775083-21kworker/2:008:01:372
2785385994029,8cyclictest23-21ksoftirqd/109:43:221
2785385994027,9cyclictest3467602-21systemd-journal07:45:001
2785385994014,18cyclictest23-21ksoftirqd/109:12:001
278538599401,25cyclictest221rcuc/108:30:021
2785385994011,16cyclictest23-21ksoftirqd/112:19:371
2785381994028,7cyclictest16950irq/224-800000007:30:010
278538199401,4cyclictest131rcu_preempt08:30:020
2785387993931,5cyclictest2775083-21kworker/2:009:04:002
2785385993929,5cyclictest23-21ksoftirqd/107:24:511
2785385993929,5cyclictest23-21ksoftirqd/107:24:511
2785385993928,7cyclictest23-21ksoftirqd/110:55:001
2785385993928,6cyclictest23-21ksoftirqd/110:55:191
2785385993928,6cyclictest23-21ksoftirqd/110:55:191
278538599392,5cyclictest131rcu_preempt10:24:541
278538599391,3cyclictest131rcu_preempt11:52:191
278538599391,3cyclictest131rcu_preempt11:52:181
2785385993912,15cyclictest23-21ksoftirqd/111:58:161
2785385993826,9cyclictest23-21ksoftirqd/111:25:021
2785385993824,11cyclictest23-21ksoftirqd/108:59:591
2785385993823,11cyclictest23-21ksoftirqd/111:31:011
2785385993817,17cyclictest23-21ksoftirqd/112:00:261
2785385993810,16cyclictest23-21ksoftirqd/109:26:591
2785385993810,16cyclictest23-21ksoftirqd/109:26:591
2785381993828,5cyclictest2788953-21munin-plugin-st08:20:010
2785381993827,5cyclictest309-21in:imuxsock09:00:510
2785385993727,8cyclictest23-21ksoftirqd/108:10:021
2785385993727,7cyclictest2789444-21sshd08:26:501
2785385993725,8cyclictest2852009-21sshd11:29:501
2785381993730,4cyclictest16950irq/224-800000008:15:010
278538199372,31cyclictest1-21systemd07:49:510
278538599363,22cyclictest23-21ksoftirqd/108:05:001
2785385993621,11cyclictest275-21dbus-daemon08:55:011
2785385993621,11cyclictest275-21dbus-daemon08:55:001
278538599356,24cyclictest23-21ksoftirqd/108:20:001
2785385993521,10cyclictest23-21ksoftirqd/108:11:501
2785385993419,12cyclictest275-21dbus-daemon07:55:011
2785381993422,8cyclictest12-21ksoftirqd/007:30:060
2785385993325,5cyclictest2786357-21latency_hist07:30:011
278538599332,3cyclictest131rcu_preempt07:10:011
2785381993324,7cyclictest12-21ksoftirqd/007:10:020
2785387993124,4cyclictest2785488-21latency_hist07:10:012
2785385993124,5cyclictest23-21ksoftirqd/108:50:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional