You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-07 - 20:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu May 07, 2026 12:45:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
242955399852,48cyclictest2506428-21ssh11:36:302
2429553998222,31cyclictest2467551-21sh10:15:312
2429549998025,11cyclictest23-21ksoftirqd/110:39:591
242955399782,39cyclictest2458985-21ssh09:58:102
242955399772,35cyclictest2502676-21ssh11:28:482
2429553997328,41cyclictest2532755-21sshd12:31:062
2429558997224,9cyclictest37-21ksoftirqd/308:00:003
242955399711,65cyclictest2456782-21ssh09:53:382
2429553997019,33cyclictest30-21ksoftirqd/209:44:152
242955399692,34cyclictest2479016-21ssh10:39:472
242954499693,51cyclictest12-21ksoftirqd/012:25:000
242955399682,60cyclictest2479576-21ssh10:40:242
242955399682,60cyclictest2479576-21ssh10:40:242
242955399681,38cyclictest2448270-21ssh09:35:392
242955399671,62cyclictest2525578-21ssh12:16:112
242955399671,62cyclictest2524587-21ssh12:14:272
242955399671,62cyclictest2512110-21ssh11:48:182
242955399662,60cyclictest2442352-21ssh09:23:432
242955399662,34cyclictest2495846-21ssh11:14:472
242955399662,34cyclictest2495846-21ssh11:14:462
242955399661,61cyclictest2497534-21ssh11:18:072
242955399661,61cyclictest2463184-21ssh10:06:342
242955399661,61cyclictest2439267-21ssh09:17:102
242955399661,61cyclictest2439267-21ssh09:17:102
242955399661,61cyclictest2436162-21ssh09:10:272
242955399661,60cyclictest2530736-21ssh12:26:562
242955399661,60cyclictest2530736-21ssh12:26:552
242955399661,60cyclictest2482167-21ssh10:45:592
242954499661,43cyclictest24865921ssh10:55:060
242955399651,61cyclictest2492316-21ssh11:07:152
242955399651,60cyclictest2535402-21ssh12:36:392
242955399651,60cyclictest2535402-21ssh12:36:382
242955399651,60cyclictest2513477-21ssh11:51:022
242955399651,60cyclictest2513477-21ssh11:51:012
242955399651,60cyclictest2509283-21ssh11:42:222
242955399651,60cyclictest2503794-21ssh11:31:042
242955399651,60cyclictest2503794-21ssh11:31:032
242955399651,60cyclictest2484949-21ssh10:51:492
242955399651,60cyclictest2484949-21ssh10:51:482
242955399651,60cyclictest2475753-21ssh10:32:452
242955399651,60cyclictest2475753-21ssh10:32:452
242955399651,60cyclictest2473179-21ssh10:27:222
242955399651,60cyclictest2461817-21ssh10:04:012
242955399651,59cyclictest2470217-21ssh10:21:112
242955399651,59cyclictest2470217-21ssh10:21:112
242955399651,59cyclictest2454536-21ssh09:48:532
2429544996510,6cyclictest131rcu_preempt11:39:590
242955399642,57cyclictest2434190-21munin-run08:40:012
242955399641,59cyclictest2528208-21ssh12:21:452
242955399641,59cyclictest2518242-21ssh12:01:002
242955399641,59cyclictest2499696-21ssh11:22:362
242955399641,59cyclictest2466006-21ssh10:12:382
242955399641,59cyclictest2447558-21ssh09:34:292
242955399632,57cyclictest2515854-21ssh11:56:012
2429553996323,36cyclictest2430667-21latency_hist07:30:022
242955399631,58cyclictest2520289-21ssh12:05:062
242955399631,58cyclictest2489195-21ssh11:00:302
242955399631,58cyclictest2489195-21ssh11:00:302
242955399631,58cyclictest2487135-21ssh10:56:242
242955399631,58cyclictest2443199-21ssh09:25:042
242955399631,56cyclictest2431547-21sshd07:49:002
242954499631,41cyclictest24770871ssh10:35:250
2429553996228,31cyclictest2430781-21latency_hist07:35:032
2429553996228,31cyclictest2430781-21latency_hist07:35:022
242955399622,55cyclictest364-21snmpd08:55:282
2429553996218,39cyclictest2431000-21sshd07:35:162
242955399621,57cyclictest2435690-21sshd09:08:012
242955399621,57cyclictest2435690-21sshd09:08:002
242954499628,6cyclictest131rcu_preempt10:15:020
242955399612,54cyclictest2431329-21munin-run07:45:012
242955399611,55cyclictest2432055-21cron07:59:592
2429544996123,34cyclictest12-21ksoftirqd/010:54:410
2429544996123,34cyclictest12-21ksoftirqd/010:54:410
2429544996114,6cyclictest16850irq/134-800000012:20:010
242954499611,36cyclictest131rcu_preempt10:00:190
242955399602,53cyclictest2434429-21munin-run08:45:012
242955399602,53cyclictest2433907-21munin-run08:35:012
242955399602,53cyclictest2432816-21sshd08:10:272
242955399602,53cyclictest2432364-21munin-run08:05:012
242955399601,55cyclictest2431796-21sshd07:53:002
242955399601,55cyclictest2431796-21sshd07:53:002
2429544996023,18cyclictest12-21ksoftirqd/011:33:130
2429544996023,18cyclictest12-21ksoftirqd/011:33:120
2429544996022,34cyclictest2457460-21cat09:55:020
242955899595,10cyclictest37-21ksoftirqd/309:57:003
2429558995913,6cyclictest131rcu_preempt10:25:053
2429558995913,5cyclictest131rcu_preempt09:15:003
242955399592,53cyclictest2433404-21munin-run08:25:002
242955399592,52cyclictest2432615-21munin-run08:10:022
242955399591,54cyclictest48-21kcompactd009:00:222
242955399591,53cyclictest2434693-21munin-run08:50:012
242955399591,53cyclictest2434693-21munin-run08:50:012
242955399591,52cyclictest48-21kcompactd007:11:392
2429544995929,14cyclictest2514791-21ssh11:54:060
2429544995929,14cyclictest2514791-21ssh11:54:050
2429544995919,35cyclictest16850irq/134-800000012:10:020
242955899585,12cyclictest37-21ksoftirqd/312:40:013
242955899585,12cyclictest37-21ksoftirqd/312:40:003
242955899584,9cyclictest37-21ksoftirqd/308:45:003
2429558995812,5cyclictest131rcu_preempt09:30:533
242955399581,53cyclictest2430294-21munin-run07:25:012
242955399581,52cyclictest48-21kcompactd008:16:322
242955399581,52cyclictest48-21kcompactd008:16:322
242955399581,51cyclictest48-21kcompactd008:50:242
2429549995838,9cyclictest23-21ksoftirqd/112:00:011
2429544995848,6cyclictest309-21in:imuxsock11:25:010
2429544995813,36cyclictest12-21ksoftirqd/011:56:070
2429544995810,6cyclictest131rcu_preempt10:45:010
2429544995810,6cyclictest131rcu_preempt10:45:010
242955899573,7cyclictest37-21ksoftirqd/312:20:013
2429558995717,6cyclictest131rcu_preempt09:42:463
2429558995714,5cyclictest131rcu_preempt11:13:523
2429558995714,5cyclictest131rcu_preempt11:13:513
242955399573,49cyclictest364-21snmpd08:27:092
242955399571,52cyclictest2429982-21sshd07:19:002
242955399571,52cyclictest2429982-21sshd07:19:002
2429544995715,38cyclictest16850irq/134-800000009:33:040
2429544995711,11cyclictest12-21ksoftirqd/007:42:410
2429544995710,34cyclictest131rcu_preempt12:36:250
2429544995710,34cyclictest131rcu_preempt12:36:240
242955899568,6cyclictest131rcu_preempt10:40:453
242955899568,6cyclictest131rcu_preempt10:40:443
242955899568,10cyclictest37-21ksoftirqd/312:12:163
2429558995614,5cyclictest131rcu_preempt11:07:283
2429558995613,5cyclictest131rcu_preempt09:45:573
2429549995613,5cyclictest131rcu_preempt10:25:371
2429544995628,13cyclictest2459100-21ssh09:58:270
2429544995628,12cyclictest2530236-21ssh12:25:510
2429544995628,12cyclictest2530236-21ssh12:25:510
2429544995628,12cyclictest2470695-21ssh10:22:110
2429544995628,12cyclictest2470695-21ssh10:22:100
2429544995619,32cyclictest16850irq/134-800000012:30:500
242955899556,9cyclictest37-21ksoftirqd/312:00:003
2429558995514,6cyclictest131rcu_preempt12:01:093
2429558995514,6cyclictest131rcu_preempt11:47:333
2429558995513,4cyclictest131rcu_preempt09:38:253
2429549995513,6cyclictest131rcu_preempt09:10:381
2429549995512,5cyclictest131rcu_preempt08:15:001
2429549995511,6cyclictest131rcu_preempt11:33:261
2429549995511,6cyclictest131rcu_preempt11:33:261
2429549995510,6cyclictest131rcu_preempt10:52:411
2429549995510,6cyclictest131rcu_preempt10:52:401
242954499555,37cyclictest131rcu_preempt11:16:400
2429544995515,37cyclictest1-21systemd09:20:020
2429544995515,37cyclictest1-21systemd09:20:010
2429558995418,6cyclictest131rcu_preempt11:31:483
2429558995418,6cyclictest131rcu_preempt11:31:473
2429558995414,5cyclictest131rcu_preempt10:15:113
2429558995410,5cyclictest131rcu_preempt10:50:143
2429558995410,5cyclictest131rcu_preempt10:50:133
2429558995410,5cyclictest131rcu_preempt10:01:273
2429549995416,9cyclictest131rcu_preempt11:18:281
2429549995413,6cyclictest131rcu_preempt10:41:361
2429549995413,6cyclictest131rcu_preempt10:41:351
2429549995412,5cyclictest131rcu_preempt10:46:321
2429549995412,5cyclictest131rcu_preempt10:02:131
2429549995410,5cyclictest131rcu_preempt12:07:101
2429544995432,12cyclictest0-21swapper/009:27:500
2429544995423,11cyclictest12-21ksoftirqd/011:01:280
2429544995423,11cyclictest12-21ksoftirqd/011:01:280
242955899539,6cyclictest131rcu_preempt11:17:563
242955899538,6cyclictest131rcu_preempt11:03:013
242955899538,6cyclictest131rcu_preempt11:03:013
242955899537,8cyclictest37-21ksoftirqd/312:30:033
242955899537,8cyclictest37-21ksoftirqd/312:30:023
242955899536,12cyclictest37-21ksoftirqd/309:25:013
242955899535,9cyclictest37-21ksoftirqd/310:30:293
242955899535,9cyclictest37-21ksoftirqd/310:30:293
2429558995330,8cyclictest37-21ksoftirqd/309:05:013
2429558995312,5cyclictest131rcu_preempt10:39:343
2429558995311,6cyclictest131rcu_preempt10:11:433
2429558995310,5cyclictest131rcu_preempt11:41:523
2429558995310,5cyclictest131rcu_preempt09:18:053
2429558995310,5cyclictest131rcu_preempt09:18:053
242954999537,5cyclictest131rcu_preempt10:22:491
242954999537,5cyclictest131rcu_preempt10:22:481
2429549995314,5cyclictest131rcu_preempt11:03:361
2429549995314,5cyclictest131rcu_preempt11:03:361
2429549995312,6cyclictest131rcu_preempt12:27:551
2429549995312,6cyclictest131rcu_preempt12:27:541
2429549995312,6cyclictest131rcu_preempt11:45:001
2429549995311,5cyclictest131rcu_preempt12:25:001
2429549995311,5cyclictest131rcu_preempt09:49:391
2429544995325,13cyclictest2523235-21ssh12:11:250
2429544995315,7cyclictest12-21ksoftirqd/011:41:540
242954499531,37cyclictest2442122-21ssh09:23:050
242955899527,12cyclictest37-21ksoftirqd/309:00:023
2429558995244,5cyclictest361rcuc/307:45:163
2429558995218,5cyclictest131rcu_preempt12:05:503
2429558995215,6cyclictest131rcu_preempt11:52:153
2429558995215,6cyclictest131rcu_preempt11:52:153
2429558995213,5cyclictest131rcu_preempt12:34:453
2429558995213,5cyclictest131rcu_preempt12:22:273
2429558995212,5cyclictest131rcu_preempt09:53:293
2429558995211,6cyclictest131rcu_preempt10:24:103
2429558995211,6cyclictest131rcu_preempt10:24:103
242954999528,11cyclictest23-21ksoftirqd/111:54:051
242954999528,11cyclictest23-21ksoftirqd/111:54:041
242954999524,10cyclictest23-21ksoftirqd/109:56:411
2429549995213,6cyclictest131rcu_preempt11:28:451
2429549995213,6cyclictest131rcu_preempt09:23:511
2429549995212,7cyclictest131rcu_preempt12:15:571
2429549995210,5cyclictest131rcu_preempt12:01:301
2429549995210,5cyclictest131rcu_preempt10:12:001
2429549995210,5cyclictest131rcu_preempt09:51:391
242954499529,6cyclictest131rcu_preempt12:05:020
242954499525,6cyclictest131rcu_preempt11:12:390
242954499525,6cyclictest131rcu_preempt11:12:380
2429544995236,12cyclictest3467602-21systemd-journal10:50:020
2429544995232,15cyclictest3467602-21systemd-journal08:15:000
2429544995225,12cyclictest2475001-21ssh10:31:020
2429544995225,12cyclictest2475001-21ssh10:31:010
2429544995215,32cyclictest16850irq/134-800000010:07:460
242955899519,5cyclictest131rcu_preempt10:05:553
242955899515,10cyclictest37-21ksoftirqd/308:25:003
2429558995115,5cyclictest131rcu_preempt11:37:083
2429558995113,5cyclictest131rcu_preempt10:46:223
2429558995112,5cyclictest131rcu_preempt09:27:333
242955899511,13cyclictest2432849-21/usr/sbin/munin08:15:013
242954999519,5cyclictest131rcu_preempt09:15:381
242954999519,5cyclictest131rcu_preempt09:15:371
242954999518,11cyclictest23-21ksoftirqd/108:45:011
242954999515,6cyclictest131rcu_preempt07:55:021
242954999515,6cyclictest131rcu_preempt07:55:011
2429549995114,6cyclictest131rcu_preempt11:20:061
2429549995112,5cyclictest131rcu_preempt11:39:091
2429549995111,6cyclictest131rcu_preempt11:13:281
2429549995111,6cyclictest131rcu_preempt11:13:271
242954499517,20cyclictest12-21ksoftirqd/009:39:460
2429544995141,7cyclictest12-21ksoftirqd/009:50:020
2429544995123,11cyclictest12-21ksoftirqd/010:19:340
2429544995114,32cyclictest16850irq/134-800000011:48:180
242955899506,8cyclictest37-21ksoftirqd/307:35:013
242955899506,8cyclictest37-21ksoftirqd/307:35:013
242955899505,7cyclictest37-21ksoftirqd/311:25:553
2429558995011,12cyclictest37-21ksoftirqd/311:22:373
2429558995010,4cyclictest131rcu_preempt08:40:013
242954999508,5cyclictest131rcu_preempt12:32:581
242954999506,5cyclictest131rcu_preempt10:55:571
242954999504,9cyclictest23-21ksoftirqd/111:47:381
242954999503,9cyclictest23-21ksoftirqd/108:20:011
242954999503,9cyclictest23-21ksoftirqd/108:20:011
2429549995014,6cyclictest131rcu_preempt09:44:541
2429549995011,6cyclictest131rcu_preempt11:05:051
2429549995011,6cyclictest131rcu_preempt09:39:341
2429549995011,5cyclictest131rcu_preempt12:38:341
2429549995011,5cyclictest131rcu_preempt12:38:331
2429549995010,6cyclictest131rcu_preempt10:16:131
2429544995021,14cyclictest2451951-21ssh09:43:280
2429544995013,32cyclictest16850irq/134-800000009:10:080
242955899499,5cyclictest131rcu_preempt10:55:383
242955899499,5cyclictest131rcu_preempt07:45:023
242955899494,8cyclictest37-21ksoftirqd/308:20:003
242955899494,8cyclictest37-21ksoftirqd/308:20:003
242954999494,11cyclictest23-21ksoftirqd/107:15:001
2429549994936,9cyclictest1-21systemd09:00:011
2429549994910,6cyclictest131rcu_preempt10:34:251
2429549994910,6cyclictest131rcu_preempt10:34:241
242954499491,41cyclictest0-21swapper/011:27:220
242955899487,7cyclictest37-21ksoftirqd/309:10:013
242955899487,7cyclictest37-21ksoftirqd/309:10:003
242955899482,10cyclictest37-21ksoftirqd/307:20:003
242955899482,10cyclictest37-21ksoftirqd/307:20:003
2429558994812,6cyclictest131rcu_preempt07:15:003
242954999489,5cyclictest131rcu_preempt10:06:171
242954999486,5cyclictest131rcu_preempt09:30:181
242954999485,5cyclictest131rcu_preempt08:25:031
242954999485,12cyclictest23-21ksoftirqd/107:35:011
242954999485,12cyclictest23-21ksoftirqd/107:35:011
242954999483,8cyclictest23-21ksoftirqd/107:40:001
2429549994812,5cyclictest131rcu_preempt09:28:591
2429549994811,6cyclictest131rcu_preempt12:14:101
2429544994813,20cyclictest16850irq/134-800000011:05:200
2429544994812,6cyclictest131rcu_preempt08:05:000
242955899478,5cyclictest131rcu_preempt07:40:013
242955899477,6cyclictest131rcu_preempt07:25:013
2429549994734,9cyclictest1-21systemd08:40:011
242954999472,10cyclictest23-21ksoftirqd/107:45:001
2429549994710,6cyclictest131rcu_preempt07:25:011
2429544994712,31cyclictest16850irq/134-800000010:26:340
242955899467,7cyclictest37-21ksoftirqd/308:35:003
242955899465,5cyclictest131rcu_preempt08:30:013
242955899465,4cyclictest131rcu_preempt07:30:013
242954499466,5cyclictest131rcu_preempt07:40:000
2429544994635,6cyclictest2429545-21latency_hist07:10:020
2429544994635,6cyclictest2429545-21latency_hist07:10:020
242955899452,21cyclictest37-21ksoftirqd/308:50:023
242955899452,21cyclictest37-21ksoftirqd/308:50:013
242954999458,6cyclictest131rcu_preempt07:18:011
242954999458,6cyclictest131rcu_preempt07:18:001
242954999458,5cyclictest131rcu_preempt08:50:031
242954999458,5cyclictest131rcu_preempt08:50:021
242954999456,8cyclictest23-21ksoftirqd/108:00:021
242954999456,6cyclictest131rcu_preempt07:30:001
2429549994529,12cyclictest3467602-21systemd-journal09:10:011
2429549994529,12cyclictest3467602-21systemd-journal09:10:001
242954499454,5cyclictest131rcu_preempt07:50:030
2429544994537,4cyclictest16850irq/134-800000009:10:010
2429544994537,4cyclictest16850irq/134-800000009:10:000
2429544994528,13cyclictest2434897-21sshd08:50:420
242954499452,37cyclictest2435189-21modprobe09:00:000
2429544994514,5cyclictest131rcu_preempt07:17:010
2429544994514,5cyclictest131rcu_preempt07:17:000
2429558994411,5cyclictest131rcu_preempt08:00:023
242955399442,37cyclictest2429640-21cat07:10:022
242955399442,37cyclictest2429640-21cat07:10:022
242954999446,6cyclictest131rcu_preempt08:35:021
2429549994436,5cyclictest23-21ksoftirqd/108:10:021
2429544994428,12cyclictest2433395-21sshd08:25:000
2429544994421,15cyclictest0-21swapper/007:12:370
2429544994420,10cyclictest12-21ksoftirqd/008:20:010
2429544994420,10cyclictest12-21ksoftirqd/008:20:000
2429558994313,10cyclictest37-21ksoftirqd/308:10:013
2429544994326,12cyclictest3467602-21systemd-journal07:30:150
2429544994326,12cyclictest3467602-21systemd-journal07:30:150
242954499431,5cyclictest131rcu_preempt08:00:010
242954499424,9cyclictest12-21ksoftirqd/008:35:020
242954499421,8cyclictest2430527-21sshd07:30:000
2429544994218,8cyclictest12-21ksoftirqd/008:10:020
242955899411,9cyclictest2431816-21/usr/sbin/munin07:55:013
242955899411,9cyclictest2431816-21/usr/sbin/munin07:55:013
242954999417,5cyclictest131rcu_preempt08:55:011
242954999414,9cyclictest23-21ksoftirqd/107:47:411
242954999414,5cyclictest131rcu_preempt08:30:001
242954999414,11cyclictest23-21ksoftirqd/109:05:011
2429549994129,8cyclictest3467602-21systemd-journal08:05:001
242954499416,6cyclictest131rcu_preempt07:55:000
242954499416,6cyclictest131rcu_preempt07:55:000
2429544994025,11cyclictest12-21ksoftirqd/007:25:000
242954499401,35cyclictest0-21swapper/008:40:000
242954999383,6cyclictest131rcu_preempt07:10:021
242954999383,6cyclictest131rcu_preempt07:10:021
242954499382,9cyclictest3467602-21systemd-journal09:02:040
2429544993813,6cyclictest131rcu_preempt08:30:010
2429544993722,11cyclictest2434780-21cat08:50:020
2429544993722,11cyclictest2434780-21cat08:50:020
2429558993628,6cyclictest37-21ksoftirqd/308:52:003
242954499341,29cyclictest2434147-21latency_hist08:40:020
2429558993023,4cyclictest2429659-21latency_hist07:10:033
2429558993023,4cyclictest2429659-21latency_hist07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional