You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-10 - 00:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 09, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
539385991322,126cyclictest0-21swapper/008:05:000
539385998311,39cyclictest19350irq/34-mmc107:55:010
539388997318,12cyclictest23-21ksoftirqd/112:15:001
53938599704,31cyclictest12-21ksoftirqd/009:20:020
53938599704,31cyclictest12-21ksoftirqd/009:20:020
53939799684,57cyclictest638519-21ssh12:40:003
53939799682,14cyclictest552719-21ssh09:25:463
53939799673,18cyclictest631539-21sh12:24:173
53939799671,13cyclictest635234-21ssh12:32:203
53939799671,13cyclictest635234-21ssh12:32:203
53939799671,13cyclictest567113-21ssh09:57:583
53939799662,17cyclictest543250-21munin-run08:25:013
53939799662,16cyclictest542414-21sshd08:06:283
53939799662,14cyclictest629472-21ssh12:19:453
53939799662,13cyclictest580676-21ssh10:28:113
53939799661,13cyclictest622660-21ssh12:04:223
539385996662,2cyclictest0-21swapper/008:30:010
539397996528,34cyclictest37-21ksoftirqd/308:50:023
539397996528,34cyclictest37-21ksoftirqd/308:50:023
53939799652,15cyclictest620310-21ssh11:59:083
53939799652,15cyclictest551282-21ssh09:22:433
53939799652,14cyclictest584584-21ssh10:36:443
539397996514,46cyclictest275-21dbus-daemon12:10:013
53939799651,15cyclictest568432-21ssh10:00:413
53939799651,14cyclictest606928-21ssh11:29:173
53939799651,14cyclictest598910-21ssh11:11:073
53939799651,14cyclictest586603-21ssh10:41:083
539388996515,6cyclictest131rcu_preempt12:05:011
53938599652,5cyclictest121ksoftirqd/012:10:040
53939799644,23cyclictest547830-21wc09:15:023
53939799643,7cyclictest540109-21sshd07:24:423
53939799642,14cyclictest558641-21ssh09:39:153
53939799642,14cyclictest558641-21ssh09:39:153
53939799642,13cyclictest615144-21ssh11:47:253
53939799641,17cyclictest583208-21ssh10:33:583
53939799641,15cyclictest571634-21ssh10:07:563
53939799641,15cyclictest571634-21ssh10:07:563
53939799641,14cyclictest596772-21ssh11:06:193
53939799641,14cyclictest591493-21ssh10:52:093
53939799641,13cyclictest604000-21ssh11:22:343
53939799641,12cyclictest578458-21ssh10:23:103
53939799641,12cyclictest573170-21ssh10:11:103
53939799641,12cyclictest562460-21ssh09:47:373
53939799641,12cyclictest562460-21ssh09:47:373
53939799641,11cyclictest593213-21ssh10:55:483
53939799632,19cyclictest575456-21ssh10:16:203
53939799632,15cyclictest548339-21ssh09:16:013
53939799632,15cyclictest548339-21ssh09:16:003
53939799632,13cyclictest555336-21ssh09:31:333
53939799632,12cyclictest632253-21ssh12:25:343
53939799632,12cyclictest626660-21ssh12:13:233
53939799631,9cyclictest540606-21sshd07:33:293
53939799631,15cyclictest544035-21munin-run08:40:013
53939799631,12cyclictest595917-21ssh11:04:453
53939799631,11cyclictest608495-21ssh11:32:373
53939599636,7cyclictest30-21ksoftirqd/212:10:032
53938899633,9cyclictest23-21ksoftirqd/111:20:001
53939799621,14cyclictest601435-21ssh11:16:463
53939799621,13cyclictest617962-21ssh11:53:513
53939799621,13cyclictest612498-21ssh11:41:243
53939799621,13cyclictest565108-21ssh09:53:413
53939799621,12cyclictest610104-21ssh11:36:083
53939599624,11cyclictest30-21ksoftirqd/210:30:002
53939799617,3cyclictest0-21swapper/308:04:053
53939799611,11cyclictest589540-21ssh10:47:503
53939799611,11cyclictest589540-21ssh10:47:503
53938599615,6cyclictest12-21ksoftirqd/009:29:180
539385996119,17cyclictest16850irq/134-800000009:37:320
539385996119,17cyclictest16850irq/134-800000009:37:320
539385996112,39cyclictest16850irq/134-800000012:00:000
53939799603,15cyclictest541159-21modprobe07:44:593
53939799601,11cyclictest560564-21ssh09:43:273
53938599606,29cyclictest12-21ksoftirqd/009:55:360
53939799594,47cyclictest541414-21munin-plugin-st07:50:003
53939799592,9cyclictest540859-21sshd07:38:283
53939799591,13cyclictest544993-21sshd08:55:293
53939799591,13cyclictest542699-21munin-run08:15:013
53938899597,12cyclictest23-21ksoftirqd/110:39:281
53938899595,8cyclictest23-21ksoftirqd/107:15:021
53938599598,28cyclictest627851-21ssh12:15:440
53939799584,46cyclictest544776-21munin-plugin-st08:55:003
539395995843,7cyclictest579245-21kworker/2:111:03:152
539395995842,8cyclictest596067-21kworker/2:212:23:222
53938899586,7cyclictest23-21ksoftirqd/110:00:021
53938899585,8cyclictest23-21ksoftirqd/110:45:021
53938899584,10cyclictest23-21ksoftirqd/111:11:221
539385995818,30cyclictest16850irq/134-800000010:07:240
539385995818,30cyclictest16850irq/134-800000010:07:240
53939799572,7cyclictest543724-21sshd08:34:563
539397995711,40cyclictest275-21dbus-daemon07:55:013
539395995742,7cyclictest540653-21kworker/2:010:58:422
53938899577,10cyclictest23-21ksoftirqd/110:17:371
53938899576,8cyclictest23-21ksoftirqd/109:24:361
53938899576,12cyclictest23-21ksoftirqd/109:51:321
539388995716,8cyclictest131rcu_preempt12:25:041
539388995711,5cyclictest131rcu_preempt12:09:531
539385995739,15cyclictest16850irq/134-800000011:35:000
53938599572,5cyclictest121ksoftirqd/011:28:390
539385995714,3cyclictest0-21swapper/010:30:020
53939799562,10cyclictest543474-21cron08:30:003
53939799561,12cyclictest48-21kcompactd007:25:493
53939799561,12cyclictest364-21snmpd07:12:133
539395995647,5cyclictest544544-21kworker/2:210:23:302
539395995642,6cyclictest540653-21kworker/2:010:52:232
539395995640,7cyclictest609433-21kworker/2:011:56:072
53938899569,6cyclictest131rcu_preempt12:34:031
53938899569,6cyclictest131rcu_preempt12:34:031
53938899565,8cyclictest23-21ksoftirqd/111:35:011
539388995616,7cyclictest131rcu_preempt09:44:521
539388995613,6cyclictest131rcu_preempt11:50:541
539388995610,6cyclictest131rcu_preempt11:27:061
539385995630,11cyclictest16850irq/134-800000012:22:550
539385995621,14cyclictest0-21swapper/010:49:410
539385995621,14cyclictest0-21swapper/010:49:410
53938599561,4cyclictest12-21ksoftirqd/009:33:040
53939799553,9cyclictest275-21dbus-daemon08:20:003
53939799553,26cyclictest3467602-21systemd-journal07:20:003
53939799552,22cyclictest544411-21wc08:45:023
53939799551,10cyclictest364-21snmpd09:09:183
539395995541,6cyclictest636308-21kworker/2:312:36:582
53938599555,13cyclictest12-21ksoftirqd/011:18:150
539385995528,12cyclictest562999-21ssh09:48:580
539385995528,12cyclictest562999-21ssh09:48:580
539385995514,15cyclictest16850irq/134-800000010:24:000
53939799545,3cyclictest0-21swapper/307:55:333
53939799545,3cyclictest0-21swapper/307:55:333
53939799542,7cyclictest364-21snmpd09:02:053
539395995413,7cyclictest131rcu_preempt09:40:362
53938899543,10cyclictest23-21ksoftirqd/112:24:331
539388995416,5cyclictest131rcu_preempt10:06:591
539388995416,5cyclictest131rcu_preempt10:06:591
539388995410,5cyclictest131rcu_preempt09:34:221
53938599547,9cyclictest12-21ksoftirqd/009:40:200
539385995425,14cyclictest611603-21ssh11:39:580
539385995419,17cyclictest16850irq/134-800000010:12:530
539385995418,18cyclictest16850irq/134-800000012:37:340
53939599539,6cyclictest131rcu_preempt11:43:542
539395995346,4cyclictest544544-21kworker/2:209:34:282
539395995344,6cyclictest544544-21kworker/2:209:51:452
539395995343,6cyclictest579245-21kworker/2:110:32:332
539395995329,10cyclictest30-21ksoftirqd/212:30:292
539395995329,10cyclictest30-21ksoftirqd/212:30:292
539395995316,14cyclictest30-21ksoftirqd/211:27:282
53938899537,5cyclictest131rcu_preempt11:47:451
53938899535,8cyclictest23-21ksoftirqd/111:05:001
539388995314,5cyclictest131rcu_preempt10:47:131
539388995314,5cyclictest131rcu_preempt10:47:121
53938899531,38cyclictest544496-21kworker/u8:109:00:021
539388995311,6cyclictest131rcu_preempt11:40:171
539388995311,6cyclictest131rcu_preempt09:49:591
539388995311,6cyclictest131rcu_preempt09:49:591
53938599539,11cyclictest12-21ksoftirqd/011:49:370
53938599536,5cyclictest16850irq/134-800000009:51:460
53938599535,7cyclictest12-21ksoftirqd/007:30:010
53938599535,10cyclictest12-21ksoftirqd/010:55:510
539385995344,5cyclictest151rcuc/011:10:030
539385995319,18cyclictest16850irq/134-800000010:04:010
539385995319,17cyclictest16850irq/134-800000010:54:550
539385995313,6cyclictest131rcu_preempt12:07:250
53939599525,8cyclictest30-21ksoftirqd/207:40:002
53939599525,7cyclictest30-21ksoftirqd/210:35:282
539395995236,8cyclictest540653-21kworker/2:010:41:492
53938899528,9cyclictest23-21ksoftirqd/110:13:281
53938899527,8cyclictest23-21ksoftirqd/109:10:481
539388995226,15cyclictest23-21ksoftirqd/107:55:011
53938899522,10cyclictest23-21ksoftirqd/112:39:021
539388995212,6cyclictest131rcu_preempt10:33:351
539388995212,6cyclictest131rcu_preempt10:01:451
53938599527,12cyclictest12-21ksoftirqd/010:36:130
53938599526,14cyclictest12-21ksoftirqd/010:17:470
539385995245,4cyclictest16850irq/134-800000009:10:010
539385995218,6cyclictest16850irq/134-800000011:06:200
539385995216,18cyclictest16850irq/134-800000012:01:550
539385995216,18cyclictest16850irq/134-800000011:42:570
539385995213,10cyclictest12-21ksoftirqd/010:30:310
539385995210,38cyclictest12-21ksoftirqd/011:22:560
53939799512,8cyclictest539477-21latency_hist07:10:023
53939599519,5cyclictest131rcu_preempt09:37:482
53939599519,5cyclictest131rcu_preempt09:37:482
539395995143,5cyclictest609433-21kworker/2:011:51:042
539395995143,5cyclictest596067-21kworker/2:212:13:432
539395995143,5cyclictest596067-21kworker/2:211:10:202
539395995143,5cyclictest579245-21kworker/2:110:49:292
539395995143,5cyclictest579245-21kworker/2:110:49:292
539395995143,5cyclictest540653-21kworker/2:010:10:192
539395995143,5cyclictest540653-21kworker/2:009:13:192
539395995142,6cyclictest544544-21kworker/2:210:00:032
539395995138,6cyclictest540653-21kworker/2:009:15:082
539395995138,6cyclictest540653-21kworker/2:009:15:082
539395995111,6cyclictest131rcu_preempt11:19:382
53938899519,7cyclictest131rcu_preempt11:21:361
53938899519,5cyclictest131rcu_preempt10:25:471
53938899518,10cyclictest23-21ksoftirqd/112:16:531
53938899516,11cyclictest23-21ksoftirqd/108:30:011
53938899515,10cyclictest23-21ksoftirqd/111:40:021
539388995112,5cyclictest131rcu_preempt09:30:011
539388995112,10cyclictest23-21ksoftirqd/109:40:021
539388995112,10cyclictest23-21ksoftirqd/109:40:021
539388995111,5cyclictest131rcu_preempt10:58:421
539388995110,5cyclictest131rcu_preempt10:25:011
53938599514,12cyclictest12-21ksoftirqd/008:40:010
539385995126,9cyclictest588152-21ssh10:45:020
539385995117,5cyclictest16850irq/134-800000009:23:150
53939599506,6cyclictest131rcu_preempt09:21:432
539395995042,5cyclictest596067-21kworker/2:212:02:562
539395995042,5cyclictest579245-21kworker/2:111:07:132
539395995042,5cyclictest544544-21kworker/2:209:59:372
539395995042,5cyclictest540653-21kworker/2:010:05:102
539395995042,5cyclictest540653-21kworker/2:010:05:102
539395995040,6cyclictest596067-21kworker/2:211:30:182
539395995039,6cyclictest411739-21kworker/2:207:31:312
539395995024,9cyclictest30-21ksoftirqd/209:29:282
53938899506,9cyclictest23-21ksoftirqd/109:19:231
53938899506,9cyclictest23-21ksoftirqd/109:19:231
539388995041,6cyclictest221rcuc/111:58:481
53938599509,5cyclictest131rcu_preempt12:27:330
539385995023,12cyclictest635264-21ssh12:32:250
539385995023,12cyclictest635264-21ssh12:32:240
539395994942,5cyclictest596067-21kworker/2:212:18:222
539395994942,5cyclictest596067-21kworker/2:211:38:382
539395994942,4cyclictest596067-21kworker/2:212:26:272
539395994941,5cyclictest609433-21kworker/2:011:47:222
539395994941,5cyclictest544544-21kworker/2:210:16:472
539395994941,5cyclictest541173-21kworker/2:108:05:052
539395994941,5cyclictest540653-21kworker/2:009:45:272
539395994941,5cyclictest540653-21kworker/2:009:45:272
539395994940,6cyclictest541173-21kworker/2:108:10:402
539395994925,9cyclictest30-21ksoftirqd/211:22:012
53938899494,8cyclictest23-21ksoftirqd/108:09:471
539388994933,12cyclictest3467602-21systemd-journal08:34:561
539388994910,6cyclictest131rcu_preempt11:07:171
53938599493,9cyclictest12-21ksoftirqd/009:10:120
539385994915,6cyclictest16850irq/134-800000011:52:200
539385994914,6cyclictest16850irq/134-800000011:04:190
539395994841,4cyclictest540653-21kworker/2:007:47:422
539395994840,5cyclictest541173-21kworker/2:108:33:342
539395994840,5cyclictest541173-21kworker/2:108:01:572
539395994840,5cyclictest541173-21kworker/2:107:51:552
539395994840,5cyclictest540653-21kworker/2:009:07:072
539395994840,5cyclictest540653-21kworker/2:008:50:222
539395994840,5cyclictest540653-21kworker/2:008:20:022
539395994840,5cyclictest538783-21kworker/2:307:24:302
53938899486,11cyclictest23-21ksoftirqd/107:45:001
53938899481,12cyclictest23-21ksoftirqd/110:51:131
53938599485,7cyclictest12-21ksoftirqd/007:10:010
539395994739,5cyclictest544544-21kworker/2:208:55:142
539395994739,5cyclictest541173-21kworker/2:108:45:172
539395994739,5cyclictest541173-21kworker/2:108:45:172
539395994739,5cyclictest541173-21kworker/2:108:42:292
539395994739,5cyclictest541173-21kworker/2:108:35:402
539395994739,5cyclictest541173-21kworker/2:107:55:172
539395994739,5cyclictest541173-21kworker/2:107:55:172
539395994739,5cyclictest540653-21kworker/2:009:01:422
539395994739,5cyclictest540653-21kworker/2:008:26:082
539395994739,5cyclictest540653-21kworker/2:008:21:112
539395994739,5cyclictest540653-21kworker/2:007:41:222
539395994739,5cyclictest538783-21kworker/2:307:15:092
539395994739,5cyclictest538783-21kworker/2:307:12:002
539395994738,6cyclictest538783-21kworker/2:307:26:242
539385994711,9cyclictest0-21swapper/007:39:040
539385994638,6cyclictest19350irq/34-mmc108:45:010
539385994633,9cyclictest1-21systemd07:35:000
539385994631,11cyclictest3467602-21systemd-journal08:00:000
539385994631,11cyclictest3467602-21systemd-journal08:00:000
53938599462,33cyclictest542799-21latency_hist08:15:020
539388994538,5cyclictest539393-21cat07:10:011
53938899451,8cyclictest23-21ksoftirqd/108:35:281
53938899443,8cyclictest23-21ksoftirqd/108:55:011
53938899443,8cyclictest23-21ksoftirqd/108:45:011
53938599443,37cyclictest539853-21munin-run07:20:010
539385994412,8cyclictest0-21swapper/007:49:480
539388994320,6cyclictest221rcuc/108:49:281
539388994320,6cyclictest221rcuc/108:49:281
539388994310,5cyclictest131rcu_preempt07:59:041
539388994310,5cyclictest131rcu_preempt07:59:041
539388994310,5cyclictest131rcu_preempt07:35:001
539385994337,3cyclictest16950irq/224-800000008:50:030
53938599433,11cyclictest12-21ksoftirqd/008:55:000
53938599431,13cyclictest545013-21sshd09:00:000
539385994311,29cyclictest0-21swapper/007:43:350
53938899427,5cyclictest131rcu_preempt08:01:281
53938899425,32cyclictest545767-21cat09:10:021
53938899417,5cyclictest131rcu_preempt07:47:281
53938899415,7cyclictest23-21ksoftirqd/107:30:011
53938899415,5cyclictest131rcu_preempt07:25:021
53938599413,9cyclictest12-21ksoftirqd/008:10:000
539385994129,8cyclictest539558-21sshd07:12:280
53938899406,5cyclictest131rcu_preempt08:25:011
539388994029,8cyclictest23-21ksoftirqd/107:39:281
53938599402,9cyclictest12-21ksoftirqd/008:17:000
53938599402,34cyclictest0-21swapper/008:35:020
539385994010,7cyclictest0-21swapper/008:48:120
53938899395,5cyclictest131rcu_preempt08:20:021
53938599393,9cyclictest12-21ksoftirqd/009:05:000
539385993925,11cyclictest540093-21sshd07:20:290
53938899381,8cyclictest23-21ksoftirqd/107:16:011
53938899381,32cyclictest309-21in:imuxsock09:00:031
539395993728,6cyclictest30-21ksoftirqd/207:10:012
539388993721,12cyclictest23-21ksoftirqd/108:11:281
539385993724,10cyclictest275-21dbus-daemon08:25:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional