You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 16:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 02, 2026 12:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18560991211,3cyclictest0-21swapper/009:20:030
1856099961,56cyclictest0-21swapper/010:56:430
18560999213,46cyclictest100307-21ssh12:04:350
1856099912,52cyclictest0-21swapper/010:35:300
18565998574,8cyclictest23-21ksoftirqd/109:25:011
18560998426,53cyclictest41960-21ssh09:49:210
18565998367,13cyclictest23-21ksoftirqd/111:15:031
18560998327,37cyclictest73491-21ssh11:02:150
18560998216,60cyclictest16850irq/134-800000009:42:160
1857099792,65cyclictest72456-21sh11:00:022
18560997938,35cyclictest3467602-21systemd-journal09:10:000
18560997933,28cyclictest97223-21ssh11:57:220
18560997932,43cyclictest95152-21ssh11:52:300
18560997827,34cyclictest67318-21ssh10:48:080
1856599776,62cyclictest55225-21rm10:20:021
1856599776,62cyclictest55225-21rm10:20:011
1856099765,43cyclictest0-21swapper/010:07:070
1856099765,43cyclictest0-21swapper/010:07:070
18560997627,44cyclictest75533-21ssh11:07:040
18560997625,47cyclictest77575-21ssh11:11:430
1856099752,55cyclictest85603-21ssh11:30:100
18560997517,43cyclictest16850irq/134-800000010:04:310
1856099751,5cyclictest61561-21munin-run10:35:000
1856099751,57cyclictest113380-21ssh12:34:570
18560997424,33cyclictest108603-21ssh12:23:330
18560997327,42cyclictest44623-21ssh09:55:120
1856599727,12cyclictest23-21ksoftirqd/110:44:591
18560997218,36cyclictest32883-21ssh09:28:190
18560997218,36cyclictest32883-21ssh09:28:190
1856099713,4cyclictest0-21swapper/011:20:010
18560997127,27cyclictest92503-21ssh11:46:140
18560997125,29cyclictest103502-21ssh12:11:380
18560997066,2cyclictest0-21swapper/012:25:020
18560997027,26cyclictest105261-21ssh12:15:360
1856099693,5cyclictest65899-21munin-run10:45:000
18560996922,31cyclictest90816-21ssh11:42:300
18560996916,48cyclictest88402-21ssh11:36:540
1856099691,4cyclictest20704-21unin-run07:50:010
18570996832,9cyclictest30-21ksoftirqd/210:24:382
1856099683,60cyclictest102688-21modprobe12:10:000
1856099683,44cyclictest3550850ssh09:34:260
18560996828,36cyclictest36511-21rm09:36:360
1857499677,39cyclictest120-21kswapd011:10:023
18560996730,32cyclictest82055-21ssh11:22:050
18565996634,8cyclictest23-21ksoftirqd/111:51:571
18560996629,33cyclictest57507-21ssh10:25:050
1856099661,4cyclictest84018-21sh11:26:370
1856099652,13cyclictest24587-21munin-run09:05:000
1856099642,9cyclictest25492-21sshd09:11:000
1856099642,13cyclictest22783-21munin-run08:30:010
18560996416,44cyclictest16850irq/134-800000010:50:170
1856099641,4cyclictest0-21swapper/009:50:180
18560996414,45cyclictest54469-21ssh10:18:300
18560996414,45cyclictest54469-21ssh10:18:300
1856599635,5cyclictest131rcu_preempt09:50:021
18565996342,16cyclictest275-21dbus-daemon10:40:001
18560996326,33cyclictest51046-21modprobe10:10:030
1856599623,9cyclictest23-21ksoftirqd/112:40:011
1856099623,5cyclictest18779-21sshd07:12:210
1856099621,57cyclictest115314-21sshd12:39:190
1856099612,13cyclictest23242-21sshd08:37:000
1856099611,8cyclictest19047-21sshd07:17:490
18570996051,5cyclictest23044-21kworker/2:209:15:582
1856599606,11cyclictest23-21ksoftirqd/111:05:021
1856099606,50cyclictest18569-21wc07:10:020
1856099606,50cyclictest18569-21wc07:10:010
1856099602,51cyclictest20169-21/usr/sbin/munin07:40:010
1856099601,7cyclictest21943-21sshd08:10:580
1856099601,4cyclictest0-21swapper/010:22:510
1856099601,17cyclictest28801-21sh09:18:520
18570995917,9cyclictest131rcu_preempt12:26:392
18570995914,7cyclictest30-21ksoftirqd/211:48:582
1856099591,17cyclictest21738-21systemd-run08:10:010
1856599582,8cyclictest23-21ksoftirqd/109:15:031
1856099583,46cyclictest21485-21unin-run08:05:010
1856099581,5cyclictest24040-21sshd08:55:000
18565995717,6cyclictest131rcu_preempt11:25:021
18565995714,6cyclictest131rcu_preempt12:10:021
1856099573,6cyclictest364-21snmpd07:29:120
1856099572,14cyclictest23024-21cut08:35:010
1856099571,8cyclictest23676-21latency_hist08:45:020
18570995620,8cyclictest30-21ksoftirqd/211:15:032
1856599569,5cyclictest131rcu_preempt12:13:401
1856599567,8cyclictest23-21ksoftirqd/110:46:391
1856599566,7cyclictest23-21ksoftirqd/112:03:121
18565995615,5cyclictest131rcu_preempt10:57:541
1856099563,46cyclictest3467602-21systemd-journal07:45:000
1856099562,14cyclictest24401-21latency_hist09:00:020
1856099562,13cyclictest22253-21tr08:20:010
1856099561,4cyclictest20912-21sshd07:51:500
18574995532,16cyclictest275-21dbus-daemon11:45:013
1857099558,4cyclictest131rcu_preempt10:54:352
1856599557,6cyclictest131rcu_preempt12:32:441
1856599555,5cyclictest131rcu_preempt09:44:511
18565995511,5cyclictest131rcu_preempt11:35:021
1856099553,9cyclictest19336-21sd-pam07:21:010
1857099546,10cyclictest30-21ksoftirqd/211:54:372
1857099545,7cyclictest30-21ksoftirqd/211:20:522
18570995436,13cyclictest1-21systemd09:55:012
1856599548,6cyclictest131rcu_preempt11:12:261
18565995415,5cyclictest131rcu_preempt09:55:511
18560995430,17cyclictest3467602-21systemd-journal08:50:010
18560995415,32cyclictest311-21rs:main0
18574995327,18cyclictest275-21dbus-daemon08:00:013
1857099536,8cyclictest30-21ksoftirqd/210:35:232
1857099534,15cyclictest30-21ksoftirqd/209:41:512
18570995336,13cyclictest1-21systemd12:00:012
18570995310,7cyclictest30-21ksoftirqd/211:45:022
1856599539,5cyclictest131rcu_preempt10:27:131
1856599536,8cyclictest23-21ksoftirqd/110:32:411
18565995314,10cyclictest23-21ksoftirqd/109:31:471
18565995311,9cyclictest23-21ksoftirqd/110:55:031
1857099527,7cyclictest30-21ksoftirqd/209:49:582
1857099527,11cyclictest30-21ksoftirqd/209:37:282
1857099526,9cyclictest30-21ksoftirqd/209:10:022
1857099526,8cyclictest30-21ksoftirqd/208:50:012
1857099526,7cyclictest30-21ksoftirqd/209:14:172
1857099525,8cyclictest30-21ksoftirqd/211:36:552
1857099525,7cyclictest30-21ksoftirqd/210:10:012
1857099525,7cyclictest30-21ksoftirqd/210:10:012
1857099524,10cyclictest30-21ksoftirqd/207:40:012
18570995221,10cyclictest30-21ksoftirqd/211:25:422
18570995210,7cyclictest30-21ksoftirqd/211:03:472
1856599527,10cyclictest23-21ksoftirqd/111:29:511
18565995234,14cyclictest275-21dbus-daemon08:40:011
18565995213,5cyclictest131rcu_preempt11:56:301
18565995211,5cyclictest131rcu_preempt12:29:271
1856099521,6cyclictest364-21snmpd07:56:160
1857099516,5cyclictest30-21ksoftirqd/212:38:242
1857099515,9cyclictest30-21ksoftirqd/212:20:332
1857099515,9cyclictest30-21ksoftirqd/210:14:132
1857099515,7cyclictest30-21ksoftirqd/207:14:592
1857099514,12cyclictest30-21ksoftirqd/208:25:002
1857099512,11cyclictest30-21ksoftirqd/210:30:042
1856599519,5cyclictest131rcu_preempt09:51:151
1856599517,6cyclictest131rcu_preempt11:40:521
1856599513,7cyclictest23-21ksoftirqd/107:20:491
18565995112,6cyclictest131rcu_preempt09:27:111
18565995112,6cyclictest131rcu_preempt09:27:111
18565995111,6cyclictest131rcu_preempt09:16:121
1856099512,5cyclictest364-21snmpd08:22:340
18574995033,13cyclictest3467602-21systemd-journal08:40:023
1857099506,8cyclictest30-21ksoftirqd/210:26:032
1857099504,8cyclictest30-21ksoftirqd/210:16:312
1857099504,8cyclictest30-21ksoftirqd/210:16:312
1857099504,6cyclictest30-21ksoftirqd/212:01:182
18570995042,5cyclictest291rcuc/209:55:532
1857099504,14cyclictest30-21ksoftirqd/209:32:042
1857099504,12cyclictest30-21ksoftirqd/207:20:012
1857099503,7cyclictest30-21ksoftirqd/209:29:502
1857099503,7cyclictest30-21ksoftirqd/209:29:492
1857099503,15cyclictest79356-21ssh11:15:512
1857099502,13cyclictest30-21ksoftirqd/209:25:022
1856599509,5cyclictest131rcu_preempt10:21:021
1856599507,8cyclictest23-21ksoftirqd/107:20:021
1856599504,6cyclictest131rcu_preempt10:07:001
1856599504,6cyclictest131rcu_preempt10:07:001
1856599501,26cyclictest68908-21kworker/u8:111:35:021
18565995011,6cyclictest131rcu_preempt08:35:011
18574994930,13cyclictest275-21dbus-daemon09:55:013
1857499492,5cyclictest131rcu_preempt08:14:593
1857099494,7cyclictest30-21ksoftirqd/212:18:312
1857099494,7cyclictest0-21swapper/210:50:012
1857099493,12cyclictest30-21ksoftirqd/211:30:352
18570994926,16cyclictest291rcuc/208:55:012
1857099492,11cyclictest30-21ksoftirqd/207:45:022
1856599498,7cyclictest131rcu_preempt12:21:391
1856599498,6cyclictest131rcu_preempt09:36:371
1856599497,9cyclictest23-21ksoftirqd/108:15:001
1856599497,11cyclictest23-21ksoftirqd/110:00:041
1856599491,10cyclictest20164-21sh07:40:001
1857099488,10cyclictest30-21ksoftirqd/208:10:022
1857099486,7cyclictest30-21ksoftirqd/211:05:372
1857099486,10cyclictest30-21ksoftirqd/212:33:302
18570994840,5cyclictest291rcuc/210:44:142
1857099482,13cyclictest30-21ksoftirqd/212:11:562
1857099481,15cyclictest23022-21munin-run08:35:012
1856599489,6cyclictest131rcu_preempt10:10:201
1856599488,7cyclictest23-21ksoftirqd/108:10:031
1856599485,6cyclictest131rcu_preempt07:10:021
1856599485,6cyclictest131rcu_preempt07:10:011
1856599483,8cyclictest23-21ksoftirqd/111:45:361
18565994832,12cyclictest23-21ksoftirqd/112:15:511
1856599482,9cyclictest23-21ksoftirqd/108:45:021
1857099478,11cyclictest30-21ksoftirqd/208:17:012
1857099474,8cyclictest30-21ksoftirqd/207:20:502
1857099474,12cyclictest30-21ksoftirqd/212:07:092
18570994713,5cyclictest131rcu_preempt09:00:012
1857099471,13cyclictest30-21ksoftirqd/208:30:022
1857099471,12cyclictest48763-21sed10:05:032
18574994631,12cyclictest1-21systemd07:25:013
18574994630,12cyclictest65881-21cron10:45:003
18574994625,16cyclictest37-21ksoftirqd/308:54:593
1857099463,13cyclictest30-21ksoftirqd/208:43:082
18570994616,9cyclictest30-21ksoftirqd/207:10:022
18570994616,9cyclictest30-21ksoftirqd/207:10:022
1857099461,12cyclictest30-21ksoftirqd/207:35:002
1856599465,9cyclictest23-21ksoftirqd/111:05:381
18565994626,14cyclictest3467602-21systemd-journal08:29:591
1857499454,9cyclictest131rcu_preempt09:30:013
1857499454,9cyclictest131rcu_preempt09:30:003
18574994534,8cyclictest37-21ksoftirqd/310:35:563
18574994528,12cyclictest37-21ksoftirqd/309:44:593
18574994523,14cyclictest20158-21cron07:40:013
18570994531,10cyclictest1-21systemd07:55:002
1857099451,8cyclictest0-21swapper/209:04:102
1856599458,11cyclictest23-21ksoftirqd/109:05:011
18565994531,10cyclictest3467602-21systemd-journal07:43:491
18574994434,7cyclictest37-21ksoftirqd/307:35:003
18574994428,13cyclictest3467602-21systemd-journal08:17:013
1857499442,3cyclictest131rcu_preempt12:22:053
18570994437,5cyclictest17846-21kworker/2:308:12:272
18565994436,5cyclictest221rcuc/107:12:201
1857499434,3cyclictest131rcu_preempt09:58:243
18574994332,8cyclictest87469-21systemctl11:35:013
18574994330,8cyclictest27064-21cron09:15:013
1857499431,3cyclictest131rcu_preempt09:48:333
1857099435,5cyclictest30-21ksoftirqd/207:29:592
18570994337,4cyclictest21495-21kworker/2:008:39:582
1856599439,5cyclictest131rcu_preempt07:45:481
18565994329,10cyclictest23-21ksoftirqd/108:50:021
1856599432,25cyclictest221rcuc/107:50:031
18565994316,22cyclictest221rcuc/109:10:001
18565994314,5cyclictest131rcu_preempt08:55:011
18565994311,6cyclictest131rcu_preempt08:20:011
18565994310,7cyclictest131rcu_preempt08:00:021
18574994229,9cyclictest111624-21sshd12:30:193
18574994228,11cyclictest3467602-21systemd-journal11:00:013
18574994228,10cyclictest3467602-21systemd-journal09:33:323
18574994227,10cyclictest309-21in:imuxsock12:05:003
1857099421,8cyclictest0-21swapper/207:59:222
18565994213,6cyclictest131rcu_preempt07:28:501
1857499413,3cyclictest131rcu_preempt10:22:213
18574994133,5cyclictest37889-21latency_hist09:40:003
18574994130,6cyclictest106964-21sendmail12:20:003
18574994126,11cyclictest37-21ksoftirqd/310:04:193
18574994126,10cyclictest37-21ksoftirqd/310:10:003
18574994126,10cyclictest37-21ksoftirqd/310:10:003
1857499412,3cyclictest131rcu_preempt11:12:143
18574994123,13cyclictest37-21ksoftirqd/309:10:013
18574994123,13cyclictest3467602-21systemd-journal10:30:003
1856599415,5cyclictest23-21ksoftirqd/108:04:491
18565994132,6cyclictest23-21ksoftirqd/109:00:011
18574994030,8cyclictest37-21ksoftirqd/312:36:473
18574994027,9cyclictest1-21systemd11:50:003
18574994027,7cyclictest37-21ksoftirqd/312:08:223
18574994026,9cyclictest309-21in:imuxsock09:05:013
18570994025,9cyclictest309-21in:imuxsock07:49:492
1857499396,5cyclictest131rcu_preempt08:50:013
1857499396,25cyclictest55049-21modprobe10:20:003
1857499396,25cyclictest55049-21modprobe10:20:003
18574993929,7cyclictest37-21ksoftirqd/312:12:553
18574993922,13cyclictest275-21dbus-daemon08:35:013
18574993917,6cyclictest131rcu_preempt08:10:033
1857499391,3cyclictest131rcu_preempt11:51:123
1857499391,3cyclictest131rcu_preempt11:04:103
1857099391,8cyclictest21555-21sed08:05:022
1856599395,5cyclictest131rcu_preempt08:25:011
1856599393,7cyclictest23-21ksoftirqd/107:35:021
1857499387,6cyclictest131rcu_preempt08:40:003
1857499385,6cyclictest131rcu_preempt10:50:503
18574993829,5cyclictest31071-21sh09:24:123
18574993828,6cyclictest309-21in:imuxsock10:10:033
18574993825,11cyclictest37-21ksoftirqd/309:19:433
18574993825,10cyclictest37-21ksoftirqd/308:05:003
1857499381,3cyclictest131rcu_preempt11:22:183
1857499381,3cyclictest131rcu_preempt11:16:573
18574993725,8cyclictest37-21ksoftirqd/310:48:183
18574993725,8cyclictest0-21swapper/311:37:193
18574993724,9cyclictest37-21ksoftirqd/311:55:443
18574993724,9cyclictest37-21ksoftirqd/307:11:083
18574993724,10cyclictest37-21ksoftirqd/307:49:303
1857499368,6cyclictest131rcu_preempt07:30:023
18574993623,10cyclictest37-21ksoftirqd/312:26:553
18574993623,10cyclictest37-21ksoftirqd/308:22:353
18574993527,5cyclictest20928-21cron07:55:003
18574993526,7cyclictest37-21ksoftirqd/311:27:243
18574993522,10cyclictest37-21ksoftirqd/309:00:013
18574993512,4cyclictest37-21ksoftirqd/307:20:013
1857499341,3cyclictest131rcu_preempt10:33:343
18574993322,9cyclictest37-21ksoftirqd/307:45:013
1857499313,5cyclictest131rcu_preempt08:30:013
18574993124,5cyclictest37-21ksoftirqd/307:10:023
18574993124,5cyclictest37-21ksoftirqd/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional