You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-17 - 22:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jul 17, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25570499941,12cyclictest0-21swapper/311:00:023
255690999286,4cyclictest12-21ksoftirqd/010:10:020
255699998238,39cyclictest1-21systemd10:20:012
255699998238,39cyclictest1-21systemd10:20:012
255690997734,20cyclictest273749-21ssh09:35:000
255690997571,2cyclictest0-21swapper/008:50:030
25569099723,28cyclictest151rcuc/011:35:020
25569099723,28cyclictest151rcuc/011:35:020
255699997128,39cyclictest329314-21sh11:31:312
255699997128,39cyclictest329314-21sh11:31:312
255699997026,8cyclictest30-21ksoftirqd/210:50:012
255699996924,41cyclictest307225-21latency_hist10:45:012
255699996827,37cyclictest344431-21sh12:03:382
25570499671,12cyclictest296490-21sshd10:22:223
255699996728,34cyclictest266786-21latency_hist09:20:022
255699996728,34cyclictest266786-21latency_hist09:20:022
25570499652,12cyclictest48-21kcompactd009:35:303
25570499641,11cyclictest48-21kcompactd011:05:083
25570499641,10cyclictest256931-21munin-run07:35:003
255699996429,31cyclictest3467602-21systemd-journal12:10:002
255699996424,11cyclictest1-21systemd08:35:012
25569099641,42cyclictest321495-21ssh11:15:020
255699996324,4cyclictest352882-21sh12:21:132
255699996320,6cyclictest352225-21latency_hist12:20:032
255690996320,39cyclictest12-21ksoftirqd/009:43:500
25570499622,7cyclictest339256-21ssh11:52:403
25570499621,8cyclictest279082-21ssh09:45:483
25570499621,7cyclictest259241-21sshd08:18:223
25570499612,7cyclictest351035-21ssh12:17:323
25570499611,8cyclictest355245-21ssh12:26:233
25570499611,8cyclictest348866-21ssh12:12:573
25570499611,8cyclictest287807-21ssh10:04:283
25570499611,17cyclictest268310-21ssh09:23:193
25570499611,14cyclictest319972-21ssh11:11:513
25570499611,13cyclictest308356-21sshd10:47:223
255699996127,30cyclictest309921-21sh10:50:302
255699996119,11cyclictest275-21dbus-daemon08:25:002
25570499602,9cyclictest3467602-21systemd-journal09:26:203
25570499602,13cyclictest48-21kcompactd010:25:033
25570499602,13cyclictest48-21kcompactd010:25:033
25570499602,13cyclictest261611-21munin-run09:05:003
25570499601,8cyclictest352376-21ssh12:20:093
25570499601,8cyclictest325151-21ssh11:22:453
25570499601,7cyclictest337534-21ssh11:49:123
25570499601,7cyclictest332272-21ssh11:38:003
25570499601,7cyclictest293621-21ssh10:16:243
25570499601,7cyclictest293621-21ssh10:16:243
25570499601,12cyclictest256441-21munin-run07:25:013
25570499601,12cyclictest256441-21munin-run07:25:013
255699996024,8cyclictest30-21ksoftirqd/209:42:272
255699996020,5cyclictest304814-21latency_hist10:40:032
25569099603,38cyclictest316654-21latency_hist11:05:010
255690996023,33cyclictest12-21ksoftirqd/011:54:160
25570499592,12cyclictest359211-21munin-run12:35:003
25570499591,7cyclictest292499-21ssh10:14:163
25570499591,7cyclictest266043-21ssh09:18:383
25570499591,7cyclictest266043-21ssh09:18:373
25570499591,16cyclictest284315-21ssh09:56:543
25570499591,12cyclictest260273-21sshd08:38:223
25570499591,10cyclictest359476-21sshd12:35:083
255699995929,26cyclictest266939-21sh09:20:082
255690995912,40cyclictest299946-21munin-plugin-st10:30:000
255690995912,40cyclictest299946-21munin-plugin-st10:30:000
25570499582,12cyclictest364-21snmpd08:47:083
25570499582,12cyclictest364-21snmpd08:47:083
25570499581,9cyclictest301677-21ssh10:33:303
25570499581,8cyclictest282157-21ssh09:52:263
25570499581,7cyclictest276515-21ssh09:40:163
255704995815,36cyclictest0-21swapper/311:20:033
25570499581,15cyclictest290356-21ssh10:09:473
25570499581,13cyclictest310233-21ssh10:51:203
255699995826,5cyclictest30-21ksoftirqd/211:39:412
25569999582,20cyclictest257477-21unin-run07:45:002
255699995821,4cyclictest261756-21latency_hist09:05:012
255699995817,9cyclictest260515-21cat08:40:022
255690995829,13cyclictest360177-21ssh12:36:410
255690995827,15cyclictest354936-21ssh12:25:340
25570499572,5cyclictest261893-21munin-run09:10:013
25570499572,15cyclictest304472-21ssh10:39:293
25570499572,12cyclictest347077-21sshd12:09:213
25570499571,11cyclictest258836-21cat08:10:033
255699995717,7cyclictest258680-21grep08:05:022
255690995727,15cyclictest270985-21ssh09:29:130
25570499562,7cyclictest315475-21sshd11:02:213
25570499561,7cyclictest329557-21ssh11:32:023
25570499561,7cyclictest329557-21ssh11:32:023
25570499561,14cyclictest342955-21ssh12:00:153
255699995625,5cyclictest313691-21sh10:58:502
255699995624,5cyclictest334222-21sh11:41:552
255699995624,4cyclictest339953-21sh11:54:162
255699995621,9cyclictest30-21ksoftirqd/209:56:062
255699995621,4cyclictest321407-21latency_hist11:15:012
255699995620,7cyclictest30-21ksoftirqd/210:10:522
255699995620,5cyclictest272585-21sh09:32:202
255690995645,6cyclictest16950irq/224-800000010:00:000
255690995637,16cyclictest16850irq/134-800000011:20:000
255690995632,18cyclictest265517-21sshd09:17:220
255690995632,18cyclictest265517-21sshd09:17:220
255690995630,18cyclictest264280-21unin-run09:15:010
25569099562,27cyclictest131rcu_preempt10:47:320
25570499552,10cyclictest335573-21true11:45:023
25570499551,8cyclictest341953-21ssh11:58:313
25570499551,8cyclictest263565-21sshd09:13:223
25570499551,5cyclictest273764-21munin-run09:35:003
255699995530,21cyclictest30-21ksoftirqd/209:25:112
255699995521,13cyclictest30-21ksoftirqd/210:28:172
255699995521,13cyclictest30-21ksoftirqd/210:28:162
255699995520,4cyclictest297210-21sh10:24:082
255699995514,11cyclictest30-21ksoftirqd/209:53:312
255690995528,11cyclictest311331-21ssh10:53:460
255690995527,13cyclictest359094-21ssh12:34:470
255690995527,12cyclictest334218-21ssh11:41:540
25569099552,37cyclictest293755-21ssh10:16:410
25569099552,37cyclictest293755-21ssh10:16:400
25570499542,14cyclictest261372-21munin-run09:00:013
25570499542,13cyclictest259762-21sshd08:26:223
25570499541,7cyclictest258503-21munin-run08:05:003
25570499541,6cyclictest328473-21munin-run11:30:013
255699995445,5cyclictest323789-21kworker/2:312:31:222
255699995425,8cyclictest30-21ksoftirqd/207:40:022
255699995425,8cyclictest30-21ksoftirqd/207:40:022
255699995425,4cyclictest275834-21sshd09:39:222
255699995420,4cyclictest341458-21sh11:57:162
255699995420,4cyclictest261866-21diskmemload12:11:502
255695995412,25cyclictest271577-21ssh09:30:021
255690995426,12cyclictest332092-21ssh11:37:370
255690995424,26cyclictest313086-21ssh10:57:280
255690995424,14cyclictest286069-21ssh10:00:260
255690995418,32cyclictest12-21ksoftirqd/010:42:240
255690995415,35cyclictest12-21ksoftirqd/009:22:270
25569099541,40cyclictest131rcu_preempt11:29:270
25570499532,6cyclictest257716-21munin-run07:50:003
25570499532,6cyclictest256692-21unin-run07:30:003
25570499532,10cyclictest305022-21ssh10:40:073
255699995321,7cyclictest357014-21latency_hist12:30:022
255699995320,5cyclictest263046-21sh09:12:072
255699995318,8cyclictest30-21ksoftirqd/212:38:012
255695995336,13cyclictest1-21systemd08:00:001
255690995344,6cyclictest16950irq/224-800000009:40:000
255690995343,7cyclictest12-21ksoftirqd/011:56:590
25570499522,6cyclictest364-21snmpd07:18:353
25570499522,6cyclictest258262-21unin-run08:00:003
25570499522,12cyclictest261067-21cron08:55:003
255699995244,5cyclictest261092-21kworker/2:310:05:022
255699995222,9cyclictest30-21ksoftirqd/208:30:022
255690995235,11cyclictest12-21ksoftirqd/008:50:010
255690995235,11cyclictest12-21ksoftirqd/008:50:010
25569099522,43cyclictest3467602-21systemd-journal09:55:000
25570499515,40cyclictest0-21swapper/308:15:003
25570499512,5cyclictest257464-21munin-run07:45:003
25570499511,11cyclictest260004-21latency_hist08:30:023
255699995126,21cyclictest315162-21sh11:01:392
255699995119,10cyclictest30-21ksoftirqd/210:31:562
255695995142,6cyclictest23-21ksoftirqd/108:35:021
255690995144,5cyclictest151rcuc/012:20:010
255690995133,13cyclictest275-21dbus-daemon07:25:010
255690995133,13cyclictest275-21dbus-daemon07:25:010
255690995126,9cyclictest279297-21ssh09:46:160
255690995123,12cyclictest301698-21ssh10:33:340
25569099511,35cyclictest292013-21ssh10:13:140
25570499503,5cyclictest364-21snmpd08:24:433
255704995023,23cyclictest255868-21sshd07:14:053
255699995019,11cyclictest30-21ksoftirqd/209:47:302
25569099505,18cyclictest304837-21latency_hist10:40:020
255690995032,15cyclictest342901-21ssh12:00:080
255690995028,15cyclictest275-21dbus-daemon08:45:000
255690995023,12cyclictest317714-21ssh11:07:080
255690994926,17cyclictest275-21dbus-daemon07:45:000
255690994925,20cyclictest12-21ksoftirqd/012:12:560
25570499482,8cyclictest257170-21cron07:40:003
25570499482,8cyclictest257170-21cron07:40:003
25570499481,12cyclictest260629-21latency_hist08:45:013
255699994840,5cyclictest269081-21kworker/2:111:22:442
255699994840,5cyclictest257742-21kworker/2:008:10:012
255699994840,5cyclictest241831-21kworker/2:307:18:102
255699994823,6cyclictest30-21ksoftirqd/207:35:022
25569099483,8cyclictest3467602-21systemd-journal10:25:000
255690994836,9cyclictest12-21ksoftirqd/007:20:020
255690994832,11cyclictest16950irq/224-800000012:23:070
25569099482,41cyclictest261972-21date09:10:010
25569099482,18cyclictest325655-21ssh11:24:000
255699994739,5cyclictest323789-21kworker/2:311:27:502
255699994738,5cyclictest261092-21kworker/2:309:00:002
255699994720,5cyclictest336566-21sh11:46:532
25569599473,3cyclictest131rcu_preempt09:23:531
255695994632,11cyclictest1-21systemd09:50:011
255690994632,9cyclictest12-21ksoftirqd/008:00:010
255690994629,13cyclictest338098-21latency_hist11:50:020
255695994531,10cyclictest1-21systemd08:50:011
255695994531,10cyclictest1-21systemd08:50:011
25569599451,4cyclictest131rcu_preempt12:01:401
255690994528,11cyclictest12-21ksoftirqd/007:35:000
255690994519,11cyclictest345875-21ssh12:06:340
25570499441,7cyclictest3467602-21systemd-journal07:51:253
25570499441,7cyclictest3467602-21systemd-journal07:51:253
25570499441,14cyclictest255768-21date07:10:013
255699994421,5cyclictest30-21ksoftirqd/207:45:332
25569999442,10cyclictest288612-21ssh10:05:492
255695994432,9cyclictest275-21dbus-daemon07:40:021
255695994432,9cyclictest275-21dbus-daemon07:40:021
255695994428,13cyclictest309-21in:imuxsock10:40:011
25569599441,3cyclictest131rcu_preempt12:20:571
255699994329,10cyclictest3467602-21systemd-journal08:16:222
25569599435,3cyclictest131rcu_preempt09:12:441
255695994325,14cyclictest1-21systemd07:35:011
25569599432,4cyclictest131rcu_preempt09:52:321
25569599432,3cyclictest131rcu_preempt11:18:311
255690994330,10cyclictest1-21systemd07:50:010
255690994329,11cyclictest258738-21cron08:10:010
255690994310,22cyclictest255692-21cat07:10:010
25569599424,3cyclictest131rcu_preempt11:56:531
25569599423,3cyclictest131rcu_preempt09:40:491
255695994233,6cyclictest261876-21ssh09:10:001
255695994231,7cyclictest1-21systemd10:00:011
255695994230,8cyclictest3467602-21systemd-journal08:24:181
255695994228,10cyclictest3467602-21systemd-journal11:31:221
255695994228,10cyclictest3467602-21systemd-journal11:31:221
25569599421,3cyclictest131rcu_preempt10:43:351
255690994236,3cyclictest16950irq/224-800000009:05:010
255690994212,6cyclictest131rcu_preempt08:05:010
25569999414,8cyclictest256679-21latency_hist07:30:022
255699994129,7cyclictest30-21ksoftirqd/209:10:012
255695994133,6cyclictest23-21ksoftirqd/111:48:471
255695994130,7cyclictest309-21in:imuxsock12:40:011
255695994129,9cyclictest23-21ksoftirqd/112:31:431
255695994129,9cyclictest23-21ksoftirqd/110:51:591
255695994127,11cyclictest23-21ksoftirqd/110:15:021
255695994126,10cyclictest23-21ksoftirqd/111:53:191
25569599411,3cyclictest131rcu_preempt09:30:451
255690994131,7cyclictest12-21ksoftirqd/007:11:050
255690994130,6cyclictest259252-21cron08:20:000
255690994127,10cyclictest275-21dbus-daemon08:30:010
255699994029,8cyclictest1-21systemd11:20:002
255699994025,11cyclictest30-21ksoftirqd/207:15:002
255695994029,8cyclictest23-21ksoftirqd/109:15:301
255695994029,8cyclictest23-21ksoftirqd/109:15:301
255695994028,9cyclictest23-21ksoftirqd/111:42:221
255695994028,8cyclictest319017-21date11:10:021
255695994026,10cyclictest23-21ksoftirqd/112:07:391
255695994025,11cyclictest295227-21cat10:20:001
255695994025,11cyclictest295227-21cat10:20:001
255695994024,11cyclictest23-21ksoftirqd/110:30:541
25569599401,35cyclictest23-21ksoftirqd/110:27:311
25569599401,35cyclictest23-21ksoftirqd/110:27:311
25569599396,7cyclictest23-21ksoftirqd/111:35:151
25569599394,26cyclictest260289-21modprobe08:40:001
255695993928,8cyclictest23-21ksoftirqd/111:00:001
255695993927,9cyclictest23-21ksoftirqd/110:03:551
255695993927,9cyclictest23-21ksoftirqd/108:20:021
255695993926,9cyclictest23-21ksoftirqd/111:13:181
255695993925,11cyclictest23-21ksoftirqd/112:10:511
255690993931,5cyclictest19350irq/34-mmc107:51:460
255690993931,5cyclictest19350irq/34-mmc107:51:460
255690993917,15cyclictest257175-21modprobe07:40:000
255690993917,15cyclictest257175-21modprobe07:40:000
255699993830,5cyclictest269081-21kworker/2:111:06:052
255699993823,11cyclictest3467602-21systemd-journal07:24:212
255699993823,11cyclictest3467602-21systemd-journal07:24:212
255695993829,7cyclictest23-21ksoftirqd/111:28:091
255695993828,6cyclictest309-21in:imuxsock07:45:001
255695993827,7cyclictest23-21ksoftirqd/112:15:131
255695993826,9cyclictest23-21ksoftirqd/107:20:001
255695993825,10cyclictest23-21ksoftirqd/108:52:221
255695993824,11cyclictest23-21ksoftirqd/110:23:221
255695993824,10cyclictest23-21ksoftirqd/111:00:351
255695993823,12cyclictest23-21ksoftirqd/110:49:481
25569099386,5cyclictest131rcu_preempt08:15:000
255690993826,8cyclictest260033-21cron08:35:000
255690993825,9cyclictest260282-21sshd08:39:220
25569099381,33cyclictest364-21snmpd08:57:020
255699993724,10cyclictest30-21ksoftirqd/207:55:022
255699993724,10cyclictest30-21ksoftirqd/207:55:022
255695993730,4cyclictest256429-21/usr/sbin/munin07:25:011
255695993730,4cyclictest256429-21/usr/sbin/munin07:25:011
255695993727,5cyclictest276163-21sh09:40:001
255695993724,8cyclictest23-21ksoftirqd/107:50:011
255695993724,10cyclictest23-21ksoftirqd/112:28:151
255699993627,7cyclictest258988-21sshd08:14:222
255695993625,8cyclictest23-21ksoftirqd/108:05:011
255695993623,8cyclictest0-21swapper/107:30:001
255695993623,10cyclictest23-21ksoftirqd/110:06:101
255695993620,12cyclictest1-21systemd07:55:011
255695993620,12cyclictest1-21systemd07:55:011
255690993626,5cyclictest256855-21latency_hist07:30:030
25569599354,3cyclictest131rcu_preempt11:20:471
255695993525,8cyclictest23-21ksoftirqd/108:45:001
25569599352,3cyclictest131rcu_preempt08:15:021
255695993523,8cyclictest23-21ksoftirqd/107:15:001
255695993518,13cyclictest3467602-21systemd-journal08:28:221
255695993424,7cyclictest23-21ksoftirqd/109:01:221
255695993422,9cyclictest23-21ksoftirqd/109:00:021
255690993426,5cyclictest259531-21cron08:25:000
255699993315,13cyclictest3467602-21systemd-journal08:45:002
255695993318,12cyclictest258785-21cat08:10:021
255699993225,4cyclictest3467562-21sshd08:48:222
255699993225,4cyclictest3467562-21sshd08:48:222
255699993223,5cyclictest309-21in:imuxsock08:00:002
25569599323,24cyclictest255697-21ls07:10:011
255699993120,8cyclictest261232-21date08:55:022
255699992820,6cyclictest30-21ksoftirqd/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional