You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-25 - 01:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Aug 24, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40017039915880,75cyclictest16850irq/134-800000010:45:030
4001703998625,56cyclictest4048901-21ssh10:35:460
400170399851,66cyclictest4092666-21ssh12:07:460
4001703998516,59cyclictest16850irq/134-800000011:36:280
4001703998429,52cyclictest0-21swapper/012:35:020
4001703998429,38cyclictest4011442-21ssh09:17:050
400170399842,77cyclictest4048532-21sed10:35:020
4001703998129,48cyclictest4041203-21ssh10:19:510
4001703998125,37cyclictest4070957-21ssh11:22:060
4001703998124,40cyclictest4094278-21ssh12:10:560
4001703998119,57cyclictest16850irq/134-800000011:09:370
400170399811,76cyclictest4022596-21ssh09:40:220
4001703998038,39cyclictest4025174-21ssh09:45:570
4001703997932,43cyclictest4066300-21ssh11:12:290
4001703997829,30cyclictest4009131-21ssh09:12:150
4001703997829,30cyclictest4009131-21ssh09:12:150
4001703997816,52cyclictest16850irq/134-800000011:47:370
4001703997728,30cyclictest4013997-21ssh09:22:280
400170399772,5cyclictest4001908-21munin-run07:15:020
400170399771,60cyclictest0-21swapper/009:52:490
400171199762,22cyclictest4013165-21ssh09:20:282
4001703997632,28cyclictest4034618-21ssh10:05:420
4001703997629,30cyclictest4044221-21ssh10:25:500
4001703997628,32cyclictest4042624-21ssh10:22:370
4001703997628,32cyclictest4042624-21ssh10:22:370
4001703997526,31cyclictest4090838-21ssh12:04:010
4001703997520,51cyclictest4106195-21ssh12:35:530
4001703997516,49cyclictest16850irq/134-800000010:01:290
4001703997512,58cyclictest16850irq/134-800000010:55:460
4001711997431,38cyclictest4100845-21munin-run12:25:002
400170399749,60cyclictest16850irq/134-800000012:29:120
4001703997424,31cyclictest4030174-21ssh09:56:290
4001703997422,48cyclictest4003729-21systemd-run07:50:000
4001703997418,51cyclictest16850irq/134-800000009:26:430
4001703997417,52cyclictest16850irq/134-800000011:25:030
4001703997417,52cyclictest16850irq/134-800000011:25:030
4001703997325,31cyclictest4096966-21ssh12:16:360
400170399731,52cyclictest0-21swapper/012:21:330
400170399727,7cyclictest16850irq/134-800000011:16:190
4001703997124,30cyclictest4085863-21ssh11:53:420
4001703997120,35cyclictest4053003-21ssh10:44:460
4001703997120,35cyclictest4053003-21ssh10:44:460
400170399711,52cyclictest4062139-21ssh11:03:440
4001703997029,37cyclictest4080032-21ssh11:40:570
4001703997021,2cyclictest0-21swapper/009:36:500
4001703997021,2cyclictest0-21swapper/009:36:500
4001711996921,41cyclictest1-21systemd12:10:002
4001703996822,31cyclictest0-21swapper/010:53:050
4001703996818,46cyclictest16850irq/134-800000009:33:500
4001703996812,51cyclictest16850irq/134-800000011:56:120
4001715996727,5cyclictest131rcu_preempt10:10:023
4001703996713,44cyclictest16850irq/134-800000008:47:090
4001711996628,34cyclictest30-21ksoftirqd/207:30:012
400171199662,17cyclictest4019862-21ssh09:35:012
4001711996617,14cyclictest275-21dbus-daemon11:55:012
400171199642,17cyclictest4038857-21cron10:15:002
400170399641,5cyclictest275-21dbus-daemon08:10:010
4001703996320,3cyclictest0-21swapper/011:33:320
4001711996227,31cyclictest4072775-21sh11:25:482
4001711996227,31cyclictest4072775-21sh11:25:482
400171199622,17cyclictest4041330-21munin-run10:20:012
4001703996219,2cyclictest0-21swapper/009:10:010
4001703996210,47cyclictest16850irq/134-800000010:11:240
4001711996123,34cyclictest3467562-21sshd09:05:082
4001711996122,35cyclictest4071308-21sh11:23:002
4001711996027,29cyclictest4082440-21sh11:46:132
4001711996021,6cyclictest30-21ksoftirqd/210:05:152
4001711996017,5cyclictest4079525-21latency_hist11:40:022
4001711996014,11cyclictest4009902-1kworker/u9:209:53:282
4001707996039,15cyclictest275-21dbus-daemon10:45:011
4001707996039,15cyclictest275-21dbus-daemon10:45:011
4001703996027,29cyclictest4003202-21systemctl07:40:000
400170399602,4cyclictest4007074-21sshd08:54:090
4001703996018,5cyclictest4002620-21sed07:25:020
4001711995825,29cyclictest30-21ksoftirqd/210:58:252
400171199582,19cyclictest4101097-21ssh12:25:052
400171199582,19cyclictest4050755-21unin-run10:40:012
4001711995819,5cyclictest4007877-21diskmemload12:36:242
4001711995815,9cyclictest4003401-21sort07:40:022
400171199581,18cyclictest4032592-21ssh10:01:382
400170399582,17cyclictest1-21systemd07:20:000
4001711995730,23cyclictest4076164-21sshd11:33:092
4001711995724,28cyclictest4057703-21sh10:54:332
4001711995722,31cyclictest4031192-21sh09:58:592
400171199572,16cyclictest1-21systemd12:15:012
4001711995719,9cyclictest4052266-1kworker/u9:111:13:322
4001711995716,7cyclictest4021265-1kworker/u9:009:40:262
400170799572,23cyclictest3467602-21systemd-journal07:15:011
400170399573,17cyclictest4005024-21munin-run08:15:010
4001703995726,27cyclictest4002937-21/usr/sbin/munin07:35:020
4001703995726,27cyclictest4002937-21/usr/sbin/munin07:35:010
400170399571,15cyclictest4006058-21cut08:35:000
4001715995624,19cyclictest37-21ksoftirqd/308:05:023
400171199563,17cyclictest4025158-21ssh09:45:552
400171199562,19cyclictest4010282-21ssh09:15:002
400171199562,19cyclictest4010282-21ssh09:15:002
4001711995621,6cyclictest4105401-21sh12:34:332
4001711995617,5cyclictest4007877-21diskmemload11:58:202
4001711995616,6cyclictest30-21ksoftirqd/212:18:232
400170399563,47cyclictest0-21swapper/007:10:030
400170399561,51cyclictest0-21swapper/008:18:170
4001703995613,9cyclictest4006586-21sh08:45:000
4001711995545,7cyclictest4005234-21kworker/2:109:26:312
4001711995517,10cyclictest30-21ksoftirqd/212:05:012
4001707995537,13cyclictest4079343-21systemctl11:40:001
4001711995415,6cyclictest30-21ksoftirqd/209:17:022
4001711995414,7cyclictest0-21swapper/210:47:522
400171199541,17cyclictest4062666-21munin-run11:05:012
4001703995432,15cyclictest3467602-21systemd-journal09:05:000
4001715995331,15cyclictest275-21dbus-daemon09:05:013
4001711995327,22cyclictest30-21ksoftirqd/210:43:412
4001711995327,22cyclictest30-21ksoftirqd/210:43:412
4001711995321,9cyclictest30-21ksoftirqd/211:06:052
4001711995315,7cyclictest4005804-21/usr/sbin/munin08:30:022
4001711995315,7cyclictest4005804-21/usr/sbin/munin08:30:022
400171199531,17cyclictest4006609-21cat08:45:022
4001703995336,13cyclictest1-21systemd08:00:010
4001703995336,13cyclictest1-21systemd08:00:010
4001703995316,8cyclictest12-21ksoftirqd/008:01:050
400170399531,2cyclictest0-21swapper/007:45:010
4001711995224,24cyclictest4022182-21sendmail09:40:002
4001711995224,24cyclictest4022182-21sendmail09:39:592
400171199522,13cyclictest1-21systemd07:35:012
400171199522,13cyclictest1-21systemd07:35:012
4001711995216,4cyclictest4044255-21sh10:25:542
4001711995215,4cyclictest0-21swapper/208:35:002
4001711995211,6cyclictest131rcu_preempt10:25:022
4001711995211,6cyclictest131rcu_preempt10:25:022
400170799521,3cyclictest131rcu_preempt10:02:471
400170399527,11cyclictest4002700-21munin-run07:30:010
4001715995133,6cyclictest37-21ksoftirqd/311:55:023
4001715995115,9cyclictest37-21ksoftirqd/310:39:563
400171199512,17cyclictest1-21systemd09:05:012
400171599504,4cyclictest131rcu_preempt10:24:013
400171599504,4cyclictest131rcu_preempt10:24:013
400171599501,7cyclictest4072202-21unin-run11:25:013
400171199501,16cyclictest4001998-21latency_hist07:15:022
4001707995029,16cyclictest275-21dbus-daemon08:35:001
400170399502,14cyclictest4006317-21munin-run08:40:000
4001703995020,25cyclictest1-21systemd07:55:010
400171199492,18cyclictest4069831-21tr11:20:002
4001711994916,4cyclictest4046971-21rm10:31:512
4001711994914,9cyclictest30-21ksoftirqd/208:10:022
4001707994926,15cyclictest4007862-21cron09:10:001
400170399491,44cyclictest4005747-21cat08:25:020
4001715994834,10cyclictest4007354-21systemctl09:00:013
4001715994829,15cyclictest275-21dbus-daemon11:20:003
4001715994811,8cyclictest131rcu_preempt11:46:393
4001711994830,13cyclictest0-21swapper/208:00:002
4001711994830,13cyclictest0-21swapper/208:00:002
400171199482,32cyclictest4081870-21ssh11:45:022
4001711994817,5cyclictest4003970-21modprobe07:54:592
400170799485,39cyclictest4103464-21ssh12:30:021
4001707994824,18cyclictest275-21dbus-daemon09:55:001
4001715994730,13cyclictest275-21dbus-daemon12:25:013
4001715994729,14cyclictest275-21dbus-daemon08:25:013
400171199472,15cyclictest4002434-21munin-run07:25:002
4001707994734,10cyclictest1-21systemd12:40:011
4001707994733,9cyclictest309-21in:imuxsock12:25:011
4001707994732,11cyclictest1-21systemd10:10:011
4001707994731,11cyclictest275-21dbus-daemon08:10:011
4001707994719,18cyclictest275-21dbus-daemon10:40:011
4001715994625,15cyclictest275-21dbus-daemon10:30:003
4001711994638,5cyclictest4001030-21kworker/2:007:50:012
400171199462,18cyclictest4002162-21munin-run07:20:002
4001707994632,10cyclictest23-21ksoftirqd/111:10:001
4001707994630,13cyclictest23-21ksoftirqd/109:50:011
4001707994623,15cyclictest275-21dbus-daemon08:45:011
400170399462,14cyclictest4007549-21latency_hist09:00:060
4001715994535,7cyclictest37-21ksoftirqd/308:15:023
4001715994534,8cyclictest37-21ksoftirqd/307:53:573
4001711994531,10cyclictest1-21systemd08:20:002
4001711994527,13cyclictest3467602-21systemd-journal08:15:012
400170799457,31cyclictest4002163-21grep07:20:001
400170799454,3cyclictest131rcu_preempt09:45:011
400170799453,3cyclictest131rcu_preempt11:00:171
4001707994528,12cyclictest275-21dbus-daemon11:45:001
400170799452,4cyclictest131rcu_preempt11:16:591
400170799451,3cyclictest131rcu_preempt09:24:491
4001707994513,7cyclictest131rcu_preempt07:35:011
4001707994513,7cyclictest131rcu_preempt07:35:001
4001707994510,26cyclictest4072178-21modprobe11:25:001
4001703994514,8cyclictest364-21snmpd08:29:360
4001703994514,8cyclictest364-21snmpd08:29:360
4001715994429,11cyclictest4079353-21tr11:40:003
400171599442,3cyclictest131rcu_preempt11:12:333
400171599441,4cyclictest131rcu_preempt09:18:473
4001711994436,5cyclictest4001030-21kworker/2:008:21:582
400170799446,31cyclictest4003723-21munin-run07:49:591
400170799446,27cyclictest4003184-21modprobe07:40:001
400170799445,31cyclictest3467602-21systemd-journal09:00:011
400170799443,3cyclictest131rcu_preempt09:14:261
400170799443,3cyclictest131rcu_preempt09:14:261
4001707994426,11cyclictest4045981-21grep10:30:001
4001707994420,16cyclictest275-21dbus-daemon08:30:011
4001707994420,16cyclictest275-21dbus-daemon08:30:011
400170799441,3cyclictest131rcu_preempt09:38:581
400170799441,3cyclictest131rcu_preempt09:38:581
4001715994330,9cyclictest37-21ksoftirqd/311:05:003
4001715994326,13cyclictest3467602-21systemd-journal07:35:003
4001715994326,13cyclictest3467602-21systemd-journal07:35:003
400170799436,3cyclictest131rcu_preempt11:26:521
400170799436,3cyclictest131rcu_preempt11:26:521
400170799435,3cyclictest131rcu_preempt08:20:001
4001707994332,8cyclictest4004512-21systemd-run08:05:001
4001707994329,7cyclictest309-21in:imuxsock08:52:091
4001707994325,12cyclictest0-21swapper/110:55:021
4001707994316,7cyclictest23-21ksoftirqd/110:32:581
4001715994229,9cyclictest3467602-21systemd-journal07:13:093
4001715994229,11cyclictest37-21ksoftirqd/309:31:153
4001715994226,12cyclictest3467602-21systemd-journal09:37:093
4001715994226,12cyclictest3467602-21systemd-journal09:37:093
400170799425,4cyclictest131rcu_preempt09:25:561
400170799422,4cyclictest131rcu_preempt10:16:521
400170799422,3cyclictest131rcu_preempt09:16:111
4001707994216,5cyclictest23-21ksoftirqd/111:00:001
400170799421,3cyclictest131rcu_preempt09:30:511
400171599415,3cyclictest131rcu_preempt12:29:583
4001715994127,10cyclictest275-21dbus-daemon07:20:013
4001715994125,12cyclictest275-21dbus-daemon12:05:013
4001715994124,14cyclictest37-21ksoftirqd/312:10:213
400171599411,3cyclictest131rcu_preempt10:20:023
4001711994112,10cyclictest30-21ksoftirqd/209:00:012
400170799416,13cyclictest4001616-21kworker/u8:107:10:021
4001707994129,9cyclictest23-21ksoftirqd/111:35:001
4001707994129,8cyclictest4005523-21sshd08:21:101
4001707994128,9cyclictest23-21ksoftirqd/111:50:021
4001707994121,11cyclictest309-21in:imuxsock12:30:021
400170799411,3cyclictest131rcu_preempt10:46:141
4001715994030,7cyclictest37-21ksoftirqd/310:48:273
4001715994028,8cyclictest4005230-21sshd08:15:183
4001715994027,10cyclictest37-21ksoftirqd/308:35:093
4001715994026,11cyclictest37-21ksoftirqd/312:34:093
4001715994023,13cyclictest275-21dbus-daemon07:30:013
4001715994022,7cyclictest37-21ksoftirqd/311:54:093
400171599401,3cyclictest131rcu_preempt12:39:103
400170799405,26cyclictest4031692-21unin-run10:00:011
400170799404,3cyclictest131rcu_preempt10:14:311
4001707994029,8cyclictest23-21ksoftirqd/112:12:481
4001707994027,10cyclictest23-21ksoftirqd/111:11:251
4001707994026,11cyclictest23-21ksoftirqd/112:06:031
400170799402,3cyclictest131rcu_preempt12:01:421
4001707994020,12cyclictest4006312-21sh08:40:001
4001715993927,9cyclictest37-21ksoftirqd/311:32:063
4001715993927,10cyclictest37-21ksoftirqd/310:44:063
4001715993927,10cyclictest37-21ksoftirqd/310:44:053
4001715993926,9cyclictest4013421-21sshd09:21:093
4001715993926,9cyclictest37-21ksoftirqd/310:57:583
400171599392,4cyclictest131rcu_preempt12:20:003
400171599392,3cyclictest131rcu_preempt11:41:433
4001711993926,10cyclictest3467602-21systemd-journal08:55:002
4001711993920,14cyclictest275-21dbus-daemon08:40:012
400171199391,6cyclictest131rcu_preempt08:50:022
400170799394,3cyclictest131rcu_preempt11:55:181
400170799393,3cyclictest131rcu_preempt10:23:441
400170799393,3cyclictest131rcu_preempt10:23:441
4001707993925,10cyclictest4007633-21systemctl09:05:001
400171599384,3cyclictest131rcu_preempt09:45:263
4001715993826,9cyclictest37-21ksoftirqd/310:05:013
4001715993823,12cyclictest4030823-21sh09:58:013
4001715993820,14cyclictest275-21dbus-daemon07:25:003
400171599381,3cyclictest131rcu_preempt11:07:333
4001707993825,10cyclictest23-21ksoftirqd/108:00:021
4001707993825,10cyclictest23-21ksoftirqd/108:00:011
4001707993823,11cyclictest23-21ksoftirqd/107:41:091
400171599379,6cyclictest131rcu_preempt09:53:393
400171599373,10cyclictest4058000-21latency_hist10:55:023
4001715993728,5cyclictest4010480-21sh09:15:033
4001715993728,5cyclictest4010480-21sh09:15:033
4001715993727,8cyclictest37-21ksoftirqd/311:28:303
4001715993727,8cyclictest37-21ksoftirqd/311:28:303
4001715993727,5cyclictest309-21in:imuxsock08:45:013
4001715993726,9cyclictest37-21ksoftirqd/309:30:013
4001715993726,9cyclictest37-21ksoftirqd/307:40:083
4001715993726,8cyclictest1-21systemd10:15:013
4001715993726,5cyclictest309-21in:imuxsock09:05:033
4001711993722,10cyclictest275-21dbus-daemon08:04:592
400170799375,3cyclictest131rcu_preempt11:50:441
400170799373,3cyclictest131rcu_preempt12:17:091
400170799372,3cyclictest131rcu_preempt07:25:021
400171599366,6cyclictest37-21ksoftirqd/308:50:043
4001715993626,7cyclictest37-21ksoftirqd/308:00:003
4001715993626,7cyclictest37-21ksoftirqd/308:00:003
400171599362,3cyclictest131rcu_preempt12:05:593
400171599361,32cyclictest37-21ksoftirqd/307:40:023
4001707993623,8cyclictest23-21ksoftirqd/107:55:001
4001707993621,10cyclictest23-21ksoftirqd/108:15:021
4001707993618,13cyclictest4002699-21systemd-run07:30:011
400171599351,9cyclictest4047723-21ssh10:33:353
400171599351,9cyclictest4022448-21ssh09:40:043
4001715993519,12cyclictest3467602-21systemd-journal07:50:003
4001715993513,7cyclictest131rcu_preempt08:30:023
4001715993513,7cyclictest131rcu_preempt08:30:023
4001711993528,4cyclictest4003467-21latency_hist07:45:022
4001715993425,5cyclictest4006114-21latency_hist08:35:003
400171199343,3cyclictest131rcu_preempt07:10:022
400170799344,3cyclictest131rcu_preempt08:50:011
4001715993223,7cyclictest37-21ksoftirqd/308:06:093
4001715993125,4cyclictest4001731-21latency_hist07:10:013
400171599302,9cyclictest4006828-21modprobe08:50:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional