You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 02:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue May 26, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
318744991171,90cyclictest0-21swapper/008:35:010
318744999186,3cyclictest150-21pr/ttyS207:40:010
318751998029,14cyclictest410538-21ssh12:06:371
318751998029,14cyclictest410538-21ssh12:06:371
318751997828,14cyclictest335902-21sh09:32:351
318751997826,15cyclictest354133-21sed10:10:021
318751997824,21cyclictest418919-21ssh12:24:111
318751997725,15cyclictest364-21snmpd09:18:431
318744997773,2cyclictest0-21swapper/007:50:010
318757997610,33cyclictest358951-21ssh10:20:023
318751997625,19cyclictest345676-21ssh09:52:501
318751997623,15cyclictest326240-21ssh09:12:301
318751997623,14cyclictest330708-21ssh09:21:441
318751997621,18cyclictest363969-21ssh10:30:201
318751997620,15cyclictest357010-21ssh10:16:091
318751997524,15cyclictest422559-21ssh12:31:321
318751997523,15cyclictest363423-21ssh10:29:341
318751997432,12cyclictest0-21swapper/112:40:001
318751997421,17cyclictest343903-21ssh09:49:111
318751997421,15cyclictest384064-21ssh11:12:051
318757997333,17cyclictest378071-21readlink10:59:593
318751997328,14cyclictest420583-21ssh12:27:321
318751997321,15cyclictest381551-21systemd11:06:581
318751997218,15cyclictest378511-21ssh11:00:261
318751997125,14cyclictest412964-21ssh12:11:391
318751997122,14cyclictest400586-21ssh11:46:041
318751997122,14cyclictest400586-21ssh11:46:031
318751997122,14cyclictest371598-21ssh10:46:131
318751997120,15cyclictest368152-21ssh10:39:221
318751997030,7cyclictest404095-21ssh11:53:291
318751997024,11cyclictest374790-21ssh10:52:591
318751997024,11cyclictest374790-21ssh10:52:591
318751997023,14cyclictest393650-21ssh11:31:481
318751997019,14cyclictest399141-21ssh11:43:111
318753996942,7cyclictest323399-21kworker/2:009:56:012
318751996923,13cyclictest359773-21ssh10:21:461
318751996917,14cyclictest333764-21ssh09:28:071
31874499696,9cyclictest121ksoftirqd/010:31:490
318751996723,16cyclictest376096-21ssh10:55:261
318751996720,14cyclictest322732-21ls08:25:031
318751996718,13cyclictest397339-21ssh11:39:401
318751996718,13cyclictest397339-21ssh11:39:391
318751996717,14cyclictest406393-21ssh11:58:151
318751996717,14cyclictest406393-21ssh11:58:141
318751996628,7cyclictest341556-21ssh09:44:221
318751996623,11cyclictest391283-21ssh11:26:511
318751996622,6cyclictest386427-21ssh11:16:491
318744996660,3cyclictest0-21swapper/009:15:030
318744996622,8cyclictest382606-21rm11:09:140
318751996520,15cyclictest347683-21ssh09:56:511
318751996516,16cyclictest416682-21ssh12:19:361
318751996516,16cyclictest416682-21ssh12:19:361
318751996451,10cyclictest349594-21ssh10:00:351
318751996444,15cyclictest318913-21sshd07:13:191
318751996444,15cyclictest318913-21sshd07:13:181
318751996423,13cyclictest0-21swapper/109:37:151
318751996423,13cyclictest0-21swapper/109:37:141
318751996422,12cyclictest388223-21ssh11:20:211
318751996420,14cyclictest407822-21ssh12:00:461
318744996425,11cyclictest12-21ksoftirqd/010:36:100
318744996425,10cyclictest377546-21rm10:58:470
318751996320,13cyclictest355251-21ssh10:12:351
318751996315,14cyclictest0-21swapper/108:35:021
31874499623,39cyclictest12-21ksoftirqd/012:35:530
318753996142,15cyclictest1-21systemd10:59:592
318751996119,8cyclictest369087-21ssh10:40:581
318751996119,8cyclictest369087-21ssh10:40:581
318751996117,13cyclictest364-21snmpd08:03:271
318751996034,20cyclictest324389-21munin-plugin-st09:00:011
318751996034,20cyclictest324389-21munin-plugin-st09:00:011
318751996034,19cyclictest324661-21cut09:04:591
31874499594,7cyclictest121ksoftirqd/012:04:150
318744995939,15cyclictest275-21dbus-daemon09:15:010
318744995929,16cyclictest356996-21ssh10:16:080
318744995918,9cyclictest12-21ksoftirqd/009:30:010
318744995912,13cyclictest12-21ksoftirqd/009:40:370
318751995842,11cyclictest320273-21munin-run07:40:011
318744995829,14cyclictest415684-21ssh12:17:210
318744995829,14cyclictest415684-21ssh12:17:200
318744995824,17cyclictest405540-21ssh11:56:170
318744995824,17cyclictest405540-21ssh11:56:160
318744995821,16cyclictest353969-21ssh10:10:010
318744995819,32cyclictest348289-21sh09:58:180
318757995739,12cyclictest1-21systemd10:30:003
318751995737,16cyclictest364-21snmpd07:52:271
31874499573,16cyclictest3467602-21systemd-journal09:35:000
318744995723,11cyclictest12-21ksoftirqd/011:23:570
318744995710,12cyclictest12-21ksoftirqd/009:49:340
318753995647,6cyclictest323399-21kworker/2:009:50:012
318744995626,15cyclictest355257-21ssh10:12:340
318744995624,16cyclictest386769-21ssh11:17:390
31874499562,38cyclictest410094-21ssh12:05:310
31874499562,38cyclictest410094-21ssh12:05:300
31874499561,49cyclictest373353-21ssh10:50:010
318753995546,6cyclictest368521-21kworker/2:311:29:102
318751995543,9cyclictest322290-21sshd08:19:191
31874499555,12cyclictest12-21ksoftirqd/010:44:540
31874499555,12cyclictest12-21ksoftirqd/010:44:530
318744995528,12cyclictest414351-21ssh12:14:550
318744995519,32cyclictest16850irq/134-800000012:25:130
31874499551,39cyclictest396884-21ssh11:38:420
31874499551,39cyclictest396884-21ssh11:38:410
318753995445,5cyclictest323399-21kworker/2:009:15:032
318751995431,18cyclictest321468-21date08:00:021
31874499547,17cyclictest321288-21tr08:00:000
318744995426,13cyclictest350652-21ssh10:03:080
318744995425,14cyclictest399501-21ssh11:44:080
318744995425,14cyclictest392005-21ssh11:28:330
31874499541,38cyclictest400825-21ssh11:46:430
31874499541,38cyclictest400825-21ssh11:46:420
318744995413,7cyclictest12-21ksoftirqd/011:02:200
31875799532,3cyclictest131rcu_preempt08:00:013
318753995345,5cyclictest385458-21kworker/2:111:35:482
318753995345,5cyclictest385458-21kworker/2:111:35:472
318753995345,5cyclictest324668-21kworker/2:210:47:262
318753995344,6cyclictest323399-21kworker/2:009:10:002
318753995335,12cyclictest3467602-21systemd-journal08:45:002
318753995318,13cyclictest356530-21ssh10:15:022
318751995341,9cyclictest3467602-21systemd-journal07:45:141
318751995341,9cyclictest323850-21sshd08:48:191
318751995340,9cyclictest322822-21sshd08:29:191
318751995340,9cyclictest322822-21sshd08:29:191
318751995327,21cyclictest323763-21latency_hist08:45:021
318751995324,25cyclictest319220-21sendmail07:20:001
318751995323,25cyclictest321756-21date08:05:021
318744995329,7cyclictest359066-21ssh10:20:120
318744995325,12cyclictest394882-21ssh11:34:330
318744995319,6cyclictest12-21ksoftirqd/009:51:210
31874499531,15cyclictest0-21swapper/012:31:190
31875799527,7cyclictest37-21ksoftirqd/311:01:213
318753995245,4cyclictest403816-21kworker/2:212:26:282
318751995240,8cyclictest319965-21sshd07:34:191
318751995239,10cyclictest364-21snmpd09:05:301
318751995228,20cyclictest322121-21cat08:15:021
318751995228,20cyclictest322121-21cat08:15:011
318744995218,8cyclictest12-21ksoftirqd/011:13:490
318757995137,9cyclictest275-21dbus-daemon12:35:023
318757995127,17cyclictest3467602-21systemd-journal09:55:003
318753995143,5cyclictest368521-21kworker/2:311:20:002
318753995143,5cyclictest324668-21kworker/2:211:02:372
318753995142,6cyclictest403816-21kworker/2:212:00:432
318753995140,7cyclictest309-21in:imuxsock08:20:002
318751995138,9cyclictest48-21kcompactd007:27:341
318751995134,12cyclictest364-21snmpd08:54:021
318751995128,18cyclictest318896-21sed07:10:021
318751995125,20cyclictest323346-21latency_hist08:40:001
31874499514,12cyclictest12-21ksoftirqd/009:36:170
31874499514,12cyclictest12-21ksoftirqd/009:36:160
318744995124,24cyclictest16950irq/224-800000007:25:020
318744995123,13cyclictest404185-21ssh11:53:480
318744995123,12cyclictest417340-21ssh12:20:350
318744995113,21cyclictest0-21swapper/010:29:170
318753995042,5cyclictest324668-21kworker/2:210:22:002
318753995042,5cyclictest324668-21kworker/2:209:35:212
318753995042,5cyclictest324668-21kworker/2:209:35:202
318753995041,5cyclictest403816-21kworker/2:212:07:172
318753995041,5cyclictest403816-21kworker/2:212:07:172
318753995012,5cyclictest131rcu_preempt12:36:552
318751995038,9cyclictest364-21snmpd07:23:181
31874499506,7cyclictest12-21ksoftirqd/007:20:010
31874499501,8cyclictest320508-21latency_hist07:45:010
318757994932,12cyclictest275-21dbus-daemon10:50:013
318753994942,5cyclictest323399-21kworker/2:009:42:292
318753994941,5cyclictest403816-21kworker/2:212:32:562
318753994941,5cyclictest403816-21kworker/2:212:19:502
318753994941,5cyclictest403816-21kworker/2:212:19:502
318753994941,5cyclictest403816-21kworker/2:211:56:462
318753994941,5cyclictest403816-21kworker/2:211:56:462
318753994941,5cyclictest324668-21kworker/2:210:34:022
318753994941,5cyclictest324668-21kworker/2:210:04:132
318753994941,5cyclictest323399-21kworker/2:009:25:532
318753994928,16cyclictest3467602-21systemd-journal11:40:202
318751994932,12cyclictest320520-21systemd-run07:45:011
31874499492,42cyclictest324387-21munin-run09:00:000
31874499492,42cyclictest324387-21munin-run09:00:000
318744994922,12cyclictest330354-21ssh09:20:510
318753994841,4cyclictest324668-21kworker/2:210:37:452
318753994840,5cyclictest368521-21kworker/2:311:49:362
318753994840,5cyclictest368521-21kworker/2:311:49:352
31875399481,31cyclictest0-21swapper/208:50:012
318744994820,12cyclictest374841-21ssh10:53:060
318744994820,12cyclictest374841-21ssh10:53:060
318744994819,9cyclictest0-21swapper/008:09:570
31875799477,8cyclictest37-21ksoftirqd/307:38:173
318757994732,11cyclictest275-21dbus-daemon09:45:013
318753994740,4cyclictest324668-21kworker/2:210:25:352
318753994740,4cyclictest324668-21kworker/2:210:09:462
318753994739,5cyclictest403816-21kworker/2:212:24:022
318753994739,5cyclictest323399-21kworker/2:009:17:442
318753994739,5cyclictest294335-21kworker/2:307:12:282
318753994739,5cyclictest294335-21kworker/2:307:12:272
318753994731,13cyclictest3467602-21systemd-journal10:20:032
318753994730,13cyclictest309-21in:imuxsock11:10:002
318744994739,5cyclictest151rcuc/008:20:010
31874499472,41cyclictest322808-21sshd08:26:190
31874499472,41cyclictest322808-21sshd08:26:180
318753994638,5cyclictest323399-21kworker/2:009:30:192
318753994638,5cyclictest294335-21kworker/2:308:40:022
318753994631,12cyclictest3467602-21systemd-journal08:25:002
31875399462,13cyclictest412514-21ssh12:10:342
318744994635,7cyclictest12-21ksoftirqd/009:10:010
318744994630,10cyclictest318745-21latency_hist07:10:020
31875799456,4cyclictest131rcu_preempt10:38:053
31875799452,3cyclictest131rcu_preempt10:30:503
31875799451,3cyclictest131rcu_preempt12:19:023
31875799451,3cyclictest131rcu_preempt12:19:023
318753994536,5cyclictest323399-21kworker/2:009:53:342
318753994524,15cyclictest30-21ksoftirqd/207:45:012
31875399452,10cyclictest385070-21ssh11:14:182
318757994427,14cyclictest3467602-21systemd-journal10:12:203
318753994438,3cyclictest323399-21kworker/2:009:20:432
318753994436,5cyclictest368521-21kworker/2:311:54:012
318753994436,5cyclictest294335-21kworker/2:307:19:052
318753994435,6cyclictest324668-21kworker/2:210:51:012
318753994435,6cyclictest324668-21kworker/2:210:51:002
31874499446,5cyclictest12-21ksoftirqd/008:15:010
31874499446,5cyclictest12-21ksoftirqd/008:15:010
31874499441,6cyclictest321107-21date07:55:020
318757994329,11cyclictest37-21ksoftirqd/309:22:203
318753994335,5cyclictest385458-21kworker/2:111:34:002
318744994335,5cyclictest151rcuc/007:35:000
318744994332,7cyclictest322609-21latency_hist08:25:010
31875799424,3cyclictest131rcu_preempt07:45:023
31875799423,6cyclictest131rcu_preempt11:06:593
31875799423,4cyclictest131rcu_preempt09:56:593
318757994231,8cyclictest3467602-21systemd-journal09:30:003
318757994229,10cyclictest37-21ksoftirqd/311:45:583
318757994229,10cyclictest37-21ksoftirqd/311:45:583
318757994228,10cyclictest3467602-21systemd-journal09:40:013
318757994228,10cyclictest3467602-21systemd-journal09:40:003
318757994227,12cyclictest37-21ksoftirqd/308:15:023
318757994227,12cyclictest37-21ksoftirqd/308:15:013
318753994234,5cyclictest323399-21kworker/2:009:00:012
318753994234,5cyclictest323399-21kworker/2:009:00:002
318753994232,6cyclictest321277-21cut08:00:002
318753994226,11cyclictest3467602-21systemd-journal07:50:002
318744994234,5cyclictest19350irq/34-mmc108:47:380
318757994130,8cyclictest413674-21sshd12:13:193
318757994130,8cyclictest3467602-21systemd-journal10:24:193
318757994125,12cyclictest275-21dbus-daemon11:53:083
318757994125,11cyclictest37-21ksoftirqd/309:46:463
31875799411,8cyclictest0-21swapper/311:38:063
31875799411,8cyclictest0-21swapper/311:38:063
31875799411,5cyclictest0-21swapper/310:05:403
31875799411,3cyclictest131rcu_preempt07:20:013
31875799411,14cyclictest387087-21sshd11:18:203
318753994135,3cyclictest368521-21kworker/2:310:40:082
318753994135,3cyclictest368521-21kworker/2:310:40:072
318753994133,5cyclictest323399-21kworker/2:009:01:182
31874499412,35cyclictest324641-21sshd09:04:200
31874499411,6cyclictest319863-21date07:30:030
31874499411,11cyclictest319120-21latency_hist07:15:030
31874499411,11cyclictest319120-21latency_hist07:15:020
318757994024,12cyclictest37-21ksoftirqd/312:07:113
318757994024,12cyclictest37-21ksoftirqd/312:07:113
318757994022,14cyclictest37-21ksoftirqd/311:30:063
318757994012,6cyclictest131rcu_preempt12:36:183
318753994034,4cyclictest294335-21kworker/2:308:35:002
318753994031,6cyclictest323399-21kworker/2:008:55:012
318744994025,10cyclictest309-21in:imuxsock08:50:030
31875799393,4cyclictest131rcu_preempt11:58:093
31875799393,4cyclictest131rcu_preempt11:58:093
318757993927,9cyclictest37-21ksoftirqd/312:03:223
318757993926,10cyclictest37-21ksoftirqd/309:14:363
318757993925,9cyclictest275-21dbus-daemon07:55:013
318757993924,12cyclictest37-21ksoftirqd/311:30:013
318757993923,12cyclictest3467602-21systemd-journal07:15:003
318757993923,12cyclictest3467602-21systemd-journal07:15:003
31875799391,4cyclictest0-21swapper/310:03:593
318753993931,5cyclictest294335-21kworker/2:307:25:002
318753993930,6cyclictest368521-21kworker/2:311:21:092
318753993926,10cyclictest30-21ksoftirqd/207:50:032
318753993925,10cyclictest3467602-21systemd-journal07:29:182
31875399392,4cyclictest131rcu_preempt08:30:012
31875399392,4cyclictest131rcu_preempt08:30:012
31874499396,6cyclictest12-21ksoftirqd/008:35:180
31874499396,6cyclictest12-21ksoftirqd/008:03:270
31875799383,3cyclictest131rcu_preempt10:52:443
31875799383,3cyclictest131rcu_preempt10:52:433
318757993825,10cyclictest37-21ksoftirqd/311:22:083
31875799382,4cyclictest131rcu_preempt11:10:523
31875799382,4cyclictest0-21swapper/310:40:413
31875799382,4cyclictest0-21swapper/310:40:403
318757993824,11cyclictest37-21ksoftirqd/308:18:203
318757993824,11cyclictest275-21dbus-daemon07:30:013
318757993824,10cyclictest37-21ksoftirqd/308:10:003
31875799381,4cyclictest131rcu_preempt08:40:003
318753993824,10cyclictest322068-21wc08:15:022
318753993824,10cyclictest322068-21wc08:15:012
318757993726,8cyclictest37-21ksoftirqd/311:40:203
318757993725,9cyclictest37-21ksoftirqd/309:15:333
318757993724,8cyclictest324387-21munin-plugin-st09:00:013
318757993724,8cyclictest324387-21munin-plugin-st09:00:003
31875799372,13cyclictest420310-21ssh12:26:583
31874499371,5cyclictest0-21swapper/008:45:020
318757993627,5cyclictest322840-21latency_hist08:30:023
318757993627,5cyclictest322840-21latency_hist08:30:013
31875799362,3cyclictest131rcu_preempt07:35:013
318757993622,10cyclictest1-21systemd09:10:013
318757993621,11cyclictest37-21ksoftirqd/312:20:053
318757993616,15cyclictest275-21dbus-daemon08:45:023
31875799361,4cyclictest131rcu_preempt09:35:023
31875399361,7cyclictest0-21swapper/207:10:022
31875399361,3cyclictest131rcu_preempt08:10:012
31875799353,4cyclictest131rcu_preempt08:35:013
31875799351,4cyclictest131rcu_preempt08:55:003
31875799342,4cyclictest131rcu_preempt07:25:023
318757993420,11cyclictest275-21dbus-daemon08:05:003
318757993418,11cyclictest275-21dbus-daemon07:50:013
31875799341,3cyclictest131rcu_preempt09:04:593
318753993428,4cyclictest320242-21/usr/sbin/munin07:40:012
318757993222,8cyclictest37-21ksoftirqd/307:10:023
318753993225,5cyclictest320031-21latency_hist07:35:012
318757993118,10cyclictest37-21ksoftirqd/308:25:013
31875399303,11cyclictest321533-21cron08:04:592
31875799291,3cyclictest131rcu_preempt08:45:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional