You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 21:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 09, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
216970799118110,6cyclictest37-21ksoftirqd/310:05:013
216970799118110,6cyclictest37-21ksoftirqd/310:05:013
2169696999691,3cyclictest0-21swapper/007:45:020
2169696999187,2cyclictest0-21swapper/010:05:030
2169696999187,2cyclictest0-21swapper/010:05:020
2169701998660,18cyclictest23-21ksoftirqd/107:45:021
2169696998176,3cyclictest0-21swapper/012:00:010
2169705997618,18cyclictest2233975-21ssh11:24:222
2169696997434,29cyclictest2261993-21kworker/u8:012:40:020
2169696997431,12cyclictest1-21systemd11:10:000
2169696997427,12cyclictest2193780-21ssh09:51:140
2169705997316,15cyclictest2261702-21ssh12:28:342
2169705997316,15cyclictest2261702-21ssh12:28:332
2169705997215,19cyclictest2225083-21ssh11:03:452
2169705997113,22cyclictest2264027-21ssh12:34:032
2169705997021,17cyclictest2226309-21sh11:06:262
2169705996912,16cyclictest2171570-21cat07:45:012
2169705996620,7cyclictest2215425-21ssh10:41:112
2169705996619,14cyclictest2235702-21ssh11:28:182
2169705996614,14cyclictest2208227-21latency_hist10:25:022
2169705996523,7cyclictest2202243-21ssh10:10:492
2169705996514,20cyclictest2231088-21ssh11:17:412
2169705996514,20cyclictest2231088-21ssh11:17:402
2169705996514,15cyclictest2212238-21ssh10:34:222
2169705996513,18cyclictest2190860-21ssh09:44:492
2169705996513,18cyclictest2190860-21ssh09:44:482
2169705996513,15cyclictest2252971-21ssh12:08:182
2169707996420,9cyclictest37-21ksoftirqd/308:35:003
2169705996417,16cyclictest2175612-21munin-run09:05:012
2169705996414,15cyclictest2236849-21ssh11:30:382
2169705996411,15cyclictest2219994-21ssh10:51:552
2169705996314,14cyclictest2181066-21ssh09:21:422
2169705996313,19cyclictest2248614-21ssh11:58:152
2169705996212,13cyclictest2218315-21ssh10:48:092
216970799614,11cyclictest37-21ksoftirqd/309:45:013
216970799614,11cyclictest37-21ksoftirqd/309:45:003
2169707996115,24cyclictest2249475-21ssh12:00:033
2169705996119,13cyclictest2200362-21ssh10:06:312
2169705996115,15cyclictest2241852-21ssh11:42:272
2169705996015,8cyclictest2183142-21ssh09:26:372
2169705996014,14cyclictest2266000-21ssh12:38:342
2169701996033,14cyclictest23-21ksoftirqd/111:39:071
216969699601,3cyclictest0-21swapper/008:00:020
216970599597,17cyclictest2175868-21diskmemload09:32:482
2169705995915,12cyclictest2245836-21ssh11:51:322
2169705995913,14cyclictest2238971-21ssh11:35:372
2169705995913,13cyclictest2197787-21ssh10:00:202
2169705995913,13cyclictest2197787-21ssh10:00:192
2169696995929,14cyclictest2246165-21ssh11:52:290
2169707995815,10cyclictest37-21ksoftirqd/310:45:483
2169707995810,11cyclictest37-21ksoftirqd/312:03:543
2169705995819,8cyclictest2177879-21ssh09:14:522
2169705995819,7cyclictest2195635-21ssh09:55:342
2169705995814,17cyclictest2192229-21sh09:47:532
2169696995854,2cyclictest0-21swapper/007:15:030
2169696995824,3cyclictest131rcu_preempt11:17:560
2169696995824,3cyclictest131rcu_preempt11:17:560
2169707995738,14cyclictest1-21systemd08:10:003
2169705995740,13cyclictest2205824-21ssh10:19:312
2169705995740,13cyclictest2205824-21ssh10:19:312
2169705995713,14cyclictest2179674-21ssh09:18:502
216970799566,16cyclictest37-21ksoftirqd/311:39:073
2169705995641,12cyclictest2250625-21ssh12:02:452
2169705995616,15cyclictest2254803-21rm12:12:312
2169705995610,16cyclictest364-21snmpd10:38:122
2169696995636,16cyclictest0-21swapper/012:25:020
216969699562,37cyclictest131rcu_preempt09:12:420
216970799558,11cyclictest37-21ksoftirqd/309:34:073
216970799555,14cyclictest37-21ksoftirqd/312:29:463
216970799555,14cyclictest37-21ksoftirqd/312:29:463
216970799555,14cyclictest37-21ksoftirqd/309:37:223
2169701995520,11cyclictest23-21ksoftirqd/111:02:001
2169696995527,25cyclictest2241847-21ssh11:42:250
2169696995523,16cyclictest2187730-21ssh09:37:240
216970799548,13cyclictest37-21ksoftirqd/311:17:353
216970799548,13cyclictest37-21ksoftirqd/311:17:343
216970799547,10cyclictest37-21ksoftirqd/309:51:523
216970799542,16cyclictest37-21ksoftirqd/309:13:183
2169707995418,8cyclictest37-21ksoftirqd/310:44:223
2169707995413,14cyclictest2263537-21sh12:32:443
216970599548,15cyclictest2173790-21latency_hist08:30:022
2169705995439,12cyclictest2188092-21ssh09:38:162
2169705995437,14cyclictest2258309-21ssh12:20:202
2169705995418,8cyclictest2228743-21ssh11:12:092
2169705995411,13cyclictest2209726-21ssh10:28:162
2169705995411,13cyclictest2209726-21ssh10:28:162
2169701995427,11cyclictest23-21ksoftirqd/109:58:291
2169696995434,4cyclictest16850irq/134-800000011:10:030
2169696995425,13cyclictest2240731-21ssh11:40:010
2169696995423,15cyclictest2244699-21ssh11:49:130
216969699541,34cyclictest12-21ksoftirqd/012:05:020
216970799537,10cyclictest37-21ksoftirqd/312:06:383
216970799536,6cyclictest37-21ksoftirqd/309:27:023
216970799534,7cyclictest37-21ksoftirqd/312:17:453
216970799534,10cyclictest37-21ksoftirqd/311:04:393
216970799532,11cyclictest37-21ksoftirqd/312:38:033
2169707995314,7cyclictest37-21ksoftirqd/312:25:013
2169707995313,5cyclictest131rcu_preempt10:55:433
2169707995310,10cyclictest37-21ksoftirqd/311:44:523
2169705995336,13cyclictest2221990-21ssh10:56:362
2169705995332,15cyclictest2171953-21cron07:54:592
2169705995311,14cyclictest2243721-21ssh11:46:482
2169696995327,10cyclictest2196003-21ssh09:56:260
216970799527,10cyclictest37-21ksoftirqd/310:16:193
216970799527,10cyclictest37-21ksoftirqd/310:16:183
216970799525,15cyclictest37-21ksoftirqd/309:20:133
216970799523,11cyclictest37-21ksoftirqd/311:24:333
216970799521,9cyclictest2245984-21ssh11:52:023
2169707995212,11cyclictest37-21ksoftirqd/310:54:283
2169705995236,13cyclictest2193932-21ssh09:51:442
216969699525,10cyclictest3467602-21systemd-journal10:09:590
2169696995248,2cyclictest0-21swapper/012:30:020
2169696995245,5cyclictest19350irq/34-mmc108:50:020
2169696995229,18cyclictest2238712-21latency_hist11:35:030
2169696995225,12cyclictest2204861-21ssh10:16:570
2169696995225,12cyclictest2204861-21ssh10:16:560
2169696995224,13cyclictest2225172-21ssh11:04:070
216969699522,3cyclictest131rcu_preempt09:42:550
216969699522,3cyclictest131rcu_preempt09:42:540
2169696995223,14cyclictest2178409-21ssh09:15:370
216969699521,3cyclictest131rcu_preempt11:21:560
216970799517,8cyclictest37-21ksoftirqd/310:28:273
216970799517,8cyclictest37-21ksoftirqd/310:28:263
216970799517,15cyclictest37-21ksoftirqd/309:19:133
216970799515,6cyclictest37-21ksoftirqd/311:05:313
216970799515,11cyclictest37-21ksoftirqd/309:59:033
216970799513,38cyclictest2211091-21kworker/u8:110:35:013
2169707995133,14cyclictest1-21systemd07:30:013
216970799513,10cyclictest37-21ksoftirqd/311:47:433
216970799511,15cyclictest37-21ksoftirqd/312:14:173
2169707995111,10cyclictest37-21ksoftirqd/310:37:383
2169705995135,12cyclictest2172763-21munin-run08:10:012
2169705995134,13cyclictest2256181-21ssh12:15:262
2169705995116,10cyclictest364-21snmpd08:42:492
2169701995112,10cyclictest23-21ksoftirqd/109:33:191
2169696995141,6cyclictest16950irq/224-800000008:19:590
2169696995133,4cyclictest16850irq/134-800000010:45:020
2169696995128,8cyclictest2210211-21ssh10:29:350
2169696995128,8cyclictest2210211-21ssh10:29:350
216969699512,26cyclictest131rcu_preempt10:40:010
216969699511,14cyclictest2219158-21cat10:50:020
216970799505,10cyclictest37-21ksoftirqd/308:30:023
2169707995042,6cyclictest37-21ksoftirqd/310:24:303
216970799501,17cyclictest346771850lldpd09:48:513
2169707995011,4cyclictest131rcu_preempt10:10:313
2169701995011,5cyclictest131rcu_preempt10:40:011
2169701995010,11cyclictest23-21ksoftirqd/110:57:391
2169696995024,11cyclictest2257194-21ssh12:18:110
216969699501,26cyclictest131rcu_preempt09:33:510
216970799498,9cyclictest37-21ksoftirqd/311:28:093
2169707994935,11cyclictest2237103-21sshd11:31:223
2169707994910,5cyclictest131rcu_preempt10:05:413
2169705994941,5cyclictest2162890-21kworker/2:008:35:012
2169705994913,13cyclictest2173264-21latency_hist08:20:022
216970199498,12cyclictest23-21ksoftirqd/111:05:371
216970199498,12cyclictest23-21ksoftirqd/109:47:581
2169701994939,6cyclictest221rcuc/112:33:201
2169701994933,12cyclictest275-21dbus-daemon07:50:011
2169701994933,12cyclictest275-21dbus-daemon07:50:011
2169701994928,17cyclictest275-21dbus-daemon09:15:021
2169701994926,7cyclictest23-21ksoftirqd/111:29:341
2169696994939,6cyclictest16850irq/134-800000010:55:370
2169696994929,15cyclictest275-21dbus-daemon10:25:010
2169696994925,4cyclictest131rcu_preempt10:10:360
2169696994925,3cyclictest131rcu_preempt09:25:210
2169696994921,13cyclictest2219554-21ssh10:50:430
216970799486,30cyclictest2174811-21kworker/u8:108:55:023
216970799481,10cyclictest2229631-21ssh11:14:243
2169705994825,18cyclictest2175320-21cron09:00:002
216970199487,5cyclictest131rcu_preempt09:37:271
2169701994823,16cyclictest23-21ksoftirqd/112:25:401
2169701994823,16cyclictest23-21ksoftirqd/112:25:391
2169701994819,17cyclictest23-21ksoftirqd/111:55:241
2169701994811,6cyclictest131rcu_preempt09:10:001
216969699481,3cyclictest131rcu_preempt12:10:470
2169705994735,9cyclictest2175827-21sshd09:05:472
2169705994734,9cyclictest2171683-21sshd07:45:382
2169705994734,9cyclictest2171683-21sshd07:45:382
2169705994734,10cyclictest2172498-21munin-run08:05:002
2169705994732,12cyclictest2174275-21sshd08:38:472
2169701994728,13cyclictest1-21systemd12:23:491
2169701994711,11cyclictest23-21ksoftirqd/110:27:371
2169701994711,11cyclictest23-21ksoftirqd/110:27:371
2169701994710,6cyclictest131rcu_preempt09:18:421
216969699473,7cyclictest2171192-21munin-plugin-st07:40:000
216969699473,7cyclictest2171192-21munin-plugin-st07:40:000
2169696994732,9cyclictest2175322-21cron09:00:000
216970799468,6cyclictest37-21ksoftirqd/307:20:023
216970799462,13cyclictest37-21ksoftirqd/307:45:003
216970799461,4cyclictest0-21swapper/308:20:013
2169705994634,8cyclictest48-21kcompactd008:21:352
2169705994631,11cyclictest2174835-21systemd-run08:50:022
216970199468,21cyclictest131rcu_preempt12:18:541
216970199467,8cyclictest23-21ksoftirqd/110:20:281
216970199465,6cyclictest131rcu_preempt12:37:141
216970199463,5cyclictest0-21swapper/111:17:461
216970199463,5cyclictest0-21swapper/111:17:451
216970199461,9cyclictest23-21ksoftirqd/110:10:201
2169701994613,10cyclictest131rcu_preempt09:00:271
216970199461,11cyclictest23-21ksoftirqd/110:18:181
216970199461,11cyclictest23-21ksoftirqd/110:18:171
216970199461,10cyclictest23-21ksoftirqd/110:41:091
2169696994639,4cyclictest16850irq/134-800000012:26:050
2169696994639,4cyclictest16850irq/134-800000012:26:040
216969699462,15cyclictest2212204-21ssh10:34:100
216969699461,6cyclictest2253649-21modprobe12:10:010
216970799451,7cyclictest0-21swapper/308:45:013
2169705994538,4cyclictest2170914-21sshd07:33:442
2169705994533,9cyclictest2171169-21sshd07:39:382
2169705994533,9cyclictest2171169-21sshd07:39:372
2169705994525,15cyclictest364-21snmpd08:13:562
216970199457,25cyclictest2241171-21kworker/u8:012:05:021
216970199455,5cyclictest131rcu_preempt10:09:331
216970199455,5cyclictest131rcu_preempt09:28:401
2169701994534,8cyclictest3467602-21systemd-journal10:34:471
2169701994531,10cyclictest23-21ksoftirqd/111:50:001
2169701994528,13cyclictest2172194-21sshd07:58:471
216970199452,5cyclictest131rcu_preempt10:53:241
2169701994519,16cyclictest23-21ksoftirqd/111:20:111
216970199451,8cyclictest23-21ksoftirqd/109:20:461
2169696994529,13cyclictest2192043-21ssh09:47:170
2169696994528,13cyclictest2234602-21ssh11:25:300
2169696994527,14cyclictest2175891-21systemd-run09:10:000
216970799449,8cyclictest37-21ksoftirqd/308:00:013
2169707994429,11cyclictest275-21dbus-daemon07:50:013
2169707994429,11cyclictest275-21dbus-daemon07:50:003
2169705994432,9cyclictest2169867-21sshd07:12:372
2169705994431,10cyclictest2170415-21sshd07:24:372
2169705994418,22cyclictest2170189-21true07:20:012
216970199448,5cyclictest131rcu_preempt12:07:371
216970199444,9cyclictest23-21ksoftirqd/111:14:271
216970199444,9cyclictest23-21ksoftirqd/109:41:451
216970199444,9cyclictest23-21ksoftirqd/109:41:441
2169701994434,7cyclictest23-21ksoftirqd/110:00:361
2169701994434,7cyclictest23-21ksoftirqd/110:00:361
216970199442,5cyclictest131rcu_preempt11:41:221
2169701994421,7cyclictest221rcuc/107:25:011
2169701994419,14cyclictest23-21ksoftirqd/109:50:381
216970199441,7cyclictest0-21swapper/108:21:471
2169701994412,5cyclictest131rcu_preempt11:34:371
216969699446,26cyclictest2169605-21kworker/u8:207:35:020
2169696994434,6cyclictest309-21in:imuxsock08:45:010
2169696994428,13cyclictest2181067-21ssh09:21:420
216970799435,8cyclictest37-21ksoftirqd/309:00:023
216970799434,13cyclictest37-21ksoftirqd/308:22:083
2169705994336,4cyclictest2172259-21munin-run08:00:012
216970199437,8cyclictest23-21ksoftirqd/111:54:121
216970199434,8cyclictest23-21ksoftirqd/108:55:481
2169701994329,8cyclictest23-21ksoftirqd/107:15:001
216970199431,4cyclictest0-21swapper/110:49:571
2169696994326,13cyclictest2172718-21sshd08:06:470
216969699431,4cyclictest2171959-21modprobe07:54:590
216969699431,38cyclictest2172467-21sshd08:01:470
216970799425,8cyclictest37-21ksoftirqd/309:05:313
216970799423,10cyclictest37-21ksoftirqd/309:05:013
216970799421,5cyclictest0-21swapper/307:35:453
216970799421,5cyclictest0-21swapper/307:35:443
2169705994230,9cyclictest364-21snmpd07:27:352
2169705994218,18cyclictest2175089-21/usr/sbin/munin08:55:002
216970199426,10cyclictest23-21ksoftirqd/112:13:291
2169701994226,13cyclictest1-21systemd08:15:011
2169696994233,5cyclictest2175080-21munin-plugin-st08:55:010
216970799414,10cyclictest37-21ksoftirqd/307:15:013
216970799413,9cyclictest37-21ksoftirqd/307:35:003
2169707994129,6cyclictest2174299-21munin-plugin-st08:40:003
2169707994129,10cyclictest37-21ksoftirqd/307:51:373
2169707994111,5cyclictest131rcu_preempt08:50:003
216970799410,6cyclictest150-21pr/ttyS208:01:243
2169696994123,13cyclictest3467602-21systemd-journal08:40:010
216970199401,8cyclictest0-21swapper/108:06:011
216970199401,7cyclictest23-21ksoftirqd/108:45:021
216970199401,6cyclictest0-21swapper/108:45:471
2169701994011,7cyclictest131rcu_preempt07:40:001
2169701994011,7cyclictest131rcu_preempt07:40:001
2169701994011,5cyclictest131rcu_preempt08:17:001
216970199401,14cyclictest23-21ksoftirqd/107:25:381
216970799391,8cyclictest0-21swapper/307:21:013
216970199392,10cyclictest23-21ksoftirqd/108:50:031
216970199391,5cyclictest150-21pr/ttyS208:34:001
2169696993933,4cyclictest16950irq/224-800000007:30:010
2169696993923,12cyclictest2172976-21sshd08:12:470
216969699391,13cyclictest2173771-21sshd08:28:460
216970799389,5cyclictest131rcu_preempt08:10:473
2169701993827,8cyclictest23-21ksoftirqd/108:05:031
216970199381,7cyclictest150-21pr/ttyS208:38:491
216970199381,5cyclictest150-21pr/ttyS207:51:391
2169696993826,8cyclictest2173515-21sshd08:21:470
216969699381,7cyclictest2170433-21/usr/sbin/munin07:25:000
216970199378,6cyclictest23-21ksoftirqd/108:26:471
216970199371,7cyclictest0-21swapper/107:30:271
2169696993731,4cyclictest16950irq/224-800000007:20:010
216970199361,11cyclictest23-21ksoftirqd/107:15:281
216969699362,4cyclictest2171731-21munin-run07:50:020
216969699362,4cyclictest2171731-21munin-run07:50:010
2169696993426,4cyclictest0-21swapper/009:05:000
2169696993322,8cyclictest12-21ksoftirqd/007:10:010
216969699331,29cyclictest0-21swapper/008:33:510
2169701993224,4cyclictest2169790-21latency_hist07:10:021
2169705993022,4cyclictest2169636-21latency_hist07:10:032
216970799282,20cyclictest2169740-21cat07:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional