You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 23:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 23, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2223156996726,4cyclictest2273380-21sh10:42:321
2223154996711,5cyclictest131rcu_preempt09:24:400
2223156996626,4cyclictest2325010-21sh12:30:481
2223156996624,11cyclictest23-21ksoftirqd/112:04:381
2223156996521,12cyclictest23-21ksoftirqd/111:45:001
222316299644,33cyclictest2227748-21kworker/u8:208:40:033
222316299644,33cyclictest2227748-21kworker/u8:208:40:023
2223154996416,10cyclictest12-21ksoftirqd/012:30:020
222315499641,60cyclictest0-21swapper/010:05:040
2223156996323,10cyclictest23-21ksoftirqd/111:46:131
2223156996322,10cyclictest23-21ksoftirqd/111:54:581
2223156996222,4cyclictest2229000-21latency_hist09:00:031
2223154996220,36cyclictest12-21ksoftirqd/010:00:000
2223160996139,6cyclictest2214980-21kworker/2:112:17:252
2223156996125,4cyclictest2256245-21sh10:06:231
2223156996125,12cyclictest2226013-21wc08:05:021
2223156996125,12cyclictest2226013-21wc08:05:011
222315499618,5cyclictest131rcu_preempt09:48:230
2223162996014,5cyclictest2319706-21grep12:20:013
2223162996013,20cyclictest2283953-21kworker/3:211:30:033
2223160996052,5cyclictest2229381-21kworker/2:009:39:512
2223160996052,5cyclictest2229381-21kworker/2:009:39:512
2223156996023,11cyclictest23-21ksoftirqd/112:28:271
2223154996030,13cyclictest2275489-21ssh10:46:580
2223154996028,15cyclictest2284505-21ssh11:06:000
2223160995940,15cyclictest1-21systemd07:20:002
2223160995934,5cyclictest2214980-21kworker/2:109:15:542
2223160995934,5cyclictest2214980-21kworker/2:109:15:532
2223156995921,6cyclictest2317547-21kworker/u8:212:24:591
2223156995914,6cyclictest131rcu_preempt10:14:431
2223154995929,14cyclictest2304283-21ssh11:47:250
2223154995920,35cyclictest1-21systemd12:40:010
2223154995915,39cyclictest12-21ksoftirqd/009:35:020
2223160995849,6cyclictest2249025-21kworker/2:011:15:022
2223156995823,4cyclictest2229328-21diskmemload12:06:371
2223154995829,12cyclictest2326458-21ssh12:34:080
2223156995723,8cyclictest23-21ksoftirqd/111:06:241
2223154995743,9cyclictest2295929-21latency_hist11:30:020
2223154995728,13cyclictest2313426-21ssh12:06:270
2223154995728,13cyclictest2277838-21ssh10:51:550
222315499572,38cyclictest2232834-21ssh09:17:160
222315499572,38cyclictest2232834-21ssh09:17:160
2223160995648,5cyclictest2214980-21kworker/2:112:13:342
2223156995623,8cyclictest23-21ksoftirqd/109:45:571
222315499567,6cyclictest131rcu_preempt07:35:260
2223154995646,6cyclictest151rcuc/010:15:000
2223154995625,15cyclictest2305829-21ssh11:50:320
2223154995621,30cyclictest12-21ksoftirqd/009:52:260
2223160995546,6cyclictest2249025-21kworker/2:011:17:432
2223160995546,6cyclictest2249025-21kworker/2:011:17:422
2223160995546,6cyclictest2214980-21kworker/2:109:31:272
2223156995523,9cyclictest23-21ksoftirqd/111:13:201
222315499553,18cyclictest2226731-21munin-run08:20:000
2223154995527,12cyclictest2264516-21ssh10:23:590
2223154995525,28cyclictest12-21ksoftirqd/009:38:220
2223154995525,28cyclictest12-21ksoftirqd/009:38:220
222315499552,49cyclictest2262573-21grep10:20:000
2223154995521,6cyclictest12-21ksoftirqd/011:16:130
2223154995521,6cyclictest12-21ksoftirqd/011:16:120
2223160995446,5cyclictest2291186-21kworker/2:311:47:072
2223160995446,5cyclictest2249025-21kworker/2:010:29:372
2223160995446,5cyclictest2229381-21kworker/2:009:43:272
2223160995446,5cyclictest2214980-21kworker/2:111:00:312
2223160995445,6cyclictest2291186-21kworker/2:311:30:222
2223160995445,6cyclictest2214980-21kworker/2:112:10:012
222315699542,6cyclictest2226462-21munin-run08:15:011
2223156995425,8cyclictest23-21ksoftirqd/108:40:251
2223156995418,12cyclictest23-21ksoftirqd/111:33:551
222315499548,22cyclictest2229328-21diskmemload09:30:020
222315499545,40cyclictest2310189-21sh12:00:010
2223154995436,14cyclictest2272111-21systemctl10:40:000
222315499543,45cyclictest3467602-21systemd-journal10:10:000
2223154995417,17cyclictest16850irq/134-800000012:13:050
2223160995346,4cyclictest2214980-21kworker/2:110:41:282
2223160995345,5cyclictest2291186-21kworker/2:311:42:032
2223160995345,5cyclictest2214980-21kworker/2:110:50:282
2223160995344,5cyclictest2214980-21kworker/2:112:38:342
2223160995343,6cyclictest2214980-21kworker/2:112:28:282
222315499533,11cyclictest12-21ksoftirqd/011:25:010
2223154995331,18cyclictest16850irq/134-800000011:45:010
2223154995324,14cyclictest2244934-21ssh09:42:460
2223154995323,14cyclictest2229713-21ssh09:10:190
2223154995320,30cyclictest12-21ksoftirqd/011:03:090
2223154995316,32cyclictest12-21ksoftirqd/011:40:030
222315499531,36cyclictest2267842-21ssh10:30:400
2223154995311,9cyclictest12-21ksoftirqd/008:15:000
2223154995310,11cyclictest12-21ksoftirqd/009:00:000
2223162995239,9cyclictest37-21ksoftirqd/310:15:013
2223162995233,14cyclictest2281496-21cron10:59:593
2223160995244,5cyclictest2249025-21kworker/2:010:11:492
2223160995244,5cyclictest2246543-21kworker/2:209:49:452
2223160995244,5cyclictest2214980-21kworker/2:112:30:182
2223160995244,5cyclictest2214980-21kworker/2:110:35:322
2223160995244,5cyclictest2214980-21kworker/2:109:57:072
2223160995235,13cyclictest3467602-21systemd-journal11:50:032
2223154995219,17cyclictest16850irq/134-800000010:42:260
2223154995214,5cyclictest131rcu_preempt12:16:160
2223154995210,9cyclictest131rcu_preempt08:25:010
2223162995134,11cyclictest3467602-21systemd-journal10:19:593
2223160995143,5cyclictest2291186-21kworker/2:311:27:122
2223160995143,5cyclictest2291186-21kworker/2:311:22:322
2223160995143,5cyclictest2249025-21kworker/2:011:09:322
2223160995143,5cyclictest2249025-21kworker/2:010:16:082
2223160995143,5cyclictest2229381-21kworker/2:009:25:132
2223160995143,5cyclictest2214980-21kworker/2:112:22:102
2223160995143,5cyclictest2214980-21kworker/2:112:02:112
2223160995143,5cyclictest2214980-21kworker/2:110:20:532
2223160995143,5cyclictest2214980-21kworker/2:110:00:172
2223160995143,5cyclictest2214980-21kworker/2:107:25:012
2223160995142,6cyclictest2291186-21kworker/2:311:39:192
2223160995142,6cyclictest2214980-21kworker/2:110:05:192
222315699513,27cyclictest2227728-21kworker/u8:009:05:021
2223156995130,16cyclictest3467602-21systemd-journal08:55:001
2223154995134,12cyclictest1-21systemd12:25:010
2223154995119,28cyclictest12-21ksoftirqd/012:03:120
222316299507,21cyclictest0-21swapper/310:40:013
2223162995043,3cyclictest371ksoftirqd/309:25:013
222316099507,6cyclictest131rcu_preempt11:00:012
2223160995043,4cyclictest2214980-21kworker/2:109:50:312
2223160995043,4cyclictest2214980-21kworker/2:109:20:252
2223160995042,5cyclictest2249025-21kworker/2:010:31:252
2223160995042,5cyclictest2214980-21kworker/2:109:13:142
2223160995042,5cyclictest2214980-21kworker/2:107:40:012
2223160995041,5cyclictest2291186-21kworker/2:311:55:222
2223160994941,5cyclictest2214980-21kworker/2:110:47:452
2223160994928,14cyclictest275-21dbus-daemon08:25:012
222316099491,8cyclictest30-21ksoftirqd/208:19:592
2223154994924,10cyclictest2281103-21ssh10:59:030
222315499491,11cyclictest12-21ksoftirqd/007:35:000
2223162994836,9cyclictest2327188-21sshd12:35:253
222316299483,27cyclictest361rcuc/307:15:023
222316299481,12cyclictest2257969-21cat10:10:023
2223160994840,5cyclictest2214980-21kworker/2:108:34:242
2223160994840,5cyclictest2214980-21kworker/2:107:55:022
2223156994814,15cyclictest221rcuc/110:00:031
222315499485,5cyclictest131rcu_preempt11:31:360
2223162994729,14cyclictest1-21systemd12:35:013
222316299471,3cyclictest131rcu_preempt11:47:333
2223160994738,5cyclictest2214980-21kworker/2:108:05:012
2223160994738,5cyclictest2214980-21kworker/2:108:05:012
222315699474,25cyclictest221rcuc/107:15:011
2223156994715,23cyclictest23-21ksoftirqd/110:20:021
2223154994731,12cyclictest3467602-21systemd-journal08:45:010
2223154994711,6cyclictest131rcu_preempt11:15:020
222316299469,7cyclictest131rcu_preempt11:05:023
2223162994630,12cyclictest275-21dbus-daemon11:25:013
222316299461,3cyclictest131rcu_preempt09:16:573
222316299461,3cyclictest131rcu_preempt09:16:563
2223160994638,5cyclictest2214980-21kworker/2:108:54:122
2223160994638,5cyclictest2214980-21kworker/2:108:40:262
2223160994638,5cyclictest2214980-21kworker/2:108:38:262
2223160994638,5cyclictest2214980-21kworker/2:108:38:262
2223160994638,4cyclictest2214980-21kworker/2:109:09:462
2223160994637,5cyclictest2214980-21kworker/2:109:02:462
222315699467,34cyclictest2243515-21sh09:40:011
222315699467,34cyclictest2243515-21sh09:40:011
2223156994622,14cyclictest2329309-21ssh12:40:011
222315499468,4cyclictest131rcu_preempt08:09:590
2223154994612,5cyclictest131rcu_preempt10:29:020
222316299454,3cyclictest131rcu_preempt09:40:023
222316299454,3cyclictest131rcu_preempt09:40:013
2223162994535,5cyclictest2228799-21modprobe09:00:003
2223162994528,14cyclictest37-21ksoftirqd/308:20:023
2223160994538,4cyclictest2214980-21kworker/2:108:46:342
2223160994537,5cyclictest2214980-21kworker/2:108:12:222
2223160994515,5cyclictest131rcu_preempt07:55:242
2223156994531,11cyclictest2223850-21sshd07:20:241
2223156994526,13cyclictest311-21rs:main1
2223156994526,13cyclictest311-21rs:main1
222315499456,10cyclictest12-21ksoftirqd/009:10:010
2223154994520,17cyclictest311-21rs:main0
2223154994511,9cyclictest12-21ksoftirqd/007:55:250
2223162994429,11cyclictest3467602-21systemd-journal09:15:013
222316299441,3cyclictest131rcu_preempt09:31:443
2223162994411,24cyclictest0-21swapper/309:10:023
2223160994436,5cyclictest2214980-21kworker/2:108:55:302
2223160994436,5cyclictest2214980-21kworker/2:107:48:072
2223156994431,10cyclictest2238531-21sshd09:29:251
2223154994438,3cyclictest150-21pr/ttyS207:55:020
2223154994432,10cyclictest12-21ksoftirqd/008:30:010
222316299436,7cyclictest37-21ksoftirqd/311:51:123
222316299435,3cyclictest131rcu_preempt10:32:373
2223162994327,11cyclictest3467602-21systemd-journal07:45:013
2223162994326,12cyclictest3467602-21systemd-journal07:50:013
222316299432,4cyclictest131rcu_preempt10:29:243
2223162994321,17cyclictest3467602-21systemd-journal10:25:013
2223162994320,18cyclictest311-21rs:main3
2223162994319,21cyclictest37-21ksoftirqd/310:53:323
2223162994312,25cyclictest37-21ksoftirqd/308:10:003
2223160994337,4cyclictest2214980-21kworker/2:107:27:002
2223160994336,4cyclictest2214980-21kworker/2:108:27:092
2223160994335,5cyclictest2212401-21kworker/2:207:11:082
2223156994327,12cyclictest275-21dbus-daemon07:30:011
2223156994327,11cyclictest3467602-21systemd-journal08:20:001
2223156994315,6cyclictest131rcu_preempt09:41:001
222315499438,5cyclictest131rcu_preempt08:50:020
222315499436,7cyclictest12-21ksoftirqd/007:10:020
222315499433,4cyclictest131rcu_preempt08:35:010
2223154994333,7cyclictest12-21ksoftirqd/007:30:000
2223154994328,10cyclictest2229074-21munin-plugin-st09:05:010
2223154994310,9cyclictest12-21ksoftirqd/008:50:270
222316299423,3cyclictest131rcu_preempt12:22:173
2223162994230,10cyclictest2285418-21apt-config11:08:033
2223162994227,10cyclictest309-21in:imuxsock11:35:263
2223162994226,11cyclictest3467602-21systemd-journal12:00:013
2223160994234,5cyclictest2214980-21kworker/2:108:09:332
2223160994234,5cyclictest2214980-21kworker/2:107:30:172
222315699424,35cyclictest2269706-21modprobe10:35:001
2223156994233,6cyclictest2276785-21latency_hist10:49:591
2223156994230,6cyclictest2241121-21sh09:35:011
2223154994212,5cyclictest131rcu_preempt07:20:000
222316299414,3cyclictest131rcu_preempt08:55:013
2223162994129,10cyclictest37-21ksoftirqd/310:02:313
2223162994128,9cyclictest37-21ksoftirqd/309:25:453
2223162994127,10cyclictest1-21systemd07:30:013
2223162994126,12cyclictest37-21ksoftirqd/309:50:333
2223162994126,10cyclictest309-21in:imuxsock09:50:003
222316299411,6cyclictest131rcu_preempt12:10:023
222316299411,24cyclictest2298415-21ssh11:35:013
2223160994131,5cyclictest3467602-21systemd-journal07:44:592
222315699418,4cyclictest131rcu_preempt11:39:591
222315699415,7cyclictest23-21ksoftirqd/110:27:341
222315699415,5cyclictest131rcu_preempt09:25:001
222315699414,4cyclictest131rcu_preempt11:22:171
222315699413,9cyclictest2272227-21date10:40:021
2223156994130,8cyclictest2281231-21sshd10:59:251
222316299409,8cyclictest131rcu_preempt07:10:023
222316299405,4cyclictest131rcu_preempt11:12:533
2223162994030,7cyclictest37-21ksoftirqd/307:25:003
2223162994026,11cyclictest37-21ksoftirqd/312:10:253
2223162994026,10cyclictest3467602-21systemd-journal07:55:253
2223162994024,12cyclictest275-21dbus-daemon09:45:013
2223162994022,12cyclictest275-21dbus-daemon09:05:003
2223162994015,6cyclictest131rcu_preempt11:41:093
2223156994032,5cyclictest2283885-21cron11:05:011
2223156994026,11cyclictest2279128-21systemd-run10:55:011
2223156994026,10cyclictest23-21ksoftirqd/112:12:301
2223156994020,14cyclictest275-21dbus-daemon08:50:011
2223154994028,8cyclictest12-21ksoftirqd/008:40:020
2223154994028,8cyclictest12-21ksoftirqd/008:40:010
2223162993929,6cyclictest309-21in:imuxsock08:25:013
222316299392,7cyclictest2226561-21latency_hist08:15:013
2223162993925,10cyclictest37-21ksoftirqd/311:17:403
2223162993925,10cyclictest37-21ksoftirqd/311:17:393
2223162993922,12cyclictest37-21ksoftirqd/308:30:003
222315699394,21cyclictest2227748-21kworker/u8:208:40:031
222315699394,21cyclictest2227748-21kworker/u8:208:40:021
222315699391,14cyclictest2291061-21ssh11:19:541
222315699391,14cyclictest2291061-21ssh11:19:541
222315499396,8cyclictest12-21ksoftirqd/007:11:500
2223154993924,11cyclictest1-21systemd07:45:000
2223154993923,12cyclictest12-21ksoftirqd/008:00:120
2223154993923,12cyclictest12-21ksoftirqd/008:00:110
2223162993828,7cyclictest37-21ksoftirqd/310:41:093
2223162993826,9cyclictest37-21ksoftirqd/309:56:463
2223162993824,10cyclictest37-21ksoftirqd/308:05:013
2223162993824,10cyclictest37-21ksoftirqd/308:05:013
2223156993828,7cyclictest23-21ksoftirqd/109:54:261
2223156993824,11cyclictest2225812-21sed08:00:021
2223156993821,9cyclictest3467602-21systemd-journal07:54:591
222315499387,8cyclictest12-21ksoftirqd/007:48:560
2223162993728,5cyclictest309-21in:imuxsock08:50:003
2223162993727,8cyclictest37-21ksoftirqd/312:26:503
2223162993722,11cyclictest2224648-21run-parts07:40:003
222315699379,6cyclictest131rcu_preempt10:24:261
2223156993730,4cyclictest2226999-21latency_hist08:25:001
2223156993730,4cyclictest2225096-21latency_hist07:45:021
2223156993728,7cyclictest23-21ksoftirqd/111:27:011
222315699372,7cyclictest23-21ksoftirqd/112:20:021
2223156993722,10cyclictest275-21dbus-daemon08:35:011
2223156993719,13cyclictest3467602-21systemd-journal07:40:001
2223162993622,11cyclictest37-21ksoftirqd/307:33:243
2223162993610,5cyclictest131rcu_preempt10:48:493
222315699367,6cyclictest131rcu_preempt07:50:011
222315699363,3cyclictest131rcu_preempt11:56:361
2223156993625,8cyclictest23-21ksoftirqd/110:03:531
2223156993622,10cyclictest309-21in:imuxsock08:30:011
2223162993528,5cyclictest37-21ksoftirqd/307:17:003
222315699353,5cyclictest131rcu_preempt09:12:521
2223162993421,10cyclictest37-21ksoftirqd/308:35:023
2223156993428,4cyclictest2223609-21latency_hist07:20:021
2223156993425,6cyclictest23-21ksoftirqd/108:08:261
2223156993326,4cyclictest2229348-21sh09:10:001
2223156993322,7cyclictest309-21in:imuxsock07:35:011
2223162993119,10cyclictest37-21ksoftirqd/307:54:593
222316299302,3cyclictest131rcu_preempt08:45:023
2223160993023,4cyclictest2223085-21latency_hist07:10:022
2223156992616,7cyclictest2223169-21cut07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional