You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-06 - 18:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Aug 06, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
250535099841,3cyclictest0-21swapper/009:00:020
250535099841,3cyclictest0-21swapper/009:00:020
2505350998374,3cyclictest150-21pr/ttyS210:30:020
250535099801,60cyclictest0-21swapper/011:10:020
2505350996827,10cyclictest12-21ksoftirqd/009:15:030
2505350996530,31cyclictest1-21systemd08:21:480
250535099652,58cyclictest2519530-21sendmail11:40:000
2505350996427,33cyclictest2512246-21latency_hist09:20:020
250535099642,28cyclictest0-21swapper/010:39:350
250536299638,38cyclictest2510745-21kworker/u8:309:00:013
2505350996320,5cyclictest2522741-21latency_hist12:40:020
2505350996233,25cyclictest16950irq/224-800000010:59:590
2505350996157,2cyclictest0-21swapper/008:55:030
2505350996123,8cyclictest3467602-21systemd-journal08:25:030
2505350996123,8cyclictest3467602-21systemd-journal08:25:030
2505350996116,12cyclictest1-21systemd10:00:010
250535799593,33cyclictest2507335-21kworker/u8:008:05:022
2505350995925,9cyclictest12-21ksoftirqd/007:45:010
2505350995916,7cyclictest2513837-21grep09:50:010
2505350995916,10cyclictest2512994-21sshd09:30:390
2505350995912,14cyclictest275-21dbus-daemon08:35:000
250535099582,27cyclictest2506075-21latency_hist07:25:020
2505357995726,24cyclictest30-21ksoftirqd/212:10:022
2505357995713,5cyclictest131rcu_preempt10:10:002
2505350995624,7cyclictest2520814-21sshd12:04:480
2505357995536,13cyclictest3467602-21systemd-journal08:35:002
2505350995522,5cyclictest1-21systemd08:40:010
2505362995438,12cyclictest275-21dbus-daemon10:20:013
2505350995424,3cyclictest0-21swapper/007:35:020
2505350995424,3cyclictest0-21swapper/007:35:010
250535099541,16cyclictest2507366-21munin-plugin-st07:50:010
250535099541,16cyclictest2507366-21munin-plugin-st07:50:000
2505357995321,11cyclictest30-21ksoftirqd/209:10:022
2505357995312,5cyclictest131rcu_preempt10:15:012
2505354995316,24cyclictest2507335-21kworker/u8:008:00:021
250535099536,41cyclictest0-21swapper/007:15:020
250535099536,41cyclictest0-21swapper/007:15:010
2505350995315,10cyclictest2507729-21cat07:55:020
2505357995243,5cyclictest2516437-21kworker/2:211:42:232
2505357995242,7cyclictest2507122-21kworker/2:108:56:102
2505357995232,14cyclictest3467602-21systemd-journal11:55:002
250535799523,12cyclictest30-21ksoftirqd/210:50:002
250535799521,10cyclictest2521345-21unin-run12:15:012
250535099523,19cyclictest2515349-21modprobe10:20:010
2505350995220,7cyclictest1-21systemd10:33:490
250535099521,5cyclictest2518364-21latency_hist11:15:020
2505362995118,9cyclictest309-21in:imuxsock07:35:013
2505362995118,9cyclictest309-21in:imuxsock07:35:013
2505357995142,6cyclictest2496452-21kworker/2:107:13:382
2505357995142,6cyclictest2496452-21kworker/2:107:13:382
2505350995121,4cyclictest2513624-21latency_hist09:45:020
2505350995121,10cyclictest12-21ksoftirqd/008:00:020
2505350995119,10cyclictest12-21ksoftirqd/009:40:020
2505362995010,14cyclictest0-21swapper/307:55:023
2505357995042,5cyclictest2516437-21kworker/2:211:37:242
2505357995041,6cyclictest2506590-21kworker/2:008:25:412
2505357995041,6cyclictest2506590-21kworker/2:008:25:402
2505357995041,5cyclictest2513827-21kworker/2:010:33:482
2505357995040,6cyclictest2507122-21kworker/2:108:51:382
2505357995039,7cyclictest2516437-21kworker/2:211:08:482
2505357995035,10cyclictest1-21systemd10:40:012
250535799501,13cyclictest30-21ksoftirqd/207:10:032
2505357995010,5cyclictest131rcu_preempt12:00:032
250535099502,23cyclictest1715541-1kworker/0:0H08:10:490
2505350995020,8cyclictest12-21ksoftirqd/011:35:030
250536299496,25cyclictest2505102-21kworker/u8:107:20:013
250535799494,9cyclictest30-21ksoftirqd/208:50:012
2505357994941,5cyclictest2516437-21kworker/2:211:31:082
2505357994941,5cyclictest2516437-21kworker/2:211:12:492
2505357994941,5cyclictest2513827-21kworker/2:010:29:492
2505357994941,5cyclictest2513045-21kworker/2:110:01:322
2505357994941,5cyclictest2513045-21kworker/2:110:01:322
2505357994941,5cyclictest2511616-21kworker/2:309:40:262
2505357994941,5cyclictest2510995-21kworker/2:209:27:122
2505357994941,5cyclictest2510995-21kworker/2:209:11:452
2505357994941,5cyclictest2507122-21kworker/2:108:41:182
2505357994941,5cyclictest2506590-21kworker/2:007:54:552
2505357994941,5cyclictest2505553-21kworker/2:207:22:302
2505357994941,5cyclictest2495397-21kworker/2:307:41:152
2505357994940,6cyclictest2507122-21kworker/2:108:19:392
2505357994939,6cyclictest2506590-21kworker/2:008:08:182
2505357994913,13cyclictest30-21ksoftirqd/208:00:012
250535499491,16cyclictest2511584-21true09:10:011
250535099498,18cyclictest16850irq/134-800000009:09:470
2505362994832,12cyclictest275-21dbus-daemon09:35:013
2505357994840,5cyclictest2520589-21kworker/2:112:30:282
2505357994840,5cyclictest2516960-21kworker/2:312:17:532
2505357994840,5cyclictest2516960-21kworker/2:312:00:302
2505357994840,5cyclictest2516437-21kworker/2:211:27:422
2505357994840,5cyclictest2513045-21kworker/2:109:51:482
2505357994840,5cyclictest2513045-21kworker/2:109:46:532
2505357994840,5cyclictest2511616-21kworker/2:309:39:412
2505357994840,5cyclictest2507122-21kworker/2:108:39:362
2505357994840,5cyclictest2506590-21kworker/2:008:10:152
2505357994840,5cyclictest2505553-21kworker/2:207:30:122
2505357994840,5cyclictest2505553-21kworker/2:207:30:112
2505357994839,6cyclictest2516437-21kworker/2:211:03:502
2505357994839,6cyclictest2516437-21kworker/2:211:03:492
2505357994838,6cyclictest2507122-21kworker/2:109:01:212
2505357994838,6cyclictest2507122-21kworker/2:109:01:212
2505350994840,5cyclictest16950irq/224-800000011:20:010
250535099482,7cyclictest2521602-21munin-run12:20:000
250535099481,4cyclictest0-21swapper/012:26:310
2505362994729,12cyclictest275-21dbus-daemon08:21:533
2505357994739,5cyclictest2520589-21kworker/2:112:39:092
2505357994739,5cyclictest2520589-21kworker/2:112:25:122
2505357994739,5cyclictest2516960-21kworker/2:312:21:092
2505357994739,5cyclictest2516960-21kworker/2:312:21:092
2505357994739,5cyclictest2516960-21kworker/2:310:55:182
2505357994739,5cyclictest2516960-21kworker/2:310:51:472
2505357994739,5cyclictest2516960-21kworker/2:310:51:462
2505357994739,5cyclictest2516437-21kworker/2:211:45:162
2505357994739,5cyclictest2516437-21kworker/2:211:21:082
2505357994739,5cyclictest2516437-21kworker/2:211:15:472
2505357994739,5cyclictest2513827-21kworker/2:010:43:392
2505357994739,5cyclictest2513827-21kworker/2:010:22:112
2505357994739,5cyclictest2513827-21kworker/2:010:15:442
2505357994739,5cyclictest2513827-21kworker/2:009:55:452
2505357994739,5cyclictest2511616-21kworker/2:309:33:192
2505357994739,5cyclictest2511616-21kworker/2:309:15:542
2505357994739,5cyclictest2507122-21kworker/2:107:48:112
2505357994739,5cyclictest2507122-21kworker/2:107:48:112
2505357994739,5cyclictest2505553-21kworker/2:207:26:402
2505357994739,5cyclictest2495397-21kworker/2:307:35:592
2505357994739,5cyclictest2495397-21kworker/2:307:15:242
2505357994738,5cyclictest2507122-21kworker/2:108:20:212
2505354994730,12cyclictest275-21dbus-daemon09:00:011
2505350994728,15cyclictest275-21dbus-daemon07:20:000
2505350994726,4cyclictest0-21swapper/009:22:030
2505350994722,9cyclictest12-21ksoftirqd/012:15:010
2505350994722,5cyclictest12-21ksoftirqd/011:30:020
2505350994721,7cyclictest12-21ksoftirqd/010:50:020
250535099471,4cyclictest0-21swapper/012:24:330
250535099471,4cyclictest0-21swapper/012:24:330
250535099471,26cyclictest0-21swapper/012:05:280
2505357994638,5cyclictest2511616-21kworker/2:309:20:322
2505354994634,9cyclictest275-21dbus-daemon12:05:011
250535499461,33cyclictest2513794-21kworker/u8:210:40:021
2505350994622,4cyclictest2514847-21latency_hist10:10:010
250535099461,4cyclictest0-21swapper/012:30:440
250535099461,29cyclictest0-21swapper/007:39:410
2505350994611,4cyclictest2516660-1kworker/dying10:55:010
2505350994611,4cyclictest2516660-1kworker/dying10:55:010
250536299458,5cyclictest131rcu_preempt10:30:013
2505354994535,6cyclictest2517181-21munin-plugin-st10:55:011
2505354994535,6cyclictest2517181-21munin-plugin-st10:55:001
2505354994533,9cyclictest2515074-21sshd10:11:481
2505354994530,11cyclictest23-21ksoftirqd/110:00:001
2505350994517,8cyclictest12-21ksoftirqd/012:00:020
2505362994429,11cyclictest3467602-21systemd-journal12:36:493
2505362994425,15cyclictest0-21swapper/312:35:013
250535099443,8cyclictest1-21systemd10:25:000
2505350994436,5cyclictest12-21ksoftirqd/008:45:030
250535099442,8cyclictest2508922-21munin-run08:20:010
250535099441,4cyclictest2519777-21sshd11:44:550
2505362994330,9cyclictest3467602-21systemd-journal10:04:483
2505362994330,9cyclictest3467602-21systemd-journal10:04:473
2505362994329,11cyclictest37-21ksoftirqd/308:05:033
2505362994328,7cyclictest309-21in:imuxsock09:05:003
2505362994328,7cyclictest309-21in:imuxsock09:04:593
2505354994334,6cyclictest23-21ksoftirqd/107:19:591
2505354994329,10cyclictest275-21dbus-daemon11:40:011
2505350994336,4cyclictest16950irq/224-800000011:55:000
2505350994328,12cyclictest12-21ksoftirqd/010:40:150
250535099431,28cyclictest0-21swapper/010:14:080
250535099431,28cyclictest0-21swapper/008:50:030
2505354994226,12cyclictest275-21dbus-daemon08:55:011
250535499421,33cyclictest2516899-21kworker/u8:211:05:021
250535499421,33cyclictest2516899-21kworker/u8:211:05:011
2505350994231,8cyclictest12-21ksoftirqd/008:10:010
250536299419,6cyclictest131rcu_preempt09:10:003
2505362994126,12cyclictest2512572-21sed09:25:013
2505362994126,11cyclictest37-21ksoftirqd/307:39:593
2505362994125,11cyclictest37-21ksoftirqd/310:45:023
2505362994123,12cyclictest3467602-21systemd-journal10:40:003
2505362994112,6cyclictest131rcu_preempt11:15:023
2505362994111,6cyclictest131rcu_preempt07:15:013
2505362994111,6cyclictest131rcu_preempt07:15:013
2505354994132,5cyclictest2516697-21latency_hist10:45:021
2505354994129,9cyclictest2522104-21sshd12:27:481
2505354994127,11cyclictest2515575-21sed10:20:031
2505354994125,12cyclictest23-21ksoftirqd/112:00:001
250535499411,3cyclictest131rcu_preempt08:35:011
2505350994127,11cyclictest2508114-21sshd08:00:160
2505362994031,7cyclictest37-21ksoftirqd/310:55:023
2505362994031,7cyclictest37-21ksoftirqd/310:55:023
2505362994028,8cyclictest2514309-21cron10:00:003
2505362994027,9cyclictest3467602-21systemd-journal11:30:483
2505362994027,10cyclictest1-21systemd09:20:013
2505362994026,7cyclictest37-21ksoftirqd/312:05:003
2505362994025,11cyclictest3467602-21systemd-journal10:30:463
2505362994025,10cyclictest3467602-21systemd-journal08:43:493
2505362994024,10cyclictest1-21systemd11:00:003
250535499409,5cyclictest131rcu_preempt12:25:021
250535499409,5cyclictest131rcu_preempt12:25:021
2505354994030,6cyclictest2510526-21munin-plugin-st08:50:001
2505354994028,9cyclictest1-21systemd11:35:011
2505354994027,8cyclictest275-21dbus-daemon10:05:011
2505354994027,8cyclictest275-21dbus-daemon10:05:011
2505354994027,10cyclictest23-21ksoftirqd/111:20:011
250535499401,3cyclictest131rcu_preempt08:45:031
250535099401,27cyclictest0-21swapper/011:45:050
2505362993928,9cyclictest37-21ksoftirqd/310:25:013
2505362993928,8cyclictest37-21ksoftirqd/311:40:013
2505362993927,10cyclictest37-21ksoftirqd/308:00:013
2505362993924,9cyclictest2508901-21cron08:20:013
250536299391,5cyclictest131rcu_preempt12:00:013
250536299391,5cyclictest131rcu_preempt10:10:013
250535499396,5cyclictest131rcu_preempt12:09:481
2505354993929,7cyclictest23-21ksoftirqd/109:25:031
2505354993926,9cyclictest1-21systemd09:40:011
2505354993923,10cyclictest275-21dbus-daemon11:30:011
2505354993922,13cyclictest3467602-21systemd-journal11:45:001
2505354993921,13cyclictest275-21dbus-daemon07:35:011
2505354993921,13cyclictest275-21dbus-daemon07:35:011
2505350993929,8cyclictest12-21ksoftirqd/010:05:030
2505350993929,8cyclictest12-21ksoftirqd/010:05:030
2505350993926,10cyclictest2518720-21sshd11:20:480
250535099391,34cyclictest2517692-21sshd11:03:480
250535099391,34cyclictest2517692-21sshd11:03:480
250535099391,34cyclictest0-21swapper/007:30:010
250536299385,6cyclictest131rcu_preempt09:55:013
250536299384,7cyclictest131rcu_preempt09:40:003
2505362993824,9cyclictest275-21dbus-daemon09:30:013
2505362993820,13cyclictest2521066-21sshd12:09:473
2505362993819,13cyclictest275-21dbus-daemon11:05:013
2505362993819,13cyclictest275-21dbus-daemon11:05:013
250535499385,22cyclictest2510745-21kworker/u8:309:45:021
250535499383,8cyclictest23-21ksoftirqd/110:25:021
250535499383,7cyclictest23-21ksoftirqd/109:30:481
2505354993828,7cyclictest2508805-21sed08:15:021
2505354993825,7cyclictest2505540-21munin-run07:15:011
2505354993825,7cyclictest2505540-21munin-run07:15:011
250535099382,32cyclictest0-21swapper/008:45:020
2505362993725,9cyclictest37-21ksoftirqd/308:30:033
2505362993725,9cyclictest1-21systemd12:15:013
250536299371,3cyclictest131rcu_preempt11:45:003
250535499375,3cyclictest131rcu_preempt08:10:001
2505354993728,7cyclictest23-21ksoftirqd/108:30:021
2505354993728,7cyclictest23-21ksoftirqd/108:30:011
2505354993725,9cyclictest2507732-21sed07:55:021
2505354993724,10cyclictest2521605-21systemctl12:20:001
2505354993723,9cyclictest23-21ksoftirqd/109:54:591
2505354993723,10cyclictest275-21dbus-daemon11:55:011
2505354993722,11cyclictest23-21ksoftirqd/112:10:411
2505362993627,7cyclictest37-21ksoftirqd/309:40:203
250535499363,9cyclictest23-21ksoftirqd/108:40:001
2505354993629,4cyclictest3467562-21sshd09:00:481
2505354993629,4cyclictest3467562-21sshd09:00:481
2505354993627,6cyclictest23-21ksoftirqd/109:20:031
2505354993626,7cyclictest23-21ksoftirqd/111:00:001
2505354993626,6cyclictest2517964-21munin-plugin-st11:10:001
2505354993625,8cyclictest2517038-21sed10:50:021
2505354993625,8cyclictest2514819-21sshd10:05:481
2505354993624,9cyclictest1-21systemd11:15:001
2505354993622,10cyclictest2522630-21cron12:40:001
2505354993617,14cyclictest275-21dbus-daemon07:40:001
2505350993621,11cyclictest12-21ksoftirqd/009:25:100
250535099361,31cyclictest2505477-21latency_hist07:10:020
2505362993528,4cyclictest3467562-21sshd12:25:483
2505362993523,9cyclictest37-21ksoftirqd/308:50:003
2505362993522,10cyclictest275-21dbus-daemon08:15:013
2505362993520,11cyclictest2509468-21sshd08:27:483
2505362993520,11cyclictest2509468-21sshd08:27:473
2505354993524,9cyclictest23-21ksoftirqd/107:25:011
2505354993523,9cyclictest23-21ksoftirqd/107:30:021
250536299348,15cyclictest2506211-21sshd07:25:033
250536299343,3cyclictest131rcu_preempt07:45:033
2505362993424,7cyclictest37-21ksoftirqd/311:30:023
2505362993420,10cyclictest37-21ksoftirqd/311:55:003
2505354993427,4cyclictest2512775-21latency_hist09:30:011
2505354993426,4cyclictest2508324-21latency_hist08:05:021
2505354993422,8cyclictest275-21dbus-daemon08:21:491
2505354993421,9cyclictest23-21ksoftirqd/111:20:391
250535499341,7cyclictest23-21ksoftirqd/112:35:011
250535099341,30cyclictest0-21swapper/009:52:420
2505362993326,4cyclictest2513814-21munin-plugin-st09:50:013
2505362993325,5cyclictest2520137-21latency_hist11:50:023
2505362993324,5cyclictest37-21ksoftirqd/312:20:013
2505362993321,9cyclictest3467602-21systemd-journal11:07:483
2505362993320,9cyclictest3467602-21systemd-journal11:24:593
250536299331,3cyclictest131rcu_preempt10:48:483
250536299331,3cyclictest131rcu_preempt09:10:163
250535499338,5cyclictest131rcu_preempt10:30:011
250535499333,24cyclictest221rcuc/108:15:021
2505354993324,6cyclictest2513812-21latency_hist09:50:021
2505354993324,5cyclictest23-21ksoftirqd/110:35:001
250536299326,4cyclictest131rcu_preempt07:29:593
2505362993223,5cyclictest2518713-21latency_hist11:20:033
2505362993218,11cyclictest2509963-21sshd08:35:483
2505362993212,15cyclictest275-21dbus-daemon10:15:003
2505354993225,4cyclictest2505279-21latency_hist07:10:011
2505354993222,7cyclictest3467602-21systemd-journal09:15:001
250536299313,5cyclictest131rcu_preempt08:55:013
2505362993122,5cyclictest309-21in:imuxsock12:21:003
2505362993122,5cyclictest309-21in:imuxsock12:20:593
2505362993122,5cyclictest2508406-21true08:10:013
2505354993124,4cyclictest2507139-21latency_hist07:45:021
250536299305,3cyclictest131rcu_preempt07:10:013
2505362993022,6cyclictest37-21ksoftirqd/307:50:023
250535499303,4cyclictest131rcu_preempt07:46:501
250535499303,4cyclictest131rcu_preempt07:46:491
2505354993019,8cyclictest2520055-21systemd-run11:50:011
250536299293,4cyclictest131rcu_preempt07:45:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional