You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-01 - 09:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Aug 01, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
80826799696,13cyclictest12-21ksoftirqd/022:50:010
808267996946,8cyclictest12-21ksoftirqd/022:10:090
808276996838,22cyclictest3467602-21systemd-journal20:35:002
808276996810,53cyclictest1-21systemd19:55:012
80826799683,29cyclictest131rcu_preempt23:21:500
80826799683,29cyclictest131rcu_preempt23:21:500
808276996719,42cyclictest275-21dbus-daemon20:55:012
808267996727,22cyclictest894776-21ssh00:00:000
808276996627,35cyclictest867704-21sh23:02:262
808276996626,35cyclictest836813-21sh21:57:112
808276996516,44cyclictest3467602-21systemd-journal22:40:022
808276996514,46cyclictest885372-21systemd-run23:40:002
80827699651,15cyclictest853639-21sshd22:32:432
80827699644,54cyclictest1-21systemd23:20:022
80827699642,14cyclictest810760-21sshd19:56:242
808276996413,43cyclictest0-21swapper/223:50:002
80826799645,7cyclictest12-21ksoftirqd/021:27:180
80826799642,9cyclictest121ksoftirqd/000:00:500
808267996312,24cyclictest131rcu_preempt23:27:540
808276996241,16cyclictest1-21systemd21:25:012
808276996222,36cyclictest871119-21latency_hist23:10:022
808276996218,39cyclictest913811-21latency_hist00:40:022
808276996218,39cyclictest913811-21latency_hist00:40:012
80827699621,13cyclictest364-21snmpd21:05:392
80826799628,7cyclictest131rcu_preempt19:40:010
80826799624,51cyclictest3467602-21systemd-journal23:20:000
80827699612,19cyclictest824190-21ssh21:30:232
80827699612,13cyclictest809798-21munin-run19:40:012
808276996119,37cyclictest842192-21sh22:08:472
80827699611,16cyclictest811325-21munin-run20:10:012
808267996125,5cyclictest0-21swapper/022:01:330
808267996118,10cyclictest12-21ksoftirqd/000:20:000
808267996118,10cyclictest12-21ksoftirqd/000:19:590
808276996017,9cyclictest847787-21sh22:20:062
80827699601,20cyclictest864340-21cat22:55:032
80827699601,20cyclictest864340-21cat22:55:022
80827699601,15cyclictest910480-21sshd00:32:562
80827699601,14cyclictest828511-21munin-run21:40:002
808276996011,4cyclictest0-21swapper/221:51:352
80827699601,13cyclictest883030-21true23:35:012
80827699601,13cyclictest883030-21true23:35:012
80826799599,6cyclictest131rcu_preempt21:49:460
808276995845,8cyclictest800493-21kworker/2:119:20:002
80827699583,10cyclictest897186-21cron00:05:002
80827699582,17cyclictest829085-21ssh21:40:442
80827699582,15cyclictest831546-21ssh21:45:522
80827699582,14cyclictest903631-21ssh00:18:402
80827699582,14cyclictest903631-21ssh00:18:392
80827699582,14cyclictest894774-21cron00:00:002
80827699582,14cyclictest859119-21ssh22:44:222
80827699582,14cyclictest859119-21ssh22:44:222
80827699581,15cyclictest907033-21ssh00:25:252
80827699581,15cyclictest907033-21ssh00:25:242
80827699574,9cyclictest880666-21unin-run23:30:012
80827699572,4cyclictest131rcu_preempt21:11:012
808276995723,28cyclictest906614-21latency_hist00:25:022
808276995720,32cyclictest813347-21latency_hist20:45:032
808272995716,18cyclictest23-21ksoftirqd/123:30:181
808272995716,18cyclictest23-21ksoftirqd/123:30:181
808267995727,13cyclictest907122-21ssh00:25:350
808267995727,13cyclictest907122-21ssh00:25:350
808267995724,9cyclictest12-21ksoftirqd/021:59:020
808267995711,23cyclictest0-21swapper/000:08:090
80827699568,9cyclictest890834-21sh23:51:202
808276995646,6cyclictest850208-21kworker/2:222:46:482
80827699562,16cyclictest811640-21latency_hist20:15:022
80827699562,14cyclictest843568-21ssh22:11:202
808276995621,31cyclictest30-21ksoftirqd/222:01:022
80827699561,16cyclictest120-21kswapd023:25:012
80827699561,16cyclictest120-21kswapd023:25:002
80826799563,5cyclictest131rcu_preempt22:26:110
808267995629,23cyclictest12-21ksoftirqd/022:42:460
808267995629,23cyclictest12-21ksoftirqd/022:42:450
80826799561,7cyclictest121ksoftirqd/000:23:320
808267995615,5cyclictest131rcu_preempt22:57:150
808267995610,9cyclictest131rcu_preempt21:32:440
808280995540,11cyclictest886538-21sshd23:42:133
80827699554,7cyclictest847498-21modprobe22:20:002
80827699553,15cyclictest899904-21ssh00:10:272
80827699552,17cyclictest822765-21ssh21:27:312
80827699552,16cyclictest814110-21wc21:00:022
80827699551,17cyclictest851766-21ssh22:28:482
80826799558,12cyclictest12-21ksoftirqd/022:55:010
80826799558,12cyclictest12-21ksoftirqd/022:55:000
808267995545,7cyclictest12-21ksoftirqd/023:32:400
808267995545,7cyclictest12-21ksoftirqd/023:32:400
808267995528,12cyclictest900365-21ssh00:11:260
808267995527,25cyclictest12-21ksoftirqd/022:15:340
808267995525,14cyclictest849848-21ssh22:24:580
808267995523,13cyclictest835880-21ssh21:55:020
808267995520,30cyclictest814439-21diskmemload23:45:030
808267995518,19cyclictest857026-21ssh22:40:020
80828099546,45cyclictest814439-21diskmemload00:15:033
80828099546,45cyclictest814439-21diskmemload00:15:023
80827699548,8cyclictest812089-21sshd20:21:432
80827699544,8cyclictest1-21systemd20:30:012
80827699542,16cyclictest897442-21ssh00:05:052
80827699542,15cyclictest871476-21ssh23:10:182
808276995419,31cyclictest865211-21sh22:57:102
80827699541,16cyclictest818263-21ssh21:18:042
80826799546,8cyclictest12-21ksoftirqd/023:40:000
80826799543,12cyclictest12-21ksoftirqd/023:54:320
808267995428,11cyclictest886911-21ssh23:43:010
808267995414,32cyclictest16850irq/134-800000021:19:390
808272995335,13cyclictest275-21dbus-daemon23:59:591
80826799534,10cyclictest812880-21munin-run20:39:590
808267995315,11cyclictest12-21ksoftirqd/022:34:260
80826799531,12cyclictest811887-21grep20:20:020
80828099523,32cyclictest120-21kswapd000:10:013
808276995222,25cyclictest309-21in:imuxsock19:25:012
808276995222,25cyclictest309-21in:imuxsock19:25:002
808276995213,3cyclictest810028-21kworker/u8:419:49:222
808272995223,13cyclictest23-21ksoftirqd/119:40:021
80827299521,5cyclictest131rcu_preempt21:35:001
80826799527,11cyclictest12-21ksoftirqd/020:05:030
808267995233,9cyclictest873481-21cron23:15:000
80826799522,8cyclictest12-21ksoftirqd/021:38:150
808267995225,7cyclictest12-21ksoftirqd/023:06:560
808267995211,5cyclictest131rcu_preempt21:14:380
80827699512,12cyclictest885646-21ssh23:40:112
808276995117,13cyclictest1-21systemd20:20:022
808276995114,11cyclictest814177-21modprobe21:05:012
808276995114,11cyclictest814177-21modprobe21:05:002
808267995128,8cyclictest867400-21ssh23:01:460
808267995124,12cyclictest841272-21ssh22:06:340
808280995016,17cyclictest828637-21rm21:40:023
80827699502,14cyclictest811079-21true20:05:012
808276995012,14cyclictest275-21dbus-daemon19:30:002
80827699501,16cyclictest812893-21tr20:39:592
808272995039,6cyclictest814439-21diskmemload00:30:021
808272995039,6cyclictest814439-21diskmemload00:30:021
80827299501,22cyclictest0-21swapper/119:25:011
80827299501,22cyclictest0-21swapper/119:25:011
80826799503,40cyclictest1-21systemd21:10:000
80826799502,14cyclictest911397-21unin-run00:35:010
808267995015,12cyclictest12-21ksoftirqd/020:55:020
80826799501,33cyclictest912021-21ssh00:36:060
80826799501,33cyclictest912021-21ssh00:36:060
808280994933,12cyclictest3467602-21systemd-journal23:55:013
808280994923,21cyclictest37-21ksoftirqd/322:27:223
808276994918,27cyclictest808635-21latency_hist19:15:022
808272994932,13cyclictest3467602-21systemd-journal19:55:011
80827299492,5cyclictest131rcu_preempt19:20:011
808272994922,19cyclictest23-21ksoftirqd/121:23:451
80827299491,5cyclictest131rcu_preempt23:17:111
808272994913,5cyclictest131rcu_preempt21:20:031
80827299491,21cyclictest0-21swapper/122:48:431
80826799498,10cyclictest12-21ksoftirqd/019:55:010
80826799494,6cyclictest12-21ksoftirqd/021:05:020
80826799494,6cyclictest12-21ksoftirqd/021:05:010
80826799492,13cyclictest12-21ksoftirqd/020:00:000
808267994916,29cyclictest12-21ksoftirqd/021:42:130
808267994911,5cyclictest131rcu_preempt20:29:440
808276994810,12cyclictest30-21ksoftirqd/219:35:022
80827299487,23cyclictest221rcuc/100:25:011
808272994827,15cyclictest275-21dbus-daemon21:30:011
808272994820,19cyclictest23-21ksoftirqd/121:36:421
80827299481,19cyclictest0-21swapper/122:13:141
80826799487,6cyclictest131rcu_preempt21:22:430
808267994831,13cyclictest3467602-21systemd-journal21:00:000
808280994726,15cyclictest894764-40logrotate00:00:003
808280994722,22cyclictest37-21ksoftirqd/322:11:143
808280994721,22cyclictest37-21ksoftirqd/322:35:433
808272994730,4cyclictest23-21ksoftirqd/100:16:411
808272994730,4cyclictest23-21ksoftirqd/100:16:411
808272994728,14cyclictest275-21dbus-daemon21:15:001
808272994720,19cyclictest23-21ksoftirqd/121:40:131
808272994719,19cyclictest23-21ksoftirqd/100:00:171
80827299471,20cyclictest0-21swapper/122:36:331
80827299471,19cyclictest0-21swapper/123:21:401
80827299471,19cyclictest0-21swapper/123:21:391
80827299471,18cyclictest0-21swapper/123:40:441
808272994711,6cyclictest131rcu_preempt21:52:071
80826799471,5cyclictest0-21swapper/019:25:030
80826799471,5cyclictest0-21swapper/019:25:020
808280994631,11cyclictest275-21dbus-daemon21:15:003
808280994621,21cyclictest37-21ksoftirqd/321:48:043
80827699461,13cyclictest808409-21latency_hist19:10:032
808272994634,8cyclictest1-21systemd23:40:011
808272994619,18cyclictest23-21ksoftirqd/123:26:151
80827299461,18cyclictest0-21swapper/121:48:051
80826799467,9cyclictest12-21ksoftirqd/019:20:000
808267994610,5cyclictest131rcu_preempt20:50:000
808267994610,5cyclictest131rcu_preempt20:50:000
808280994532,9cyclictest37-21ksoftirqd/323:26:353
808280994519,22cyclictest37-21ksoftirqd/322:48:523
80827299456,4cyclictest131rcu_preempt23:51:521
808272994535,6cyclictest23-21ksoftirqd/123:15:011
808272994528,13cyclictest1-21systemd20:35:021
808272994519,19cyclictest23-21ksoftirqd/123:08:421
80827299451,19cyclictest0-21swapper/100:36:101
80827299451,19cyclictest0-21swapper/100:36:101
80827299451,19cyclictest0-21swapper/100:06:101
80827299451,18cyclictest0-21swapper/122:34:331
80826799458,5cyclictest131rcu_preempt20:15:000
808267994538,4cyclictest811321-21latency_hist20:10:020
808267994534,7cyclictest309-21in:imuxsock19:35:010
808267994529,13cyclictest812100-21sshd20:24:430
808280994434,5cyclictest824189-21sh21:30:243
808280994430,11cyclictest37-21ksoftirqd/321:56:413
808280994427,8cyclictest37-21ksoftirqd/300:38:443
808280994427,8cyclictest37-21ksoftirqd/300:38:443
808280994427,13cyclictest809779-21tr19:40:013
808280994425,9cyclictest37-21ksoftirqd/322:21:443
808280994421,19cyclictest37-21ksoftirqd/323:31:083
808280994421,19cyclictest37-21ksoftirqd/323:31:083
808280994420,20cyclictest37-21ksoftirqd/323:49:483
808280994419,21cyclictest37-21ksoftirqd/322:00:253
808276994414,6cyclictest131rcu_preempt20:50:022
808276994414,6cyclictest131rcu_preempt20:50:022
808272994428,12cyclictest275-21dbus-daemon20:55:491
808272994428,11cyclictest275-21dbus-daemon20:20:011
808272994415,18cyclictest131rcu_preempt00:35:001
80827299441,16cyclictest0-21swapper/122:51:531
80827299441,16cyclictest0-21swapper/122:51:531
80827299441,16cyclictest0-21swapper/122:09:411
80827299441,15cyclictest0-21swapper/122:00:271
808272994410,23cyclictest0-21swapper/119:55:021
80828099436,33cyclictest814439-21diskmemload00:25:003
808280994334,6cyclictest878286-21cron23:25:003
808280994334,6cyclictest878286-21cron23:24:593
808280994329,9cyclictest1-21systemd20:45:003
808280994325,14cyclictest275-21dbus-daemon20:20:013
808280994319,20cyclictest37-21ksoftirqd/321:52:153
80827299438,10cyclictest23-21ksoftirqd/122:26:511
80827299431,17cyclictest0-21swapper/100:12:181
80827299431,15cyclictest0-21swapper/122:41:041
80827299431,15cyclictest0-21swapper/122:41:041
80826799433,36cyclictest0-21swapper/019:45:010
80828099425,4cyclictest131rcu_preempt22:30:583
808280994229,10cyclictest1-21systemd00:35:003
808280994217,21cyclictest37-21ksoftirqd/323:15:343
80827299428,4cyclictest131rcu_preempt23:01:031
80827299423,7cyclictest23-21ksoftirqd/119:15:011
808272994230,9cyclictest849932-21sed22:25:011
808267994228,10cyclictest810277-21cron19:50:010
80828099419,23cyclictest37-21ksoftirqd/322:43:513
80828099419,23cyclictest37-21ksoftirqd/322:43:503
80828099413,3cyclictest131rcu_preempt00:27:233
80828099413,3cyclictest131rcu_preempt00:27:223
808280994123,3cyclictest37-21ksoftirqd/322:20:013
808280994117,20cyclictest37-21ksoftirqd/319:29:593
80827299418,17cyclictest23-21ksoftirqd/120:29:431
808272994129,8cyclictest23-21ksoftirqd/122:20:011
808272994127,10cyclictest309-21in:imuxsock19:50:001
808272994126,11cyclictest23-21ksoftirqd/120:40:021
808272994123,3cyclictest23-21ksoftirqd/122:59:181
808272994116,15cyclictest23-21ksoftirqd/120:05:021
808272994110,6cyclictest131rcu_preempt21:55:181
80826799417,22cyclictest808274-21latency_hist19:10:010
80828099401,6cyclictest131rcu_preempt23:06:163
808272994031,6cyclictest23-21ksoftirqd/120:15:021
80828099394,5cyclictest131rcu_preempt22:51:383
80828099394,5cyclictest131rcu_preempt22:51:373
808280993927,8cyclictest37-21ksoftirqd/320:54:593
808280993926,10cyclictest37-21ksoftirqd/321:19:533
80828099392,5cyclictest131rcu_preempt21:20:283
808272993911,16cyclictest23-21ksoftirqd/119:35:011
80826799391,35cyclictest0-21swapper/020:35:010
808280993830,5cyclictest0-21swapper/323:15:023
808280993826,9cyclictest830932-21pluginstate21:45:013
80828099382,5cyclictest131rcu_preempt22:08:043
808280993825,10cyclictest37-21ksoftirqd/323:00:013
808280993825,10cyclictest37-21ksoftirqd/300:02:283
80828099382,4cyclictest131rcu_preempt21:26:413
808280993810,20cyclictest37-21ksoftirqd/323:02:563
808272993828,7cyclictest23-21ksoftirqd/119:45:001
808272993827,8cyclictest3467602-21systemd-journal20:44:591
808272993811,5cyclictest131rcu_preempt20:55:001
808272993810,6cyclictest131rcu_preempt23:45:241
80826799383,31cyclictest0-21swapper/020:42:250
80828099372,5cyclictest131rcu_preempt23:39:163
808280993718,7cyclictest131rcu_preempt00:10:193
80827299379,6cyclictest131rcu_preempt21:06:361
80827299378,6cyclictest131rcu_preempt20:21:361
80827299372,6cyclictest131rcu_preempt19:30:001
808272993726,8cyclictest23-21ksoftirqd/120:47:351
808272993726,8cyclictest23-21ksoftirqd/120:47:351
808280993623,10cyclictest813052-21sed20:40:013
808280993622,10cyclictest275-21dbus-daemon19:45:003
808272993626,5cyclictest309-21in:imuxsock21:05:001
808272993626,5cyclictest309-21in:imuxsock21:05:001
808267993625,8cyclictest12-21ksoftirqd/019:15:030
808267993624,9cyclictest12-21ksoftirqd/019:30:000
808280993525,7cyclictest37-21ksoftirqd/319:25:023
808280993525,7cyclictest37-21ksoftirqd/319:25:013
808280993520,11cyclictest1-21systemd20:05:023
808272993521,9cyclictest309-21in:imuxsock20:10:021
80828099345,5cyclictest131rcu_preempt20:10:023
808280993424,7cyclictest37-21ksoftirqd/320:25:013
808280993423,8cyclictest810766-21sshd19:57:233
808280993419,11cyclictest37-21ksoftirqd/320:30:003
808280993418,11cyclictest275-21dbus-daemon19:20:003
808280993326,4cyclictest813396-21latency_hist20:50:023
808280993326,4cyclictest813396-21latency_hist20:50:013
808280993324,6cyclictest810583-21latency_hist19:55:023
808280993315,14cyclictest275-21dbus-daemon19:50:013
808272993324,6cyclictest808205-21latency_hist19:10:021
808280993223,5cyclictest37-21ksoftirqd/321:06:443
808280993221,8cyclictest812810-21latency_hist20:35:033
808280993219,9cyclictest309-21in:imuxsock20:10:023
808280993215,3cyclictest37-21ksoftirqd/321:05:023
808280993215,3cyclictest37-21ksoftirqd/321:05:013
808276993222,7cyclictest30-21ksoftirqd/219:45:022
808280993021,7cyclictest37-21ksoftirqd/319:35:013
808280992919,6cyclictest309-21in:imuxsock19:15:013
808280992820,6cyclictest37-21ksoftirqd/319:10:013
80828099281,19cyclictest0-21swapper/320:56:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional