You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-27 - 23:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Jul 27, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3523987991033,4cyclictest131rcu_preempt11:50:022
352398299688,22cyclictest3611689-21ssh12:00:211
3523982996717,11cyclictest23-21ksoftirqd/109:10:031
3523982996717,10cyclictest23-21ksoftirqd/110:55:011
3523982996711,14cyclictest3624387-21ssh12:27:151
3523982996616,13cyclictest23-21ksoftirqd/109:51:201
3523982996616,11cyclictest23-21ksoftirqd/110:13:441
3523991996528,9cyclictest37-21ksoftirqd/309:23:253
3523991996528,9cyclictest37-21ksoftirqd/309:23:253
352398299652,18cyclictest364-21snmpd12:14:321
3523982996519,11cyclictest23-21ksoftirqd/110:17:211
3523982996517,10cyclictest23-21ksoftirqd/110:06:301
3523991996454,7cyclictest37-21ksoftirqd/311:55:033
3523982996416,9cyclictest23-21ksoftirqd/110:43:521
3523982996416,10cyclictest23-21ksoftirqd/109:20:181
3523982996416,10cyclictest23-21ksoftirqd/109:20:171
352397599641,5cyclictest131rcu_preempt09:50:000
3523982996317,9cyclictest23-21ksoftirqd/112:21:001
3523982996317,10cyclictest23-21ksoftirqd/110:03:211
3523982996316,10cyclictest23-21ksoftirqd/111:10:551
3523982996315,12cyclictest23-21ksoftirqd/107:55:011
3523982996314,15cyclictest3533136-21ssh09:15:531
3523982996314,11cyclictest23-21ksoftirqd/110:30:011
3523975996340,18cyclictest16850irq/134-800000009:30:000
3523975996313,39cyclictest12-21ksoftirqd/012:04:430
3523982996216,11cyclictest23-21ksoftirqd/110:48:351
3523982996215,11cyclictest23-21ksoftirqd/109:43:561
3523982996212,14cyclictest3628623-21ssh12:36:021
3523982996212,10cyclictest23-21ksoftirqd/111:54:091
3523982996120,7cyclictest23-21ksoftirqd/111:47:391
3523982996117,9cyclictest23-21ksoftirqd/109:50:011
3523982996115,11cyclictest23-21ksoftirqd/111:23:381
3523982996113,9cyclictest23-21ksoftirqd/112:18:251
3523982996112,14cyclictest3584463-21sh11:03:151
352397599615,5cyclictest131rcu_preempt11:48:130
3523975996141,6cyclictest12-21ksoftirqd/010:42:050
3523975996133,13cyclictest3557667-21ssh10:07:060
3523975996122,8cyclictest12-21ksoftirqd/009:10:000
3523987996037,5cyclictest3524985-21kworker/2:010:17:102
352398299604,20cyclictest3600401-21ssh11:36:341
352398299604,20cyclictest3581598-21ssh10:57:111
352398299604,16cyclictest3528705-21latency_hist08:40:011
3523982996014,13cyclictest3598885-21ssh11:33:341
3523982996014,10cyclictest23-21ksoftirqd/111:44:141
3523982996013,10cyclictest23-21ksoftirqd/109:33:311
3523975996031,13cyclictest3576218-21ssh10:45:500
3523975996026,28cyclictest0-21swapper/012:28:010
3523975996025,32cyclictest12-21ksoftirqd/012:37:500
3523991995911,15cyclictest37-21ksoftirqd/309:40:023
3523982995940,13cyclictest275-21dbus-daemon10:40:011
3523982995915,10cyclictest23-21ksoftirqd/109:55:571
3523982995913,10cyclictest23-21ksoftirqd/109:27:491
3523975995929,25cyclictest131rcu_preempt09:54:480
3523975995929,15cyclictest3543455-21ssh09:37:340
3523975995928,15cyclictest3586194-21ssh11:06:450
3523991995815,11cyclictest37-21ksoftirqd/310:54:573
3523987995850,5cyclictest3544583-21kworker/2:211:54:472
3523982995839,14cyclictest1-21systemd09:05:001
3523982995817,11cyclictest23-21ksoftirqd/112:34:561
3523982995817,11cyclictest23-21ksoftirqd/112:34:561
3523982995813,10cyclictest23-21ksoftirqd/110:25:001
3523982995812,9cyclictest23-21ksoftirqd/111:05:111
3523982995812,10cyclictest23-21ksoftirqd/109:37:361
3523975995820,34cyclictest12-21ksoftirqd/009:43:510
3523991995716,9cyclictest37-21ksoftirqd/309:54:483
3523991995711,6cyclictest131rcu_preempt08:05:023
3523982995714,10cyclictest23-21ksoftirqd/111:17:511
3523982995711,10cyclictest23-21ksoftirqd/111:27:171
352397599574,10cyclictest12-21ksoftirqd/010:15:020
352397599573,37cyclictest131rcu_preempt09:17:150
3523975995729,11cyclictest12-21ksoftirqd/010:20:010
3523975995726,21cyclictest3569013-21ssh10:30:450
3523975995723,5cyclictest3623281-21latency_hist12:25:020
3523975995717,36cyclictest12-21ksoftirqd/011:18:270
352399199566,6cyclictest131rcu_preempt09:49:593
352399199561,4cyclictest131rcu_preempt09:00:003
3523991995612,6cyclictest131rcu_preempt09:26:003
3523982995611,13cyclictest3532446-21ssh09:14:541
3523975995635,16cyclictest275-21dbus-daemon11:00:010
3523975995629,12cyclictest3554891-21ssh10:01:100
3523975995617,3cyclictest12-21ksoftirqd/009:34:200
3523991995517,6cyclictest131rcu_preempt10:21:533
3523991995512,13cyclictest0-21swapper/311:10:013
3523982995537,14cyclictest275-21dbus-daemon07:40:001
3523982995514,7cyclictest23-21ksoftirqd/111:55:061
3523975995527,13cyclictest3620747-21ssh12:19:500
3523975995527,13cyclictest3598998-21ssh11:33:580
3523975995522,29cyclictest12-21ksoftirqd/012:10:030
3523991995422,12cyclictest37-21ksoftirqd/309:19:553
3523975995427,11cyclictest3626085-21ssh12:30:340
3523975995427,11cyclictest3626085-21ssh12:30:340
3523975995426,13cyclictest3610267-21ssh11:57:340
3523975995426,12cyclictest3530835-21ssh09:11:050
352397599542,16cyclictest12-21ksoftirqd/009:55:520
3523975995415,34cyclictest12-21ksoftirqd/011:15:000
3523991995315,6cyclictest131rcu_preempt12:18:343
3523991995314,5cyclictest131rcu_preempt09:43:113
3523991995312,11cyclictest131rcu_preempt11:03:023
3523987995344,6cyclictest3544583-21kworker/2:209:49:302
3523987995344,5cyclictest3524985-21kworker/2:010:50:012
3523987995343,6cyclictest3544583-21kworker/2:212:34:002
3523987995343,6cyclictest3544583-21kworker/2:212:34:002
3523975995325,12cyclictest3583295-21ssh11:00:310
3523975995316,33cyclictest12-21ksoftirqd/010:53:260
3523991995213,9cyclictest131rcu_preempt12:14:223
3523987995244,5cyclictest3544583-21kworker/2:210:58:352
3523987995242,6cyclictest3544583-21kworker/2:212:25:222
3523987995242,6cyclictest3524985-21kworker/2:010:35:382
3523982995210,8cyclictest3570295-21ssh10:33:411
3523975995227,10cyclictest3617160-21ssh12:11:590
352399199517,6cyclictest131rcu_preempt07:35:013
352399199517,6cyclictest131rcu_preempt07:35:003
3523987995143,5cyclictest3544583-21kworker/2:211:55:502
3523987995143,5cyclictest3524985-21kworker/2:009:26:112
3523987995142,6cyclictest3524985-21kworker/2:009:15:022
3523987995141,5cyclictest3524985-21kworker/2:008:50:022
3523987995121,10cyclictest30-21ksoftirqd/211:06:362
3523982995111,7cyclictest23-21ksoftirqd/112:08:291
352397599519,37cyclictest0-21swapper/009:25:010
352397599519,37cyclictest0-21swapper/009:25:000
352397599513,4cyclictest131rcu_preempt11:21:270
352399199506,9cyclictest37-21ksoftirqd/312:30:003
352399199503,12cyclictest37-21ksoftirqd/310:12:563
352399199501,5cyclictest131rcu_preempt07:40:023
3523991995010,10cyclictest37-21ksoftirqd/308:45:013
3523987995043,4cyclictest3524985-21kworker/2:010:31:532
3523987995042,5cyclictest3524985-21kworker/2:010:44:062
3523987995042,5cyclictest3524985-21kworker/2:010:25:562
3523987995042,5cyclictest3524985-21kworker/2:010:05:022
3523987995041,6cyclictest3544583-21kworker/2:212:16:432
3523987995041,5cyclictest3524985-21kworker/2:010:24:152
3523987995039,7cyclictest3524985-21kworker/2:009:17:142
3523987995039,6cyclictest3609008-21kworker/2:112:05:472
3523987995034,12cyclictest1-21systemd10:55:002
3523982995027,16cyclictest275-21dbus-daemon08:00:011
3523982995027,16cyclictest275-21dbus-daemon08:00:011
3523982995011,8cyclictest23-21ksoftirqd/108:25:021
3523975995045,3cyclictest0-21swapper/008:35:010
3523975995018,24cyclictest3566289-21latency_hist10:25:020
3523975995014,32cyclictest12-21ksoftirqd/008:25:010
3523975995010,6cyclictest131rcu_preempt11:45:020
352399199498,7cyclictest37-21ksoftirqd/310:20:013
3523991994927,7cyclictest37-21ksoftirqd/312:33:023
3523991994927,7cyclictest37-21ksoftirqd/312:33:023
3523991994922,10cyclictest37-21ksoftirqd/312:01:213
3523991994914,5cyclictest131rcu_preempt10:06:373
352399199491,12cyclictest37-21ksoftirqd/311:48:183
3523987994944,3cyclictest3544583-21kworker/2:211:00:572
3523987994941,5cyclictest3544583-21kworker/2:212:12:102
3523987994941,5cyclictest3544583-21kworker/2:211:15:532
3523987994941,5cyclictest3524985-21kworker/2:011:35:482
3523987994940,6cyclictest3524985-21kworker/2:008:10:022
3523987994940,5cyclictest3544583-21kworker/2:211:13:362
3523987994940,5cyclictest3524985-21kworker/2:010:10:012
3523987994940,5cyclictest3524985-21kworker/2:009:45:022
3523975994921,13cyclictest3595015-21ssh11:25:080
3523975994920,19cyclictest0-21swapper/011:36:340
352399199486,4cyclictest131rcu_preempt11:34:553
3523991994838,6cyclictest37-21ksoftirqd/309:10:023
352399199482,7cyclictest3467602-21systemd-journal10:25:123
352399199482,5cyclictest131rcu_preempt10:41:223
3523991994821,7cyclictest361rcuc/310:55:383
3523991994817,6cyclictest131rcu_preempt11:30:013
3523991994815,5cyclictest131rcu_preempt11:20:113
3523991994812,5cyclictest131rcu_preempt08:50:013
3523991994810,6cyclictest131rcu_preempt12:23:213
3523987994841,4cyclictest3524985-21kworker/2:011:40:262
3523987994840,5cyclictest3544583-21kworker/2:209:55:442
3523987994840,5cyclictest3524985-21kworker/2:011:34:062
3523987994840,5cyclictest3524985-21kworker/2:011:24:142
3523987994840,5cyclictest3524985-21kworker/2:009:37:212
3523987994840,5cyclictest3524985-21kworker/2:009:32:322
3523987994839,6cyclictest3524985-21kworker/2:008:20:002
3523987994838,6cyclictest3544583-21kworker/2:209:55:022
352397599484,11cyclictest3608967-21munin-run11:55:010
3523975994811,28cyclictest12-21ksoftirqd/010:35:080
352399199479,20cyclictest131rcu_preempt10:47:283
3523991994733,8cyclictest37-21ksoftirqd/311:45:013
352399199472,9cyclictest37-21ksoftirqd/307:10:023
352399199472,9cyclictest37-21ksoftirqd/307:10:023
3523991994714,6cyclictest131rcu_preempt10:39:443
3523991994712,6cyclictest131rcu_preempt09:05:023
3523991994710,8cyclictest37-21ksoftirqd/311:35:523
3523991994710,6cyclictest131rcu_preempt10:34:583
3523987994740,4cyclictest3524985-21kworker/2:010:10:552
3523987994738,5cyclictest3544583-21kworker/2:212:35:422
3523987994738,5cyclictest3544583-21kworker/2:212:22:012
3523975994732,10cyclictest3527543-21cat08:19:590
352397599471,35cyclictest12-21ksoftirqd/010:26:090
3523991994613,8cyclictest131rcu_preempt11:55:423
3523991994613,5cyclictest131rcu_preempt11:13:403
3523991994612,6cyclictest131rcu_preempt08:10:023
3523987994638,5cyclictest3524985-21kworker/2:009:25:032
3523987994638,5cyclictest3524985-21kworker/2:009:25:032
3523982994627,13cyclictest275-21dbus-daemon08:20:011
352399199457,6cyclictest131rcu_preempt08:40:003
352399199454,6cyclictest37-21ksoftirqd/311:17:473
3523991994531,10cyclictest3614288-21sshd12:05:473
3523991994529,12cyclictest3467602-21systemd-journal08:00:013
3523991994529,12cyclictest3467602-21systemd-journal08:00:003
3523991994515,20cyclictest131rcu_preempt10:01:013
3523991994512,8cyclictest131rcu_preempt09:58:293
3523991994512,5cyclictest131rcu_preempt09:30:103
3523987994537,5cyclictest3544583-21kworker/2:212:00:152
3523987994524,14cyclictest275-21dbus-daemon07:50:012
3523987994524,14cyclictest275-21dbus-daemon07:50:002
3523982994531,8cyclictest309-21in:imuxsock08:30:011
352397599459,6cyclictest131rcu_preempt07:15:020
3523975994530,10cyclictest3528561-21cron08:40:010
352399199449,5cyclictest131rcu_preempt08:25:033
3523991994413,6cyclictest131rcu_preempt12:39:393
3523987994436,5cyclictest3524985-21kworker/2:011:27:462
3523987994436,5cyclictest3524985-21kworker/2:008:05:022
3523982994430,11cyclictest3467602-21systemd-journal08:03:311
3523975994428,12cyclictest3529197-21sed08:50:010
3523975994426,13cyclictest275-21dbus-daemon08:10:000
3523975994413,26cyclictest151rcuc/007:20:010
352399199437,5cyclictest131rcu_preempt08:35:023
3523991994312,5cyclictest131rcu_preempt09:13:423
3523991994311,6cyclictest131rcu_preempt07:20:023
3523991994310,8cyclictest131rcu_preempt07:42:473
3523987994335,5cyclictest3524985-21kworker/2:007:50:302
3523987994335,5cyclictest3498732-21kworker/2:207:25:022
352398299436,4cyclictest131rcu_preempt08:45:251
3523975994326,14cyclictest3529582-21sshd08:57:470
3523975994324,14cyclictest3467718-21lldpd08:45:010
3523975994323,14cyclictest275-21dbus-daemon07:40:000
3523975994312,22cyclictest3523968-21cyclictest07:10:020
3523975994312,22cyclictest3523968-21cyclictest07:10:010
3523991994228,9cyclictest309-21in:imuxsock08:55:013
3523991994228,9cyclictest309-21in:imuxsock08:55:013
352398799428,13cyclictest30-21ksoftirqd/208:45:012
3523987994234,5cyclictest3524985-21kworker/2:009:05:022
3523987994234,5cyclictest3524985-21kworker/2:007:40:002
3523987994234,5cyclictest3498732-21kworker/2:207:11:552
3523987994231,8cyclictest3467602-21systemd-journal07:15:022
3523982994230,9cyclictest23-21ksoftirqd/107:45:011
352397599427,31cyclictest3525792-21latency_hist07:45:020
352397599423,9cyclictest131rcu_preempt07:30:010
3523975994235,4cyclictest16950irq/224-800000008:30:000
3523975994215,10cyclictest12-21ksoftirqd/009:05:020
352397599421,37cyclictest0-21swapper/008:05:010
3523975994211,6cyclictest131rcu_preempt08:00:020
3523975994211,6cyclictest131rcu_preempt08:00:010
352399199417,5cyclictest131rcu_preempt08:20:013
3523991994134,5cyclictest37-21ksoftirqd/307:54:403
3523991994113,18cyclictest131rcu_preempt07:30:023
3523987994134,5cyclictest3524985-21kworker/2:008:51:372
3523987994134,5cyclictest3524985-21kworker/2:008:51:362
3523987994124,13cyclictest3467602-21systemd-journal09:05:312
3523982994130,7cyclictest3524709-21munin-plugin-st07:25:001
3523982994128,9cyclictest23-21ksoftirqd/109:00:001
3523982994126,8cyclictest3467602-21systemd-journal07:15:001
3523982994111,6cyclictest23-21ksoftirqd/107:32:351
3523982994111,6cyclictest23-21ksoftirqd/107:32:341
3523975994117,16cyclictest12-21ksoftirqd/008:10:030
3523975994117,16cyclictest12-21ksoftirqd/008:10:020
3523975994113,6cyclictest131rcu_preempt07:25:020
352399199401,5cyclictest150-21pr/ttyS208:13:413
352399199401,5cyclictest150-21pr/ttyS208:13:403
3523987994032,5cyclictest3515796-21kworker/2:307:30:002
3523987994025,12cyclictest3467602-21systemd-journal07:31:372
3523987994025,12cyclictest3467602-21systemd-journal07:31:372
352398299405,26cyclictest3527281-21grep08:15:011
352398299405,26cyclictest3527281-21grep08:15:001
3523982994024,11cyclictest3529368-21munin-run08:55:021
3523982994024,11cyclictest3529368-21munin-run08:55:011
3523975994016,19cyclictest131rcu_preempt08:55:020
3523975994016,19cyclictest131rcu_preempt08:55:020
352399199391,5cyclictest150-21pr/ttyS207:14:263
352399199390,7cyclictest150-21pr/ttyS207:24:243
3523987993931,5cyclictest3524985-21kworker/2:007:42:292
352398299394,25cyclictest3525994-21munin-run07:50:011
352398299394,25cyclictest3525994-21munin-run07:50:011
3523982993928,6cyclictest309-21in:imuxsock07:30:001
3523987993827,8cyclictest3524112-21sed07:10:032
3523982993828,7cyclictest3528295-21sshd08:31:371
3523982993825,9cyclictest23-21ksoftirqd/108:45:001
3523982993824,10cyclictest3467602-21systemd-journal07:17:161
3523975993822,13cyclictest3526210-21sshd07:50:080
3523975993813,5cyclictest131rcu_preempt07:35:010
3523975993813,5cyclictest131rcu_preempt07:35:010
352399199371,6cyclictest150-21pr/ttyS208:20:533
3523987993727,7cyclictest30-21ksoftirqd/208:14:072
3523987993727,7cyclictest30-21ksoftirqd/208:14:072
3523987993725,8cyclictest275-21dbus-daemon08:25:012
352399199361,6cyclictest0-21swapper/307:46:383
352399199361,6cyclictest0-21swapper/307:46:373
352398799363,4cyclictest131rcu_preempt09:00:002
3523987993628,5cyclictest3498732-21kworker/2:207:10:022
3523975993627,6cyclictest12-21ksoftirqd/007:50:020
3523975993627,6cyclictest12-21ksoftirqd/007:50:010
3523987993522,9cyclictest1-21systemd08:35:002
3523987993426,5cyclictest3524985-21kworker/2:008:26:482
3523982993319,10cyclictest364-21snmpd08:05:401
3523987993222,8cyclictest3515796-21kworker/2:308:00:012
3523987993222,8cyclictest3515796-21kworker/2:308:00:002
3523982993214,15cyclictest3523912-21latency_hist07:10:021
3523982993214,15cyclictest3523912-21latency_hist07:10:011
352398799311,4cyclictest131rcu_preempt08:40:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional