You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 17:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Mar 30, 2026 12:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
329347799108104,2cyclictest0-21swapper/008:30:020
3293477998783,2cyclictest0-21swapper/008:45:020
3293477998271,5cyclictest0-21swapper/009:10:010
3293477998236,41cyclictest16850irq/134-800000009:35:000
3293477998035,10cyclictest3353388-21sed11:15:020
3293477997226,14cyclictest3302861-21ssh09:17:190
3293484997036,10cyclictest3281868-21kworker/2:009:56:272
3293477997065,3cyclictest0-21swapper/007:50:030
329348099689,8cyclictest23-21ksoftirqd/110:05:021
329348099689,8cyclictest23-21ksoftirqd/110:05:021
3293480996826,9cyclictest23-21ksoftirqd/111:20:011
3293477996629,32cyclictest3319002-21latency_hist09:55:030
329348099656,12cyclictest23-21ksoftirqd/109:30:021
3293477996522,14cyclictest3350375-21ssh11:08:170
3293477996516,3cyclictest131rcu_preempt11:37:240
329348099646,8cyclictest23-21ksoftirqd/112:40:011
3293480996430,8cyclictest23-21ksoftirqd/111:30:531
3293484996049,8cyclictest3342464-21kworker/2:111:30:552
329348099601,9cyclictest23-21ksoftirqd/111:15:001
3293484995935,6cyclictest3308234-21kworker/2:210:35:552
329348899581,19cyclictest0-21swapper/307:20:013
3293484995850,5cyclictest3281868-21kworker/2:009:54:352
3293477995828,15cyclictest3385856-21ssh12:30:440
3293477995828,15cyclictest3367374-21ssh11:47:490
3293477995819,34cyclictest3311510-21kworker/u8:109:40:010
3293477995813,10cyclictest12-21ksoftirqd/010:55:430
3293488995712,12cyclictest37-21ksoftirqd/310:35:023
3293488995710,6cyclictest131rcu_preempt07:50:023
329348099577,7cyclictest23-21ksoftirqd/112:15:011
329348099575,15cyclictest23-21ksoftirqd/110:10:011
329348099574,12cyclictest23-21ksoftirqd/112:00:001
3293477995732,9cyclictest12-21ksoftirqd/012:40:010
3293477995727,14cyclictest3337207-21ssh10:37:320
3293488995619,7cyclictest37-21ksoftirqd/311:05:033
329348499569,6cyclictest131rcu_preempt11:40:002
3293484995648,4cyclictest3342464-21kworker/2:112:15:452
329348099569,10cyclictest23-21ksoftirqd/109:50:041
329348099568,11cyclictest23-21ksoftirqd/112:30:011
329348099565,9cyclictest23-21ksoftirqd/112:21:471
3293477995628,12cyclictest3385303-21ssh12:29:510
329347799561,39cyclictest3378045-21ssh12:12:500
3293488995511,6cyclictest131rcu_preempt12:30:023
3293484995548,4cyclictest3281868-21kworker/2:009:48:492
3293484995546,6cyclictest3308234-21kworker/2:210:03:482
3293484995546,6cyclictest3308234-21kworker/2:210:03:472
3293484995540,7cyclictest3342464-21kworker/2:112:26:572
3293484995512,6cyclictest131rcu_preempt12:00:012
329348099555,9cyclictest23-21ksoftirqd/109:25:001
329347799555,4cyclictest131rcu_preempt12:00:130
3293477995536,14cyclictest1-21systemd10:00:000
3293477995531,8cyclictest12-21ksoftirqd/009:50:000
3293477995527,13cyclictest3358200-21ssh11:26:070
3293477995524,15cyclictest3327457-21ssh10:15:010
3293477995524,15cyclictest3327457-21ssh10:15:000
329347799551,3cyclictest131rcu_preempt09:41:210
3293484995446,5cyclictest3308234-21kworker/2:210:15:242
329348099548,8cyclictest23-21ksoftirqd/112:19:481
3293480995420,9cyclictest23-21ksoftirqd/110:18:551
3293477995427,12cyclictest3307881-21ssh09:29:070
329347799542,4cyclictest131rcu_preempt11:20:000
3293477995424,15cyclictest3332007-21ssh10:25:040
329347799542,3cyclictest131rcu_preempt08:50:030
329347799542,3cyclictest131rcu_preempt08:50:020
3293488995340,10cyclictest0-21swapper/311:55:023
3293488995315,10cyclictest131rcu_preempt11:07:213
3293488995311,9cyclictest150-21pr/ttyS211:20:573
3293484995345,5cyclictest3342464-21kworker/2:111:02:392
3293484995345,5cyclictest3342464-21kworker/2:110:52:032
3293484995345,5cyclictest3308234-21kworker/2:210:14:072
3293484995345,5cyclictest3308234-21kworker/2:210:14:062
3293484995345,5cyclictest3281868-21kworker/2:009:25:142
3293484995345,5cyclictest3281868-21kworker/2:009:21:562
3293484995338,7cyclictest3342464-21kworker/2:112:35:552
3293484995335,13cyclictest1-21systemd07:35:022
329348499531,3cyclictest0-21swapper/207:15:022
329348099532,14cyclictest23-21ksoftirqd/110:25:021
3293477995346,3cyclictest0-21swapper/007:35:000
3293477995345,6cyclictest12-21ksoftirqd/011:59:060
3293477995326,12cyclictest3327889-21ssh10:15:330
3293477995326,11cyclictest3339757-21ssh10:43:300
3293477995326,11cyclictest3339757-21ssh10:43:300
3293477995321,29cyclictest12-21ksoftirqd/012:21:170
329348899529,6cyclictest131rcu_preempt09:49:213
329348899526,9cyclictest37-21ksoftirqd/309:44:593
329348899524,4cyclictest131rcu_preempt12:20:013
329348499525,8cyclictest131rcu_preempt12:35:022
3293484995244,5cyclictest3308234-21kworker/2:210:28:082
3293484995244,5cyclictest3308234-21kworker/2:210:05:512
3293484995244,5cyclictest3281868-21kworker/2:009:40:482
3293484995244,5cyclictest3281868-21kworker/2:009:37:512
3293484995243,6cyclictest3281868-21kworker/2:010:21:332
3293484995243,5cyclictest3281868-21kworker/2:010:46:342
329348499521,8cyclictest3298983-21latency_hist08:55:012
329348099528,5cyclictest131rcu_preempt07:35:001
3293480995210,6cyclictest131rcu_preempt09:00:011
329347799524,42cyclictest3297051-21sendmail-msp08:20:010
329347799521,4cyclictest131rcu_preempt12:05:140
3293488995141,6cyclictest37-21ksoftirqd/310:30:013
3293488995132,14cyclictest275-21dbus-daemon08:00:003
329348499516,6cyclictest131rcu_preempt11:00:032
3293484995143,5cyclictest3342464-21kworker/2:111:49:292
3293484995143,5cyclictest3342464-21kworker/2:111:23:052
3293484995143,5cyclictest3281868-21kworker/2:009:33:162
3293484995142,6cyclictest3308234-21kworker/2:210:33:012
3293484995142,6cyclictest3281868-21kworker/2:010:44:562
3293484995142,6cyclictest3281868-21kworker/2:010:44:562
3293484995142,5cyclictest3342464-21kworker/2:111:19:282
3293484995140,6cyclictest3342464-21kworker/2:111:43:342
329348099518,8cyclictest23-21ksoftirqd/111:24:111
329348099517,9cyclictest23-21ksoftirqd/110:40:001
329348099516,10cyclictest23-21ksoftirqd/109:16:051
329348099514,11cyclictest23-21ksoftirqd/109:33:091
329348099513,9cyclictest23-21ksoftirqd/109:36:031
3293480995131,16cyclictest1-21systemd08:50:021
3293480995131,16cyclictest1-21systemd08:50:011
329348099513,10cyclictest23-21ksoftirqd/111:28:291
3293480995111,6cyclictest131rcu_preempt11:52:551
3293480995111,5cyclictest131rcu_preempt11:44:101
3293477995144,4cyclictest3295724-21latency_hist07:55:010
3293477995127,3cyclictest131rcu_preempt11:41:390
329347799512,4cyclictest131rcu_preempt10:47:130
329348899507,5cyclictest131rcu_preempt12:40:013
3293488995029,16cyclictest3467602-21systemd-journal09:30:013
3293488995011,6cyclictest131rcu_preempt10:36:023
329348899501,11cyclictest3386853-21sh12:33:113
3293484995043,4cyclictest3342464-21kworker/2:111:52:082
3293484995042,5cyclictest3342464-21kworker/2:112:11:452
3293484995042,5cyclictest3342464-21kworker/2:112:08:342
3293484995042,5cyclictest3342464-21kworker/2:112:00:102
3293484995042,5cyclictest3342464-21kworker/2:111:13:572
3293484995042,5cyclictest3297823-21kworker/2:309:12:372
3293484995039,8cyclictest3342464-21kworker/2:111:05:142
329348499501,9cyclictest3295228-21cut07:45:002
329348099509,9cyclictest23-21ksoftirqd/112:05:011
329348099505,9cyclictest23-21ksoftirqd/112:32:211
329348099505,11cyclictest23-21ksoftirqd/111:05:011
3293480995011,5cyclictest131rcu_preempt11:39:131
3293480995011,5cyclictest131rcu_preempt10:28:231
3293480995010,5cyclictest131rcu_preempt09:54:401
3293488994911,9cyclictest37-21ksoftirqd/309:50:223
329348499497,5cyclictest131rcu_preempt09:20:012
3293484994941,5cyclictest3342464-21kworker/2:112:21:412
3293484994941,5cyclictest3342464-21kworker/2:111:29:172
3293484994941,5cyclictest3281868-21kworker/2:008:00:012
3293484994941,5cyclictest3281868-21kworker/2:007:50:022
329348499493,5cyclictest131rcu_preempt08:40:002
329348099497,6cyclictest131rcu_preempt11:10:021
329348099495,12cyclictest23-21ksoftirqd/110:52:381
3293480994929,14cyclictest275-21dbus-daemon07:30:011
329348099492,12cyclictest23-21ksoftirqd/107:45:011
3293477994936,7cyclictest3299098-21sendmail09:00:010
3293477994926,7cyclictest3361481-21ssh11:34:160
3293477994924,18cyclictest275-21dbus-daemon10:55:000
3293477994920,13cyclictest3324111-21ssh10:06:420
3293488994829,14cyclictest275-21dbus-daemon10:50:013
3293488994811,6cyclictest131rcu_preempt08:20:213
3293484994840,5cyclictest3281868-21kworker/2:007:51:172
329348099489,6cyclictest131rcu_preempt11:49:291
329348099488,5cyclictest131rcu_preempt12:05:501
329348099487,8cyclictest23-21ksoftirqd/110:55:421
3293480994811,5cyclictest131rcu_preempt09:15:001
3293477994830,15cyclictest3379563-21ssh12:16:010
329347799481,3cyclictest131rcu_preempt10:30:420
329348899472,4cyclictest131rcu_preempt10:04:533
329348899472,4cyclictest131rcu_preempt10:04:533
3293488994718,20cyclictest361rcuc/310:05:563
3293488994716,24cyclictest131rcu_preempt10:25:023
3293488994713,7cyclictest131rcu_preempt09:25:013
3293488994712,6cyclictest131rcu_preempt12:10:013
3293488994711,9cyclictest131rcu_preempt12:00:013
3293484994738,5cyclictest3281868-21kworker/2:007:25:342
3293484994738,5cyclictest3281868-21kworker/2:007:25:012
329348099477,7cyclictest23-21ksoftirqd/110:00:021
329348099474,8cyclictest23-21ksoftirqd/110:45:011
329348099474,8cyclictest23-21ksoftirqd/110:45:011
329348099473,10cyclictest23-21ksoftirqd/110:31:481
3293480994728,14cyclictest275-21dbus-daemon08:30:001
3293480994713,5cyclictest131rcu_preempt10:48:451
329347799472,4cyclictest131rcu_preempt09:13:210
3293477994721,11cyclictest3369407-21ssh11:52:380
329348899469,7cyclictest131rcu_preempt10:10:103
329348899469,7cyclictest131rcu_preempt10:10:103
329348899464,10cyclictest37-21ksoftirqd/311:00:023
329348899463,7cyclictest37-21ksoftirqd/308:40:023
329348899463,6cyclictest131rcu_preempt12:05:013
3293488994631,12cyclictest3353263-21systemd-run11:15:003
329348899462,7cyclictest0-21swapper/309:30:273
329348899462,6cyclictest37-21ksoftirqd/310:43:153
329348899462,6cyclictest37-21ksoftirqd/310:43:143
329348899462,5cyclictest131rcu_preempt09:40:023
329348899461,9cyclictest0-21swapper/309:12:403
3293488994615,15cyclictest37-21ksoftirqd/308:49:333
3293488994615,15cyclictest37-21ksoftirqd/308:49:323
3293484994639,4cyclictest3281868-21kworker/2:008:26:352
3293484994638,5cyclictest3281868-21kworker/2:008:11:462
3293484994638,5cyclictest3281868-21kworker/2:007:18:242
329348099466,9cyclictest23-21ksoftirqd/110:10:251
329348099466,9cyclictest23-21ksoftirqd/110:10:251
329348099466,7cyclictest23-21ksoftirqd/109:10:001
329348099465,8cyclictest23-21ksoftirqd/108:25:011
3293480994633,9cyclictest1-21systemd08:00:001
3293480994631,11cyclictest3467602-21systemd-journal08:51:311
3293480994613,6cyclictest131rcu_preempt08:05:021
3293480994610,10cyclictest23-21ksoftirqd/107:40:001
3293477994621,9cyclictest3331170-21ssh10:23:230
3293477994615,8cyclictest12-21ksoftirqd/011:22:520
329348899453,6cyclictest131rcu_preempt07:40:023
3293488994532,9cyclictest275-21dbus-daemon09:55:493
329348899452,5cyclictest131rcu_preempt11:35:003
3293488994517,6cyclictest131rcu_preempt11:20:013
3293488994514,9cyclictest131rcu_preempt07:25:003
3293488994514,19cyclictest37-21ksoftirqd/309:18:283
3293484994537,5cyclictest3297823-21kworker/2:309:08:162
329348099459,4cyclictest131rcu_preempt08:40:011
329348099453,8cyclictest23-21ksoftirqd/108:20:001
3293477994527,14cyclictest3347245-21ssh11:00:360
329348899449,6cyclictest131rcu_preempt10:53:363
329348899449,5cyclictest131rcu_preempt12:14:493
3293488994433,8cyclictest37-21ksoftirqd/311:30:013
3293488994415,17cyclictest37-21ksoftirqd/310:16:593
3293488994414,19cyclictest131rcu_preempt08:20:013
3293488994413,6cyclictest131rcu_preempt09:05:003
3293488994412,6cyclictest131rcu_preempt11:43:423
3293488994410,6cyclictest131rcu_preempt11:40:013
3293484994436,5cyclictest3281868-21kworker/2:008:23:592
3293484994435,6cyclictest3297823-21kworker/2:308:56:232
3293484994425,15cyclictest3297033-21systemctl08:20:002
3293484994410,5cyclictest131rcu_preempt09:05:002
329348099449,6cyclictest131rcu_preempt09:41:191
3293480994429,10cyclictest3467602-21systemd-journal07:50:011
329347799443,7cyclictest275-21dbus-daemon07:20:000
3293477994431,9cyclictest3294955-21systemctl07:39:590
3293477994418,3cyclictest131rcu_preempt10:03:540
3293477994418,3cyclictest131rcu_preempt10:03:530
329347799441,39cyclictest3304381-21ssh09:20:290
3293488994329,10cyclictest309-21in:imuxsock08:00:023
3293488994322,15cyclictest275-21dbus-daemon09:10:003
3293488994318,21cyclictest37-21ksoftirqd/311:46:503
3293488994314,19cyclictest131rcu_preempt07:10:013
329348499436,12cyclictest0-21swapper/208:35:012
3293484994335,5cyclictest3281868-21kworker/2:008:03:142
3293484994334,6cyclictest3281868-21kworker/2:008:08:132
329348099438,6cyclictest23-21ksoftirqd/108:35:011
329348099435,10cyclictest23-21ksoftirqd/107:54:591
329348099431,13cyclictest3299386-21unin-run09:05:021
329348899423,9cyclictest37-21ksoftirqd/312:22:583
3293488994217,21cyclictest37-21ksoftirqd/308:09:073
3293480994225,12cyclictest3467602-21systemd-journal07:20:001
3293480994225,12cyclictest275-21dbus-daemon07:25:001
3293477994229,10cyclictest151rcuc/007:10:020
329347799422,6cyclictest1-21systemd08:05:020
3293477994226,12cyclictest3467602-21systemd-journal07:55:020
3293477994221,16cyclictest275-21dbus-daemon08:25:000
3293488994115,6cyclictest131rcu_preempt08:45:013
3293484994134,4cyclictest3281868-21kworker/2:008:44:072
329348099416,8cyclictest23-21ksoftirqd/108:45:001
329348099410,5cyclictest150-21pr/ttyS208:07:031
3293477994126,11cyclictest12-21ksoftirqd/007:26:440
3293477994123,12cyclictest309-21in:imuxsock07:25:010
3293488994022,12cyclictest3467602-21systemd-journal08:15:003
3293484994032,5cyclictest3281868-21kworker/2:008:49:462
3293484994032,5cyclictest3281868-21kworker/2:008:49:452
3293480994022,13cyclictest1-21systemd07:15:011
3293477994033,4cyclictest16950irq/224-800000008:10:000
329347799402,5cyclictest3298843-21munin-run08:55:000
3293477994025,12cyclictest3295201-21sshd07:41:170
3293488993929,7cyclictest3293624-21sshd07:11:183
3293488993923,4cyclictest37-21ksoftirqd/307:40:173
3293488993911,5cyclictest131rcu_preempt08:55:213
3293477993931,5cyclictest3297554-21latency_hist08:30:000
329347799391,10cyclictest3293707-21cat07:15:010
3293488993830,4cyclictest3297565-21sh08:30:003
329348899381,5cyclictest131rcu_preempt08:55:003
3293488993814,15cyclictest131rcu_preempt07:52:393
3293484993816,18cyclictest131rcu_preempt07:40:002
329348899371,5cyclictest131rcu_preempt07:35:023
3293488993712,16cyclictest37-21ksoftirqd/307:27:293
329348899367,15cyclictest37-21ksoftirqd/308:30:373
329348099352,9cyclictest23-21ksoftirqd/108:15:011
329347799358,23cyclictest19350irq/34-mmc108:40:020
3293477993524,8cyclictest3296728-21sshd08:10:140
329347799341,4cyclictest131rcu_preempt09:05:020
329348099323,19cyclictest3293502-21grep07:10:011
329348499292,5cyclictest131rcu_preempt07:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional