You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 17:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Apr 05, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9400709913496,32cyclictest37-21ksoftirqd/309:35:023
9400709911487,11cyclictest37-21ksoftirqd/308:15:013
940061997717,9cyclictest23-21ksoftirqd/111:12:121
94006199717,17cyclictest1018525-21ssh11:58:491
94006199708,18cyclictest1005576-21ssh11:28:271
940061996916,13cyclictest23-21ksoftirqd/110:05:021
94006199684,24cyclictest941652-21sed07:40:021
940056996864,2cyclictest0-21swapper/009:00:020
94006199669,17cyclictest970088-21ssh10:05:261
94006199667,19cyclictest988541-21ssh10:48:421
940061996616,12cyclictest1012684-21/usr/sbin/munin11:45:021
940061996616,12cyclictest1012684-21/usr/sbin/munin11:45:011
940061996614,12cyclictest23-21ksoftirqd/111:54:421
940061996614,12cyclictest23-21ksoftirqd/111:54:421
94006199658,17cyclictest989277-21ssh10:50:061
94006199654,21cyclictest980833-21ssh10:30:251
94006199654,21cyclictest980833-21ssh10:30:251
94005699656,45cyclictest19350irq/34-mmc108:10:030
94006199648,18cyclictest956524-21ssh09:34:121
940061996416,11cyclictest23-21ksoftirqd/110:19:571
940061996415,9cyclictest23-21ksoftirqd/109:29:221
94005699648,37cyclictest12-21ksoftirqd/011:20:250
94005699641,60cyclictest0-21swapper/008:55:020
94006699639,40cyclictest291rcuc/208:00:012
940061996316,10cyclictest23-21ksoftirqd/112:00:231
94006199628,17cyclictest1000130-21ssh11:15:301
940061996214,11cyclictest23-21ksoftirqd/112:21:311
940061996213,10cyclictest23-21ksoftirqd/112:10:411
940061996212,13cyclictest23-21ksoftirqd/110:59:061
940056996236,8cyclictest12-21ksoftirqd/011:00:020
94006199618,15cyclictest976587-21ssh10:20:301
94006199616,17cyclictest951931-21ssh09:23:161
94006199615,17cyclictest1033829-21ssh12:34:251
940061996116,9cyclictest23-21ksoftirqd/110:25:241
940061996115,10cyclictest23-21ksoftirqd/109:13:011
940061996113,10cyclictest23-21ksoftirqd/112:07:091
94006199606,15cyclictest1008665-21ssh11:35:141
94006199603,21cyclictest1026575-21ssh12:17:241
940061996016,10cyclictest23-21ksoftirqd/109:44:581
940061996012,9cyclictest23-21ksoftirqd/109:48:201
940061996012,14cyclictest985564-21ssh10:41:261
940056996016,36cyclictest12-21ksoftirqd/010:27:070
94006199595,18cyclictest941308-21systemd-run07:35:011
940056995927,14cyclictest1035762-21ssh12:39:060
940066995849,6cyclictest945218-21kworker/2:309:34:512
940061995814,11cyclictest23-21ksoftirqd/112:36:421
940056995853,3cyclictest150-21pr/ttyS210:05:020
94005699583,22cyclictest997570-21ssh11:10:010
940056995829,13cyclictest988397-21ssh10:48:210
940056995818,12cyclictest12-21ksoftirqd/011:49:440
940056995810,42cyclictest3467602-21systemd-journal08:40:000
94006199573,21cyclictest994609-21rm11:02:481
94006199573,18cyclictest940832-21latency_hist07:25:011
940061995716,9cyclictest23-21ksoftirqd/111:30:261
940061995716,9cyclictest23-21ksoftirqd/111:30:251
940061995713,9cyclictest23-21ksoftirqd/108:55:011
940061995713,11cyclictest23-21ksoftirqd/110:13:191
940061995713,10cyclictest23-21ksoftirqd/109:38:411
940061995712,9cyclictest23-21ksoftirqd/112:29:041
940056995746,7cyclictest954750-21systemd09:30:020
940056995731,10cyclictest975254-21ssh10:17:370
940056995719,33cyclictest982608-21latency_hist10:35:020
940056995719,33cyclictest982608-21latency_hist10:35:020
940066995648,5cyclictest976292-21kworker/2:011:37:422
940066995647,5cyclictest945218-21kworker/2:309:22:032
94006199568,16cyclictest984767-21munin-run10:40:001
940061995612,9cyclictest23-21ksoftirqd/111:22:401
94006199561,14cyclictest965152-21ssh09:54:181
94006199561,14cyclictest965152-21ssh09:54:171
940056995627,13cyclictest984539-21ssh10:39:260
940056995625,4cyclictest131rcu_preempt09:45:130
940066995547,5cyclictest976292-21kworker/2:011:35:032
940066995547,5cyclictest976292-21kworker/2:011:35:022
940061995535,14cyclictest1-21systemd07:45:001
940061995513,7cyclictest23-21ksoftirqd/109:10:021
940061995513,7cyclictest23-21ksoftirqd/109:10:011
940061995512,14cyclictest949927-21sh09:18:501
940061995510,8cyclictest23-21ksoftirqd/111:06:501
940070995434,15cyclictest275-21dbus-daemon10:45:013
940066995446,5cyclictest976292-21kworker/2:012:30:022
940066995446,5cyclictest976292-21kworker/2:011:00:012
940066995445,5cyclictest976292-21kworker/2:012:33:042
940066995444,5cyclictest954740-21kworker/2:209:55:012
940066995444,5cyclictest954740-21kworker/2:209:55:002
940066995443,6cyclictest945218-21kworker/2:310:00:012
94006199546,14cyclictest943633-21munin-run08:19:591
940061995417,7cyclictest23-21ksoftirqd/109:58:581
94005699545,30cyclictest131rcu_preempt10:00:040
940056995426,13cyclictest1009980-21ssh11:38:440
940056995425,13cyclictest1033351-21ssh12:33:200
940056995422,28cyclictest12-21ksoftirqd/012:20:320
94005699541,38cyclictest1001797-21ssh11:19:490
940070995336,13cyclictest1-21systemd09:20:013
940066995344,6cyclictest976292-21kworker/2:011:59:532
940066995343,6cyclictest976292-21kworker/2:010:40:012
940066995314,18cyclictest870140-21kworker/2:107:15:022
940056995326,11cyclictest956742-21ssh09:34:410
940056995318,29cyclictest12-21ksoftirqd/011:53:000
940056995318,29cyclictest12-21ksoftirqd/011:53:000
940070995236,12cyclictest1-21systemd10:15:003
940066995244,5cyclictest945218-21kworker/2:310:17:362
940066995243,6cyclictest945218-21kworker/2:310:21:102
94006199524,14cyclictest943906-21munin-run08:25:001
940056995247,2cyclictest0-21swapper/007:35:020
94005699524,5cyclictest131rcu_preempt10:40:410
940056995227,11cyclictest948475-21ssh09:15:030
940070995137,10cyclictest1-21systemd12:10:013
940066995143,5cyclictest976292-21kworker/2:012:22:322
940066995143,5cyclictest976292-21kworker/2:011:52:102
940066995143,5cyclictest976292-21kworker/2:011:52:092
940066995143,5cyclictest976292-21kworker/2:010:53:352
940066995143,5cyclictest976292-21kworker/2:010:28:202
940066995142,5cyclictest976292-21kworker/2:012:18:552
940066995142,5cyclictest954740-21kworker/2:209:42:162
940066995134,12cyclictest1-21systemd10:35:002
940066995134,12cyclictest1-21systemd10:35:002
94006199512,13cyclictest1012957-21ssh11:45:091
940056995141,7cyclictest12-21ksoftirqd/012:20:020
94005699511,4cyclictest131rcu_preempt12:27:120
940056995114,33cyclictest12-21ksoftirqd/010:11:220
940066995042,5cyclictest976292-21kworker/2:012:10:052
940066995042,5cyclictest976292-21kworker/2:011:05:452
940066995041,6cyclictest976292-21kworker/2:011:11:442
940066995041,6cyclictest976292-21kworker/2:010:48:002
940066995041,5cyclictest945218-21kworker/2:309:13:482
94006199506,16cyclictest943417-21sed08:15:021
94005699509,6cyclictest131rcu_preempt09:11:380
94005699509,20cyclictest131rcu_preempt10:51:270
940056995043,4cyclictest16950irq/224-800000010:00:000
940056995042,5cyclictest940063-21ls07:10:010
940056995034,5cyclictest12-21ksoftirqd/010:23:000
940066994941,5cyclictest976292-21kworker/2:011:23:322
940066994941,5cyclictest976292-21kworker/2:011:01:522
940066994941,5cyclictest945218-21kworker/2:310:11:552
940066994940,6cyclictest976292-21kworker/2:011:43:142
940066994940,6cyclictest976292-21kworker/2:011:43:142
940066994940,6cyclictest976292-21kworker/2:011:26:372
940066994940,5cyclictest945218-21kworker/2:309:35:082
94005699494,4cyclictest131rcu_preempt11:26:070
94005699493,14cyclictest3467602-21systemd-journal09:53:490
94005699493,14cyclictest3467602-21systemd-journal09:53:480
940056994924,10cyclictest1024012-21ssh12:11:250
940066994840,5cyclictest976292-21kworker/2:012:00:562
940066994840,5cyclictest976292-21kworker/2:011:46:322
940066994840,5cyclictest945218-21kworker/2:309:16:502
940066994839,6cyclictest954740-21kworker/2:209:49:522
940066994839,5cyclictest945218-21kworker/2:310:06:242
940061994833,9cyclictest3467602-21systemd-journal07:15:011
94006199481,42cyclictest3467602-21systemd-journal08:34:181
940061994810,8cyclictest23-21ksoftirqd/107:17:011
940056994832,10cyclictest1019107-21grep12:00:010
94007099476,6cyclictest37-21ksoftirqd/311:01:293
940066994739,5cyclictest945218-21kworker/2:309:26:582
940066994738,6cyclictest976292-21kworker/2:012:05:052
940066994737,6cyclictest939643-21kworker/2:007:35:002
94005699479,6cyclictest131rcu_preempt11:05:000
94005699477,7cyclictest12-21ksoftirqd/012:10:010
940056994735,9cyclictest12-21ksoftirqd/011:15:020
940056994721,11cyclictest958615-21ssh09:39:110
94007099464,3cyclictest131rcu_preempt11:34:433
94007099464,3cyclictest131rcu_preempt11:34:423
940070994632,9cyclictest275-21dbus-daemon08:10:013
940070994629,12cyclictest1-21systemd08:50:003
940070994628,14cyclictest980796-21sshd10:30:203
940070994628,14cyclictest980796-21sshd10:30:193
94006699467,6cyclictest131rcu_preempt10:44:182
940066994640,3cyclictest945218-21kworker/2:310:04:352
940066994638,5cyclictest976292-21kworker/2:012:38:002
940066994638,5cyclictest939643-21kworker/2:007:55:002
940066994632,10cyclictest1-21systemd09:10:002
940066994632,10cyclictest1-21systemd09:10:002
94005699463,9cyclictest941549-21latency_hist07:40:010
940056994638,5cyclictest16950irq/224-800000008:00:000
940056994630,12cyclictest12-21ksoftirqd/009:10:000
940056994630,12cyclictest12-21ksoftirqd/009:10:000
940056994628,14cyclictest960470-21ssh09:43:160
940056994627,15cyclictest12-21ksoftirqd/012:01:300
940056994611,15cyclictest1012668-21ssh11:45:010
940056994611,15cyclictest1012668-21ssh11:45:000
940070994527,13cyclictest3467602-21systemd-journal09:25:003
940066994539,3cyclictest976292-21kworker/2:011:17:132
940066994536,5cyclictest939643-21kworker/2:008:00:202
940066994536,5cyclictest939643-21kworker/2:007:50:012
940066994530,12cyclictest3467602-21systemd-journal08:05:022
940061994521,17cyclictest945958-21cron09:04:591
940056994529,11cyclictest275-21dbus-daemon08:25:000
940070994428,12cyclictest3467602-21systemd-journal10:40:003
940070994422,15cyclictest275-21dbus-daemon12:20:003
94007099441,3cyclictest131rcu_preempt11:45:033
940066994435,6cyclictest939643-21kworker/2:008:35:012
940066994431,9cyclictest3467602-21systemd-journal08:27:192
940056994434,7cyclictest945970-21munin-plugin-st09:05:010
940056994417,12cyclictest951277-21ssh09:21:340
94005699441,38cyclictest945170-21sshd08:49:160
94007099437,24cyclictest37-21ksoftirqd/308:05:023
940070994330,9cyclictest3467602-21systemd-journal08:35:033
940070994330,10cyclictest37-21ksoftirqd/312:14:063
940070994329,11cyclictest37-21ksoftirqd/312:29:493
940070994329,11cyclictest37-21ksoftirqd/308:45:013
940070994310,5cyclictest131rcu_preempt11:54:083
940070994310,5cyclictest131rcu_preempt11:54:073
940066994334,5cyclictest939643-21kworker/2:007:40:022
94006199434,7cyclictest23-21ksoftirqd/108:48:191
940056994334,6cyclictest12-21ksoftirqd/011:35:020
940056994334,6cyclictest12-21ksoftirqd/011:35:010
94005699431,8cyclictest0-21swapper/007:54:200
94005699431,12cyclictest941842-21unin-run07:45:010
94007099426,19cyclictest0-21swapper/311:15:453
940070994232,7cyclictest963446-21cat09:50:023
940070994228,9cyclictest275-21dbus-daemon09:10:023
940070994228,9cyclictest275-21dbus-daemon09:10:013
940070994228,11cyclictest37-21ksoftirqd/311:40:033
94007099421,21cyclictest0-21swapper/310:29:113
940066994231,6cyclictest939643-21kworker/2:007:20:002
94006699422,34cyclictest275-21dbus-daemon08:45:002
94006199423,29cyclictest945702-21munin-run09:00:011
940061994225,13cyclictest23-21ksoftirqd/108:40:011
94006199421,15cyclictest0-21swapper/107:55:001
94005699425,7cyclictest131rcu_preempt08:30:000
94007099415,3cyclictest131rcu_preempt11:10:013
94007099414,7cyclictest37-21ksoftirqd/312:35:353
940070994128,10cyclictest1-21systemd09:00:013
940070994126,12cyclictest0-21swapper/309:30:013
94007099411,4cyclictest131rcu_preempt11:11:083
94006699413,7cyclictest131rcu_preempt07:10:022
940066994133,5cyclictest939643-21kworker/2:008:50:032
940066994133,5cyclictest939643-21kworker/2:008:10:272
940066994133,5cyclictest939643-21kworker/2:007:20:312
940066994128,9cyclictest309-21in:imuxsock08:54:182
940061994128,10cyclictest364-21snmpd07:27:351
940061994125,13cyclictest942841-21sshd08:04:091
940056994134,4cyclictest16950irq/224-800000008:14:590
940056994130,8cyclictest12-21ksoftirqd/007:20:010
94007099405,5cyclictest131rcu_preempt11:00:013
94007099401,4cyclictest131rcu_preempt10:15:583
94007099401,3cyclictest131rcu_preempt10:07:573
94007099401,3cyclictest0-21swapper/310:53:393
940066994033,4cyclictest945218-21kworker/2:309:02:262
940066994032,5cyclictest939643-21kworker/2:008:36:192
940066994031,5cyclictest939643-21kworker/2:008:20:022
940061994026,9cyclictest309-21in:imuxsock08:30:011
94005699402,5cyclictest275-21dbus-daemon07:15:020
94005699401,37cyclictest0-21swapper/008:05:010
94007099399,6cyclictest131rcu_preempt08:00:033
94007099394,4cyclictest131rcu_preempt12:24:433
940070993925,11cyclictest37-21ksoftirqd/312:02:213
940070993924,11cyclictest37-21ksoftirqd/309:35:293
94007099391,4cyclictest131rcu_preempt10:02:243
94007099391,3cyclictest131rcu_preempt07:35:003
940061993927,9cyclictest944915-21sshd08:43:201
940061993927,9cyclictest942560-21sshd07:59:091
940061993923,12cyclictest943084-21sshd08:07:191
940056993925,9cyclictest310-21in:imklog07:25:020
94007099383,3cyclictest131rcu_preempt11:55:533
940070993827,9cyclictest37-21ksoftirqd/312:32:303
940070993819,15cyclictest275-21dbus-daemon11:39:593
94007099381,4cyclictest131rcu_preempt10:25:003
94007099381,3cyclictest131rcu_preempt09:44:563
940061993826,9cyclictest942050-21sshd07:47:201
94007099376,11cyclictest131rcu_preempt10:47:543
94007099373,3cyclictest131rcu_preempt11:21:013
940070993724,9cyclictest275-21dbus-daemon07:50:013
940070993723,10cyclictest942427-21wc07:55:013
94007099371,5cyclictest0-21swapper/311:46:223
94007099371,3cyclictest131rcu_preempt09:13:253
94007099371,3cyclictest131rcu_preempt08:55:013
94006699371,6cyclictest0-21swapper/207:45:002
94005699373,8cyclictest12-21ksoftirqd/008:19:590
94005699372,31cyclictest944948-21systemd-run08:45:000
94005699371,33cyclictest0-21swapper/007:28:340
94007099368,6cyclictest131rcu_preempt11:25:293
940070993625,8cyclictest940460-21sed07:15:023
940070993620,12cyclictest37-21ksoftirqd/308:27:433
94007099361,3cyclictest131rcu_preempt09:56:123
940056993628,5cyclictest274-21cron08:35:000
940056993625,8cyclictest12-21ksoftirqd/007:48:190
940070993526,5cyclictest963575-21sh09:50:063
940070993526,5cyclictest963575-21sh09:50:063
94007099352,3cyclictest131rcu_preempt08:35:013
940070993518,12cyclictest275-21dbus-daemon07:20:013
940066993526,5cyclictest945218-21kworker/2:309:00:032
940066993521,11cyclictest30-21ksoftirqd/207:29:592
940070993424,7cyclictest944046-21latency_hist08:25:033
940070993322,9cyclictest37-21ksoftirqd/307:40:013
940070993321,9cyclictest275-21dbus-daemon09:05:003
94007099324,3cyclictest131rcu_preempt07:45:033
94007099322,3cyclictest131rcu_preempt07:10:023
940070993220,10cyclictest37-21ksoftirqd/307:30:013
94007099321,3cyclictest131rcu_preempt07:25:013
94006699323,7cyclictest30-21ksoftirqd/208:21:022
94007099311,3cyclictest131rcu_preempt08:19:193
94006199315,20cyclictest940064-21sort07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional