You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 19:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Apr 03, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32535899109105,2cyclictest0-21swapper/011:35:010
325358991081,103cyclictest0-21swapper/011:20:030
32535899901,3cyclictest150-21pr/ttyS209:00:030
325367998167,12cyclictest396650-21ssh11:40:042
325367998167,12cyclictest396650-21ssh11:40:032
325367998121,24cyclictest403909-21ssh11:56:552
325358998176,3cyclictest0-21swapper/010:55:000
325367998020,19cyclictest365345-21ssh10:27:402
325367997622,16cyclictest391632-21ssh11:28:382
325367997530,14cyclictest377859-21ssh10:56:402
325367997424,16cyclictest383003-21ssh11:08:452
325367997420,16cyclictest357286-21ssh10:09:132
325358997431,10cyclictest3467602-21systemd-journal11:05:000
325367997318,20cyclictest408052-21ssh12:06:292
325367997217,20cyclictest416469-21ssh12:25:572
325367997119,15cyclictest410146-21ssh12:11:182
325358997167,2cyclictest0-21swapper/009:20:040
325367997024,14cyclictest355077-21ssh10:04:082
325367997021,16cyclictest358878-21rm10:12:452
325367997018,21cyclictest337062-21ssh09:22:182
325367997018,14cyclictest420855-21rm12:36:002
325367997016,20cyclictest347252-21ssh09:45:412
325358997012,45cyclictest357599-21sh10:10:020
325367996922,16cyclictest387866-21systemd-run11:20:012
325367996919,17cyclictest360821-21ssh10:17:142
325367996919,16cyclictest332955-21ssh09:12:562
325367996829,13cyclictest325569-21sshd07:13:522
325367996722,14cyclictest326893-21munin-run07:40:012
325367996722,14cyclictest326893-21munin-run07:40:012
325367996721,14cyclictest330988-21sed08:55:022
325367996618,15cyclictest374664-21ssh10:49:272
325367996617,15cyclictest328489-21cut08:10:022
325367996616,18cyclictest389550-21ssh11:23:532
325367996517,14cyclictest419426-21ssh12:33:012
325367996517,14cyclictest419426-21ssh12:33:012
32535899654,48cyclictest325361-21latency_hist07:10:020
325367996447,10cyclictest311031-21kworker/2:309:20:012
325367996418,13cyclictest341870-21ssh09:33:292
325367996418,13cyclictest341870-21ssh09:33:292
325367996318,15cyclictest362628-21ssh10:21:182
325367996316,14cyclictest384041-21ssh11:10:472
325367996216,16cyclictest327195-21latency_hist07:45:012
325367996215,14cyclictest338348-21apt-config09:25:042
32535899622,36cyclictest422474-21ssh12:40:010
325367996021,13cyclictest406330-21ssh12:02:362
325367996021,13cyclictest406330-21ssh12:02:362
325367996015,13cyclictest344421-21ssh09:39:262
325369995913,6cyclictest131rcu_preempt12:11:563
325369995911,6cyclictest131rcu_preempt09:14:513
325367995918,13cyclictest376051-21ssh10:52:352
325367995913,13cyclictest368693-21ssh10:35:152
325367995912,13cyclictest331564-21diskmemload09:41:262
325367995912,13cyclictest331564-21diskmemload09:41:262
325369995810,6cyclictest131rcu_preempt09:55:583
325367995843,11cyclictest397698-21ssh11:42:412
325367995840,15cyclictest367736-21ssh10:33:192
325367995839,15cyclictest380925-21ssh11:03:512
325362995831,17cyclictest23-21ksoftirqd/110:53:131
325358995848,6cyclictest16850irq/134-800000011:59:590
32536999576,10cyclictest37-21ksoftirqd/311:07:373
325367995745,9cyclictest350766-21ssh09:54:072
325367995738,15cyclictest413194-21ssh12:18:382
325367995718,13cyclictest352847-21ssh09:58:592
325367995716,13cyclictest0-21swapper/208:21:502
325362995710,6cyclictest131rcu_preempt10:50:021
325358995740,13cyclictest275-21dbus-daemon08:20:000
325358995727,14cyclictest409286-21ssh12:09:360
325358995725,15cyclictest418425-21ssh12:30:230
325358995725,15cyclictest418425-21ssh12:30:230
325358995720,32cyclictest16850irq/134-800000010:47:250
32536999567,8cyclictest131rcu_preempt11:11:483
325369995614,6cyclictest131rcu_preempt10:40:163
325369995612,6cyclictest131rcu_preempt09:23:313
325369995611,9cyclictest37-21ksoftirqd/309:20:003
325369995610,39cyclictest342498-21unin-run09:35:013
325369995610,39cyclictest342498-21unin-run09:35:013
325367995642,10cyclictest328926-21cron08:17:002
325367995639,13cyclictest414774-21ssh12:22:042
325367995639,13cyclictest399358-21ssh11:46:252
325367995637,15cyclictest371015-21ssh10:40:352
325367995635,17cyclictest392471-21ssh11:30:122
325358995632,19cyclictest348955-21ssh09:50:000
325358995629,12cyclictest354609-21ssh10:02:530
32536999557,11cyclictest37-21ksoftirqd/311:17:113
325369995536,13cyclictest1-21systemd07:45:003
32536999551,12cyclictest37-21ksoftirqd/312:05:453
325367995543,9cyclictest401488-21ssh11:51:122
325367995539,11cyclictest326107-21sshd07:23:412
325367995539,11cyclictest326107-21sshd07:23:402
325362995515,10cyclictest131rcu_preempt12:09:441
32536299551,41cyclictest422597-21ssh12:40:011
32535899555,6cyclictest131rcu_preempt11:43:180
325358995527,13cyclictest416230-21ssh12:25:170
325358995526,14cyclictest390963-21ssh11:27:030
32535899552,37cyclictest345395-21ssh09:41:190
32535899552,37cyclictest345395-21ssh09:41:190
325358995517,4cyclictest12-21ksoftirqd/011:08:410
32535899551,38cyclictest361370-21ssh10:18:340
32536999548,5cyclictest131rcu_preempt11:35:153
32536999548,5cyclictest131rcu_preempt11:35:153
32536999547,9cyclictest37-21ksoftirqd/311:50:213
325369995415,6cyclictest131rcu_preempt10:48:003
325369995414,16cyclictest131rcu_preempt12:18:463
325369995413,5cyclictest131rcu_preempt09:49:153
325369995410,6cyclictest131rcu_preempt10:18:073
325369995410,6cyclictest131rcu_preempt09:28:583
325362995412,6cyclictest131rcu_preempt11:05:001
325362995411,5cyclictest131rcu_preempt11:08:051
325358995435,4cyclictest16850irq/134-800000012:25:000
325358995427,12cyclictest352325-21ssh09:57:350
325358995426,13cyclictest378907-21ssh10:59:170
325358995425,14cyclictest402831-21ssh11:54:430
32536999534,8cyclictest37-21ksoftirqd/312:00:013
325369995316,6cyclictest131rcu_preempt11:01:113
325369995313,6cyclictest131rcu_preempt09:40:003
325369995313,5cyclictest131rcu_preempt10:27:493
325369995312,6cyclictest131rcu_preempt10:25:023
325367995339,10cyclictest328676-21sshd08:12:422
325367995338,10cyclictest326634-21munin-run07:35:002
32536299538,5cyclictest131rcu_preempt10:45:001
32536299538,5cyclictest131rcu_preempt10:08:191
32536299536,6cyclictest131rcu_preempt12:15:451
32536299534,21cyclictest23-21ksoftirqd/112:02:411
32536299534,21cyclictest23-21ksoftirqd/112:02:411
32536999529,6cyclictest131rcu_preempt10:32:503
32536999529,5cyclictest131rcu_preempt12:02:463
32536999529,5cyclictest131rcu_preempt12:02:463
32536999525,5cyclictest131rcu_preempt09:41:593
32536999525,5cyclictest131rcu_preempt09:41:593
325369995216,7cyclictest131rcu_preempt12:35:343
325369995214,5cyclictest131rcu_preempt11:45:103
325369995214,5cyclictest131rcu_preempt10:36:303
325369995213,5cyclictest131rcu_preempt10:11:413
325369995213,5cyclictest131rcu_preempt09:05:023
325369995211,5cyclictest131rcu_preempt11:26:433
325367995237,12cyclictest329712-21sshd08:30:412
325367995236,13cyclictest329460-21sshd08:25:412
325362995230,17cyclictest275-21dbus-daemon12:30:001
325362995212,6cyclictest131rcu_preempt09:20:011
325358995225,11cyclictest409656-21ssh12:10:060
325358995224,24cyclictest385540-21ssh11:14:440
325358995222,15cyclictest396394-21ssh11:39:500
325358995222,15cyclictest396394-21ssh11:39:500
32536999519,5cyclictest131rcu_preempt12:32:283
32536999519,5cyclictest131rcu_preempt12:32:273
32536999516,10cyclictest37-21ksoftirqd/307:40:003
32536999516,10cyclictest37-21ksoftirqd/307:39:593
32536999513,11cyclictest37-21ksoftirqd/312:29:523
325369995116,6cyclictest131rcu_preempt11:41:173
325369995114,6cyclictest131rcu_preempt11:24:443
325369995114,5cyclictest131rcu_preempt10:57:353
325369995113,6cyclictest131rcu_preempt11:34:153
325369995111,16cyclictest131rcu_preempt12:22:403
325367995135,13cyclictest364-21snmpd07:17:352
32536299517,5cyclictest131rcu_preempt11:35:361
32536299517,5cyclictest131rcu_preempt11:35:361
32536299513,9cyclictest23-21ksoftirqd/111:45:011
325362995135,12cyclictest1-21systemd07:50:011
32536299512,10cyclictest23-21ksoftirqd/112:13:331
32535899513,43cyclictest327929-21munin-run08:00:010
325358995126,21cyclictest3508521ssh09:54:190
325358995123,12cyclictest370481-21ssh10:39:550
325358995119,27cyclictest12-21ksoftirqd/010:28:550
325358995112,35cyclictest131rcu_preempt09:34:220
325358995112,35cyclictest131rcu_preempt09:34:220
32536999508,5cyclictest131rcu_preempt10:00:103
32536999505,8cyclictest37-21ksoftirqd/310:07:123
32536999505,13cyclictest350681-21ssh09:53:533
32536999504,7cyclictest37-21ksoftirqd/308:54:143
325367995036,10cyclictest330290-21munin-run08:45:012
325367995024,21cyclictest330126-21latency_hist08:40:022
325367995024,21cyclictest330126-21latency_hist08:40:022
32536299508,6cyclictest131rcu_preempt07:40:001
32536299508,6cyclictest131rcu_preempt07:40:001
32536299504,7cyclictest23-21ksoftirqd/110:34:101
325362995038,8cyclictest221rcuc/110:15:001
325362995014,5cyclictest131rcu_preempt11:59:151
325362995012,22cyclictest131rcu_preempt09:43:561
325362995012,22cyclictest131rcu_preempt09:43:561
32535899504,31cyclictest131rcu_preempt09:36:150
325358995025,9cyclictest363850-21ssh10:24:200
325358995011,35cyclictest131rcu_preempt09:25:110
32536999496,11cyclictest37-21ksoftirqd/310:54:353
325369994912,5cyclictest131rcu_preempt07:30:003
325367994936,9cyclictest328212-21munin-run08:05:012
325367994935,10cyclictest331074-21munin-run09:00:022
32536299497,11cyclictest23-21ksoftirqd/109:52:311
325362994939,7cyclictest221rcuc/112:21:531
32536299492,11cyclictest23-21ksoftirqd/111:20:011
325362994914,10cyclictest23-21ksoftirqd/111:14:561
325362994912,5cyclictest131rcu_preempt10:28:121
325362994911,5cyclictest131rcu_preempt08:55:021
325362994910,6cyclictest131rcu_preempt10:24:561
325358994941,4cyclictest16950irq/224-800000009:10:020
325358994916,16cyclictest335635-21ssh09:19:200
32535899491,42cyclictest387829-21munin-run11:20:000
32536999483,10cyclictest37-21ksoftirqd/308:20:013
325367994840,4cyclictest327960-21munin-run08:00:022
325367994838,6cyclictest331546-21sshd09:09:412
325367994835,10cyclictest364-21snmpd07:52:422
32536299489,6cyclictest131rcu_preempt07:35:021
32536299483,7cyclictest23-21ksoftirqd/109:25:591
32536299483,4cyclictest131rcu_preempt10:59:351
325362994831,12cyclictest275-21dbus-daemon10:40:011
32536299481,8cyclictest23-21ksoftirqd/110:00:551
325362994814,5cyclictest131rcu_preempt07:45:011
325362994814,10cyclictest23-21ksoftirqd/111:27:261
325362994813,5cyclictest131rcu_preempt12:33:101
325362994813,5cyclictest131rcu_preempt12:33:101
325362994813,5cyclictest131rcu_preempt09:38:311
325362994810,6cyclictest131rcu_preempt11:20:041
325362994810,6cyclictest131rcu_preempt09:33:521
325362994810,6cyclictest131rcu_preempt09:33:521
32535899484,5cyclictest131rcu_preempt10:12:360
325358994830,15cyclictest16850irq/134-800000012:17:010
325358994830,14cyclictest326889-21cat07:40:000
325358994830,14cyclictest326889-21cat07:40:000
32536999473,9cyclictest37-21ksoftirqd/307:33:093
32536999473,9cyclictest37-21ksoftirqd/307:13:413
325369994711,6cyclictest131rcu_preempt08:05:003
325367994735,9cyclictest330512-21sshd08:49:412
325367994733,10cyclictest364-21snmpd09:00:362
32536299476,5cyclictest131rcu_preempt09:57:191
325362994715,5cyclictest131rcu_preempt11:30:111
325362994711,6cyclictest131rcu_preempt10:19:501
325358994718,26cyclictest12-21ksoftirqd/007:20:020
325358994714,29cyclictest12-21ksoftirqd/007:27:370
32536999463,8cyclictest37-21ksoftirqd/307:55:023
325369994611,6cyclictest131rcu_preempt09:10:023
325369994611,5cyclictest131rcu_preempt08:00:013
325367994628,14cyclictest3467602-21systemd-journal07:28:542
32536299466,9cyclictest23-21ksoftirqd/109:20:551
32536299466,6cyclictest131rcu_preempt11:52:401
32536299466,5cyclictest131rcu_preempt09:46:491
32536299462,7cyclictest23-21ksoftirqd/111:46:171
325362994620,6cyclictest221rcuc/108:50:031
325362994610,6cyclictest131rcu_preempt09:00:001
325358994629,14cyclictest405926-21ssh12:01:310
325358994629,14cyclictest405926-21ssh12:01:300
32535899462,3cyclictest0-21swapper/010:44:350
325358994622,20cyclictest12-21ksoftirqd/011:46:510
32536999455,7cyclictest37-21ksoftirqd/308:49:073
32536999455,10cyclictest37-21ksoftirqd/308:30:003
325369994511,8cyclictest131rcu_preempt08:58:353
32536299458,5cyclictest131rcu_preempt09:14:461
325362994526,13cyclictest275-21dbus-daemon08:20:011
32536999449,5cyclictest131rcu_preempt08:31:413
32536999443,10cyclictest37-21ksoftirqd/307:17:413
325369994410,5cyclictest131rcu_preempt07:21:413
325369994410,5cyclictest131rcu_preempt07:21:413
325367994432,9cyclictest364-21snmpd07:45:552
32536299444,6cyclictest131rcu_preempt09:05:021
325362994431,10cyclictest3467602-21systemd-journal08:30:331
325362994427,13cyclictest3467602-21systemd-journal08:15:011
325362994419,6cyclictest221rcuc/108:05:011
325358994436,4cyclictest16950irq/224-800000009:10:230
325369994332,6cyclictest309-21in:imuxsock08:10:013
32536299439,5cyclictest131rcu_preempt07:20:001
325358994328,11cyclictest275-21dbus-daemon08:05:010
32535899432,36cyclictest0-21swapper/007:15:010
325358994315,13cyclictest367736-21ssh10:33:190
32536999426,9cyclictest37-21ksoftirqd/308:25:013
32536999425,6cyclictest37-21ksoftirqd/308:37:513
32536999425,6cyclictest37-21ksoftirqd/308:37:513
32536999424,6cyclictest131rcu_preempt08:43:173
325362994210,5cyclictest131rcu_preempt08:36:531
325362994210,5cyclictest131rcu_preempt08:36:531
325362994210,5cyclictest131rcu_preempt07:22:401
325362994210,5cyclictest131rcu_preempt07:22:391
32535899422,6cyclictest330024-21sendmail-msp08:40:000
32535899422,6cyclictest330024-21sendmail-msp08:39:590
32536999411,7cyclictest0-21swapper/307:45:353
325367994120,15cyclictest325437-21cat07:10:022
32536299415,5cyclictest131rcu_preempt08:00:011
325362994131,7cyclictest23-21ksoftirqd/107:27:411
32536299412,5cyclictest131rcu_preempt08:45:001
325362994122,13cyclictest275-21dbus-daemon08:30:001
325362994113,6cyclictest131rcu_preempt08:25:011
32535899412,6cyclictest328452-21munin-run08:10:010
325358994111,6cyclictest327140-21sshd07:43:410
325369994025,11cyclictest37-21ksoftirqd/308:15:013
32536299402,10cyclictest23-21ksoftirqd/108:10:011
325362993910,6cyclictest131rcu_preempt09:08:221
32535899392,9cyclictest327646-21sshd07:53:580
325358993928,6cyclictest19350irq/34-mmc108:45:000
325358993927,8cyclictest330509-21sshd08:49:130
325358993926,10cyclictest12-21ksoftirqd/007:50:020
325358993925,11cyclictest330746-21sshd08:50:410
32536299387,5cyclictest131rcu_preempt07:53:411
32535899382,7cyclictest331044-21munin-run09:00:010
325358993825,9cyclictest12-21ksoftirqd/007:34:290
32536299379,5cyclictest131rcu_preempt07:14:281
325358993726,8cyclictest12-21ksoftirqd/008:30:010
325362993614,9cyclictest131rcu_preempt07:10:021
325358993623,9cyclictest326097-21sshd07:20:410
325358993623,9cyclictest326097-21sshd07:20:410
32535899361,32cyclictest0-21swapper/008:23:500
325358993525,7cyclictest328850-21latency_hist08:15:030
325358993523,9cyclictest12-21ksoftirqd/008:33:040
325369992916,8cyclictest37-21ksoftirqd/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional