You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 16:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Mon Jun 08, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22129899115101,10cyclictest23-21ksoftirqd/110:45:021
2212939910041,55cyclictest16950irq/224-800000007:50:020
2212939910041,55cyclictest16950irq/224-800000007:50:020
22130499966,6cyclictest37-21ksoftirqd/310:05:023
221293998158,18cyclictest12-21ksoftirqd/008:35:020
221293998158,18cyclictest12-21ksoftirqd/008:35:010
22129399814,54cyclictest295237-21sh11:40:000
221293997928,40cyclictest12-21ksoftirqd/010:05:000
22129899761,8cyclictest131rcu_preempt07:35:021
221293997617,7cyclictest16850irq/134-800000009:43:410
221293997527,13cyclictest290199-21ssh11:28:560
22129399752,38cyclictest249211-21ssh09:57:560
221304997428,21cyclictest3590952-1kworker/3:2H10:40:033
221298997018,14cyclictest3467602-21systemd-journal11:10:011
22129899692,24cyclictest48-21kcompactd012:22:381
221293996917,17cyclictest16850irq/134-800000012:10:190
221298996825,40cyclictest3467562-21sshd08:56:121
221293996820,25cyclictest12-21ksoftirqd/012:04:450
221293996818,18cyclictest16850irq/134-800000009:29:420
221293996818,17cyclictest16850irq/134-800000010:27:020
221293996726,10cyclictest292667-21ssh11:34:260
221304996561,2cyclictest0-21swapper/312:20:013
221302996530,7cyclictest224615-21kworker/2:110:00:022
221298996521,5cyclictest226603-21latency_hist08:50:031
221298996521,5cyclictest226603-21latency_hist08:50:021
22129399657,23cyclictest221292-21latency_hist07:10:010
221293996557,3cyclictest0-21swapper/009:00:020
221298996420,5cyclictest293972-21sh11:37:081
221298996318,6cyclictest23-21ksoftirqd/111:25:161
221298996223,5cyclictest292016-21sh11:32:481
221298996219,5cyclictest250589-21sh10:00:331
221293996228,9cyclictest12-21ksoftirqd/012:10:010
221293996228,9cyclictest12-21ksoftirqd/012:10:010
221302996140,8cyclictest256928-21kworker/2:311:37:312
22129899613,17cyclictest223317-21munin-run07:50:011
22129899613,17cyclictest223317-21munin-run07:50:001
221298996120,4cyclictest260987-21sh10:23:501
22129399611,44cyclictest282209-21ssh11:10:460
221298996021,5cyclictest227467-21diskmemload12:13:341
221293996031,14cyclictest285729-21ssh11:18:560
22129899592,19cyclictest233305-21ssh09:22:411
221298995918,5cyclictest296698-21sh11:43:131
221298995917,13cyclictest272694-21munin-run10:50:011
22129899591,20cyclictest1-21systemd10:10:001
221293995923,26cyclictest12-21ksoftirqd/012:20:010
221293995920,19cyclictest309515-21kworker/u8:112:21:200
221298995826,28cyclictest23-21ksoftirqd/109:37:551
221298995821,5cyclictest245247-21sh09:49:121
221298995817,7cyclictest0-21swapper/110:57:581
221293995829,13cyclictest256763-21ssh10:14:420
221293995829,13cyclictest256763-21ssh10:14:420
221298995725,29cyclictest23-21ksoftirqd/112:25:261
221298995725,29cyclictest23-21ksoftirqd/112:25:261
221298995722,11cyclictest23-21ksoftirqd/111:11:361
22129899572,19cyclictest241357-21ssh09:40:151
221298995720,4cyclictest274-21cron08:40:001
221298995715,9cyclictest23-21ksoftirqd/112:37:161
221293995727,15cyclictest298898-21ssh11:47:590
221293995727,13cyclictest319964-21ssh12:35:000
221302995647,6cyclictest256928-21kworker/2:310:57:282
221298995625,27cyclictest23-21ksoftirqd/111:02:131
221298995624,28cyclictest23-21ksoftirqd/107:55:021
22129899562,18cyclictest227213-21munin-plugin-st09:05:011
22129899562,18cyclictest227213-21munin-plugin-st09:05:001
221293995622,16cyclictest23425450rm09:25:010
22129399561,32cyclictest131rcu_preempt11:40:280
221293995610,19cyclictest131rcu_preempt10:16:350
221302995544,6cyclictest256928-21kworker/2:310:47:112
221302995541,11cyclictest3467602-21systemd-journal08:50:012
221302995541,11cyclictest3467602-21systemd-journal08:50:002
221298995522,6cyclictest23-21ksoftirqd/111:21:031
221298995522,6cyclictest23-21ksoftirqd/111:21:031
22129899552,14cyclictest231971-21modprobe09:20:001
221293995539,10cyclictest3467602-21systemd-journal12:00:010
221293995522,24cyclictest0-21swapper/010:58:180
221293995514,24cyclictest0-21swapper/011:53:460
221293995512,13cyclictest1-21systemd08:40:010
221298995421,5cyclictest225855-21latency_hist08:35:021
221298995419,8cyclictest23-21ksoftirqd/109:54:401
221298995418,5cyclictest300250-21sh11:50:531
221298995418,4cyclictest225121-21sshd08:23:121
221298995413,8cyclictest1-21systemd12:05:001
221293995426,13cyclictest246659-21ssh09:52:080
221304995315,5cyclictest131rcu_preempt12:00:013
221302995345,5cyclictest226994-21kworker/2:309:36:532
221302995337,12cyclictest1-21systemd08:55:002
221302995334,14cyclictest3467602-21systemd-journal09:45:002
22129899532,17cyclictest275103-21ssh10:55:031
221298995321,12cyclictest23-21ksoftirqd/107:45:021
221298995321,10cyclictest23-21ksoftirqd/107:15:001
221298995317,9cyclictest23-21ksoftirqd/110:38:511
22129399532,43cyclictest0-21swapper/011:25:010
22129399532,43cyclictest0-21swapper/011:25:010
221293995322,14cyclictest241038-21ssh09:40:010
221293995318,17cyclictest16850irq/134-800000011:05:500
22129399531,3cyclictest131rcu_preempt10:50:020
22130499527,5cyclictest131rcu_preempt09:37:553
221304995238,10cyclictest37-21ksoftirqd/308:50:023
221304995213,7cyclictest131rcu_preempt09:28:583
221302995244,5cyclictest247819-21kworker/2:010:27:472
221302995243,5cyclictest268232-21kworker/2:112:30:302
22129899527,23cyclictest223800-21kworker/u8:308:15:011
22129899522,17cyclictest264441-21ssh10:31:231
221298995216,9cyclictest23-21ksoftirqd/112:06:221
221298995216,9cyclictest23-21ksoftirqd/112:06:221
221298995213,11cyclictest23-21ksoftirqd/109:31:561
221293995225,7cyclictest0-21swapper/010:09:540
22129399522,15cyclictest275-21dbus-daemon10:55:010
22129399521,4cyclictest320992-21ssh12:36:570
221304995129,16cyclictest275-21dbus-daemon12:10:013
221304995129,16cyclictest275-21dbus-daemon12:10:003
221302995143,5cyclictest256928-21kworker/2:311:48:312
221302995143,5cyclictest247819-21kworker/2:010:08:452
221302995142,6cyclictest268232-21kworker/2:112:36:102
221298995121,11cyclictest23-21ksoftirqd/109:12:321
221298995121,11cyclictest23-21ksoftirqd/109:12:311
221298995116,8cyclictest23-21ksoftirqd/110:26:501
221293995118,6cyclictest16850irq/134-800000012:25:390
221293995118,6cyclictest16850irq/134-800000012:25:390
221293995116,18cyclictest16850irq/134-800000010:24:050
22129399511,3cyclictest131rcu_preempt10:45:020
22130499507,5cyclictest131rcu_preempt11:00:583
221304995039,7cyclictest37-21ksoftirqd/308:35:023
22130499501,8cyclictest37-21ksoftirqd/310:35:033
22130499501,19cyclictest0-21swapper/311:15:483
221302995042,5cyclictest299697-21kworker/2:212:15:142
221302995042,5cyclictest247819-21kworker/2:010:11:272
221302995042,5cyclictest247819-21kworker/2:010:11:262
221302995041,5cyclictest299697-21kworker/2:212:05:542
221302995041,5cyclictest299697-21kworker/2:212:05:542
221302995041,5cyclictest256928-21kworker/2:311:10:482
221298995021,4cyclictest224955-21latency_hist08:20:021
221298995019,27cyclictest227467-21diskmemload09:28:441
221298995017,7cyclictest23-21ksoftirqd/110:18:061
221298995014,9cyclictest23-21ksoftirqd/110:13:361
221298995014,9cyclictest23-21ksoftirqd/110:13:361
22129899501,30cyclictest221rcuc/111:20:021
221293995020,7cyclictest266682-21ssh10:36:180
221293995016,6cyclictest16850irq/134-800000009:11:210
221293995016,6cyclictest16850irq/134-800000009:11:200
221304994930,13cyclictest275-21dbus-daemon08:00:003
221304994928,15cyclictest3467602-21systemd-journal10:15:013
221304994928,15cyclictest3467602-21systemd-journal10:15:003
221304994916,11cyclictest37-21ksoftirqd/307:35:023
221304994916,11cyclictest37-21ksoftirqd/307:35:013
221304994913,6cyclictest131rcu_preempt11:09:593
221302994942,4cyclictest299697-21kworker/2:212:04:092
221302994942,4cyclictest247819-21kworker/2:010:21:162
221302994942,4cyclictest226994-21kworker/2:309:23:582
221302994941,5cyclictest268232-21kworker/2:111:53:202
221302994941,5cyclictest224615-21kworker/2:109:52:112
221302994940,6cyclictest268232-21kworker/2:112:29:132
221302994940,6cyclictest268232-21kworker/2:112:29:132
221298994917,7cyclictest23-21ksoftirqd/107:17:351
221298994917,7cyclictest23-21ksoftirqd/107:17:341
221298994916,8cyclictest23-21ksoftirqd/111:46:101
221293994920,25cyclictest12-21ksoftirqd/008:48:120
221293994920,25cyclictest12-21ksoftirqd/008:48:110
221304994835,9cyclictest37-21ksoftirqd/309:50:013
221304994833,11cyclictest3467602-21systemd-journal10:35:033
221304994812,6cyclictest131rcu_preempt09:35:013
221304994812,22cyclictest227467-21diskmemload11:45:023
221302994842,3cyclictest224615-21kworker/2:109:10:332
221302994842,3cyclictest224615-21kworker/2:109:10:332
221302994840,5cyclictest299697-21kworker/2:212:11:022
221302994840,5cyclictest268232-21kworker/2:110:52:122
221302994840,5cyclictest256928-21kworker/2:311:07:572
221302994840,5cyclictest256928-21kworker/2:311:00:482
221302994840,5cyclictest256928-21kworker/2:310:36:202
221302994840,5cyclictest226994-21kworker/2:309:16:042
221302994839,5cyclictest256928-21kworker/2:311:24:322
221302994839,5cyclictest256928-21kworker/2:311:24:322
221302994833,11cyclictest3467602-21systemd-journal08:45:012
22129399485,7cyclictest12-21ksoftirqd/010:30:130
221293994816,24cyclictest278875-21rm11:03:410
221293994816,22cyclictest230678-21rm09:16:550
22130499479,14cyclictest37-21ksoftirqd/311:40:013
22130499476,6cyclictest131rcu_preempt08:25:013
22130499475,9cyclictest37-21ksoftirqd/312:31:243
221304994738,7cyclictest37-21ksoftirqd/307:30:003
221304994737,8cyclictest37-21ksoftirqd/311:14:573
221304994737,7cyclictest37-21ksoftirqd/309:17:333
22130499471,5cyclictest131rcu_preempt11:24:573
22130499471,5cyclictest131rcu_preempt11:24:573
221304994711,6cyclictest131rcu_preempt12:40:003
221304994711,6cyclictest131rcu_preempt11:25:073
221302994741,3cyclictest226994-21kworker/2:309:26:062
221302994739,5cyclictest256928-21kworker/2:311:16:372
221302994739,5cyclictest256928-21kworker/2:310:34:082
221302994739,5cyclictest247819-21kworker/2:010:19:492
22130299472,14cyclictest303753-21ssh11:59:002
221298994718,7cyclictest23-21ksoftirqd/108:54:191
221298994715,8cyclictest23-21ksoftirqd/112:32:271
221298994713,8cyclictest309-21in:imuxsock08:05:001
22130499464,9cyclictest37-21ksoftirqd/308:17:133
22130499463,5cyclictest131rcu_preempt09:43:463
221304994635,8cyclictest37-21ksoftirqd/311:50:063
221304994610,5cyclictest131rcu_preempt12:30:023
221304994610,5cyclictest131rcu_preempt12:30:013
221302994638,5cyclictest256928-21kworker/2:311:28:022
221302994638,5cyclictest226994-21kworker/2:309:31:592
221302994638,5cyclictest224615-21kworker/2:109:48:262
22130499455,7cyclictest37-21ksoftirqd/312:10:393
22130499453,7cyclictest37-21ksoftirqd/310:05:183
221304994514,5cyclictest131rcu_preempt10:20:023
221304994513,6cyclictest131rcu_preempt11:00:013
221304994512,5cyclictest131rcu_preempt10:55:003
221304994512,28cyclictest37-21ksoftirqd/310:30:013
221304994511,7cyclictest131rcu_preempt07:45:023
221302994537,5cyclictest268232-21kworker/2:112:20:152
221302994529,12cyclictest275-21dbus-daemon08:15:012
221293994524,17cyclictest237326-21ssh09:31:330
22129399452,38cyclictest1-21systemd08:45:000
22130499447,7cyclictest37-21ksoftirqd/307:40:003
22130499447,11cyclictest37-21ksoftirqd/309:25:003
221304994434,8cyclictest37-21ksoftirqd/307:47:123
221304994434,8cyclictest37-21ksoftirqd/307:47:123
221304994432,8cyclictest37-21ksoftirqd/308:10:003
22130499441,7cyclictest0-21swapper/309:14:233
22130499441,7cyclictest0-21swapper/309:14:223
221304994414,6cyclictest131rcu_preempt11:32:163
221304994410,6cyclictest131rcu_preempt09:50:353
221302994436,5cyclictest256928-21kworker/2:311:41:392
221302994436,5cyclictest220821-21kworker/2:307:24:432
221302994435,5cyclictest224615-21kworker/2:110:03:002
22129399442,38cyclictest226783-21ls08:55:020
221304994330,10cyclictest37-21ksoftirqd/308:40:003
221304994329,11cyclictest37-21ksoftirqd/312:00:493
221304994329,11cyclictest37-21ksoftirqd/311:45:193
22130499431,5cyclictest0-21swapper/310:25:013
221302994335,5cyclictest220821-21kworker/2:307:13:032
22130299433,3cyclictest131rcu_preempt08:05:032
221302994329,10cyclictest3467602-21systemd-journal08:35:022
221298994329,11cyclictest3467602-21systemd-journal11:59:591
221298994324,15cyclictest223827-21run-parts08:00:001
221298994318,6cyclictest23-21ksoftirqd/109:59:501
221293994330,9cyclictest3467602-21systemd-journal07:15:030
221293994330,9cyclictest3467602-21systemd-journal07:15:030
221293994327,13cyclictest244056-21ssh09:46:200
221293994322,16cyclictest275-21dbus-daemon08:25:000
221293994317,22cyclictest12-21ksoftirqd/007:40:020
22130499427,7cyclictest131rcu_preempt07:20:023
22130499427,7cyclictest131rcu_preempt07:20:013
221304994231,8cyclictest37-21ksoftirqd/312:25:023
221304994215,5cyclictest131rcu_preempt10:45:083
221304994212,5cyclictest131rcu_preempt09:59:093
221302994235,4cyclictest224615-21kworker/2:109:05:392
221302994234,5cyclictest256928-21kworker/2:310:40:182
221302994234,5cyclictest224615-21kworker/2:108:30:092
221302994234,5cyclictest224615-21kworker/2:108:30:092
22130299423,3cyclictest131rcu_preempt11:33:592
221302994232,6cyclictest222774-21cron07:40:002
221302994226,12cyclictest275-21dbus-daemon07:50:012
221302994226,12cyclictest275-21dbus-daemon07:50:012
221293994229,9cyclictest3467602-21systemd-journal07:58:120
22129399421,9cyclictest227590-21latency_hist09:10:020
22130499414,8cyclictest37-21ksoftirqd/308:30:023
22130499414,8cyclictest37-21ksoftirqd/308:30:013
22130499413,13cyclictest37-21ksoftirqd/308:44:593
221304994120,16cyclictest275-21dbus-daemon07:25:013
22130499411,5cyclictest131rcu_preempt08:50:023
22130499411,5cyclictest131rcu_preempt08:50:023
221298994123,13cyclictest275-21dbus-daemon08:45:011
22129399412,6cyclictest221628-21wc07:15:010
221293994125,12cyclictest224081-21sshd08:03:130
221304994028,8cyclictest37-21ksoftirqd/309:10:013
221302994030,6cyclictest309-21in:imuxsock08:10:002
221302994030,5cyclictest309-21in:imuxsock09:00:002
221302994028,9cyclictest275-21dbus-daemon07:20:012
221302994028,9cyclictest275-21dbus-daemon07:20:012
221298994026,11cyclictest23-21ksoftirqd/112:15:241
22129899401,4cyclictest131rcu_preempt07:29:591
221298994012,11cyclictest0-21swapper/108:25:361
221298994012,11cyclictest0-21swapper/108:25:351
22130499399,5cyclictest131rcu_preempt09:05:023
22130499399,5cyclictest131rcu_preempt09:05:013
221302993931,5cyclictest220821-21kworker/2:307:10:012
221293993932,5cyclictest19350irq/34-mmc107:55:020
221293993927,9cyclictest224905-21pluginstate08:20:020
22129399392,6cyclictest225396-21munin-run08:30:010
22129399392,6cyclictest225396-21munin-run08:30:010
22129399391,34cyclictest223160-21latency_hist07:45:020
221304993815,19cyclictest361rcuc/308:35:033
221302993811,4cyclictest131rcu_preempt08:19:412
221293993826,9cyclictest3467602-21systemd-journal08:06:120
221293993821,11cyclictest1-21systemd07:25:000
22129399381,34cyclictest0-21swapper/007:30:010
221293993813,16cyclictest0-21swapper/008:10:310
221304993728,6cyclictest37-21ksoftirqd/307:10:023
221304993727,8cyclictest37-21ksoftirqd/307:55:023
221304993719,13cyclictest37-21ksoftirqd/308:14:593
221304993710,6cyclictest131rcu_preempt09:00:013
221302993727,7cyclictest30-21ksoftirqd/208:00:012
221298993710,6cyclictest131rcu_preempt07:35:011
221298993710,6cyclictest131rcu_preempt07:35:011
22129399373,5cyclictest222513-21sshd07:34:120
22129399373,5cyclictest222513-21sshd07:34:110
221302993625,7cyclictest30-21ksoftirqd/208:24:112
22129899368,5cyclictest131rcu_preempt08:10:001
221298993622,11cyclictest221296-21grep07:10:001
22130499356,5cyclictest131rcu_preempt08:04:583
221304993524,7cyclictest37-21ksoftirqd/307:15:003
221302993525,7cyclictest30-21ksoftirqd/208:30:032
221302993525,7cyclictest30-21ksoftirqd/208:30:022
221302993521,11cyclictest30-21ksoftirqd/209:05:022
221302993521,11cyclictest30-21ksoftirqd/209:05:012
221298993520,11cyclictest275-21dbus-daemon08:35:011
221293993519,4cyclictest19350irq/34-mmc109:05:030
221293993519,4cyclictest19350irq/34-mmc109:05:020
221302993422,8cyclictest275-21dbus-daemon07:55:002
221302993323,7cyclictest222276-21modprobe07:30:002
221302993223,7cyclictest30-21ksoftirqd/207:45:002
221302993221,6cyclictest0-21swapper/207:35:012
221302993221,6cyclictest0-21swapper/207:35:002
22129899323,3cyclictest131rcu_preempt07:25:021
221298993019,8cyclictest227489-21sh09:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional