You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 10:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon May 11, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3588703999285,3cyclictest0-21swapper/021:15:020
3588703999285,3cyclictest0-21swapper/021:15:020
3588703998884,2cyclictest0-21swapper/023:45:010
3588703998528,39cyclictest3602131-21ssh21:25:180
3588703998324,41cyclictest3686471-21ssh00:23:500
3588703998324,41cyclictest3686471-21ssh00:23:490
358870399815,58cyclictest0-21swapper/022:13:200
358870399811,78cyclictest0-21swapper/019:45:020
358870399811,78cyclictest0-21swapper/019:45:010
358870399801,65cyclictest3692944-21ssh00:37:360
3588703998016,58cyclictest16850irq/134-800000021:50:100
3588703998016,58cyclictest16850irq/134-800000021:50:090
3588703997822,51cyclictest3680783-21ssh00:11:260
3588703997715,57cyclictest16850irq/134-800000000:28:200
3588703997521,34cyclictest3637276-21ssh22:39:440
3588703997414,55cyclictest16850irq/134-800000022:44:420
3588703997414,55cyclictest16850irq/134-800000022:44:410
358870399733,64cyclictest0-21swapper/021:43:260
3588703997330,39cyclictest3691181-21ssh00:34:000
358870399732,55cyclictest3674118-21ssh23:57:290
3588717997229,13cyclictest150-21pr/ttyS221:10:063
3588717997229,13cyclictest150-21pr/ttyS221:10:053
358870399728,5cyclictest16850irq/134-800000021:37:160
3588703997245,14cyclictest16850irq/134-800000022:10:030
358870399721,15cyclictest3657307-21ssh23:21:580
358870399719,13cyclictest3671588-21ssh23:52:040
3588703997130,34cyclictest275-21dbus-daemon23:20:000
3588703997123,44cyclictest3684512-21ssh00:19:460
358870399703,7cyclictest3593823-21sshd20:48:260
358870899696,45cyclictest3663494-21ssh23:35:021
3588703996929,22cyclictest3670244-21ssh23:49:350
3588703996912,6cyclictest16850irq/134-800000022:04:500
3588703996817,35cyclictest0-21swapper/022:55:230
358870399681,63cyclictest0-21swapper/022:18:050
3588703996811,6cyclictest16850irq/134-800000022:50:220
3588703996811,6cyclictest16850irq/134-800000022:50:210
358870399673,16cyclictest3606565-21cron21:35:010
3588703996729,21cyclictest3660267-21ssh23:28:190
3588703996720,30cyclictest3617319-21ssh21:57:440
3588717996549,14cyclictest3588699-21cyclictest22:50:023
3588703996519,42cyclictest16850irq/134-800000023:01:370
3588703996519,42cyclictest16850irq/134-800000023:01:370
3588703996518,28cyclictest16850irq/134-800000023:06:040
358870399651,49cyclictest0-21swapper/022:47:520
358870399651,3cyclictest3632646-21systemd-run22:30:000
358870399651,2cyclictest0-21swapper/021:21:330
358870899641,48cyclictest3687003-21ssh00:25:021
358870899641,48cyclictest3687003-21ssh00:25:011
358870399646,12cyclictest3634221-21ssh22:33:150
3588703996434,26cyclictest3591731-21sshd20:05:250
358870399641,8cyclictest3613660-21latency_hist21:50:020
3588703996417,30cyclictest0-21swapper/023:31:120
3588703996215,42cyclictest3675970-21ssh00:01:130
3588703996214,32cyclictest3679560-21ssh00:09:090
3588717996147,10cyclictest48-21kcompactd023:24:363
3588708996140,12cyclictest23-21ksoftirqd/122:50:021
3588703996120,38cyclictest3597585-21ssh21:15:250
358870399602,54cyclictest3467602-21systemd-journal20:54:270
3588703996024,33cyclictest0-21swapper/023:37:070
358870399602,18cyclictest3594907-21systemd-run21:10:020
3588717995946,9cyclictest48-21kcompactd020:31:283
3588703995926,29cyclictest0-21swapper/019:55:000
3588703995915,39cyclictest1-21systemd20:35:010
3588703995911,14cyclictest3628468-21ssh22:20:510
3588717995845,9cyclictest48-21kcompactd019:28:143
3588717995845,9cyclictest48-21kcompactd019:28:133
3588717995812,14cyclictest3594753-21cat21:05:013
3588713995847,6cyclictest3635009-21kworker/2:100:04:502
358870399582,52cyclictest3592897-21latency_hist20:30:030
358870399582,15cyclictest3654057-21cat23:15:030
3588717995739,14cyclictest1-21systemd20:55:003
3588717995712,13cyclictest3594884-21diskmemload21:19:543
358871799570,19cyclictest3685257-21ssh00:20:473
358871799570,19cyclictest3685257-21ssh00:20:473
3588713995748,6cyclictest3601810-21kworker/2:221:36:522
3588713995734,18cyclictest3467602-21systemd-journal21:56:252
3588703995725,27cyclictest16950irq/224-800000021:00:010
3588717995644,8cyclictest48-21kcompactd021:49:523
3588717995643,9cyclictest48-21kcompactd000:11:583
3588713995647,5cyclictest3594127-21kworker/2:122:08:392
3588713995646,6cyclictest3635009-21kworker/2:123:22:422
3588717995520,10cyclictest37-21ksoftirqd/323:53:233
358871799551,17cyclictest3643416-21rm22:52:293
358871799551,17cyclictest3643416-21rm22:52:283
3588713995547,5cyclictest3635009-21kworker/2:100:14:462
3588713995547,5cyclictest3627991-21kworker/2:000:31:492
3588713995546,6cyclictest3627991-21kworker/2:022:37:252
3588713995546,6cyclictest3601810-21kworker/2:222:29:042
3588713995546,6cyclictest3594127-21kworker/2:122:15:442
3588703995532,21cyclictest0-21swapper/019:55:270
358870399552,13cyclictest3592535-21sshd20:22:270
3588717995441,9cyclictest48-21kcompactd021:29:473
3588713995446,5cyclictest3594127-21kworker/2:121:25:002
3588713995445,6cyclictest3635009-21kworker/2:100:29:052
3588713995445,6cyclictest3601810-21kworker/2:221:32:572
3588713995444,7cyclictest3635009-21kworker/2:100:21:442
3588713995444,7cyclictest3635009-21kworker/2:100:21:442
3588713995443,7cyclictest3594127-21kworker/2:121:15:432
3588713995443,6cyclictest3635009-21kworker/2:123:16:372
358870399543,8cyclictest3589962-21unin-run19:35:010
358870399542,8cyclictest275-21dbus-daemon21:05:000
3588703995413,34cyclictest0-21swapper/019:19:590
358870399541,16cyclictest3592282-21cron20:19:590
358870399541,14cyclictest3588880-21sshd19:12:270
3588717995333,16cyclictest3467602-21systemd-journal00:35:013
3588713995345,5cyclictest3635009-21kworker/2:123:39:402
3588713995345,5cyclictest3627991-21kworker/2:022:56:012
3588713995345,5cyclictest3601810-21kworker/2:222:23:012
3588713995344,6cyclictest3635009-21kworker/2:123:45:582
3588713995344,6cyclictest3601810-21kworker/2:222:31:322
3588713995344,5cyclictest3594127-21kworker/2:122:04:462
358870399533,45cyclictest3593342-21sendmail-msp20:40:010
3588713995244,5cyclictest3635009-21kworker/2:123:59:222
3588713995244,5cyclictest3635009-21kworker/2:123:03:202
3588713995244,5cyclictest3635009-21kworker/2:123:03:192
3588713995244,5cyclictest3592046-21kworker/2:321:10:022
3588713995244,5cyclictest3576158-21kworker/2:119:50:012
3588713995243,6cyclictest3627991-21kworker/2:000:37:452
3588713995243,5cyclictest3635009-21kworker/2:100:20:002
3588713995235,13cyclictest275-21dbus-daemon21:30:012
358870399521,8cyclictest3591529-21unin-run20:05:000
3588717995135,11cyclictest120-21kswapd000:40:013
358871799511,20cyclictest3683826-21sh00:18:053
358871799511,16cyclictest3634103-21rm22:32:543
3588713995144,5cyclictest3594127-21kworker/2:122:11:442
3588713995143,5cyclictest3635009-21kworker/2:123:50:512
3588713995143,5cyclictest3635009-21kworker/2:100:07:462
3588713995143,5cyclictest3627991-21kworker/2:023:10:412
3588713995143,5cyclictest3627991-21kworker/2:023:06:452
3588713995143,5cyclictest3627991-21kworker/2:022:53:322
3588713995143,5cyclictest3627991-21kworker/2:022:53:322
3588713995142,6cyclictest3601810-21kworker/2:221:51:022
3588713995142,6cyclictest3601810-21kworker/2:221:51:022
3588713995141,6cyclictest3635009-21kworker/2:123:30:282
358870399513,7cyclictest3590365-21latency_hist19:40:010
358870399512,44cyclictest0-21swapper/019:10:020
3588717995043,4cyclictest3601635-21ssh21:24:343
3588717995037,9cyclictest48-21kcompactd020:23:423
3588717995030,14cyclictest3625516-21sh22:15:023
358871799502,12cyclictest3611337-21latency_hist21:45:013
358871799500,14cyclictest3653745-21ssh23:14:353
3588713995042,5cyclictest3635009-21kworker/2:123:40:232
3588713995042,5cyclictest3627991-21kworker/2:022:44:592
3588713995042,5cyclictest3627991-21kworker/2:022:44:592
3588713995042,5cyclictest3601810-21kworker/2:221:47:292
3588713995041,6cyclictest3627991-21kworker/2:022:48:432
3588713995041,6cyclictest3592046-21kworker/2:321:10:482
3588713995041,6cyclictest3592046-21kworker/2:321:10:472
3588713995041,6cyclictest3576158-21kworker/2:120:00:002
3588713995028,16cyclictest3467602-21systemd-journal19:19:592
3588708995032,12cyclictest275-21dbus-daemon23:20:011
358871799498,13cyclictest364-21snmpd23:32:143
3588717994937,9cyclictest3667403-21sshd23:43:263
358871799491,15cyclictest3589607-21latency_hist19:25:023
358871799491,14cyclictest3638751-21sh22:42:393
358871799491,14cyclictest3638751-21sh22:42:383
358871799490,15cyclictest3687906-21ssh00:26:363
358871799490,15cyclictest3607031-21ssh21:35:443
3588713994941,5cyclictest3635009-21kworker/2:123:25:542
358870399492,3cyclictest0-21swapper/019:25:020
358870399491,15cyclictest3593575-21sshd20:43:260
358870399491,15cyclictest3589675-21sshd19:28:260
358870399491,15cyclictest3589675-21sshd19:28:250
3588717994839,7cyclictest150-21pr/ttyS220:48:353
3588717994836,9cyclictest3654153-21ssh23:15:103
3588717994836,9cyclictest3617029-21ssh21:56:583
3588717994833,10cyclictest48-21kcompactd019:33:133
3588717994833,10cyclictest3467602-21systemd-journal23:50:023
3588717994831,13cyclictest275-21dbus-daemon22:30:003
3588713994840,6cyclictest3601810-21kworker/2:221:42:372
3588713994840,5cyclictest3592046-21kworker/2:320:20:012
3588717994737,8cyclictest150-21pr/ttyS222:57:143
3588717994736,8cyclictest150-21pr/ttyS223:01:273
3588717994736,8cyclictest150-21pr/ttyS223:01:263
3588717994734,10cyclictest3678735-21ssh00:07:163
3588713994739,5cyclictest3592046-21kworker/2:320:45:142
3588713994739,5cyclictest3576158-21kworker/2:119:23:412
3588713994738,5cyclictest3592046-21kworker/2:320:25:022
3588713994738,5cyclictest3576158-21kworker/2:119:31:242
3588708994732,12cyclictest275-21dbus-daemon19:25:011
3588708994729,13cyclictest3594362-21systemctl21:00:001
3588708994722,12cyclictest275-21dbus-daemon21:15:011
3588708994722,12cyclictest275-21dbus-daemon21:15:011
358870399472,13cyclictest3590708-21sshd19:47:260
3588717994635,9cyclictest3659629-21ssh23:26:473
3588717994635,9cyclictest150-21pr/ttyS222:05:283
3588717994634,9cyclictest3635410-21ssh22:35:273
3588717994634,8cyclictest3467602-21systemd-journal21:34:253
3588717994634,7cyclictest3651611-21latency_hist23:10:013
3588717994633,10cyclictest150-21pr/ttyS219:54:303
3588717994632,11cyclictest150-21pr/ttyS220:26:533
3588717994632,11cyclictest150-21pr/ttyS219:57:273
358871799460,17cyclictest3657336-21kworker/u8:123:39:333
3588713994638,5cyclictest3592046-21kworker/2:321:01:182
3588713994638,5cyclictest3592046-21kworker/2:320:30:112
3588713994638,5cyclictest3576158-21kworker/2:119:35:462
3588708994636,6cyclictest3591767-21munin-plugin-st20:10:001
3588708994633,10cyclictest1-21systemd21:05:011
3588708994631,11cyclictest3467602-21systemd-journal22:21:251
3588708994627,14cyclictest309-21in:imuxsock22:35:011
3588717994535,8cyclictest150-21pr/ttyS220:40:123
3588717994534,9cyclictest3673309-21ssh23:55:373
3588717994534,9cyclictest150-21pr/ttyS220:58:543
3588717994534,8cyclictest150-21pr/ttyS219:38:283
3588717994533,9cyclictest3619755-21ssh22:02:443
3588717994531,11cyclictest150-21pr/ttyS220:16:283
3588713994537,5cyclictest3576158-21kworker/2:119:41:292
3588713994537,5cyclictest3576158-21kworker/2:119:41:292
3588713994537,5cyclictest3576158-21kworker/2:119:12:192
3588708994532,10cyclictest3467602-21systemd-journal19:31:261
3588708994529,11cyclictest309-21in:imuxsock20:10:021
358870899451,29cyclictest0-21swapper/100:00:031
358870399451,12cyclictest3591983-21modprobe20:10:270
3588717994439,3cyclictest3628328-21ssh22:20:333
3588717994433,9cyclictest150-21pr/ttyS219:45:103
3588717994433,8cyclictest150-21pr/ttyS221:52:313
3588717994433,8cyclictest150-21pr/ttyS221:52:303
3588717994433,8cyclictest150-21pr/ttyS220:00:233
3588717994432,9cyclictest150-21pr/ttyS220:38:593
3588717994432,9cyclictest150-21pr/ttyS220:10:063
3588717994432,9cyclictest150-21pr/ttyS219:10:473
3588717994432,10cyclictest150-21pr/ttyS222:15:183
3588717994431,10cyclictest150-21pr/ttyS220:06:443
3588717994429,11cyclictest3467602-21systemd-journal19:40:023
3588717994429,11cyclictest3467602-21systemd-journal19:40:023
358871799442,12cyclictest3594884-21diskmemload00:01:003
3588713994437,4cyclictest3576158-21kworker/2:119:50:262
3588713994436,5cyclictest3592046-21kworker/2:320:40:202
3588713994436,5cyclictest3576158-21kworker/2:120:05:262
3588713994435,6cyclictest3592046-21kworker/2:320:56:432
3588713994435,6cyclictest3576158-21kworker/2:120:02:202
3588708994431,9cyclictest1-21systemd20:05:011
3588717994333,8cyclictest150-21pr/ttyS221:05:263
3588717994330,10cyclictest150-21pr/ttyS219:15:573
3588713994336,4cyclictest3592046-21kworker/2:320:28:542
3588708994331,7cyclictest3604335-21sh21:30:011
3588708994329,9cyclictest3639788-21munin-plugin-st22:45:011
3588708994329,9cyclictest3639788-21munin-plugin-st22:45:001
3588713994234,5cyclictest3592046-21kworker/2:320:52:322
3588713994234,5cyclictest3576158-21kworker/2:120:14:172
3588708994232,7cyclictest23-21ksoftirqd/122:25:031
3588708994232,6cyclictest3636261-21ssh22:37:261
3588708994230,9cyclictest3614582-21sshd21:51:571
3588708994230,9cyclictest3614582-21sshd21:51:561
3588708994225,13cyclictest3689462-21date00:30:021
3588708994221,15cyclictest275-21dbus-daemon19:55:001
3588713994134,4cyclictest3592046-21kworker/2:320:35:202
358870899416,20cyclictest23-21ksoftirqd/123:50:021
358870899413,3cyclictest131rcu_preempt22:50:131
358870899413,3cyclictest131rcu_preempt22:50:121
3588708994132,7cyclictest23-21ksoftirqd/123:24:141
3588708994130,8cyclictest23-21ksoftirqd/100:10:001
3588708994129,8cyclictest3467602-21systemd-journal23:09:271
3588708994125,12cyclictest23-21ksoftirqd/100:19:071
3588708994125,11cyclictest275-21dbus-daemon23:40:011
3588708994028,10cyclictest23-21ksoftirqd/122:04:381
3588708994027,10cyclictest23-21ksoftirqd/122:17:171
3588708994021,13cyclictest3467602-21systemd-journal19:45:011
3588708994021,13cyclictest3467602-21systemd-journal19:45:001
358870899401,3cyclictest131rcu_preempt21:44:401
3588713993925,10cyclictest3589683-21cron19:30:012
3588713993925,10cyclictest3589683-21cron19:30:002
358870899394,3cyclictest131rcu_preempt23:27:441
3588708993931,5cyclictest3593032-21sshd20:31:271
3588708993929,6cyclictest23-21ksoftirqd/120:45:011
3588708993927,9cyclictest23-21ksoftirqd/121:19:101
3588708993924,11cyclictest23-21ksoftirqd/100:39:241
3588708993923,11cyclictest3467602-21systemd-journal19:15:021
3588708993923,11cyclictest309-21in:imuxsock20:40:001
3588708993921,13cyclictest275-21dbus-daemon21:50:011
3588708993920,13cyclictest275-21dbus-daemon19:30:011
3588708993920,13cyclictest275-21dbus-daemon19:30:011
358870899384,3cyclictest131rcu_preempt00:35:001
3588708993825,10cyclictest23-21ksoftirqd/123:10:051
358870899381,4cyclictest131rcu_preempt23:04:081
358870899381,4cyclictest131rcu_preempt23:04:081
358870899381,4cyclictest131rcu_preempt00:05:001
3588708993729,5cyclictest3617347-21sh21:57:481
3588708993728,5cyclictest3672808-21sh23:55:001
3588708993727,7cyclictest23-21ksoftirqd/121:20:341
3588708993726,8cyclictest23-21ksoftirqd/120:25:001
3588708993721,12cyclictest275-21dbus-daemon20:00:011
3588708993720,12cyclictest275-21dbus-daemon20:30:021
3588708993720,12cyclictest275-21dbus-daemon19:50:011
358870899371,7cyclictest131rcu_preempt23:00:001
3588708993629,5cyclictest23-21ksoftirqd/100:11:471
3588708993626,6cyclictest23-21ksoftirqd/121:30:431
3588708993623,10cyclictest23-21ksoftirqd/121:35:511
3588708993619,12cyclictest3589176-21systemctl19:20:001
358870899354,3cyclictest131rcu_preempt22:10:401
3588708993527,5cyclictest3595027-21sh21:10:021
3588708993524,9cyclictest23-21ksoftirqd/123:41:061
358870899352,3cyclictest131rcu_preempt22:06:091
3588708993425,6cyclictest3588760-21latency_hist19:10:011
3588708993324,6cyclictest23-21ksoftirqd/119:40:031
3588708993223,7cyclictest23-21ksoftirqd/120:55:011
358870899321,4cyclictest131rcu_preempt20:20:001
3588717993124,4cyclictest3588845-21latency_hist19:10:023
3588708993023,4cyclictest3593827-21sshd20:49:261
3588713992922,4cyclictest3588645-21latency_hist19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional