You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 03:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 02, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2384659998448,24cyclictest23-21ksoftirqd/122:00:011
2384665997324,6cyclictest309-21in:imuxsock19:25:013
2384659997263,7cyclictest23-21ksoftirqd/121:30:031
2384655997217,16cyclictest2480021-21ssh00:28:150
2384665996925,6cyclictest2390292-21sendmail21:00:013
2384665996922,14cyclictest275-21dbus-daemon21:55:013
238466599683,23cyclictest3467602-21systemd-journal19:59:593
2384665996830,10cyclictest3467602-21systemd-journal21:41:253
2384655996823,16cyclictest2385613-21sshd19:27:450
2384655996630,5cyclictest2434346-21ssh22:46:250
2384659996419,8cyclictest23-21ksoftirqd/123:45:021
2384655996413,18cyclictest2408171-21ssh21:48:220
2384665996326,12cyclictest37-21ksoftirqd/300:16:443
2384665996325,10cyclictest37-21ksoftirqd/323:08:573
2384659996311,10cyclictest23-21ksoftirqd/121:40:011
2384655996323,9cyclictest2410076-21ssh21:52:260
2384655996314,15cyclictest2404581-21date21:40:020
2384655996220,38cyclictest120-21kswapd023:50:000
2384655996219,11cyclictest2421850-21ssh22:18:490
2384655996215,12cyclictest2442651-21latency_hist23:05:020
238466599615,20cyclictest2422290-21rm22:19:593
2384665996119,9cyclictest0-21swapper/321:10:023
2384665996119,9cyclictest0-21swapper/321:10:013
2384665996029,5cyclictest1-21systemd20:35:013
238465999604,10cyclictest23-21ksoftirqd/121:45:001
2384659996011,9cyclictest23-21ksoftirqd/100:35:021
2384659996011,9cyclictest23-21ksoftirqd/100:35:021
2384655996011,18cyclictest2427094-21latency_hist22:30:020
2384665995930,4cyclictest2395257-21sh21:19:453
2384659995914,10cyclictest23-21ksoftirqd/120:20:031
2384655995920,15cyclictest2412342-21sh21:57:330
2384655995919,36cyclictest48-21kcompactd000:00:300
2384655995916,15cyclictest2416706-21sh22:07:100
2384655995916,15cyclictest2416706-21sh22:07:100
2384655995913,14cyclictest2386149-21/usr/sbin/munin19:40:000
2384665995814,22cyclictest275-21dbus-daemon23:50:013
2384663995841,13cyclictest1-21systemd00:00:002
2384663995841,13cyclictest1-21systemd00:00:002
2384659995819,9cyclictest23-21ksoftirqd/100:25:021
2384659995819,9cyclictest23-21ksoftirqd/100:25:021
2384655995830,11cyclictest2438121-21ssh22:55:000
2384655995821,4cyclictest2446770-21ssh23:14:110
2384655995817,38cyclictest120-21kswapd022:45:010
2384655995817,11cyclictest2424644-21munin-run22:25:010
2384665995741,10cyclictest2456063-21munin-plugin-st23:34:593
2384665995736,16cyclictest275-21dbus-daemon21:30:003
2384663995719,9cyclictest30-21ksoftirqd/223:10:022
2384663995710,6cyclictest131rcu_preempt23:12:372
2384655995731,11cyclictest2393136-21munin-run21:14:590
2384655995729,12cyclictest2447449-21ssh23:15:240
2384655995718,6cyclictest16850irq/134-800000021:18:140
2384663995615,16cyclictest2417888-21munin-run22:10:012
2384663995615,16cyclictest2417888-21munin-run22:10:012
238465999569,8cyclictest23-21ksoftirqd/123:13:171
238465599565,45cyclictest2467307-40mandb00:00:000
238465599565,45cyclictest2467307-40mandb00:00:000
2384655995641,12cyclictest1-21systemd22:15:000
238465599562,23cyclictest0-21swapper/023:05:300
2384655995620,17cyclictest16850irq/134-800000022:59:590
2384655995619,6cyclictest16850irq/134-800000023:44:080
2384655995615,10cyclictest2429653-21sh22:35:480
2384655995612,13cyclictest2388843-21sed20:30:020
2384663995517,7cyclictest131rcu_preempt23:45:032
238465999557,8cyclictest23-21ksoftirqd/100:30:021
238465999557,6cyclictest23-21ksoftirqd/122:40:231
238465999554,11cyclictest23-21ksoftirqd/121:00:011
2384659995513,13cyclictest23-21ksoftirqd/122:34:591
2384655995518,6cyclictest16850irq/134-800000023:24:460
2384655995518,6cyclictest16850irq/134-800000023:24:460
2384655995518,31cyclictest2454394-21ssh23:30:510
2384655995513,37cyclictest2481146-21ssh00:30:250
2384655995513,37cyclictest2481146-21ssh00:30:240
2384655995512,14cyclictest2389607-21latency_hist20:45:020
2384655995512,14cyclictest2389607-21latency_hist20:45:010
238466399544,8cyclictest30-21ksoftirqd/223:31:052
2384663995413,6cyclictest131rcu_preempt00:21:092
2384663995413,6cyclictest131rcu_preempt00:21:092
238465999544,4cyclictest131rcu_preempt00:05:031
2384655995442,8cyclictest16950irq/224-800000021:30:010
238465599544,11cyclictest131rcu_preempt20:20:030
2384655995427,12cyclictest2464505-21ssh23:53:380
2384665995336,12cyclictest275-21dbus-daemon00:25:013
2384665995336,12cyclictest275-21dbus-daemon00:25:013
238466399538,10cyclictest30-21ksoftirqd/200:40:002
238466399536,10cyclictest30-21ksoftirqd/222:40:012
238466399535,10cyclictest30-21ksoftirqd/222:20:032
2384663995345,5cyclictest2460630-21kworker/2:300:17:252
2384663995311,11cyclictest30-21ksoftirqd/221:58:102
238465999538,5cyclictest131rcu_preempt22:28:211
238465999537,10cyclictest23-21ksoftirqd/123:05:391
238465999536,9cyclictest23-21ksoftirqd/122:01:521
238465999535,11cyclictest23-21ksoftirqd/122:52:551
2384655995326,12cyclictest2452284-21ssh23:26:140
2384655995318,31cyclictest2427772-21sh22:31:430
2384655995318,31cyclictest2406859-21latency_hist21:45:020
2384655995311,37cyclictest2474667-21sh00:16:140
238466399529,7cyclictest30-21ksoftirqd/223:25:072
2384663995244,5cyclictest2429110-21kworker/2:123:24:102
2384663995244,5cyclictest2429110-21kworker/2:123:24:102
2384663995244,5cyclictest2429110-21kworker/2:123:19:412
2384663995244,5cyclictest2388762-21kworker/2:321:19:392
2384663995243,6cyclictest2387718-21kworker/2:020:21:242
2384663995234,14cyclictest275-21dbus-daemon21:25:002
2384663995214,8cyclictest30-21ksoftirqd/221:37:362
238465999527,7cyclictest23-21ksoftirqd/123:25:011
238465999527,7cyclictest23-21ksoftirqd/123:25:001
238465999526,6cyclictest23-21ksoftirqd/122:50:021
238465999526,6cyclictest23-21ksoftirqd/100:09:471
238465999523,14cyclictest23-21ksoftirqd/123:30:001
2384659995212,6cyclictest23-21ksoftirqd/123:57:591
2384659995212,6cyclictest23-21ksoftirqd/123:57:581
2384659995210,9cyclictest0-21swapper/123:20:011
238465599529,12cyclictest2477754-21ssh00:23:170
238465599529,12cyclictest2477754-21ssh00:23:170
2384655995239,10cyclictest2385090-21sshd19:19:250
2384655995213,14cyclictest2390824-21diskmemload00:38:320
2384655995210,38cyclictest0-21swapper/019:44:590
238466399516,10cyclictest30-21ksoftirqd/221:43:102
238466399514,8cyclictest30-21ksoftirqd/221:46:012
2384663995143,5cyclictest2388762-21kworker/2:322:31:342
2384663995142,6cyclictest2451610-21kworker/2:023:39:522
2384663995142,6cyclictest2388762-21kworker/2:322:01:072
2384663995142,5cyclictest2388762-21kworker/2:321:32:252
2384663995114,5cyclictest131rcu_preempt21:53:252
238465999519,5cyclictest131rcu_preempt23:37:121
238465999519,11cyclictest23-21ksoftirqd/119:10:031
238465999516,7cyclictest23-21ksoftirqd/123:05:001
238465999516,7cyclictest23-21ksoftirqd/121:50:031
238465999515,8cyclictest23-21ksoftirqd/100:37:171
238465999514,7cyclictest23-21ksoftirqd/121:23:541
2384655995136,11cyclictest2387430-21sshd20:02:250
2384655995136,11cyclictest2387430-21sshd20:02:240
2384655995135,12cyclictest2384860-21munin-run19:15:020
2384655995135,12cyclictest2384860-21munin-run19:15:010
2384655995134,13cyclictest2390569-21munin-run21:05:000
2384655995111,12cyclictest131rcu_preempt20:10:020
2384665995033,11cyclictest1-21systemd00:30:003
238466399509,6cyclictest131rcu_preempt22:29:532
238466399506,7cyclictest30-21ksoftirqd/222:57:152
2384663995042,5cyclictest2460630-21kworker/2:323:54:492
2384663995042,5cyclictest2387718-21kworker/2:022:12:372
2384663995042,5cyclictest2387718-21kworker/2:021:13:552
2384663995042,5cyclictest2386702-21kworker/2:319:52:522
2384663995042,5cyclictest2385396-21kworker/2:119:45:232
2384663995041,6cyclictest2460630-21kworker/2:323:47:122
2384663995041,6cyclictest2387718-21kworker/2:022:20:112
2384663995041,5cyclictest2375496-21kworker/2:219:36:302
238466399504,14cyclictest30-21ksoftirqd/220:20:012
2384663995012,5cyclictest131rcu_preempt00:13:192
2384659995042,5cyclictest23-21ksoftirqd/100:17:171
2384659995018,8cyclictest23-21ksoftirqd/121:05:021
2384659995012,11cyclictest23-21ksoftirqd/122:16:311
238465599508,14cyclictest2390824-21diskmemload00:14:220
2384655995035,11cyclictest2389017-21munin-run20:35:020
2384655995028,18cyclictest364-21snmpd23:37:140
238465599502,14cyclictest2389854-21latency_hist20:50:010
2384655995015,31cyclictest0-21swapper/022:00:200
238466599493,32cyclictest2386675-21cut19:50:003
2384665994924,14cyclictest0-21swapper/322:40:003
238466399498,6cyclictest131rcu_preempt22:44:102
2384663994943,4cyclictest2476249-21kworker/2:100:28:082
2384663994942,4cyclictest2375496-21kworker/2:219:14:062
2384663994942,4cyclictest2375496-21kworker/2:219:14:052
2384663994941,5cyclictest2460630-21kworker/2:300:03:092
2384663994941,5cyclictest2433615-21kworker/2:223:01:002
2384663994941,5cyclictest2429110-21kworker/2:122:50:102
2384663994941,5cyclictest2429110-21kworker/2:122:45:212
2384663994941,5cyclictest2388762-21kworker/2:320:57:522
2384663994941,5cyclictest2387718-21kworker/2:021:25:162
2384663994941,5cyclictest2387718-21kworker/2:020:45:482
2384663994941,5cyclictest2385396-21kworker/2:120:03:232
2384663994941,5cyclictest2385396-21kworker/2:120:03:232
2384663994941,5cyclictest2375496-21kworker/2:219:16:262
238466399491,9cyclictest30-21ksoftirqd/200:35:012
238466399491,9cyclictest30-21ksoftirqd/200:35:012
238466399491,12cyclictest30-21ksoftirqd/220:39:482
238465999496,7cyclictest23-21ksoftirqd/121:31:231
238465999496,11cyclictest2454783-21ssh23:31:531
238465999494,9cyclictest23-21ksoftirqd/121:12:551
238465999494,8cyclictest23-21ksoftirqd/121:50:221
238465999494,10cyclictest23-21ksoftirqd/122:15:001
238465999491,9cyclictest0-21swapper/123:54:351
238465999491,6cyclictest0-21swapper/123:46:211
2384659994911,7cyclictest364-21snmpd19:36:111
2384659994910,10cyclictest309-21in:imuxsock20:40:001
238465599496,39cyclictest2401315-21ssh21:33:040
2384655994941,5cyclictest16950irq/224-800000021:00:010
2384655994930,14cyclictest3467602-21systemd-journal20:54:590
2384663994841,5cyclictest2433615-21kworker/2:200:09:412
2384663994840,5cyclictest2388762-21kworker/2:320:53:262
2384663994840,5cyclictest2387718-21kworker/2:021:05:392
2384663994840,5cyclictest2387718-21kworker/2:021:05:392
2384663994840,5cyclictest2387718-21kworker/2:020:27:322
2384663994840,5cyclictest2386702-21kworker/2:319:55:092
2384663994840,5cyclictest2385396-21kworker/2:120:09:092
2384663994840,5cyclictest2385396-21kworker/2:119:43:482
2384663994840,5cyclictest2375496-21kworker/2:219:31:012
238465999485,7cyclictest23-21ksoftirqd/122:37:301
2384655994840,4cyclictest2387925-21sshd20:10:260
2384655994829,15cyclictest2470010-21ssh00:05:360
2384665994733,7cyclictest309-21in:imuxsock19:15:013
2384665994733,7cyclictest309-21in:imuxsock19:15:013
2384665994723,19cyclictest2467302-40ogrotate00:00:003
2384665994723,19cyclictest2467302-40ogrotate00:00:003
2384663994739,5cyclictest2388762-21kworker/2:320:40:272
2384663994739,5cyclictest2388762-21kworker/2:320:40:272
2384663994739,5cyclictest2387718-21kworker/2:021:02:332
2384663994739,5cyclictest2387718-21kworker/2:020:30:182
2384663994739,5cyclictest2387718-21kworker/2:020:11:332
2384663994739,5cyclictest2375496-21kworker/2:219:26:572
2384663994739,5cyclictest2375496-21kworker/2:219:20:242
238465999475,8cyclictest23-21ksoftirqd/122:56:481
238465999475,6cyclictest23-21ksoftirqd/120:45:021
238465999475,6cyclictest23-21ksoftirqd/120:45:011
238465999473,8cyclictest23-21ksoftirqd/122:21:141
2384659994713,4cyclictest131rcu_preempt00:10:311
2384665994628,12cyclictest275-21dbus-daemon21:40:013
238465999469,9cyclictest23-21ksoftirqd/119:27:461
238465999466,9cyclictest23-21ksoftirqd/120:24:591
238465999466,12cyclictest23-21ksoftirqd/121:18:171
238465999465,7cyclictest23-21ksoftirqd/122:06:071
238465999465,7cyclictest23-21ksoftirqd/122:06:071
238465999461,7cyclictest2386726-21sed19:50:011
2384655994632,11cyclictest2389232-21sshd20:38:250
2384655994611,30cyclictest12-21ksoftirqd/021:10:010
2384655994611,30cyclictest12-21ksoftirqd/021:10:010
2384655994610,33cyclictest0-21swapper/021:23:020
2384665994529,11cyclictest3467602-21systemd-journal21:15:003
2384665994524,15cyclictest1-21systemd22:25:013
2384665994518,20cyclictest275-21dbus-daemon19:35:013
2384665994511,24cyclictest2442648-21modprobe23:05:003
238465999456,10cyclictest23-21ksoftirqd/120:50:021
238465999454,11cyclictest23-21ksoftirqd/119:18:241
2384659994538,5cyclictest23-21ksoftirqd/120:05:011
2384659994538,5cyclictest23-21ksoftirqd/120:05:001
2384659994534,8cyclictest23-21ksoftirqd/120:15:011
2384655994537,5cyclictest2388513-21munin-run20:25:010
2384655994530,11cyclictest364-21snmpd19:46:080
2384655994526,15cyclictest364-21snmpd19:21:220
2384665994411,6cyclictest131rcu_preempt23:55:023
238465999447,9cyclictest23-21ksoftirqd/119:55:031
238465999442,9cyclictest23-21ksoftirqd/119:15:021
238465999442,9cyclictest23-21ksoftirqd/119:15:011
2384659994424,13cyclictest275-21dbus-daemon19:25:011
238465999441,8cyclictest0-21swapper/120:30:241
238465999441,6cyclictest0-21swapper/120:55:021
238465999441,10cyclictest23-21ksoftirqd/121:10:021
238465999441,10cyclictest23-21ksoftirqd/121:10:011
2384655994429,11cyclictest48-21kcompactd019:31:480
2384655994429,11cyclictest2386901-21sshd19:52:490
2384655994417,22cyclictest2387205-21sendmail-msp20:00:000
2384665994332,6cyclictest309-21in:imuxsock22:45:013
2384665994328,10cyclictest309-21in:imuxsock23:17:003
238465999436,7cyclictest23-21ksoftirqd/119:45:021
238465999433,11cyclictest23-21ksoftirqd/119:59:251
2384665994232,6cyclictest2389536-21/usr/sbin/munin20:45:013
2384665994232,6cyclictest2389536-21/usr/sbin/munin20:45:013
2384665994227,12cyclictest275-21dbus-daemon23:45:003
2384659994211,7cyclictest0-21swapper/120:30:011
238466599414,27cyclictest2415609-21cut22:05:003
2384665994130,8cyclictest2428078-21sshd22:32:243
2384665994124,11cyclictest3467602-21systemd-journal20:25:013
2384665994122,13cyclictest275-21dbus-daemon21:05:013
2384665994119,18cyclictest3467602-21systemd-journal19:55:013
2384665994113,6cyclictest131rcu_preempt00:00:273
2384665994111,7cyclictest131rcu_preempt22:59:523
238465999416,6cyclictest23-21ksoftirqd/119:31:241
238465999413,8cyclictest23-21ksoftirqd/120:10:011
2384665994024,12cyclictest3467602-21systemd-journal23:36:233
238466599401,5cyclictest131rcu_preempt22:10:013
238466599401,5cyclictest131rcu_preempt22:10:003
238466399409,5cyclictest131rcu_preempt19:10:022
238466599395,25cyclictest2402128-21cut21:35:003
238466599392,9cyclictest2425258-21ssh22:26:103
2384665993928,6cyclictest2389786-21munin-plugin-st20:50:003
2384665993927,9cyclictest37-21ksoftirqd/319:43:243
2384665993927,9cyclictest37-21ksoftirqd/300:10:393
2384665993923,12cyclictest37-21ksoftirqd/319:20:003
238466599387,4cyclictest150-21pr/ttyS200:40:003
238466599384,10cyclictest37-21ksoftirqd/320:09:263
238466599382,8cyclictest0-21swapper/322:11:483
2384665993827,7cyclictest311-21rs:main3
2384665993824,11cyclictest37-21ksoftirqd/323:21:393
2384665993824,11cyclictest37-21ksoftirqd/323:21:393
238466599381,11cyclictest2433945-21ssh22:45:203
238466599373,8cyclictest48-21kcompactd021:20:353
2384665993727,7cyclictest2445808-21sh23:11:513
2384665993727,6cyclictest2406889-21sh21:45:043
2384665993725,10cyclictest37-21ksoftirqd/323:25:353
2384665993723,11cyclictest37-21ksoftirqd/300:09:253
2384665993716,16cyclictest275-21dbus-daemon20:15:013
2384655993722,11cyclictest2384695-21grep19:10:020
238466599366,6cyclictest131rcu_preempt20:35:263
2384665993626,8cyclictest37-21ksoftirqd/319:40:033
2384665993624,9cyclictest37-21ksoftirqd/300:32:053
2384665993624,9cyclictest37-21ksoftirqd/300:32:053
2384665993621,10cyclictest309-21in:imuxsock21:59:253
2384665993614,19cyclictest37-21ksoftirqd/320:30:023
238466599358,5cyclictest131rcu_preempt20:01:163
238466599358,5cyclictest131rcu_preempt20:01:163
2384665993527,6cyclictest37-21ksoftirqd/320:20:003
2384665993517,14cyclictest275-21dbus-daemon19:30:013
2384665993223,7cyclictest37-21ksoftirqd/320:53:203
2384665992821,4cyclictest2384797-21cat19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional