You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 20:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 24, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25843082920,8sleep10-21swapper/107:50:201
26190942910,7sleep24497-21PK-Backend09:30:002
26190942910,7sleep24497-21PK-Backend09:30:002
197399860,73rtkit-daemon0-21swapper/207:09:112
26138232790,4sleep24497-21PK-Backend09:15:122
256869727739,13sleep10-21swapper/107:05:501
26026732750,4sleep04497-21PK-Backend08:40:290
197399750,60rtkit-daemon0-21swapper/307:07:203
256873126344,13sleep00-21swapper/007:06:160
197399600,9rtkit-daemon1972-21rtkit-daemon11:55:481
26455342580,13sleep20-21swapper/210:45:102
2569181995811,23cyclictest0-21swapper/307:17:023
197399580,9rtkit-daemon1972-21rtkit-daemon10:43:331
197399570,9rtkit-daemon1972-21rtkit-daemon12:34:530
197399570,9rtkit-daemon1972-21rtkit-daemon12:22:161
256918199551,28cyclictest2647262-21latency_hist10:50:013
197399550,8rtkit-daemon1972-21rtkit-daemon09:25:331
197399550,8rtkit-daemon1972-21rtkit-daemon09:25:331
197399550,10rtkit-daemon1972-21rtkit-daemon11:38:581
26093902540,16sleep013-21ksoftirqd/009:00:250
197399530,9rtkit-daemon1972-21rtkit-daemon10:00:571
197399530,8rtkit-daemon1972-21rtkit-daemon10:41:270
197399530,8rtkit-daemon1972-21rtkit-daemon09:42:471
197399520,8rtkit-daemon1972-21rtkit-daemon11:34:161
25943402510,6chrt4497-21PK-Backend08:20:032
256918199517,28cyclictest2649269-21cat10:55:153
256918199502,44cyclictest2583549-21apt07:50:013
256918199501,45cyclictest1137-21runrttasks12:02:093
2569181994917,28cyclictest2661571-21cat11:30:203
2569181994917,27cyclictest438-20systemd-journal10:31:213
2569181994917,27cyclictest3037-21gmain09:25:033
2569181994917,27cyclictest3037-21gmain09:25:023
2569181994916,28cyclictest2606114-21grep08:50:283
256918199491,42cyclictest2612858-21ntp_states09:10:243
256918199482,30cyclictest26163562/usr/sbin/munin09:20:263
2569181994819,25cyclictest2647734-21hddtemp_smartct10:50:203
2569181994817,26cyclictest2650785-21awk11:00:133
2569181994817,26cyclictest2650785-21awk11:00:123
256918199481,42cyclictest2673273-21/usr/sbin/munin12:05:133
256918199481,42cyclictest1054-21snmpd07:10:253
256918199481,42cyclictest1054-21snmpd07:10:243
2569181994813,29cyclictest2575875-21processes07:25:263
2569181994717,25cyclictest2598199-21cstates08:30:293
2569181994716,26cyclictest2654748-21cut11:10:203
2569181994715,27cyclictest2637157-21cut10:20:223
256918199471,42cyclictest1137-21runrttasks12:15:383
256918199471,41cyclictest2665117-21irqstats11:40:203
256918199471,41cyclictest2593148-21diskstats08:15:173
256918199471,31cyclictest2681346-21taskset12:25:493
25869732460,13sleep342-21ksoftirqd/307:56:043
2569181994617,25cyclictest2643295-21/usr/sbin/munin10:40:003
2569181994614,28cyclictest2607924-21cat08:55:313
2569181994614,28cyclictest2601290-21pkexec08:38:073
2569181994614,27cyclictest2634331-21seq10:11:113
2569181994611,30cyclictest2678739-21expr12:20:143
256918199452,38cyclictest2663655-21ntp_states11:35:263
2569181994520,21cyclictest2610514-21cpuspeed_turbos09:05:123
256918199451,40cyclictest2627756-20unin-run09:55:023
256918199451,38cyclictest2574192-21python307:20:313
2569181994513,28cyclictest2615190-21seq09:18:443
2569181994512,28cyclictest2628221-21cat09:55:153
2569181994512,28cyclictest2588465-21switchtime08:00:283
256918199451,16cyclictest26197132sh09:30:173
2569181994423,18cyclictest2657434-21cron11:20:013
2569181994415,24cyclictest2601846-21cpuspeed_turbos08:40:143
2569181994415,24cyclictest2595227-21tr08:20:223
256918199441,38cyclictest2682692-21ntp_offset12:30:243
2569181994413,26cyclictest2675715-21netstat12:10:253
2569181994413,26cyclictest2609243-21missed_timers09:00:233
197399440,35rtkit-daemon2660054-21sed11:25:220
197399440,35rtkit-daemon2660054-21sed11:25:220
25997952430,14sleep2331rcuc/208:35:032
256918199435,34cyclictest2345-21gdbus10:19:563
256918199432,36cyclictest2669185-21users11:50:303
256918199432,36cyclictest2669185-21users11:50:293
2569181994317,22cyclictest2638979-21missed_timers10:25:213
2569181994312,27cyclictest2621500-21basename09:35:233
2569181994311,28cyclictest2666670-21expr11:45:163
2569181994311,28cyclictest2579250-21netstat07:35:243
197399430,34rtkit-daemon4497-21PK-Backend12:05:220
256918199429,28cyclictest438-20systemd-journal08:29:593
256918199422,35cyclictest2626171-21date09:50:023
2569181994217,21cyclictest2652613-21cpuspeed_turbos11:05:133
2569181994215,23cyclictest2583787-21/usr/sbin/munin07:50:203
2569181994215,22cyclictest2589282-21/usr/sbin/munin08:05:123
256918199421,38cyclictest2591538-21/usr/sbin/munin08:10:223
256918199421,37cyclictest2644505-21cat10:40:293
256918199421,36cyclictest2631425-21/usr/sbin/munin10:05:263
2569181994210,28cyclictest2580574-21cat07:40:143
197399420,33rtkit-daemon2569158-21cyclictest10:12:211
256918199416,30cyclictest2657822-21cat11:20:143
2569181994115,22cyclictest2659633-21cpuspeed_turbos11:25:153
2569181994115,22cyclictest2659633-21cpuspeed_turbos11:25:143
2569181994115,22cyclictest2629936-21cstates10:00:173
2569181994115,22cyclictest1137-21runrttasks12:39:103
256918199411,36cyclictest2578315-21basename07:35:003
256918199411,36cyclictest0-21swapper/311:56:083
2569181994110,28cyclictest42-21ksoftirqd/308:47:193
256916899411,17cyclictest2606650-21perf08:55:001
197399410,34rtkit-daemon4497-21PK-Backend12:14:561
256918199401,24cyclictest2624015-21chrt09:42:003
2569176994023,13cyclictest2629876-21tr10:00:162
2569176994022,13cyclictest34-21ksoftirqd/208:22:122
256916899402,33cyclictest2638661-21expr10:25:131
2569168994023,12cyclictest438-20systemd-journal12:25:281
2569163994026,10cyclictest2647817-21/usr/sbin/munin10:50:220
2569163994019,17cyclictest2637402-21ntp_states10:20:270
26684742390,6chrt0-21swapper/011:50:150
26684742390,6chrt0-21swapper/011:50:150
26594852390,6sleep126-21ksoftirqd/111:25:141
26594852390,6sleep126-21ksoftirqd/111:25:131
25709002390,5sleep00-21swapper/007:15:010
25709002390,5sleep00-21swapper/007:15:000
2569168993920,14cyclictest2575407-21df07:25:161
2569168993911,11cyclictest1137-21runrttasks11:50:021
2569163993922,13cyclictest2632037-21sshd10:05:200
26455302380,6chrt0-21swapper/110:45:101
2569176993825,10cyclictest2671309-21ls12:00:002
2569176993821,13cyclictest34-21ksoftirqd/211:52:482
2569176993821,13cyclictest34-21ksoftirqd/211:52:482
2569176993820,13cyclictest2642620-21pluginstate10:35:282
256917699381,15cyclictest26491092cpu10:55:142
256916399381,13cyclictest0-21swapper/008:08:350
197399380,30rtkit-daemon13-21ksoftirqd/012:12:390
197399380,27rtkit-daemon4497-21PK-Backend08:08:102
2569176993726,7cyclictest2585030-21runrttasks07:52:042
2569176993722,11cyclictest2575440-21fw_packets07:25:182
2569176993720,13cyclictest2626997-21ntpq09:50:242
2569176993719,14cyclictest34-21ksoftirqd/208:44:582
256917699371,13cyclictest4497-21PK-Backend11:17:382
2569168993724,9cyclictest2655711-21idleruntime-cro11:15:011
2569168993720,6cyclictest26-21ksoftirqd/107:55:231
2569168993720,13cyclictest2614149-21expr09:15:141
256916899371,13cyclictest25712151acpi07:15:051
256916899370,34cyclictest1972-21rtkit-daemon09:05:171
256916399372,25cyclictest25869722lsb_release07:56:040
2569163993722,11cyclictest2093-21JS10:49:090
2569163993721,12cyclictest2592276-21sshd08:12:200
2569163993719,14cyclictest2587940-21df08:00:150
2569163993719,13cyclictest2638743-21df_inode10:25:150
256916399371,32cyclictest2634384-21chrt10:11:510
197399370,31rtkit-daemon0-21swapper/109:45:141
197399370,28rtkit-daemon26-21ksoftirqd/110:56:051
197399370,24rtkit-daemon0-21swapper/207:49:202
256917699362,30cyclictest2635288-21diskstats10:15:172
2569176993619,13cyclictest34-21ksoftirqd/208:02:472
256916899363,29cyclictest2668902-21wc11:50:251
256916899363,29cyclictest2668902-21wc11:50:251
2569168993620,12cyclictest2642619-21awk10:35:271
2569168993618,13cyclictest438-20systemd-journal08:05:001
2569168993615,16cyclictest2590178-21python308:05:281
256916899361,31cyclictest2598937-21/usr/sbin/munin08:30:471
2569168993613,14cyclictest26-21ksoftirqd/109:30:131
2569163993622,11cyclictest2657435-21cron11:20:010
2569163993621,11cyclictest2595213-21ntp_kernel_err08:20:220
2569163993620,12cyclictest2624191-21sshd09:44:190
256916399361,7cyclictest2190-21gvfs-udisks2-vo12:29:560
197399360,28rtkit-daemon1941-21systemd10:33:470
197399360,24rtkit-daemon0-21swapper/208:27:062
26014722350,7sleep02601466-21turbostat08:40:010
2569176993520,11cyclictest2676476-21cron12:15:002
2569176993519,12cyclictest2632688-21sshd10:07:202
2569176993519,12cyclictest2577820-21sed07:30:312
2569176993519,11cyclictest438-20systemd-journal08:59:542
2569176993518,13cyclictest34-21ksoftirqd/210:26:302
2569176993518,13cyclictest34-21ksoftirqd/209:12:302
2569176993516,15cyclictest2677161-21which12:15:182
2569168993520,11cyclictest2682831-21smartctl12:30:291
2569168993520,11cyclictest2593541-21awk08:15:261
256916899351,9cyclictest0-21swapper/110:16:231
2569168993519,12cyclictest2613036-21grep09:10:281
2569168993518,13cyclictest26-21ksoftirqd/108:43:181
2569168993518,12cyclictest26-21ksoftirqd/107:45:201
2569168993517,6cyclictest26-21ksoftirqd/111:25:001
2569163993524,7cyclictest2678678-21tr12:20:140
2569163993521,10cyclictest2670250-21idleruntime11:55:200
2569163993521,10cyclictest13-21ksoftirqd/011:45:270
2569163993520,11cyclictest2684208-21cat12:35:220
2569163993519,12cyclictest2646267-21ntp_kernel_err10:45:220
2569163993519,12cyclictest2630960-21sshd10:02:210
2569163993519,12cyclictest2604215-21ntp_states08:45:250
2569163993518,13cyclictest13-21ksoftirqd/009:27:090
2569163993518,13cyclictest13-21ksoftirqd/009:27:080
256916399351,11cyclictest2581825-21turbostat07:45:000
197399350,28rtkit-daemon1941-21systemd10:37:160
256917699344,10cyclictest34-21ksoftirqd/212:35:002
2569176993426,5cyclictest4497-21PK-Backend12:06:552
2569176993424,6cyclictest2648792-21latency_hist10:55:012
2569176993423,7cyclictest2573477-21grep07:20:162
2569176993419,11cyclictest2609813-21sshd09:01:192
2569176993419,11cyclictest2603729-21cat08:45:152
256917699341,8cyclictest4497-21PK-Backend11:06:522
2569168993425,5cyclictest26-21ksoftirqd/111:05:001
2569168993425,5cyclictest26-21ksoftirqd/111:04:591
256916899341,9cyclictest4497-21PK-Backend12:04:371
2569168993419,11cyclictest2674461-21sshd12:06:201
2569168993417,13cyclictest26-21ksoftirqd/110:25:001
2569168993417,13cyclictest1-21systemd09:40:001
256916899341,30cyclictest0-21swapper/110:31:091
256916399342,9cyclictest2652249-21sh11:04:590
256916399342,9cyclictest2652249-21sh11:04:590
2569163993425,6cyclictest2626431-21cpuspeed_turbos09:50:150
256916399342,10cyclictest2665012-21diskstats11:40:160
256916399341,9cyclictest4497-21PK-Backend09:10:280
2569163993419,11cyclictest2672251-21sshd12:00:230
2569163993419,11cyclictest2611234-21ps09:05:250
2569163993419,11cyclictest2606231-21cat08:50:280
2569163993418,12cyclictest2614065-21expr09:15:140
2569163993418,11cyclictest2619445-21cat09:30:150
2569163993417,13cyclictest2574063-21ntp_states07:20:280
2569163993414,17cyclictest4497-21PK-Backend07:35:000
256916399341,30cyclictest4497-21PK-Backend08:15:240
2569163993410,5cyclictest141rcu_preempt11:37:120
2569163993410,5cyclictest141rcu_preempt07:16:420
25864912330,5sleep2331rcuc/207:55:242
256917699339,5cyclictest141rcu_preempt09:07:142
2569176993323,6cyclictest2569489-21cpuspeed_turbos07:10:122
2569176993323,6cyclictest2569489-21cpuspeed_turbos07:10:112
2569176993322,7cyclictest2679080-21memory12:20:232
2569176993318,11cyclictest2665037-21seq11:40:172
2569176993318,11cyclictest2620834-21pluginstate09:35:012
2569176993316,13cyclictest2633531-21expr10:10:162
2569176993316,13cyclictest2623554-21seq09:40:252
2569176993314,15cyclictest2663662-21ntp_states11:35:272
2569176993314,15cyclictest2591404-21df08:10:162
256916899335,24cyclictest2676919-21cpuspeed_turbos12:15:121
2569168993323,6cyclictest26-21ksoftirqd/111:05:291
2569168993323,5cyclictest26-21ksoftirqd/109:50:491
2569168993319,10cyclictest2657472-21systemd-run11:20:001
2569168993319,10cyclictest2603996-21sed08:45:201
256916899331,8cyclictest4497-21PK-Backend09:22:311
2569168993317,12cyclictest2683828-21tr12:35:141
2569168993317,11cyclictest438-20systemd-journal07:43:191
2569168993313,17cyclictest2570853-21unin-run07:15:011
2569168993313,17cyclictest2570853-21unin-run07:15:011
256916899331,14cyclictest4497-21PK-Backend10:54:181
2569163993317,12cyclictest2654568-21expr11:10:150
2569163993315,13cyclictest438-20systemd-journal11:09:590
2569163993314,14cyclictest13-21ksoftirqd/011:30:150
256916399331,29cyclictest4497-21PK-Backend08:32:240
256916399331,29cyclictest0-21swapper/009:55:470
256916399331,29cyclictest0-21swapper/009:45:080
2569176993224,5cyclictest0-21swapper/211:26:172
2569176993224,5cyclictest0-21swapper/211:26:172
2569176993220,8cyclictest2666524-21tr11:45:142
2569176993219,9cyclictest2069-21gnome-shell07:35:092
2569176993217,11cyclictest2661355-21grep11:30:162
2569176993217,11cyclictest2651379-21cut11:00:242
2569176993217,11cyclictest2651379-21cut11:00:242
2569176993215,13cyclictest2658081-21df11:20:152
2569176993215,13cyclictest2621483-21which09:35:232
2569176993215,12cyclictest438-20systemd-journal12:28:192
256917699321,4cyclictest0-21swapper/208:32:232
256917699321,29cyclictest0-21swapper/212:39:262
256917699321,28cyclictest0-21swapper/209:23:412
256917699321,28cyclictest0-21swapper/207:41:072
2569168993222,7cyclictest2596365-21tr08:25:141
2569168993222,5cyclictest26-21ksoftirqd/107:30:191
2569168993219,9cyclictest26-21ksoftirqd/108:10:221
256916899321,8cyclictest4497-21PK-Backend11:40:051
2569168993218,10cyclictest2609506-21cut09:00:281
2569168993216,12cyclictest2631953-21cut10:05:191
2569168993216,12cyclictest2607532-21sed08:55:251
2569168993215,12cyclictest26-21ksoftirqd/107:20:151
256916899321,4cyclictest4497-21PK-Backend08:37:201
256916899321,28cyclictest0-21swapper/107:36:031
2569163993222,5cyclictest13-21ksoftirqd/007:49:140
2569163993220,8cyclictest2677560-21grep12:15:250
2569163993217,11cyclictest2615664-21ls09:20:140
2569163993216,12cyclictest2658647-21sed11:20:290
2569163993216,12cyclictest2607557-21cat08:55:250
256916399321,4cyclictest4497-21PK-Backend07:52:150
256916399321,29cyclictest0-21swapper/008:26:020
256916399321,28cyclictest4497-21PK-Backend07:36:380
2569176993125,4cyclictest0-21swapper/212:00:582
2569176993115,11cyclictest2625008-21awk09:45:172
256917699311,4cyclictest0-21swapper/210:42:112
2569176993114,5cyclictest34-21ksoftirqd/210:31:302
256917699311,27cyclictest0-21swapper/209:58:422
2569168993120,7cyclictest26-21ksoftirqd/108:21:091
2569163993115,12cyclictest2635779-21sensors_temp10:15:270
256916399311,4cyclictest4497-21PK-Backend07:28:100
197399310,24rtkit-daemon4497-21PK-Backend07:15:522
256917699301,4cyclictest0-21swapper/210:21:132
2569168993020,5cyclictest26-21ksoftirqd/109:55:251
2569163993015,11cyclictest1-21systemd09:40:010
2569176992922,4cyclictest0-21swapper/211:11:152
2569176992917,8cyclictest2606668-21awk08:55:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional