You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 20:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 31, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
241341522710,6sleep00-21swapper/007:05:290
241341522710,6sleep00-21swapper/007:05:290
197399900,72rtkit-daemon0-21swapper/307:05:283
197399900,72rtkit-daemon0-21swapper/307:05:283
24435662830,4sleep34497-21PK-Backend08:30:313
24620912760,5sleep22462093-21tr09:25:152
24328292680,5sleep1251rcuc/108:00:271
241268626842,18sleep20-21swapper/207:05:162
241268626842,18sleep20-21swapper/207:05:162
241354526344,14sleep10-21swapper/107:07:121
241354526344,14sleep10-21swapper/107:07:121
197399630,8rtkit-daemon1972-21rtkit-daemon07:44:222
197399630,18rtkit-daemon1972-21rtkit-daemon07:51:272
24188122600,21sleep342-21ksoftirqd/307:20:253
25157702580,6chrt4497-21PK-Backend11:55:302
197399580,9rtkit-daemon1972-21rtkit-daemon07:56:552
24277622550,6sleep10-21swapper/107:45:261
24277622550,6sleep10-21swapper/107:45:261
241393299545,44cyclictest2506974-21grep11:30:280
2413942995317,31cyclictest2470148-21grep09:45:293
2413939995342,8cyclictest34-21ksoftirqd/208:10:392
2413932995318,30cyclictest2493651-21cut10:55:020
2413932995316,25cyclictest13-21ksoftirqd/012:23:160
2413932995315,32cyclictest2423414-21wc07:35:140
241393299531,47cyclictest2428964-21df07:50:160
2413932995220,28cyclictest1033-21gdbus09:12:490
2413932995220,27cyclictest2069-21gnome-shell10:22:590
2413932995216,32cyclictest2435999-21kernelversion08:10:190
2413932995215,32cyclictest2504548-21cat11:25:150
2413932995214,33cyclictest2751-21gdbus07:23:310
2413932995214,33cyclictest2450766-21sshd08:54:000
2413932995214,32cyclictest2442663-21grep08:30:140
2413932995212,34cyclictest2420127-21expr07:25:150
197399520,8rtkit-daemon1972-21rtkit-daemon08:36:012
2413932995115,31cyclictest2414255-21grep07:10:140
2413932995113,33cyclictest2464441-21latency09:30:210
2413942995011,12cyclictest1074-21lldpd12:33:053
2413942995011,12cyclictest1074-21lldpd12:33:053
2413932995024,21cyclictest2522214-21hddtemp_smartct12:15:170
2413932995020,26cyclictest2426829-21/usr/sbin/munin07:45:080
2413932995020,26cyclictest2426829-21/usr/sbin/munin07:45:080
2413932995018,27cyclictest2497591-21diskstats11:05:170
2413932995017,29cyclictest2495659-21tr11:00:150
2413932995016,30cyclictest2473649-21ls09:55:310
2413932995015,30cyclictest438-20systemd-journal07:41:290
2413932995013,32cyclictest2515693-21which11:55:280
2413942994911,33cyclictest2524081-21expr12:20:133
2413932994918,26cyclictest2485722-21missed_timers10:30:230
2413932994917,27cyclictest2458834-21hddtemp_smartct09:15:190
2413932994916,28cyclictest2466711-21timerwakeupswit09:35:300
2413932994913,31cyclictest2511320-21cut11:45:020
2413932994912,32cyclictest2476197-21cpuspeed10:05:130
2413932994910,11cyclictest2468041-21cat09:40:220
2413932994816,28cyclictest2504171-21latency_hist11:25:010
2413932994816,28cyclictest1137-21runrttasks11:14:110
2413932994816,27cyclictest2434499-21ntpq08:05:240
2413932994815,28cyclictest2490962-21needreboot10:45:210
2413932994814,29cyclictest2483550-21tr10:25:140
2413932994813,31cyclictest2527927-21missed_timers12:30:220
2413932994813,31cyclictest2527927-21missed_timers12:30:220
2413932994813,31cyclictest2471188-21expr09:50:140
2413932994812,31cyclictest2451522-21cpuspeed_turbos08:55:150
2413932994811,31cyclictest2525675-21cpuspeed_turbos12:25:160
197399480,6rtkit-daemon1972-21rtkit-daemon08:52:053
25169752470,6sleep34497-21PK-Backend12:00:133
24492902470,17sleep234-21ksoftirqd/208:50:022
2413942994718,24cyclictest2422154-21missed_timers07:30:243
2413942994717,26cyclictest4497-21PK-Backend10:31:003
2413942994711,31cyclictest2492919-21switchtime10:50:283
2413939994734,9cyclictest34-21ksoftirqd/209:12:472
2413939994731,13cyclictest34-21ksoftirqd/210:50:132
2413939994710,20cyclictest34-21ksoftirqd/210:35:012
241393699473,16cyclictest2486732-21sed10:35:021
241393699472,19cyclictest2478372-21kernelversion10:10:221
241393299475,37cyclictest2440952-21cstates08:25:150
2413932994715,28cyclictest2529967-21switchtime12:35:300
2413932994715,27cyclictest2488451-21latency_hist10:40:020
2413932994715,27cyclictest2453325-21cpuspeed_turbos09:00:150
2413932994715,27cyclictest2453325-21cpuspeed_turbos09:00:150
2413932994713,30cyclictest2448035-21df08:45:170
241394299468,12cyclictest438-20systemd-journal10:40:003
241394299461,7cyclictest2505313-21python311:25:293
2413942994612,29cyclictest2457467-21proc_pri09:10:293
241393999460,42cyclictest1972-21rtkit-daemon08:32:472
241393299468,10cyclictest2476016-21cut10:05:010
2413932994616,25cyclictest2514824-21latency_hist11:55:010
2413932994614,27cyclictest2463583-21apt-key09:30:010
2413932994614,27cyclictest2455469-21latency09:05:200
241393299461,39cyclictest2069-21gnome-shell11:35:110
2413932994613,29cyclictest1137-21runrttasks12:11:010
2413932994613,28cyclictest2519879-21sed12:10:010
2413932994613,28cyclictest2516812-21cpuspeed_turbos12:00:130
2413942994516,25cyclictest2503687-21timerwakeupswit11:20:303
2413942994514,26cyclictest2472178-21systemd-udevd09:52:413
241394299451,39cyclictest2476306-21cut10:05:133
241393699458,14cyclictest1046-21lldpd11:20:091
2413936994529,11cyclictest2494200-21iwlist10:55:221
241393699452,15cyclictest2499510-21sed11:10:221
241393299456,36cyclictest1137-21runrttasks09:49:400
2413932994516,25cyclictest2416535-21cstates07:15:130
2413932994513,27cyclictest2444929-21ntpq08:35:240
2413932994511,10cyclictest2493951-21cstates10:55:170
2413942994419,22cyclictest2415570-21latency_hist07:15:013
2413942994418,23cyclictest4497-21PK-Backend09:24:483
2413942994418,22cyclictest2420878-21timerwakeupswit07:25:303
2413942994417,24cyclictest4497-21PK-Backend11:31:123
2413942994415,26cyclictest4497-21PK-Backend09:04:313
2413942994415,26cyclictest4497-21PK-Backend09:04:313
2413939994433,8cyclictest34-21ksoftirqd/211:35:132
2413939994422,6cyclictest34-21ksoftirqd/207:10:042
2413932994413,27cyclictest2439220-21cstates08:20:130
2413932994411,10cyclictest2431088-21open_inodes07:55:240
2413932994410,11cyclictest2489218-21tr10:40:220
2413942994315,24cyclictest2481258-21latency_hist10:20:013
2413942994314,25cyclictest2481573-21/usr/sbin/munin10:20:263
241393299439,11cyclictest2512950-21cat11:50:010
24649962420,9sleep10-21swapper/109:30:331
241394299429,11cyclictest2439586-21cat08:20:203
241394299427,11cyclictest2456405-21ls09:10:003
241394299422,7cyclictest2424073-21/usr/sbin/munin07:35:263
2413942994214,24cyclictest4497-21PK-Backend10:41:383
2413942994214,24cyclictest4497-21PK-Backend07:19:203
2413942994212,27cyclictest2478919-21timerwakeupswit10:10:313
241393699423,35cyclictest0-21swapper/107:40:101
2413936994228,10cyclictest2421281-21cron07:30:001
241393299429,10cyclictest2501491-21processes11:15:260
241393299429,10cyclictest2501491-21processes11:15:250
241393299429,10cyclictest1074-21lldpd07:30:090
2413932994215,23cyclictest2446748-21processes08:40:260
241393299421,4cyclictest0-21swapper/009:23:340
241393299421,13cyclictest2478617-21grep10:10:250
25112812410,6sleep10-21swapper/111:45:011
25105502410,16sleep234-21ksoftirqd/211:40:292
24406272410,16sleep1251rcuc/108:25:021
241394299418,6cyclictest0-21swapper/308:50:023
241394299412,7cyclictest2495863-21diskstats11:00:183
2413942994126,12cyclictest0-21swapper/311:37:543
2413942994115,22cyclictest2525556-21/usr/sbin/munin12:25:263
2413942994114,15cyclictest42-21ksoftirqd/307:53:293
241394299411,36cyclictest0-21swapper/309:30:193
2413942994111,27cyclictest2435702-21cstates08:10:133
2413939994115,11cyclictest2502140-21apt-get11:20:012
2413939994115,11cyclictest2502140-21apt-get11:20:012
241393999411,11cyclictest2450848-21cron08:55:002
241393299414,12cyclictest2432142-21cpuspeed_turbos08:00:140
2413932994114,24cyclictest1137-21runrttasks08:19:230
2413932994111,5cyclictest0-21swapper/010:19:220
2413942994024,13cyclictest4497-21PK-Backend12:05:443
2413942994024,12cyclictest2425257-21tr07:40:123
2413942994014,22cyclictest2512818-21sshd11:49:343
241394299401,11cyclictest2499911-21expr11:10:313
241393999406,19cyclictest34-21ksoftirqd/210:25:282
2413939994020,6cyclictest141rcu_preempt10:44:552
2413939994010,6cyclictest141rcu_preempt10:59:132
2413936994026,10cyclictest2459888-21cron09:20:001
241393699401,19cyclictest2452946-21/usr/sbin/munin09:00:011
24224212390,5sleep10-21swapper/107:30:291
241394299397,8cyclictest0-21swapper/308:15:273
241394299397,7cyclictest42-21ksoftirqd/312:00:003
241394299394,8cyclictest2529853-21wc12:35:293
241394299394,30cyclictest2467162-21systemd09:40:003
241394299391,9cyclictest924-21NetworkManager08:00:203
241394299391,35cyclictest0-21swapper/309:25:173
241394299391,12cyclictest2501351-21cut11:15:223
241394299391,12cyclictest2501351-21cut11:15:223
241394299391,12cyclictest2428235-21taskset07:49:073
241394299391,12cyclictest2428235-21taskset07:49:073
241394299391,11cyclictest2495325-21cut11:00:013
241394299391,11cyclictest2474549-21grep10:00:143
241394299391,11cyclictest2431639-21chrt07:58:273
2413939993928,8cyclictest34-21ksoftirqd/207:38:582
2413939993926,9cyclictest34-21ksoftirqd/210:05:012
241393999392,33cyclictest2473766-21/usr/sbin/munin09:55:322
241393999391,19cyclictest2513267-21/usr/sbin/munin11:50:142
241393999391,14cyclictest24409322tr08:25:152
2413936993921,13cyclictest2143-21upowerd09:52:421
2413936993911,16cyclictest2751-21gdbus11:35:111
24476202380,5sleep10-21swapper/108:45:021
241394299388,7cyclictest2459103-21head09:15:233
241394299386,6cyclictest2497736-21wc11:05:213
2413942993818,14cyclictest42-21ksoftirqd/308:25:403
2413942993817,17cyclictest4497-21PK-Backend08:55:273
2413942993814,6cyclictest141rcu_preempt10:30:003
241394299381,12cyclictest2467577-21tr09:40:143
241394299381,10cyclictest2522615-21grep12:15:253
241393999386,5cyclictest141rcu_preempt11:12:002
241393999385,16cyclictest34-21ksoftirqd/208:57:342
2413939993829,5cyclictest34-21ksoftirqd/212:34:232
2413939993829,5cyclictest34-21ksoftirqd/212:34:232
2413939993822,12cyclictest2465387-21cron09:35:002
241393999382,11cyclictest438-20systemd-journal12:20:002
241393999381,20cyclictest2416262-21munin-node07:15:062
241393999381,16cyclictest2456619-21chrt09:10:022
241393699388,10cyclictest2474124-21pool-udisksd09:59:561
2413936993823,12cyclictest2518480-21cpuspeed_turbos12:05:121
2413936993823,11cyclictest26-21ksoftirqd/109:45:211
241394299372,9cyclictest2520297-21cstates12:10:163
241394299371,13cyclictest2473897-21chrt09:57:033
241394299371,11cyclictest2490876-21chrt10:45:203
241394299371,10cyclictest2444395-21tr08:35:143
2413939993727,6cyclictest2519821-21ls12:10:002
2413939993721,13cyclictest34-21ksoftirqd/207:33:102
241393999371,15cyclictest1054-21snmpd07:45:222
241393999371,15cyclictest1054-21snmpd07:45:222
241393699376,9cyclictest0-21swapper/111:05:291
2413936993722,11cyclictest923-21dbus-daemon08:10:371
2413936993722,11cyclictest2528787-21/usr/sbin/munin12:35:001
2413936993722,11cyclictest2528787-21/usr/sbin/munin12:34:591
2413936993720,14cyclictest2455495-21sed09:05:201
2413936993715,16cyclictest2442583-21cat08:30:141
241394299365,27cyclictest4497-21PK-Backend11:53:493
241394299361,10cyclictest2435090-21chrt08:08:213
241393999367,10cyclictest34-21ksoftirqd/210:45:202
2413939993620,12cyclictest2503124-21if_err_enp2s011:20:212
2413939993614,7cyclictest141rcu_preempt08:05:002
241393999361,18cyclictest2419360-21chrt07:23:092
241393999361,14cyclictest1137-21runrttasks08:05:212
2413936993622,10cyclictest26-21ksoftirqd/110:00:251
2413936993621,11cyclictest2457943-21systemd-udevd09:12:471
2413936993620,12cyclictest2417654-21sshd07:18:291
2413936993619,12cyclictest2441121-21entropy08:25:181
241393699361,8cyclictest2527033-21cron12:30:011
2413936993618,14cyclictest2512847-21mktemp11:49:481
241394299352,12cyclictest2446523-21taskset08:40:213
241394299351,10cyclictest2510948-21chrt11:42:473
241393999358,5cyclictest141rcu_preempt10:05:042
241393999356,10cyclictest34-21ksoftirqd/212:00:262
2413939993525,6cyclictest34-21ksoftirqd/208:16:292
2413939993519,5cyclictest34-21ksoftirqd/210:20:282
2413939993513,6cyclictest141rcu_preempt08:25:022
241393999351,20cyclictest2523781-21/usr/sbin/munin12:20:292
2413939993511,7cyclictest141rcu_preempt09:37:262
241393999351,16cyclictest2479124-21sshd10:12:282
241393699357,9cyclictest0-21swapper/112:20:201
2413936993523,9cyclictest2161-21gmain12:20:021
2413936993521,10cyclictest1043-21gdbus07:38:561
2413936993521,10cyclictest1043-21gdbus07:23:291
2413936993520,11cyclictest2431095-21pluginstate07:55:251
2413936993520,11cyclictest2429862-21seq07:52:491
2413936993519,12cyclictest2480176-21sed10:15:211
2413936993519,12cyclictest2461553-21seq09:24:121
2413936993519,11cyclictest438-20systemd-journal09:30:001
2413936993518,13cyclictest2496492-21sed11:00:301
2413936993518,12cyclictest2520433-21expr12:10:161
2413936993517,13cyclictest2506311-21grep11:30:161
25015622340,5sleep10-21swapper/111:15:281
25015622340,5sleep10-21swapper/111:15:281
2413939993425,5cyclictest34-21ksoftirqd/209:41:002
2413939993419,11cyclictest25206712sleep212:10:232
2413939993419,11cyclictest2512041-21cat11:45:222
2413939993418,12cyclictest2504945-21cut11:25:222
241393999341,18cyclictest34-21ksoftirqd/209:20:212
241393999341,16cyclictest2498241-21sort11:05:302
241393999341,12cyclictest924-21NetworkManager09:01:212
241393999341,12cyclictest924-21NetworkManager09:01:212
241393999341,11cyclictest2470836-21sort09:50:112
2413936993427,5cyclictest26-21ksoftirqd/110:38:251
2413936993422,8cyclictest2516747-21grep12:00:121
2413936993419,11cyclictest2477577-21seq10:09:521
2413936993419,11cyclictest2449495-21wc08:50:121
2413936993417,12cyclictest2434488-21ntpq08:05:241
2413936993415,14cyclictest2492301-21expr10:50:141
241393699341,30cyclictest0-21swapper/111:58:101
2413936993412,17cyclictest2484239-21python310:25:281
25261662330,8chrt0-21swapper/212:25:232
241393999334,9cyclictest0-21swapper/208:40:232
241393999334,25cyclictest2506368-21cut11:30:152
2413939993316,5cyclictest34-21ksoftirqd/211:00:222
2413939993312,6cyclictest141rcu_preempt09:45:132
241393999331,17cyclictest2415609-21munin-node10:36:122
241393999331,15cyclictest2419976-21grep07:25:142
241393999331,11cyclictest2529210-21cut12:35:162
2413936993320,9cyclictest2514752-21sed11:55:021
241393699331,8cyclictest438-20systemd-journal10:42:291
2413936993317,12cyclictest2505156-21ntpq11:25:251
2413936993316,12cyclictest2437387-21cat08:15:141
2413936993316,12cyclictest2414518-21basename07:10:191
2413936993312,17cyclictest2529760-21tr12:35:261
2413936993312,17cyclictest2481959-21df10:20:141
2413939993214,14cyclictest2458801-21df09:15:162
2413939993211,6cyclictest141rcu_preempt10:15:242
2413936993216,12cyclictest2453678-21interrupts09:00:211
2413936993216,12cyclictest2453678-21interrupts09:00:211
2413936993216,11cyclictest2468109-21ntpq09:40:231
2413936993215,13cyclictest2466643-21cat09:35:301
2413936993211,17cyclictest2448450-21/usr/sbin/munin08:45:271
2413936993210,17cyclictest2491934-21sed10:50:011
2413936993112,14cyclictest2445225-21timerwakeupswit08:35:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional