You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 18:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 15, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
256115722780,7sleep12561153-21ntp_states07:05:241
256192321360,12sleep3421ksoftirqd/307:10:013
1623991070,26rtkit-daemon2197014-21PK-Backend09:40:173
25669782900,7sleep31753-21gnome-shell07:20:243
25669782900,7sleep31753-21gnome-shell07:20:233
162399890,10rtkit-daemon1753-21gnome-shell11:25:082
25610062800,16sleep234-21ksoftirqd/207:05:222
162399800,6rtkit-daemon2197014-21PK-Backend09:56:222
162399780,5rtkit-daemon0-21swapper/211:00:042
26023522770,4sleep3411rcuc/309:00:293
162399760,5rtkit-daemon0-21swapper/210:50:552
162399750,6rtkit-daemon2197014-21PK-Backend08:31:342
162399750,5rtkit-daemon0-21swapper/211:46:372
162399750,5rtkit-daemon0-21swapper/211:38:532
256165727453,14sleep00-21swapper/007:07:340
162399730,5rtkit-daemon2197014-21PK-Backend08:26:262
162399700,6rtkit-daemon0-21swapper/207:55:382
26422502670,5sleep00-21swapper/010:55:010
162399600,10rtkit-daemon1753-21gnome-shell08:17:193
256199399570,53cyclictest1622-21rtkit-daemon09:06:320
2562003995619,11cyclictest2595132-21cat08:40:232
25773262550,15sleep234-21ksoftirqd/207:50:202
2562003995517,12cyclictest2676145-21sshd12:27:472
2562003995416,12cyclictest2585172-21cat08:10:302
162399540,8rtkit-daemon1622-21rtkit-daemon08:50:511
2562003995322,9cyclictest2633535-21mktemp10:30:022
2561993995329,20cyclictest1-21systemd07:35:010
2562003995220,7cyclictest1069-21runrttasks09:26:422
2562003995217,13cyclictest2583538-40dpkg08:06:132
256199399513,4cyclictest141rcu_preempt09:42:010
26350872490,14sleep0131ksoftirqd/010:33:360
256200399491,25cyclictest1753-21gnome-shell08:23:022
162399490,6rtkit-daemon0-21swapper/308:10:253
162399490,35rtkit-daemon1753-21gnome-shell10:43:172
2562003994825,5cyclictest0-21swapper/208:04:042
256199399481,15cyclictest25802742grep08:00:010
256200399472,24cyclictest1753-21gnome-shell07:12:512
256200399472,23cyclictest1753-21gnome-shell11:23:432
162399470,34rtkit-daemon0-21swapper/312:26:213
25691082460,8sleep30-21swapper/307:25:313
2562003994617,9cyclictest1753-21gnome-shell09:40:012
2562003994616,11cyclictest2629651-21sshd10:19:472
256200399461,24cyclictest1753-21gnome-shell11:57:152
2561996994626,15cyclictest3242253-21nautilus10:30:011
26636352450,6sleep22197014-21PK-Backend11:50:342
2562007994526,7cyclictest42-21ksoftirqd/308:06:213
2562007994525,7cyclictest42-21ksoftirqd/311:00:223
2562007994510,17cyclictest1753-21gnome-shell09:20:283
2561996994527,13cyclictest2653160-21apt-get11:25:001
2561996994527,13cyclictest2633163-21apt-get10:30:091
2561996994523,17cyclictest2576058-21python307:45:301
162399450,34rtkit-daemon2197014-21PK-Backend08:19:502
26773912440,7chrt2197014-21PK-Backend12:30:302
26773912440,7chrt2197014-21PK-Backend12:30:302
26277042440,8chrt2197014-21PK-Backend10:12:220
26277042440,8chrt2197014-21PK-Backend10:12:220
2562007994423,6cyclictest42-21ksoftirqd/311:05:253
2562007994422,6cyclictest42-21ksoftirqd/309:10:163
2562007994416,10cyclictest2676906-21cat12:30:183
2562007994416,10cyclictest2676906-21cat12:30:183
2562003994434,7cyclictest1753-21gnome-shell09:42:162
2561993994424,16cyclictest1753-21gnome-shell12:21:070
25751942430,7sleep342-21ksoftirqd/307:45:153
256200399437,18cyclictest1753-21gnome-shell11:30:162
256199699436,6cyclictest141rcu_preempt09:42:531
162399430,6rtkit-daemon2197014-21PK-Backend10:23:233
26194752420,16sleep234-21ksoftirqd/209:50:152
2562007994224,6cyclictest42-21ksoftirqd/309:31:283
2562007994223,6cyclictest42-21ksoftirqd/309:52:053
2562007994223,6cyclictest42-21ksoftirqd/307:50:153
2562007994222,16cyclictest42-21ksoftirqd/307:35:023
2561996994229,8cyclictest438-20systemd-journal09:49:551
256199699421,38cyclictest0-21swapper/109:25:271
2561996994212,12cyclictest26-21ksoftirqd/107:12:121
162399420,30rtkit-daemon1753-21gnome-shell09:47:152
26502802410,7sleep20-21swapper/211:15:152
25736372410,7sleep32197014-21PK-Backend07:40:153
2562007994123,6cyclictest42-21ksoftirqd/308:57:073
2562007994123,5cyclictest42-21ksoftirqd/311:33:543
2562007994123,5cyclictest42-21ksoftirqd/308:50:033
2562007994122,6cyclictest42-21ksoftirqd/309:05:403
2562007994122,16cyclictest42-21ksoftirqd/308:39:243
2562007994115,22cyclictest2634113-40lsb_release10:30:203
2562003994122,15cyclictest3242253-21nautilus12:35:042
2562003994122,14cyclictest2593398-21cat08:35:222
2561996994123,14cyclictest2623460-21ntp_states10:00:291
2561996994118,11cyclictest141rcu_preempt10:48:471
2561996994118,11cyclictest141rcu_preempt10:48:471
2561996994113,22cyclictest2594447-21wc08:40:021
2561996994113,11cyclictest26-21ksoftirqd/108:45:291
256199399416,19cyclictest13-21ksoftirqd/010:25:260
2561993994120,16cyclictest3242253-21nautilus09:00:010
256199399411,16cyclictest141rcu_preempt09:03:490
2562007994029,8cyclictest42-21ksoftirqd/310:55:003
2562007994023,6cyclictest42-21ksoftirqd/311:42:303
2562007994023,6cyclictest42-21ksoftirqd/307:15:013
2562007994023,13cyclictest2666363-21grep12:00:153
2562007994021,6cyclictest42-21ksoftirqd/312:39:413
2562007994020,16cyclictest42-21ksoftirqd/310:45:003
2562007994019,13cyclictest42-21ksoftirqd/310:15:023
2562007994019,13cyclictest42-21ksoftirqd/310:15:023
256200399401,9cyclictest2574920-21dump-pmu-power07:45:002
2562003994017,20cyclictest2610290-21awk09:25:012
2561996994028,8cyclictest2568785-21grep07:25:231
2561996994026,10cyclictest2651393-21sshd11:19:461
2561996994025,11cyclictest2641880-21sshd10:51:461
256199699402,34cyclictest1753-21gnome-shell12:00:151
2561996994021,15cyclictest2581968-21pool-org.gnome.08:05:021
256199699401,14cyclictest26061412uname09:10:281
2561993994030,6cyclictest2567314-21awk07:20:300
2561993994030,6cyclictest2567314-21awk07:20:300
2561993994025,11cyclictest2595914-21sshd08:43:480
2561993994022,13cyclictest438-20systemd-journal10:05:010
256199399401,13cyclictest141rcu_preempt11:38:390
2561993994010,9cyclictest13-21ksoftirqd/011:40:500
162399400,29rtkit-daemon2197014-21PK-Backend12:05:262
2562007993925,9cyclictest438-20systemd-journal10:05:013
2562007993922,6cyclictest42-21ksoftirqd/311:16:493
2562007993922,6cyclictest42-21ksoftirqd/308:23:233
256200799391,8cyclictest2197014-21PK-Backend10:16:353
2562003993927,9cyclictest2634731-21timerandwakeup10:30:342
256200399392,33cyclictest1753-21gnome-shell09:18:492
2562003993921,14cyclictest2643912-40apt-check10:57:192
2562003993910,9cyclictest0-21swapper/211:05:222
2561996993925,10cyclictest2663913-21sshd11:51:461
2561996993924,10cyclictest2626100-21cat10:10:021
2561996993918,17cyclictest2645682-21switchtime11:00:361
2561996993917,17cyclictest2653585-21mktemp11:25:031
2561996993912,12cyclictest2668046-21cpuspeed_turbos12:05:131
2561993993928,7cyclictest2678817-21cstates12:35:190
256199399392,7cyclictest0-21swapper/012:00:230
2561993993922,12cyclictest855-21dbus-daemon08:55:000
2561993993922,12cyclictest2628498-21cat10:15:150
2561993993917,16cyclictest1-21systemd11:30:010
162399390,5rtkit-daemon0-21swapper/210:14:172
162399390,5rtkit-daemon0-21swapper/210:14:172
162399390,27rtkit-daemon0-21swapper/207:20:442
162399390,27rtkit-daemon0-21swapper/207:20:442
2562007993827,7cyclictest1069-21runrttasks11:40:023
256200799382,32cyclictest1753-21gnome-shell08:40:583
256200799382,31cyclictest1753-21gnome-shell09:16:373
2562007993822,12cyclictest2663383-21grep11:50:263
256200799381,8cyclictest1753-21gnome-shell12:14:533
2562007993815,7cyclictest141rcu_preempt12:15:203
2562007993814,6cyclictest391irq_work/310:07:043
256200399385,29cyclictest2640435-21perf-exec10:50:002
256200399385,29cyclictest2640435-21perf-exec10:50:002
256200399384,17cyclictest1753-21gnome-shell07:26:242
256200399382,32cyclictest1753-21gnome-shell10:38:512
256199699389,12cyclictest26-21ksoftirqd/107:30:241
2561996993822,12cyclictest2661452-21interrupts11:45:211
2561996993821,13cyclictest2572145-21irqstats07:35:231
2561996993821,12cyclictest438-20systemd-journal12:19:591
2561996993816,18cyclictest2679996-21sshd12:38:471
2561996993816,18cyclictest2660864-21latency_hist11:45:021
2561996993811,23cyclictest2565905-21seq07:16:391
2561993993826,8cyclictest1824-21JS14:31:440
2561993993819,14cyclictest2614368-21df09:35:160
2561993993819,14cyclictest2591457-21df08:30:160
2561993993818,16cyclictest1069-21runrttasks10:24:090
256199399381,34cyclictest0-21swapper/009:19:410
256199399381,14cyclictest26716532expr12:15:150
162399380,27rtkit-daemon2197014-21PK-Backend12:11:132
162399380,27rtkit-daemon0-21swapper/207:39:132
162399380,26rtkit-daemon0-21swapper/209:10:482
25997192370,7sleep32197014-21PK-Backend08:55:033
256200799373,30cyclictest1753-21gnome-shell09:27:263
2562007993721,5cyclictest42-21ksoftirqd/312:25:013
2562007993719,13cyclictest2642601-21gpgv10:55:023
2562007993718,6cyclictest42-21ksoftirqd/307:55:163
2562007993717,15cyclictest438-20systemd-journal11:12:473
2562007993713,6cyclictest391irq_work/309:49:373
256200399377,7cyclictest0-21swapper/209:09:492
2562003993721,12cyclictest2625156-21ntp_kernel_err10:05:252
2562003993720,13cyclictest2566218-21pool-org.gnome.07:20:012
256200399371,21cyclictest175350gnome-shell08:45:302
256199699378,11cyclictest26-21ksoftirqd/108:28:431
2561996993725,8cyclictest1753-21gnome-shell09:58:361
2561996993723,10cyclictest2644170-21cron11:00:001
2561996993719,14cyclictest2603708-21irqstats09:05:231
2561996993719,13cyclictest2675952-21users12:25:331
256199699371,4cyclictest0-21swapper/108:16:571
2561996993711,13cyclictest26-21ksoftirqd/109:50:181
256199399379,5cyclictest2499055-21kworker/0:008:16:050
256199399372,8cyclictest2640303-21sshd10:48:480
256199399372,8cyclictest2640303-21sshd10:48:480
2561993993725,7cyclictest1753-21gnome-shell07:47:450
2561993993723,9cyclictest1753-21gnome-shell08:39:280
2561993993723,10cyclictest13-21ksoftirqd/009:20:260
256199399372,30cyclictest0-21swapper/011:30:090
2561993993722,11cyclictest13-21ksoftirqd/010:57:230
2561993993721,12cyclictest2677350-21ntp_states12:30:290
2561993993721,12cyclictest2677350-21ntp_states12:30:280
2561993993721,12cyclictest2563406-21cat07:10:360
2561993993719,13cyclictest2576909-21awk07:50:130
2561993993718,15cyclictest13-21ksoftirqd/007:29:020
2561993993718,14cyclictest3242253-21nautilus10:10:030
256199399371,33cyclictest0-21swapper/011:45:280
256199399371,31cyclictest2637941-21ntpq10:40:270
2561993993711,7cyclictest17531gnome-shell11:50:430
162399370,26rtkit-daemon0-21swapper/207:33:012
2562007993619,12cyclictest438-20systemd-journal12:10:003
2562007993617,15cyclictest3242255-21gdbus07:20:003
256200799361,15cyclictest2652571-21python311:20:293
2562003993620,12cyclictest2623223-21cut10:00:242
256199699368,14cyclictest26-21ksoftirqd/110:15:301
256199699366,11cyclictest141rcu_preempt12:30:141
256199699366,11cyclictest141rcu_preempt12:30:141
256199699363,5cyclictest141rcu_preempt08:08:161
2561996993620,12cyclictest855-21dbus-daemon11:35:021
2561996993620,12cyclictest2626875-21if_enp2s010:10:191
2561996993620,12cyclictest2626875-21if_enp2s010:10:191
256199699361,6cyclictest141rcu_preempt07:42:541
256199699361,31cyclictest2585354-21sshd08:11:481
256199399366,26cyclictest1069-21runrttasks08:08:340
2561993993624,8cyclictest2636942-21perf10:40:000
256199399362,30cyclictest1753-21gnome-shell07:42:100
2561993993622,10cyclictest13-21ksoftirqd/008:25:120
2561993993620,12cyclictest2613479-21seq09:31:540
256199399361,8cyclictest2621469-21kernelversion09:55:240
256199399361,8cyclictest1358-1kworker/u9:112:10:010
256199399361,18cyclictest1069-21runrttasks11:16:410
162399360,25rtkit-daemon0-21swapper/211:10:282
162399360,25rtkit-daemon0-21swapper/210:21:502
2562007993520,11cyclictest42-21ksoftirqd/309:37:203
2562007993519,11cyclictest2621628-21cat09:55:243
2562003993526,6cyclictest1753-21gnome-shell09:00:582
256200399352,10cyclictest1753-21gnome-shell11:44:292
256200399351,9cyclictest2197014-21PK-Backend08:51:322
256199699359,10cyclictest26-21ksoftirqd/107:55:281
256199699357,6cyclictest141rcu_preempt12:25:011
256199699357,6cyclictest141rcu_preempt10:25:001
256199699352,6cyclictest0-21swapper/108:20:231
2561996993520,11cyclictest2614229-21expr09:35:141
2561996993519,12cyclictest2648944-21ntp_kernel_pll_11:10:251
2561996993519,12cyclictest2594843-21expr08:40:161
2561996993519,11cyclictest2602436-21cat09:00:321
2561996993517,14cyclictest2607948-21python309:15:371
2561996993515,15cyclictest2576877-20journal-offline07:50:011
2561996993510,6cyclictest141rcu_preempt09:33:531
256199399354,10cyclictest141rcu_preempt09:10:110
2561993993528,4cyclictest2197014-21PK-Backend08:03:520
2561993993526,6cyclictest0-21swapper/009:46:200
2561993993524,7cyclictest1753-21gnome-shell09:54:350
2561993993522,8cyclictest1753-21gnome-shell11:58:550
2561993993520,11cyclictest2645481-21ntp_kernel_pll_11:00:290
2561993993518,13cyclictest2675303-21if_enp2s012:25:200
2561993993517,13cyclictest13-21ksoftirqd/011:13:050
256199399351,15cyclictest2573192-21/usr/sbin/munin07:40:000
256199399351,13cyclictest2611263-21ntp_kernel_pll_09:25:250
256200799342,9cyclictest2654947-40lsb_release11:27:213
2562007993423,6cyclictest42-21ksoftirqd/311:55:123
2562007993423,6cyclictest42-21ksoftirqd/308:30:183
2562007993422,8cyclictest1753-21gnome-shell07:38:333
2562007993419,11cyclictest42-21ksoftirqd/311:45:123
2562003993420,10cyclictest2612613-21expr09:30:162
256200399341,5cyclictest0-21swapper/208:58:292
256199699345,10cyclictest141rcu_preempt07:21:541
256199699345,10cyclictest141rcu_preempt07:21:531
2561996993424,7cyclictest2600471-21/usr/sbin/munin08:55:251
256199699342,15cyclictest2664448-21/usr/sbin/munin11:55:121
2561996993418,12cyclictest2657752-21cat11:35:241
2561996993417,12cyclictest2669795-21cut12:10:161
2561996993417,12cyclictest2608520-21cron09:24:551
2561996993416,13cyclictest2636558-21uptime10:35:331
2561996993413,16cyclictest1-21systemd11:10:021
256199399348,10cyclictest141rcu_preempt12:12:370
256199399342,5cyclictest0-21swapper/011:22:200
256199399342,5cyclictest0-21swapper/008:21:170
2561993993417,13cyclictest13-21ksoftirqd/011:05:120
2562007993322,7cyclictest1753-21gnome-shell08:28:403
2562007993319,10cyclictest2636933-21cron10:40:003
2562007993318,11cyclictest42-21ksoftirqd/310:25:353
2562007993314,6cyclictest2639930-21sed10:45:333
2562007993314,6cyclictest2639930-21sed10:45:333
256200399331,8cyclictest2197014-21PK-Backend12:21:372
256199699332,26cyclictest2591838-21ntp_states08:30:271
256199699331,29cyclictest0-21swapper/110:41:451
2561993993318,11cyclictest2565134-21sort07:15:200
256199399331,14cyclictest141rcu_preempt08:15:010
2562007993218,10cyclictest2580385-21dpkg08:00:103
2562003993216,12cyclictest2575773-21memory07:45:242
256200399319,18cyclictest1753-21gnome-shell12:19:272
2562003993117,10cyclictest2666318-21expr12:00:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional