You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 03:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jan 10, 2026 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36952872930,8sleep10-21swapper/119:10:131
197399930,8rtkit-daemon1972-21rtkit-daemon20:52:003
197399910,9rtkit-daemon1972-21rtkit-daemon19:17:533
197399910,8rtkit-daemon1972-21rtkit-daemon00:34:463
197399880,53rtkit-daemon0-21swapper/319:06:433
36937422710,19sleep013-21ksoftirqd/019:05:180
37775542640,6sleep30-21swapper/323:00:233
197399630,10rtkit-daemon1972-21rtkit-daemon23:23:522
369434926140,14sleep20-21swapper/219:06:252
369314426140,14sleep10-21swapper/119:05:041
369479699600,57cyclictest1972-21rtkit-daemon21:12:412
197399600,9rtkit-daemon1972-21rtkit-daemon22:57:053
197399600,9rtkit-daemon1972-21rtkit-daemon20:22:492
37393502590,16sleep013-21ksoftirqd/021:14:560
197399590,9rtkit-daemon1972-21rtkit-daemon19:54:470
37288702570,6chrt0-21swapper/020:45:020
369479699570,54cyclictest1972-21rtkit-daemon23:44:502
197399560,9rtkit-daemon1972-21rtkit-daemon22:18:333
37626212550,8sleep30-21swapper/322:20:183
37578482540,7sleep1251rcuc/122:05:261
37176682540,21sleep013-21ksoftirqd/020:14:550
197399530,8rtkit-daemon1972-21rtkit-daemon21:44:552
197399530,8rtkit-daemon1972-21rtkit-daemon19:34:170
37967982520,7sleep03796800-21rt-features23:55:250
37967982520,7sleep03796800-21rt-features23:55:240
37109232520,5sleep20-21swapper/219:55:012
37605252510,7sleep04497-21PK-Backend22:15:110
197399510,38rtkit-daemon13-21ksoftirqd/021:07:480
37258652500,15chrt34-21ksoftirqd/220:35:222
36996742500,5sleep1251rcuc/119:20:231
3694801995019,26cyclictest3794688-21irqstats23:50:193
369479699501,45cyclictest1972-21rtkit-daemon22:54:242
369479699501,45cyclictest1972-21rtkit-daemon22:54:242
369479699501,12cyclictest141rcu_preempt21:45:112
369479699501,12cyclictest141rcu_preempt21:45:112
197399490,33rtkit-daemon4497-21PK-Backend23:05:161
37810302480,6chrt3781028-21head23:10:220
369479699485,17cyclictest0-21swapper/219:55:142
369479699482,19cyclictest1033-21gdbus22:30:032
3694793994824,13cyclictest26-21ksoftirqd/122:56:081
3694793994824,13cyclictest26-21ksoftirqd/122:00:001
3694789994815,11cyclictest3735140-21sed21:00:290
38096272470,18chrt34-21ksoftirqd/200:32:012
37135072470,8sleep2341ksoftirqd/220:00:252
37034542470,4chrt0-21swapper/119:30:291
3694801994717,6cyclictest141rcu_preempt23:20:283
3694796994715,7cyclictest3797867-21cpuspeed_turbos00:00:142
3694796994712,9cyclictest34-21ksoftirqd/223:50:202
3694789994724,14cyclictest13-21ksoftirqd/020:30:000
3694789994724,13cyclictest13-21ksoftirqd/021:35:270
3694789994723,13cyclictest13-21ksoftirqd/022:32:030
3694789994721,13cyclictest13-21ksoftirqd/019:20:250
3694801994616,11cyclictest3767525-21ls22:35:013
3694796994615,14cyclictest141rcu_preempt21:40:022
369479699461,41cyclictest1972-21rtkit-daemon20:56:572
3694793994622,14cyclictest26-21ksoftirqd/123:30:021
197399460,33rtkit-daemon4497-21PK-Backend23:34:420
3694801994512,13cyclictest141rcu_preempt00:16:103
369479699452,19cyclictest3699484-21latency19:20:192
3694793994523,14cyclictest26-21ksoftirqd/121:10:571
369479399451,14cyclictest3760137-21dump-pmu-power22:15:001
3694789994523,13cyclictest13-21ksoftirqd/022:56:040
369479699441,21cyclictest3763929-21cpuspeed22:25:122
369479699441,19cyclictest3734814-21needreboot21:00:222
3694796994410,11cyclictest3770732-21timerwakeupswit22:40:282
3694793994422,13cyclictest26-21ksoftirqd/119:35:191
3694793994421,14cyclictest26-21ksoftirqd/121:25:021
3694789994422,13cyclictest13-21ksoftirqd/022:10:010
197399440,32rtkit-daemon4497-21PK-Backend22:13:510
369480199432,22cyclictest3779586-21cat23:05:283
3694801994313,9cyclictest42-21ksoftirqd/300:25:253
3694801994313,9cyclictest42-21ksoftirqd/300:25:253
369479699435,11cyclictest34-21ksoftirqd/222:37:122
369479699431,23cyclictest3737383-21apt-config21:10:012
369479699431,20cyclictest3806129-21cron00:25:002
369479699431,20cyclictest3752826-21timerwakeupswit21:50:282
369479699431,19cyclictest3706005-21cpuspeed_turbos19:40:122
3694793994321,14cyclictest26-21ksoftirqd/122:40:241
3694793994320,14cyclictest26-21ksoftirqd/123:30:141
3694789994321,8cyclictest13-21ksoftirqd/020:35:020
3694789994320,14cyclictest13-21ksoftirqd/020:55:020
197399430,31rtkit-daemon4497-21PK-Backend23:41:541
369480199426,19cyclictest141rcu_preempt20:40:013
369480199424,17cyclictest3789902-21switchtime23:35:273
369480199422,22cyclictest3702445-21cpuspeed19:30:133
3694801994217,7cyclictest42-21ksoftirqd/319:10:103
3694801994217,10cyclictest42-21ksoftirqd/320:27:423
3694801994215,7cyclictest3724130-21ntpq20:30:243
3694801994214,11cyclictest3764573-21netstat22:25:223
369479699427,17cyclictest3761390-21timerandwakeup22:15:282
369479699422,20cyclictest3779024-21hddtemp_smartct23:05:172
369479699421,17cyclictest3800087-21grep00:05:222
369479699421,17cyclictest3800087-21grep00:05:212
3694789994229,9cyclictest13-21ksoftirqd/000:16:370
3694789994219,13cyclictest13-21ksoftirqd/019:38:490
197399420,29rtkit-daemon0-21swapper/020:05:420
38050402410,5chrt0-21swapper/300:20:143
3694801994113,11cyclictest3791544-21proc_pri23:40:243
369480199411,19cyclictest3801733-21/usr/sbin/munin00:10:203
3694801994111,6cyclictest141rcu_preempt19:30:003
3694801994110,17cyclictest42-21ksoftirqd/319:35:173
369479699413,15cyclictest3802726-21date00:15:012
3694796994124,13cyclictest34-21ksoftirqd/222:06:132
369479699412,10cyclictest3807905-21munin-run00:30:012
369479699412,10cyclictest3807905-21munin-run00:30:002
369479699411,19cyclictest1-21systemd22:00:002
369479699411,18cyclictest0-21swapper/221:26:052
369479699411,12cyclictest3789958-21switchtime23:35:272
369479699411,12cyclictest3704634-21wc19:35:192
3694796994110,5cyclictest141rcu_preempt20:08:582
369479399419,5cyclictest141rcu_preempt21:55:021
369479399418,6cyclictest141rcu_preempt21:20:021
369479399413,6cyclictest2069-21gnome-shell20:16:581
369479399413,6cyclictest2069-21gnome-shell20:16:581
3694793994125,11cyclictest3713508-21/usr/sbin/munin20:00:251
3694793994124,13cyclictest26-21ksoftirqd/119:45:121
3694793994123,6cyclictest26-21ksoftirqd/123:25:021
3694793994123,5cyclictest26-21ksoftirqd/123:50:011
3694793994123,13cyclictest26-21ksoftirqd/122:55:021
3694793994123,13cyclictest26-21ksoftirqd/122:55:021
369479399411,9cyclictest3795100-21grep23:50:271
3694789994123,6cyclictest13-21ksoftirqd/021:25:020
3694789994123,13cyclictest13-21ksoftirqd/023:50:000
3694789994123,13cyclictest13-21ksoftirqd/020:40:000
3694789994122,14cyclictest13-21ksoftirqd/022:55:010
3694789994122,14cyclictest13-21ksoftirqd/022:55:010
3694789994121,6cyclictest13-21ksoftirqd/020:45:230
197399410,28rtkit-daemon0-21swapper/020:20:500
37728672400,7sleep34497-21PK-Backend22:50:003
37573502400,5sleep3411rcuc/322:05:173
3694801994023,12cyclictest438-20systemd-journal21:36:563
369480199401,19cyclictest3797441-40updatedb.mlocat00:00:023
369480199401,19cyclictest3797441-40updatedb.mlocat00:00:023
3694801994011,4cyclictest141rcu_preempt21:45:003
3694801994010,6cyclictest141rcu_preempt20:45:273
3694796994023,13cyclictest34-21ksoftirqd/221:33:212
369479699402,19cyclictest3696460-21unin-run19:15:012
369479699402,18cyclictest3742845-21munin-run21:25:012
369479699401,20cyclictest3797637-21latency_hist00:00:032
369479699401,17cyclictest3803274-21awk00:15:172
369479699401,16cyclictest0-21swapper/220:40:182
369479699401,15cyclictest0-21swapper/220:45:242
369479699401,13cyclictest3760141-21idleruntime-cro22:15:002
369479699401,13cyclictest3755924-21awk22:00:212
369479699401,12cyclictest3703278-21ls19:30:282
369479399406,7cyclictest141rcu_preempt20:45:311
3694789994023,6cyclictest13-21ksoftirqd/021:00:020
369478999401,8cyclictest3789731-21awk23:35:210
3694789994017,18cyclictest3753252-21unin-run21:55:000
197399400,28rtkit-daemon0-21swapper/021:20:000
37491962390,7sleep0161rcuc/021:40:270
369480199397,6cyclictest141rcu_preempt23:45:183
369480199393,17cyclictest141rcu_preempt19:20:183
369480199392,19cyclictest3726658-21apt-get20:40:093
3694801993910,6cyclictest141rcu_preempt00:00:213
3694801993910,5cyclictest141rcu_preempt00:05:113
3694801993910,5cyclictest141rcu_preempt00:05:113
3694801993910,14cyclictest141rcu_preempt21:05:193
3694801993910,14cyclictest141rcu_preempt19:50:203
369479699395,9cyclictest34-21ksoftirqd/220:10:102
3694796993924,11cyclictest3702104-21cron19:30:002
3694796993922,13cyclictest34-21ksoftirqd/223:27:502
369479699392,16cyclictest3696471-21munin-node19:15:062
369479699391,19cyclictest3777292-21awk23:00:182
369479699391,16cyclictest0-21swapper/200:00:022
369479699391,12cyclictest3724271-21gzip20:30:272
369479699391,12cyclictest3722407-21timerandwakeup20:25:282
3694793993922,5cyclictest26-21ksoftirqd/119:45:001
3694793993922,12cyclictest438-20systemd-journal19:59:551
369478999396,11cyclictest3694783-21cyclictest00:05:010
3694789993923,11cyclictest3713766-21cat20:00:300
3694789993922,6cyclictest13-21ksoftirqd/023:10:000
3694789993920,6cyclictest13-21ksoftirqd/021:30:010
369478999391,7cyclictest3698641-21cron19:20:000
197399390,28rtkit-daemon0-21swapper/023:41:240
369480199389,14cyclictest141rcu_preempt22:15:013
369480199389,14cyclictest141rcu_preempt20:15:013
369480199388,15cyclictest141rcu_preempt19:55:233
369480199384,12cyclictest141rcu_preempt21:20:193
3694801993812,9cyclictest42-21ksoftirqd/320:25:013
369480199381,20cyclictest3750768-21ntp_states21:45:223
369480199381,20cyclictest3750768-21ntp_states21:45:223
3694801993811,13cyclictest141rcu_preempt19:50:023
369479699389,5cyclictest141rcu_preempt20:16:552
369479699389,5cyclictest141rcu_preempt20:16:552
3694796993822,5cyclictest34-21ksoftirqd/222:48:512
3694796993821,6cyclictest34-21ksoftirqd/219:46:122
3694796993819,5cyclictest34-21ksoftirqd/222:20:092
369479699381,18cyclictest0-21swapper/223:20:012
369479699381,18cyclictest0-21swapper/223:20:012
369479699381,16cyclictest0-21swapper/223:50:012
369479399385,8cyclictest26-21ksoftirqd/100:25:001
369479399384,11cyclictest26-21ksoftirqd/120:37:261
369479399382,32cyclictest3771778-21df_inode22:45:161
3694793993823,10cyclictest3714945-21cat20:05:191
3694793993821,6cyclictest26-21ksoftirqd/100:30:131
3694793993819,14cyclictest26-21ksoftirqd/100:35:411
369479399381,33cyclictest3769293-21seq22:38:461
369478999387,5cyclictest141rcu_preempt21:32:220
369478999385,6cyclictest141rcu_preempt22:20:180
3694789993823,11cyclictest3785986-21if_enp2s023:25:190
3694789993822,5cyclictest13-21ksoftirqd/000:35:000
3694789993822,12cyclictest3810147-21find00:35:100
3694789993821,6cyclictest13-21ksoftirqd/000:23:590
3694789993821,5cyclictest13-21ksoftirqd/021:50:000
3694789993821,5cyclictest13-21ksoftirqd/021:50:000
3694789993821,13cyclictest13-21ksoftirqd/019:10:280
3694789993820,14cyclictest13-21ksoftirqd/022:30:020
3694789993820,14cyclictest13-21ksoftirqd/000:30:020
3694789993820,14cyclictest13-21ksoftirqd/000:30:020
3694789993820,13cyclictest13-21ksoftirqd/000:10:190
369480199379,14cyclictest141rcu_preempt21:25:133
369480199378,14cyclictest141rcu_preempt21:50:133
369480199378,13cyclictest141rcu_preempt21:00:233
369480199377,15cyclictest141rcu_preempt19:40:133
369480199376,13cyclictest141rcu_preempt21:11:003
3694801993722,11cyclictest976-21rs:main3
3694801993721,12cyclictest3746044-21sshd21:31:563
369480199371,32cyclictest3769438-21pluginstate22:40:013
3694801993711,6cyclictest141rcu_preempt22:40:133
3694796993712,9cyclictest34-21ksoftirqd/222:55:212
369479699371,20cyclictest0-21swapper/200:35:282
369479399373,5cyclictest141rcu_preempt19:29:561
3694793993720,6cyclictest26-21ksoftirqd/123:05:001
3694793993720,5cyclictest26-21ksoftirqd/121:25:121
3694793993719,6cyclictest26-21ksoftirqd/122:05:001
3694793993719,13cyclictest26-21ksoftirqd/120:40:191
3694789993722,11cyclictest3755928-21ntpq22:00:210
3694789993720,5cyclictest13-21ksoftirqd/023:50:130
3694789993719,6cyclictest13-21ksoftirqd/022:00:010
3694789993719,14cyclictest13-21ksoftirqd/023:05:010
3694789993711,4cyclictest141rcu_preempt22:40:130
369480199366,6cyclictest141rcu_preempt23:35:003
369480199364,17cyclictest141rcu_preempt00:35:253
3694796993620,5cyclictest34-21ksoftirqd/220:54:002
369479399368,5cyclictest141rcu_preempt22:30:041
369479399368,5cyclictest141rcu_preempt21:45:141
369479399368,5cyclictest141rcu_preempt21:45:141
369479399368,5cyclictest141rcu_preempt21:35:131
369479399367,5cyclictest141rcu_preempt00:26:531
369479399367,5cyclictest141rcu_preempt00:26:531
369479399366,11cyclictest26-21ksoftirqd/120:30:541
3694793993625,7cyclictest3736040-21cpuspeed_turbos21:05:121
3694793993622,10cyclictest3803024-21tr00:15:131
3694793993620,5cyclictest26-21ksoftirqd/120:30:021
3694793993620,11cyclictest3790146-21sshd23:35:561
3694793993619,6cyclictest26-21ksoftirqd/120:15:001
3694793993619,6cyclictest26-21ksoftirqd/119:15:071
3694793993619,5cyclictest26-21ksoftirqd/121:30:141
369479399361,12cyclictest3797412-21latency_hist00:00:021
369479399361,12cyclictest3797412-21latency_hist00:00:021
369479399361,12cyclictest141rcu_preempt19:50:461
369478999366,6cyclictest141rcu_preempt00:05:130
369478999366,6cyclictest141rcu_preempt00:05:130
3694789993619,12cyclictest13-21ksoftirqd/020:20:010
3694789993619,12cyclictest13-21ksoftirqd/020:20:010
3694789993614,17cyclictest3701517-21/usr/sbin/munin19:25:270
369480199356,7cyclictest141rcu_preempt23:30:013
369480199355,6cyclictest141rcu_preempt22:05:003
3694796993519,5cyclictest34-21ksoftirqd/221:16:062
3694796993519,11cyclictest438-20systemd-journal23:35:002
369479399353,9cyclictest26-21ksoftirqd/121:03:421
3694793993524,7cyclictest3765359-21idleruntime-cro22:30:011
3694793993523,8cyclictest3782646-21chrt23:15:221
3694793993523,8cyclictest3782646-21chrt23:15:221
3694793993518,6cyclictest26-21ksoftirqd/123:10:291
3694793993518,6cyclictest26-21ksoftirqd/120:52:191
369479399351,29cyclictest4497-21PK-Backend22:20:331
3694793993511,5cyclictest3799392-21/usr/sbin/munin00:05:261
3694793993511,5cyclictest3799392-21/usr/sbin/munin00:05:261
369478999357,6cyclictest141rcu_preempt19:55:180
369478999357,5cyclictest141rcu_preempt23:23:310
369478999352,7cyclictest0-21swapper/022:47:560
3694789993520,10cyclictest3708096-21expr19:45:190
369480199348,5cyclictest141rcu_preempt22:55:023
369480199348,5cyclictest141rcu_preempt22:55:013
3694801993418,11cyclictest3782798-21awk23:15:243
3694801993418,11cyclictest3782798-21awk23:15:233
369480199341,29cyclictest4497-21PK-Backend20:55:213
3694793993420,10cyclictest3732388-21/usr/sbin/munin20:55:191
3694793993417,6cyclictest26-21ksoftirqd/100:11:221
3694793993417,13cyclictest26-21ksoftirqd/122:18:431
369478999347,5cyclictest141rcu_preempt22:35:020
369479699331,9cyclictest0-21swapper/223:10:292
369479399332,11cyclictest141rcu_preempt21:43:501
3694793993317,5cyclictest26-21ksoftirqd/100:00:291
369480199329,5cyclictest141rcu_preempt20:19:053
369480199329,5cyclictest141rcu_preempt20:19:053
369480199325,6cyclictest141rcu_preempt21:15:093
369480199324,5cyclictest141rcu_preempt23:15:023
369480199321,27cyclictest3714125-21chrt20:03:523
3694793993216,11cyclictest3719679-21acpi20:20:101
369478999321,9cyclictest4497-21PK-Backend19:40:120
3694789993217,11cyclictest3782836-21sh23:15:250
3694789993217,11cyclictest3782836-21sh23:15:250
369480199311,6cyclictest4497-21PK-Backend21:55:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional