You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 03:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Jan 03, 2026 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
390535321290,4sleep30-21swapper/321:25:143
1973991040,9rtkit-daemon1972-21rtkit-daemon20:01:090
197399940,9rtkit-daemon1972-21rtkit-daemon23:28:060
38648072920,5sleep04497-21PK-Backend19:30:130
38754112910,7sleep34497-21PK-Backend20:00:143
197399900,9rtkit-daemon1972-21rtkit-daemon23:54:330
197399900,9rtkit-daemon1972-21rtkit-daemon22:23:150
197399890,9rtkit-daemon1972-21rtkit-daemon00:01:290
197399880,9rtkit-daemon1972-21rtkit-daemon21:31:470
197399880,9rtkit-daemon1972-21rtkit-daemon21:16:060
197399880,9rtkit-daemon1972-21rtkit-daemon20:56:120
197399880,9rtkit-daemon1972-21rtkit-daemon20:16:140
197399880,9rtkit-daemon1972-21rtkit-daemon20:16:140
197399880,8rtkit-daemon1972-21rtkit-daemon21:05:350
197399870,9rtkit-daemon1972-21rtkit-daemon23:33:000
197399870,9rtkit-daemon1972-21rtkit-daemon20:26:100
197399870,9rtkit-daemon1972-21rtkit-daemon20:07:440
38941522790,5sleep33894150-21apt20:55:003
385666827437,14sleep10-21swapper/119:07:181
38717352710,5sleep14497-21PK-Backend19:50:141
385656727052,13sleep00-21swapper/019:06:010
39454542640,14sleep1261ksoftirqd/123:20:031
39367232640,7sleep24497-21PK-Backend22:55:202
385655026348,9sleep30-21swapper/319:05:473
3857011996224,7cyclictest141rcu_preempt19:59:591
385658926240,14sleep20-21swapper/219:06:192
197399610,9rtkit-daemon1972-21rtkit-daemon20:01:241
385700799570,53cyclictest1972-21rtkit-daemon00:23:170
197399570,8rtkit-daemon1972-21rtkit-daemon22:05:141
39314292550,6chrt4497-21PK-Backend22:40:152
38980512540,24sleep342-21ksoftirqd/321:05:023
38980512540,24sleep342-21ksoftirqd/321:05:023
197399530,8rtkit-daemon1972-21rtkit-daemon23:08:080
197399520,8rtkit-daemon1972-21rtkit-daemon22:11:491
3857011995013,11cyclictest438-20systemd-journal00:40:001
3857011995013,11cyclictest438-20systemd-journal00:39:591
38959282490,6chrt4497-21PK-Backend21:00:021
3857017994927,14cyclictest42-21ksoftirqd/322:49:593
3857015994811,6cyclictest141rcu_preempt19:40:002
3857015994736,8cyclictest34-21ksoftirqd/219:18:482
3857017994618,6cyclictest1033-21gdbus00:09:363
3857011994624,13cyclictest26-21ksoftirqd/100:10:301
3857011994624,13cyclictest26-21ksoftirqd/100:10:291
3857011994623,14cyclictest26-21ksoftirqd/120:07:501
3857017994514,16cyclictest42-21ksoftirqd/322:05:273
3857015994523,12cyclictest34-21ksoftirqd/220:00:022
3857011994534,8cyclictest26-21ksoftirqd/121:35:011
38876782440,4sleep00-21swapper/020:35:220
38591142440,7sleep30-21swapper/319:15:043
3857017994418,8cyclictest42-21ksoftirqd/319:54:513
385701599445,17cyclictest141rcu_preempt22:49:592
3857015994423,14cyclictest34-21ksoftirqd/221:30:132
3857011994423,6cyclictest26-21ksoftirqd/122:30:231
3857007994422,14cyclictest13-21ksoftirqd/019:55:020
39178702430,6sleep10-21swapper/122:00:211
39072272430,7sleep34497-21PK-Backend21:30:183
385701799439,8cyclictest141rcu_preempt19:55:293
3857011994325,7cyclictest26-21ksoftirqd/119:15:051
385701199431,14cyclictest38648082ls19:30:141
3857007994328,11cyclictest13-21ksoftirqd/019:35:270
390351724219,11sleep126-21ksoftirqd/121:20:151
3857017994218,6cyclictest3865583-21switchtime19:30:283
385701599426,6cyclictest141rcu_preempt22:00:012
3857015994221,12cyclictest34-21ksoftirqd/222:05:222
3857015994219,14cyclictest34-21ksoftirqd/222:32:142
3857011994228,10cyclictest3911765-21perf21:45:001
385701199421,21cyclictest3867156-21/usr/sbin/munin19:35:251
3857007994224,13cyclictest3890315-40apt-check20:40:530
3857007994219,14cyclictest13-21ksoftirqd/019:50:020
3857017994124,5cyclictest42-21ksoftirqd/319:25:223
3857017994123,14cyclictest42-21ksoftirqd/323:05:433
3857017994122,7cyclictest42-21ksoftirqd/322:15:173
3857017994122,5cyclictest42-21ksoftirqd/300:05:003
3857017994119,18cyclictest3879989-21cron20:15:003
3857015994130,8cyclictest34-21ksoftirqd/200:09:362
3857015994121,13cyclictest34-21ksoftirqd/220:05:142
3857015994120,14cyclictest34-21ksoftirqd/219:40:132
3857015994120,13cyclictest34-21ksoftirqd/223:35:212
3857015994120,13cyclictest34-21ksoftirqd/219:24:552
385701199419,7cyclictest39421202df_inode23:10:151
3857011994124,6cyclictest26-21ksoftirqd/120:45:291
3857011994124,6cyclictest26-21ksoftirqd/120:27:391
3857011994124,5cyclictest26-21ksoftirqd/100:32:061
3857011994123,6cyclictest26-21ksoftirqd/119:25:091
3857011994123,5cyclictest26-21ksoftirqd/100:20:281
3857011994123,14cyclictest3951168-21cat23:35:211
3857011994123,14cyclictest26-21ksoftirqd/123:52:291
3857011994123,14cyclictest26-21ksoftirqd/122:55:111
3857011994123,14cyclictest26-21ksoftirqd/120:35:211
385700799418,14cyclictest141rcu_preempt00:20:010
3857007994124,6cyclictest13-21ksoftirqd/021:55:180
3857007994123,14cyclictest13-21ksoftirqd/022:20:000
3857007994118,14cyclictest13-21ksoftirqd/020:50:020
3857007994110,27cyclictest438-20systemd-journal19:18:490
3857017994023,5cyclictest42-21ksoftirqd/320:15:323
3857017994023,5cyclictest42-21ksoftirqd/320:15:323
385701799402,34cyclictest3921049-21df_inode22:10:123
3857017994022,6cyclictest42-21ksoftirqd/323:35:203
3857017994022,6cyclictest42-21ksoftirqd/323:00:153
3857017994022,6cyclictest42-21ksoftirqd/300:35:223
3857017994022,6cyclictest42-21ksoftirqd/300:35:223
3857017994022,6cyclictest42-21ksoftirqd/300:32:323
3857017994022,5cyclictest42-21ksoftirqd/323:15:013
3857015994021,6cyclictest34-21ksoftirqd/222:50:222
3857015994021,11cyclictest34-21ksoftirqd/222:20:242
3857015994020,13cyclictest34-21ksoftirqd/200:22:572
3857015994020,12cyclictest34-21ksoftirqd/220:05:022
3857015994019,14cyclictest34-21ksoftirqd/220:15:352
3857015994019,14cyclictest34-21ksoftirqd/220:15:342
385701199406,11cyclictest26-21ksoftirqd/120:50:191
3857011994023,6cyclictest26-21ksoftirqd/123:56:471
3857011994023,6cyclictest26-21ksoftirqd/123:45:191
3857011994023,6cyclictest26-21ksoftirqd/122:46:261
3857011994023,6cyclictest26-21ksoftirqd/121:45:121
3857011994023,6cyclictest26-21ksoftirqd/121:35:211
3857011994023,6cyclictest26-21ksoftirqd/121:15:301
3857011994023,6cyclictest26-21ksoftirqd/120:10:251
3857011994023,6cyclictest26-21ksoftirqd/119:20:261
3857011994023,6cyclictest26-21ksoftirqd/100:25:121
3857011994023,5cyclictest26-21ksoftirqd/120:20:251
3857011994023,13cyclictest26-21ksoftirqd/121:55:171
3857011994023,13cyclictest26-21ksoftirqd/119:43:021
3857011994022,6cyclictest26-21ksoftirqd/122:40:161
3857011994022,6cyclictest26-21ksoftirqd/122:21:411
3857011994022,6cyclictest26-21ksoftirqd/119:10:291
3857011994022,6cyclictest26-21ksoftirqd/100:05:201
385700799408,16cyclictest3873924-21load19:55:200
385700799408,14cyclictest141rcu_preempt22:40:290
3857007994022,13cyclictest13-21ksoftirqd/019:20:160
3857007994013,11cyclictest141rcu_preempt00:35:270
3857007994013,11cyclictest141rcu_preempt00:35:270
3857007994010,4cyclictest141rcu_preempt23:45:380
38874042390,5sleep2331rcuc/220:35:142
38617162390,7chrt0-21swapper/319:20:233
385701799396,17cyclictest0-21swapper/320:40:293
3857017993923,12cyclictest42-21ksoftirqd/323:40:113
3857017993922,7cyclictest42-21ksoftirqd/320:32:093
3857017993922,6cyclictest42-21ksoftirqd/321:59:573
3857017993922,5cyclictest42-21ksoftirqd/322:30:303
3857017993922,5cyclictest42-21ksoftirqd/321:41:373
3857017993921,6cyclictest42-21ksoftirqd/322:03:013
3857017993921,6cyclictest42-21ksoftirqd/320:55:223
3857017993921,6cyclictest42-21ksoftirqd/320:24:103
3857015993925,6cyclictest34-21ksoftirqd/220:20:192
3857015993922,13cyclictest3909315-21ntp_states21:35:252
3857015993921,6cyclictest34-21ksoftirqd/200:00:022
3857015993921,13cyclictest438-20systemd-journal21:29:382
3857015993919,13cyclictest34-21ksoftirqd/219:51:242
3857011993923,5cyclictest26-21ksoftirqd/120:40:261
3857011993922,6cyclictest26-21ksoftirqd/123:43:111
3857011993922,6cyclictest26-21ksoftirqd/123:30:121
3857011993922,6cyclictest26-21ksoftirqd/123:25:291
3857011993922,6cyclictest26-21ksoftirqd/123:05:161
3857011993922,6cyclictest26-21ksoftirqd/123:00:191
3857011993922,6cyclictest26-21ksoftirqd/100:00:141
3857011993922,5cyclictest26-21ksoftirqd/122:53:221
3857011993922,5cyclictest26-21ksoftirqd/122:25:251
3857011993922,5cyclictest26-21ksoftirqd/121:09:111
3857011993922,5cyclictest26-21ksoftirqd/121:02:311
3857011993922,5cyclictest26-21ksoftirqd/121:02:311
3857011993921,6cyclictest26-21ksoftirqd/100:15:251
3857011993920,5cyclictest26-21ksoftirqd/121:50:201
385700799399,14cyclictest141rcu_preempt21:45:000
385700799399,13cyclictest141rcu_preempt22:05:220
385700799398,6cyclictest141rcu_preempt19:15:010
385700799398,6cyclictest141rcu_preempt00:00:010
3857007993922,6cyclictest13-21ksoftirqd/019:25:120
3857007993922,13cyclictest3928478-21python322:30:280
3857007993922,12cyclictest13-21ksoftirqd/022:55:150
3857007993921,14cyclictest13-21ksoftirqd/023:15:010
3857007993918,9cyclictest13-21ksoftirqd/021:03:190
3857007993918,9cyclictest13-21ksoftirqd/021:03:180
385700799391,19cyclictest3944972-40lsb_release23:17:010
39221792380,6sleep0161rcuc/022:15:010
3857017993829,6cyclictest3903096-21cron21:20:013
3857017993824,11cyclictest42-21ksoftirqd/321:45:233
3857017993824,11cyclictest42-21ksoftirqd/320:25:233
3857017993823,11cyclictest3935979-21cron22:55:013
3857017993822,5cyclictest42-21ksoftirqd/322:24:073
3857017993822,11cyclictest3927325-21cron22:30:013
3857017993819,6cyclictest42-21ksoftirqd/321:50:193
3857015993824,6cyclictest34-21ksoftirqd/222:35:302
3857015993821,13cyclictest34-21ksoftirqd/200:00:152
3857015993821,12cyclictest438-20systemd-journal21:20:392
3857015993820,5cyclictest34-21ksoftirqd/223:03:572
3857015993819,5cyclictest34-21ksoftirqd/219:10:142
3857011993821,6cyclictest26-21ksoftirqd/122:15:131
3857011993821,6cyclictest26-21ksoftirqd/120:30:161
3857011993821,6cyclictest26-21ksoftirqd/119:45:251
3857011993821,5cyclictest26-21ksoftirqd/122:35:201
3857011993821,5cyclictest26-21ksoftirqd/121:10:141
3857011993821,5cyclictest26-21ksoftirqd/120:15:271
3857011993821,5cyclictest26-21ksoftirqd/120:15:261
385700799389,15cyclictest13-21ksoftirqd/021:30:010
3857007993821,13cyclictest13-21ksoftirqd/020:30:280
3857007993815,14cyclictest13-21ksoftirqd/000:30:340
3857007993810,10cyclictest3914479-21awk21:50:220
39457222370,5chrt0-21swapper/223:20:152
3857017993720,6cyclictest42-21ksoftirqd/300:23:003
3857017993720,5cyclictest42-21ksoftirqd/323:58:123
3857017993720,5cyclictest42-21ksoftirqd/323:21:083
3857017993720,5cyclictest42-21ksoftirqd/322:35:123
3857017993720,13cyclictest3857935-21ntpq19:10:253
3857017993718,6cyclictest42-21ksoftirqd/323:15:283
385701599375,6cyclictest141rcu_preempt00:15:012
385701599375,6cyclictest141rcu_preempt00:15:012
385701599375,5cyclictest141rcu_preempt21:05:232
385701599372,30cyclictest3956986-21users23:50:292
3857015993722,11cyclictest923-21dbus-daemon00:29:572
3857015993721,13cyclictest34-21ksoftirqd/220:30:322
3857015993721,12cyclictest3950400-21sshd23:34:382
385701199376,9cyclictest26-21ksoftirqd/121:26:541
3857011993719,6cyclictest26-21ksoftirqd/123:20:281
385700799378,6cyclictest141rcu_preempt22:00:180
385700799374,11cyclictest13-21ksoftirqd/022:35:090
3857007993724,9cyclictest13-21ksoftirqd/000:10:310
3857007993724,9cyclictest13-21ksoftirqd/000:10:300
3857007993712,5cyclictest141rcu_preempt22:45:190
3857007993711,6cyclictest141rcu_preempt22:30:020
3857007993711,6cyclictest141rcu_preempt21:45:240
3857017993623,9cyclictest42-21ksoftirqd/323:25:233
3857017993619,5cyclictest42-21ksoftirqd/321:05:253
385701799361,8cyclictest3888765-21kworker/3:221:20:133
385701799361,14cyclictest0-21swapper/320:35:163
385701599368,13cyclictest141rcu_preempt21:49:572
385701599367,13cyclictest141rcu_preempt23:05:302
3857015993626,6cyclictest3890934-21/usr/sbin/munin20:45:302
3857015993625,7cyclictest3866140-21cron19:35:002
3857015993623,9cyclictest2093-21JS10:49:092
3857015993619,13cyclictest34-21ksoftirqd/223:18:332
3857015993619,12cyclictest34-21ksoftirqd/200:20:022
3857015993618,14cyclictest34-21ksoftirqd/223:40:582
3857015993618,14cyclictest34-21ksoftirqd/222:27:092
385700799366,14cyclictest141rcu_preempt22:50:120
3857007993626,6cyclictest13-21ksoftirqd/020:15:010
3857007993625,6cyclictest13-21ksoftirqd/000:25:140
3857007993618,14cyclictest13-21ksoftirqd/020:55:020
3857007993612,11cyclictest923-21dbus-daemon00:09:380
385700799361,20cyclictest3868328-21cpuspeed_turbos19:40:140
385701799355,13cyclictest0-21swapper/319:44:383
385701799355,10cyclictest42-21ksoftirqd/320:05:213
3857017993525,6cyclictest1081-21ntpd23:30:243
3857017993520,11cyclictest3954883-21ntp_kernel_err23:45:243
3857017993519,12cyclictest42-21ksoftirqd/300:25:143
3857017993519,12cyclictest3963619-21awk00:10:263
3857017993519,12cyclictest3963619-21awk00:10:253
3857017993519,12cyclictest3955896-21awk23:50:123
3857017993519,12cyclictest3908768-21cstates21:35:153
3857017993519,12cyclictest3870535-21needreboot19:45:243
3857017993518,5cyclictest42-21ksoftirqd/322:55:303
385701799351,30cyclictest3900092-21cpuspeed_turbos21:10:133
3857017993510,8cyclictest141rcu_preempt19:38:123
3857015993526,6cyclictest1054-21snmpd20:27:352
3857015993525,6cyclictest3922790-21diskstats22:15:172
3857015993525,6cyclictest34-21ksoftirqd/220:40:572
3857015993519,12cyclictest34-21ksoftirqd/223:15:022
385701599351,8cyclictest0-21swapper/220:55:152
3857015993518,14cyclictest34-21ksoftirqd/221:10:392
3857015993518,13cyclictest34-21ksoftirqd/200:35:212
3857015993518,13cyclictest34-21ksoftirqd/200:35:212
3857015993518,13cyclictest34-21ksoftirqd/200:30:252
3857015993517,14cyclictest34-21ksoftirqd/221:50:202
3857015993517,13cyclictest34-21ksoftirqd/223:45:252
3857015993517,13cyclictest34-21ksoftirqd/221:40:232
385700799359,10cyclictest3900808-21switchtime21:10:280
3857007993520,11cyclictest3882055-21awk20:20:130
3857007993517,14cyclictest13-21ksoftirqd/023:35:110
3857007993516,7cyclictest141rcu_preempt21:21:300
385700799351,21cyclictest1137-21runrttasks23:00:510
385701799344,12cyclictest0-21swapper/320:49:493
385701799341,29cyclictest3965339-21tr00:15:253
3857015993424,6cyclictest1054-21snmpd20:13:432
3857015993419,11cyclictest34-21ksoftirqd/222:00:182
3857015993417,14cyclictest34-21ksoftirqd/222:14:232
3857015993417,14cyclictest34-21ksoftirqd/221:00:112
3857015993417,14cyclictest34-21ksoftirqd/221:00:102
3857015993417,13cyclictest34-21ksoftirqd/223:25:542
3857015993417,13cyclictest34-21ksoftirqd/219:45:202
3857015993416,6cyclictest34-21ksoftirqd/221:15:032
3857015993410,5cyclictest141rcu_preempt19:27:382
385700799348,5cyclictest141rcu_preempt21:35:020
385700799347,6cyclictest141rcu_preempt23:25:010
3857007993411,6cyclictest141rcu_preempt23:40:270
385701799311,26cyclictest3931070-21apt22:40:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional