You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 15:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Jan 03, 2026 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41680432990,8sleep3438-20systemd-journal08:20:003
414270228061,13sleep00-21swapper/007:07:260
414289027338,13sleep30-21swapper/307:09:553
414144327347,16sleep10-21swapper/107:05:141
197399730,5rtkit-daemon0-21swapper/008:14:060
197399690,5rtkit-daemon0-21swapper/008:58:180
414282726343,14sleep20-21swapper/207:09:052
197399630,4rtkit-daemon0-21swapper/109:22:180
197399590,8rtkit-daemon1972-21rtkit-daemon12:22:232
197399580,7rtkit-daemon1972-21rtkit-daemon11:33:052
197399570,8rtkit-daemon1972-21rtkit-daemon12:20:343
197399560,9rtkit-daemon1972-21rtkit-daemon11:58:453
197399560,8rtkit-daemon1972-21rtkit-daemon11:09:043
276992550,7sleep14497-21PK-Backend10:50:211
197399550,9rtkit-daemon1972-21rtkit-daemon11:39:113
197399540,9rtkit-daemon1972-21rtkit-daemon12:17:363
197399540,9rtkit-daemon1972-21rtkit-daemon12:17:363
4143041995312,13cyclictest4161522-21grep08:00:153
197399530,8rtkit-daemon1972-21rtkit-daemon12:36:003
197399530,8rtkit-daemon1972-21rtkit-daemon12:36:003
534222520,6chrt0-21swapper/012:00:300
197399520,8rtkit-daemon1972-21rtkit-daemon11:24:563
647852510,6sleep20-21swapper/212:35:092
647852510,6sleep20-21swapper/212:35:092
121422510,5chrt0-21swapper/010:05:290
41658952490,16chrt42-21ksoftirqd/308:10:303
41552152490,6sleep0161rcuc/007:40:290
4143037994821,12cyclictest141rcu_preempt07:41:502
414304199472,30cyclictest42-21ksoftirqd/312:00:243
4143035994722,20cyclictest4155625-21cron07:45:001
379462470,8sleep00-21swapper/011:20:020
41580812460,7sleep14497-21PK-Backend07:50:201
414304199467,17cyclictest4183865-21/usr/sbin/munin09:05:173
4143041994624,8cyclictest42-21ksoftirqd/311:43:223
4143041994622,7cyclictest42-21ksoftirqd/311:34:043
382392460,15chrt34-21ksoftirqd/211:20:122
41512992450,8sleep34497-21PK-Backend07:30:233
4143041994523,8cyclictest42-21ksoftirqd/307:11:013
4143041994522,8cyclictest42-21ksoftirqd/308:57:093
4143041994521,7cyclictest42-21ksoftirqd/308:22:563
4143041994520,9cyclictest42-21ksoftirqd/309:33:283
4143041994520,7cyclictest42-21ksoftirqd/310:05:273
41546602440,5sleep3411rcuc/307:40:203
4143041994423,7cyclictest42-21ksoftirqd/308:44:383
4143041994423,7cyclictest42-21ksoftirqd/307:17:453
4143041994422,7cyclictest42-21ksoftirqd/312:07:333
4143041994422,7cyclictest42-21ksoftirqd/311:00:403
32222440,6chrt0-21swapper/009:40:250
414304199439,19cyclictest42-21ksoftirqd/307:35:293
414304199433,10cyclictest42-21ksoftirqd/311:00:023
4143041994322,7cyclictest42-21ksoftirqd/311:54:213
4143041994322,7cyclictest42-21ksoftirqd/311:54:213
4143041994322,7cyclictest42-21ksoftirqd/309:23:133
4143041994322,7cyclictest42-21ksoftirqd/308:27:323
4143041994322,7cyclictest42-21ksoftirqd/307:26:423
4143041994321,7cyclictest42-21ksoftirqd/310:29:373
4143041994314,10cyclictest42-21ksoftirqd/312:28:383
414303099433,23cyclictest52159-21sh12:00:010
4143030994324,6cyclictest13-21ksoftirqd/010:25:000
197399430,35rtkit-daemon4144746-21munin-node11:48:152
4143041994222,7cyclictest42-21ksoftirqd/311:19:443
4143041994221,7cyclictest42-21ksoftirqd/310:22:173
4143041994212,11cyclictest42-21ksoftirqd/309:11:023
4143037994213,15cyclictest141rcu_preempt11:00:022
4143030994224,14cyclictest13-21ksoftirqd/011:30:010
4143030994224,14cyclictest13-21ksoftirqd/011:30:010
414304199415,18cyclictest42-21ksoftirqd/310:30:153
4143041994121,16cyclictest24724-21sed10:40:283
4143041994115,22cyclictest62834-21cpu12:30:133
4143041994112,11cyclictest42-21ksoftirqd/309:43:063
4143037994119,18cyclictest4177455-21irqstats08:45:202
4143030994123,14cyclictest13-21ksoftirqd/008:35:260
4143030994123,14cyclictest13-21ksoftirqd/007:25:020
4143030994122,14cyclictest13-21ksoftirqd/011:30:030
4143030994115,6cyclictest141rcu_preempt09:35:000
158202410,4sleep10-21swapper/110:15:281
158202410,4sleep10-21swapper/110:15:281
414304199408,17cyclictest42-21ksoftirqd/311:29:513
414304199408,17cyclictest42-21ksoftirqd/311:29:513
414304199406,9cyclictest4497-21PK-Backend12:10:153
4143041994028,9cyclictest42-21ksoftirqd/310:38:483
4143041994011,8cyclictest42-21ksoftirqd/310:50:013
4143041994011,11cyclictest42-21ksoftirqd/307:56:223
4143041994010,11cyclictest42-21ksoftirqd/308:07:243
4143037994014,6cyclictest141rcu_preempt07:15:022
414303599404,6cyclictest141rcu_preempt07:15:011
4143035994022,5cyclictest26-21ksoftirqd/111:09:571
4143035994022,14cyclictest26-21ksoftirqd/108:18:581
4143035994022,13cyclictest438-20systemd-journal11:18:191
4143035994022,13cyclictest26-21ksoftirqd/109:01:071
4143030994022,6cyclictest13-21ksoftirqd/007:30:150
4143030994022,14cyclictest13-21ksoftirqd/007:25:190
4143030994022,13cyclictest13-21ksoftirqd/009:45:140
414303099401,7cyclictest46972-21turbostat11:45:000
643392396,29sleep264604-21cut12:35:022
41562792390,6sleep0161rcuc/007:45:160
414304199399,12cyclictest42-21ksoftirqd/307:45:383
414304199399,11cyclictest42-21ksoftirqd/307:23:413
414304199397,8cyclictest42-21ksoftirqd/309:05:023
414304199396,17cyclictest42-21ksoftirqd/308:50:303
414304199394,14cyclictest42-21ksoftirqd/309:50:263
414304199394,14cyclictest42-21ksoftirqd/309:50:253
4143041993930,6cyclictest411rcuc/310:11:273
4143041993924,11cyclictest4157925-21cstates07:50:153
4143041993920,7cyclictest42-21ksoftirqd/311:46:283
4143041993919,8cyclictest42-21ksoftirqd/309:56:453
414304199391,14cyclictest9371-21taskset10:00:153
414303599396,6cyclictest141rcu_preempt07:15:141
4143035993921,13cyclictest26-21ksoftirqd/111:03:541
4143035993920,13cyclictest26-21ksoftirqd/108:24:071
4143035993912,7cyclictest141rcu_preempt09:29:591
414303599391,14cyclictest4497-21PK-Backend12:14:231
414303099399,8cyclictest13-21ksoftirqd/012:15:350
414303099399,8cyclictest13-21ksoftirqd/012:15:350
4143030993922,5cyclictest13-21ksoftirqd/012:35:020
4143030993917,18cyclictest6464-21sed09:50:240
4143030993917,18cyclictest6464-21sed09:50:240
414303099391,33cyclictest30889-21apt-config11:00:030
414303099391,16cyclictest4167091-20journal-offline08:15:210
414304199386,8cyclictest42-21ksoftirqd/311:10:133
414304199384,11cyclictest42-21ksoftirqd/308:35:023
414303799387,5cyclictest141rcu_preempt08:50:292
414303799386,9cyclictest34-21ksoftirqd/209:57:182
4143037993825,9cyclictest34-21ksoftirqd/207:35:302
4143037993822,11cyclictest438-20systemd-journal09:40:182
4143037993820,13cyclictest27281-21awk10:50:142
4143037993814,11cyclictest141rcu_preempt10:35:092
4143037993813,5cyclictest141rcu_preempt10:08:172
4143037993812,6cyclictest141rcu_preempt11:00:142
414303599386,5cyclictest141rcu_preempt12:05:131
4143035993822,11cyclictest26-21ksoftirqd/109:42:071
4143035993821,13cyclictest4189865-21awk09:20:251
4143035993820,5cyclictest26-21ksoftirqd/110:55:221
414303599381,6cyclictest41761862switchtime08:40:291
414303099387,11cyclictest438-20systemd-journal11:15:000
4143030993823,11cyclictest4164574-21cron08:10:000
4143030993822,5cyclictest13-21ksoftirqd/007:15:260
414303099382,22cyclictest4160030-21needreboot07:55:240
4143030993822,12cyclictest13-21ksoftirqd/010:30:270
4143030993821,5cyclictest13-21ksoftirqd/012:05:300
4143030993820,14cyclictest13-21ksoftirqd/010:45:000
4143030993818,17cyclictest2263-21gsd-media-keys10:38:480
4143030993814,7cyclictest141rcu_preempt12:40:000
4143030993814,7cyclictest141rcu_preempt12:40:000
414303099381,20cyclictest39736-21cron11:25:000
197399380,31rtkit-daemon42-21ksoftirqd/310:54:163
414304199375,9cyclictest42-21ksoftirqd/310:15:033
414304199375,9cyclictest42-21ksoftirqd/310:15:023
414304199372,20cyclictest42-21ksoftirqd/309:18:183
4143041993718,7cyclictest42-21ksoftirqd/309:47:033
414303799374,14cyclictest4497-21PK-Backend12:06:532
414303799372,12cyclictest59092-21munin-run12:20:002
414303799372,12cyclictest59092-21munin-run12:20:002
4143037993720,13cyclictest34-21ksoftirqd/209:36:372
4143037993720,13cyclictest34-21ksoftirqd/208:06:192
4143037993720,12cyclictest4166577-21awk08:15:112
4143037993712,15cyclictest34-21ksoftirqd/211:20:032
414303599373,5cyclictest141rcu_preempt08:35:001
4143035993720,13cyclictest26-21ksoftirqd/112:28:361
4143035993720,13cyclictest26-21ksoftirqd/111:44:541
4143035993720,13cyclictest26-21ksoftirqd/109:08:391
4143035993720,12cyclictest438-20systemd-journal09:45:421
4143035993718,15cyclictest4151634-21sed07:30:291
4143035993718,14cyclictest26-21ksoftirqd/110:01:151
414303599371,16cyclictest4497-21PK-Backend07:47:341
414303599371,14cyclictest4497-21PK-Backend11:26:581
414303599371,14cyclictest4497-21PK-Backend11:26:581
414303599371,14cyclictest4497-21PK-Backend09:19:301
4143030993726,6cyclictest13-21ksoftirqd/009:30:010
4143030993725,8cyclictest2093-21JS10:49:090
4143030993722,11cyclictest923-21dbus-daemon10:00:000
4143030993721,5cyclictest13-21ksoftirqd/008:35:000
4143030993721,13cyclictest13-21ksoftirqd/011:45:130
4143030993721,12cyclictest13-21ksoftirqd/012:15:010
4143030993720,5cyclictest13-21ksoftirqd/008:26:100
4143030993720,13cyclictest13-21ksoftirqd/008:45:280
4143030993720,12cyclictest13-21ksoftirqd/011:35:300
4143030993719,5cyclictest13-21ksoftirqd/008:25:000
4143030993719,14cyclictest13-21ksoftirqd/009:00:260
414303099371,16cyclictest61049-21date12:25:020
398532370,5sleep1251rcuc/111:25:021
41820262360,5chrt4497-21PK-Backend09:00:021
414304199369,15cyclictest42-21ksoftirqd/308:47:263
414304199361,20cyclictest4174721-21runrttasks08:36:533
414304199361,19cyclictest42-21ksoftirqd/309:37:383
414304199361,15cyclictest4191927-21taskset09:25:393
414303799366,5cyclictest141rcu_preempt11:35:242
4143037993622,10cyclictest34-21ksoftirqd/212:10:252
4143037993621,11cyclictest52098-21sshd11:59:192
4143037993621,11cyclictest4161120-21dpkg08:00:022
4143037993620,12cyclictest4172946-21sshd08:31:182
4143037993618,14cyclictest34-21ksoftirqd/209:49:252
4143037993618,13cyclictest438-20systemd-journal10:14:552
4143037993618,13cyclictest34-21ksoftirqd/212:00:322
4143037993618,13cyclictest34-21ksoftirqd/211:50:102
4143037993618,13cyclictest34-21ksoftirqd/211:50:102
4143037993617,14cyclictest4185993-21df09:10:192
4143037993614,5cyclictest141rcu_preempt08:25:272
4143037993613,5cyclictest141rcu_preempt07:25:142
4143037993612,6cyclictest141rcu_preempt07:50:202
4143037993610,6cyclictest141rcu_preempt07:15:172
414303599366,6cyclictest141rcu_preempt07:56:301
414303599362,20cyclictest0-21swapper/110:21:371
414303599361,9cyclictest0-21swapper/111:39:381
4143035993618,14cyclictest4161755-21sshd08:00:181
4143035993618,13cyclictest26-21ksoftirqd/108:05:531
4143035993610,6cyclictest141rcu_preempt11:15:001
414303099362,7cyclictest4144327-21/usr/sbin/munin07:10:300
4143030993623,8cyclictest956-21systemd-logind09:38:410
4143030993620,12cyclictest4187285-21sort09:15:020
4143030993618,6cyclictest13-21ksoftirqd/011:50:240
4143030993618,6cyclictest13-21ksoftirqd/011:50:240
414303099361,13cyclictest4159087-21dump-pmu-power07:54:590
4143030993610,13cyclictest141rcu_preempt10:30:010
414303799359,5cyclictest141rcu_preempt08:04:092
414303799359,5cyclictest141rcu_preempt07:20:112
4143037993520,11cyclictest25896-21cat10:45:142
4143037993519,12cyclictest10739-21apt-get10:05:012
4143037993519,11cyclictest438-20systemd-journal10:17:002
4143037993519,11cyclictest438-20systemd-journal10:17:002
4143037993518,12cyclictest4173935-21ls08:35:192
4143037993517,13cyclictest34-21ksoftirqd/207:31:172
4143037993511,6cyclictest141rcu_preempt10:30:142
414303599356,10cyclictest26-21ksoftirqd/110:49:281
414303599351,9cyclictest4497-21PK-Backend09:39:251
4143035993519,12cyclictest4147571-21df07:20:171
414303599351,8cyclictest0-21swapper/109:56:591
4143035993518,5cyclictest26-21ksoftirqd/108:49:491
4143035993518,12cyclictest59475-21wc12:20:111
414303599351,5cyclictest141rcu_preempt08:55:001
414303599351,15cyclictest4497-21PK-Backend10:35:511
414303099353,14cyclictest16479-21chrt10:20:010
414303099353,14cyclictest16479-21chrt10:20:010
4143030993525,6cyclictest13-21ksoftirqd/008:50:190
4143030993525,5cyclictest13-21ksoftirqd/008:40:280
4143030993524,6cyclictest13-21ksoftirqd/011:05:170
4143030993524,6cyclictest13-21ksoftirqd/010:55:010
4143030993524,6cyclictest13-21ksoftirqd/009:05:240
4143030993524,6cyclictest13-21ksoftirqd/008:00:060
4143030993520,11cyclictest4187876-21grep09:15:200
4143030993517,6cyclictest13-21ksoftirqd/012:25:140
197399350,28rtkit-daemon0-21swapper/211:09:462
4143037993423,7cyclictest25498-21date10:45:022
4143037993418,12cyclictest40570-21cat11:25:202
4143037993418,12cyclictest40570-21cat11:25:202
4143037993413,5cyclictest141rcu_preempt10:20:292
414303799341,30cyclictest4497-21PK-Backend09:20:402
414303599343,6cyclictest5831-21cpu09:50:151
414303599343,6cyclictest5831-21cpu09:50:151
4143035993423,7cyclictest4185766-21tr09:10:171
4143035993420,10cyclictest26-21ksoftirqd/107:25:281
414303599341,9cyclictest4497-21PK-Backend08:38:481
4143035993419,11cyclictest1-21systemd11:35:001
4143035993416,6cyclictest26-21ksoftirqd/110:14:281
4143030993424,6cyclictest13-21ksoftirqd/010:59:410
4143030993423,6cyclictest27022-21sh10:50:000
4143030993423,6cyclictest13-21ksoftirqd/010:00:230
414303799338,5cyclictest141rcu_preempt10:25:022
4143037993322,7cyclictest4187563-21cstates09:15:142
4143037993318,11cyclictest4191687-21python309:25:282
4143037993318,11cyclictest4165573-21ntpq08:10:242
4143037993318,11cyclictest35530-21grep11:10:282
4143037993315,15cyclictest4497-21PK-Backend11:45:002
414303599335,5cyclictest141rcu_preempt10:27:181
414303599334,25cyclictest4171536-21basename08:30:001
414303599331,8cyclictest4497-21PK-Backend10:07:181
4143035993318,11cyclictest49873-21sed11:50:291
4143035993318,11cyclictest49873-21sed11:50:291
4143035993318,11cyclictest47405-21grep11:45:131
4143035993318,11cyclictest20496-21expr10:30:141
4143035993317,12cyclictest58522-21grep12:15:301
4143035993317,12cyclictest58522-21grep12:15:301
4143030993319,10cyclictest4152659-21ls07:35:130
414303799327,6cyclictest141rcu_preempt12:25:242
414303799323,5cyclictest141rcu_preempt08:40:142
4143037993215,12cyclictest4169420-21awk08:20:322
414303799321,28cyclictest0-21swapper/209:30:592
414303799321,28cyclictest0-21swapper/209:06:362
414303599327,5cyclictest141rcu_preempt09:30:161
414303599322,5cyclictest141rcu_preempt11:56:321
4143035993225,5cyclictest0-21swapper/107:35:191
4143035993224,5cyclictest4497-21PK-Backend12:36:281
4143035993224,5cyclictest4497-21PK-Backend12:36:281
4143035993215,12cyclictest438-20systemd-journal12:03:191
414303599321,14cyclictest4497-21PK-Backend08:11:191
4143037993119,7cyclictest4182562-21sh09:00:182
4143037993113,13cyclictest34-21ksoftirqd/208:57:052
4143035993117,10cyclictest63319-21cut12:30:201
414303799301,9cyclictest6695-21smartctl09:50:312
414303799301,9cyclictest6695-21smartctl09:50:312
4143037993016,10cyclictest4156118-21ls07:45:132
414303599301,13cyclictest4497-21PK-Backend10:43:361
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional