You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-05 - 07:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Jul 05, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1425329279,9sleep00-21swapper/019:07:000
1421729065,9sleep30-21swapper/319:06:363
1423728764,13sleep10-21swapper/119:06:491
1235628270,8sleep20-21swapper/219:05:092
64682570,1sleep3381rcuc/300:20:143
1457499285,1cyclictest0-21swapper/200:38:252
14574992725,2cyclictest0-21swapper/223:55:132
1457099260,26cyclictest0-21swapper/122:38:471
14574992522,3cyclictest0-21swapper/200:30:342
14576992422,1cyclictest0-21swapper/323:32:473
14574992424,0cyclictest0-21swapper/222:38:232
1457099240,2cyclictest0-21swapper/121:15:121
1457699231,1cyclictest0-21swapper/321:43:033
1457099231,1cyclictest0-21swapper/120:10:181
1456899230,0cyclictest0-21swapper/021:33:340
14574992217,5cyclictest0-21swapper/222:00:332
1457499220,21cyclictest7101-21diskmemload22:15:232
1457099220,22cyclictest0-21swapper/121:13:081
14574992120,1cyclictest0-21swapper/220:20:132
14576992019,1cyclictest0-21swapper/323:14:093
1457699200,19cyclictest0-21swapper/322:09:403
1457499201,2cyclictest1162-21munin-run19:50:002
1457499200,1cyclictest55550irq/126-enp2s0-21:53:022
14570992017,2cyclictest0-21swapper/122:30:011
1457099200,19cyclictest0-21swapper/123:25:401
1456899200,2cyclictest0-21swapper/021:27:330
14576991917,1cyclictest0-21swapper/323:53:433
1457699190,17cyclictest0-21swapper/323:04:103
14574991917,2cyclictest0-21swapper/223:49:022
14574991915,4cyclictest0-21swapper/222:11:032
1457499190,17cyclictest0-21swapper/223:29:122
1457099192,1cyclictest141rcu_preempt20:40:131
1457099190,1cyclictest0-21swapper/122:09:391
1457099190,1cyclictest0-21swapper/100:31:201
1457099190,18cyclictest0-21swapper/122:00:461
1457099190,16cyclictest0-21swapper/123:53:431
1456899195,7cyclictest3316-21perf22:05:010
1456899190,19cyclictest0-21swapper/000:32:420
1456899190,0cyclictest0-21swapper/000:17:570
14574991815,3cyclictest0-21swapper/221:14:522
14574991815,3cyclictest0-21swapper/200:08:232
1457499180,1cyclictest0-21swapper/221:36:442
1457099185,2cyclictest0-21swapper/122:44:531
14570991812,4cyclictest38150irq/131-snd_hda22:58:441
1457099181,13cyclictest0-21swapper/123:20:121
1457099180,5cyclictest5870-21ssh00:00:271
1457099180,18cyclictest0-21swapper/123:09:371
1457099180,0cyclictest0-21swapper/123:32:471
14568991815,2cyclictest13-21ksoftirqd/023:45:520
1456899180,2cyclictest0-21swapper/022:14:400
1457699174,8cyclictest19240-21apt-get21:55:133
1457699170,1cyclictest0-21swapper/323:46:193
1457699170,1cyclictest0-21swapper/300:17:233
1457699170,11cyclictest0-21swapper/322:40:293
1457499170,16cyclictest0-21swapper/200:01:242
14570991714,1cyclictest0-21swapper/122:23:491
1456899170,2cyclictest0-21swapper/020:10:130
1456899170,0cyclictest0-21swapper/022:26:290
1457699169,4cyclictest141rcu_preempt00:32:063
1457699165,5cyclictest0-21swapper/321:00:123
1457699162,12cyclictest23503-21date22:54:593
14576991611,4cyclictest32142-21cron20:55:013
14570991613,2cyclictest0-21swapper/121:34:331
1457099160,15cyclictest0-21swapper/123:46:321
1457099160,15cyclictest0-21swapper/121:23:021
14568991616,0cyclictest0-21swapper/000:28:280
1456899160,15cyclictest0-21swapper/000:06:200
1457699150,14cyclictest0-21swapper/322:21:023
1457699150,14cyclictest0-21swapper/300:04:023
14574991511,4cyclictest0-21swapper/223:31:232
14574991511,4cyclictest0-21swapper/221:34:442
1457499150,15cyclictest56550irq/124-enp0s3119:21:232
1457099153,11cyclictest7101-21diskmemload00:09:081
14570991513,1cyclictest29853-21ssh00:15:071
1457099151,1cyclictest0-21swapper/122:53:241
1457099150,1cyclictest0-21swapper/100:38:001
1457099150,14cyclictest0-21swapper/100:25:241
1456899153,6cyclictest13930-21idleruntime-cro00:25:010
14568991513,1cyclictest0-21swapper/021:19:490
14568991511,3cyclictest26221-21sort23:15:200
1456899150,1cyclictest141rcu_preempt23:09:240
1457699146,4cyclictest452-21systemd-logind00:06:493
1457699141,0cyclictest141rcu_preempt23:09:373
14576991410,3cyclictest11390-21ssh22:28:313
1457699140,1cyclictest0-21swapper/321:54:573
1457699140,12cyclictest0-21swapper/322:34:483
14574991411,3cyclictest0-21swapper/200:17:242
14574991411,2cyclictest20137-21df23:50:162
14574991411,2cyclictest11136-21grep20:10:142
1457499140,13cyclictest31073-21wc22:20:252
1457499140,13cyclictest0-21swapper/221:41:322
1457499140,12cyclictest2044-21grep19:50:172
1457099143,3cyclictest0-21swapper/123:35:301
14570991412,1cyclictest25-21ksoftirqd/120:20:131
14570991410,3cyclictest25-21ksoftirqd/122:49:571
1457099140,14cyclictest0-21swapper/123:14:081
1456899141,7cyclictest7094-21munin-run21:10:000
14568991413,1cyclictest13-21ksoftirqd/023:11:550
14568991413,1cyclictest0-21swapper/000:36:520
14568991410,3cyclictest23091-21ls00:10:240
1456899140,7cyclictest23323-21idleruntime-cro22:55:000
1456899140,14cyclictest0-21swapper/021:12:270
1457699134,8cyclictest7101-21diskmemload21:39:473
1457699134,6cyclictest6842-21/usr/sbin/munin22:45:213
1457699133,3cyclictest0-21swapper/321:30:143
1457699131,8cyclictest0-21swapper/322:58:373
14576991310,2cyclictest18877-21grep19:15:263
1457699130,1cyclictest9628-21grep21:10:273
1457699130,1cyclictest0-21swapper/320:15:213
1457699130,1cyclictest0-21swapper/320:15:203
1457699130,13cyclictest0-21swapper/321:18:253
1457499139,3cyclictest18083-21tr23:10:202
1457499139,3cyclictest10309-21grep20:05:272
1457499131,8cyclictest29772-21latency_hist20:50:002
14574991312,1cyclictest0-21swapper/222:31:382
14574991310,2cyclictest4559-21cut19:55:172
14574991310,2cyclictest17684-21ssh21:15:232
14574991310,2cyclictest13647-21grep20:15:142
14574991310,2cyclictest13647-21grep20:15:132
1457499130,13cyclictest0-21swapper/221:22:332
1457099139,1cyclictest0-21swapper/123:57:461
14570991312,1cyclictest0-21swapper/122:31:281
1457099130,1cyclictest0-21swapper/121:58:141
1457099130,13cyclictest0-21swapper/121:45:261
1457099130,10cyclictest0-21swapper/121:52:131
1456899139,3cyclictest20324-21grep19:20:160
1456899139,3cyclictest18890-21grep19:15:260
1456899133,2cyclictest141rcu_preempt23:25:310
1456899132,9cyclictest14997-21wc22:49:590
14568991312,1cyclictest0-21swapper/022:05:250
14568991310,2cyclictest27869-21wc23:35:210
14568991310,2cyclictest20775-21uname23:50:190
14568991310,2cyclictest18872-21wc21:35:180
14568991310,2cyclictest16924-21turbostat19:15:000
1456899130,12cyclictest7101-21diskmemload23:55:130
1457699129,2cyclictest5954-21ssh23:41:293
1457699129,2cyclictest32416-21chrt20:55:133
1457699129,2cyclictest30953-21rm23:56:033
1457699129,2cyclictest26575-21wc21:20:253
1457699129,2cyclictest24964-21sh00:11:393
1457699129,2cyclictest22179-21grep20:30:273
1457699129,2cyclictest21864-21date19:25:013
1457699129,2cyclictest21630-21grep22:15:173
1457699129,2cyclictest1862-21rm22:03:553
1457699129,2cyclictest15557-21wc00:25:193
1457699129,2cyclictest15292-21ssh23:28:183
1457699129,2cyclictest14796-21ssh22:10:493
1457699129,2cyclictest12605-21sed20:10:263
1457699121,2cyclictest141rcu_preempt21:25:253
1457699121,1cyclictest0-21swapper/323:15:453
14576991210,1cyclictest6760-21ssh21:47:213
1457499129,2cyclictest8754-21chrt22:26:322
1457499129,2cyclictest6835-21grep00:20:172
1457499129,2cyclictest5928-21cut21:05:192
1457499129,2cyclictest31230-21ssh22:40:152
1457499129,2cyclictest30787-21sed20:50:182
1457499129,2cyclictest2804-21sed21:45:162
1457499129,2cyclictest22563-21wc00:29:592
1457499129,2cyclictest19802-21grep20:25:252
1457499129,2cyclictest17451-21grep19:15:142
1457499129,2cyclictest1461-21sensors23:00:252
1457499124,5cyclictest225-21systemd-journal22:00:012
1457499120,8cyclictest0-21swapper/200:10:122
1457499120,2cyclictest0-21swapper/223:41:412
1457499120,11cyclictest29769-21sort19:40:162
1457499120,11cyclictest2077-21cron23:40:002
1457099129,3cyclictest23762-21taskset19:26:231
1457099129,2cyclictest4023-21grep19:55:141
1457099129,2cyclictest32061-21taskset20:54:031
1457099129,2cyclictest23940-21ssh21:38:391
1457099128,3cyclictest26377-21chrt19:34:251
1457099128,1cyclictest25-21ksoftirqd/122:12:261
1457099127,1cyclictest648-21tune2fs20:55:171
1457099125,5cyclictest25-21ksoftirqd/120:25:121
1457099125,3cyclictest7101-21diskmemload22:15:051
1457099120,11cyclictest7101-21diskmemload23:15:471
1457099120,11cyclictest21884-21chrt00:10:141
1457099120,11cyclictest0-21swapper/100:22:121
1456899129,2cyclictest3869-21wc21:00:250
1456899129,2cyclictest31159-21grep22:20:240
1456899129,2cyclictest28361-21tail19:35:250
1456899129,2cyclictest24555-21tr19:30:150
1456899129,2cyclictest24114-21grep20:35:200
1456899129,2cyclictest22364-21basename22:35:140
1456899129,2cyclictest18856-21ssh23:30:180
1456899128,3cyclictest3106-21sed21:45:170
1456899128,3cyclictest2864-21wc19:50:260
1456899128,3cyclictest15090-21sed20:15:260
1456899128,3cyclictest15090-21sed20:15:260
1456899124,7cyclictest475-21in:imuxsock22:59:540
1456899122,2cyclictest0-21swapper/022:40:150
1456899120,11cyclictest11448-21sort21:50:150
1456899120,11cyclictest0-21swapper/023:03:100
1457699119,2cyclictest6805-21taskset21:06:333
1457699119,1cyclictest38552sleep323:20:593
1457699119,1cyclictest31940-21expr19:45:153
1457699119,1cyclictest31318-21sh23:37:323
1457699119,1cyclictest28732-21cat20:45:183
1457699119,1cyclictest26657-21taskset19:35:033
1457699119,1cyclictest24570-21ssh22:35:423
1457699119,1cyclictest23456-21cat19:25:243
1457699119,1cyclictest16024-21grep19:10:213
1457699118,2cyclictest7538-21cut20:00:243
1457699118,2cyclictest31202-21ssh00:35:043
1457699118,2cyclictest26772-21wc20:40:233
1457699118,2cyclictest18398-21tr20:25:143
1457699118,2cyclictest15866-21cut20:20:153
1457699113,1cyclictest0-21swapper/319:55:133
1457699110,6cyclictest0-21swapper/319:30:143
1457499119,1cyclictest0-21swapper/223:18:002
1457499118,2cyclictest6671-21sh22:45:042
1457499118,2cyclictest5737-21head22:05:252
1457499118,2cyclictest32450-21grep20:55:142
1457499118,2cyclictest27423-21basename20:45:012
1457499118,2cyclictest24334-21chrt19:30:142
1457499118,2cyclictest23641-21sed19:25:262
1457499118,2cyclictest20880-21ssh22:53:072
1457499118,2cyclictest1776-21ssh21:25:172
1457499118,2cyclictest16901-21cron19:15:002
1457499113,7cyclictest56550irq/124-enp0s3119:36:042
1457499113,4cyclictest225-21systemd-journal20:37:452
1457499110,10cyclictest7101-21diskmemload23:21:332
1457099119,2cyclictest2479-21rm23:00:571
1457099119,1cyclictest28004-21expr20:45:141
1457099119,1cyclictest16447-21tail19:10:261
1457099119,1cyclictest14014-21cut20:15:161
1457099119,1cyclictest14014-21cut20:15:151
1457099118,2cyclictest890-21cut21:25:131
1457099118,2cyclictest7813-21taskset23:42:551
1457099118,2cyclictest6217-21head21:05:291
1457099118,2cyclictest32082-21tune2fs19:45:151
1457099118,2cyclictest28314-21ssh21:40:301
1457099118,2cyclictest27148-21expr19:35:151
1457099118,2cyclictest17953-21head19:15:161
1457099111,2cyclictest225-21systemd-journal20:35:001
1456899118,2cyclictest8960-21ssh00:02:350
1456899118,2cyclictest8846-21expr20:05:150
1456899118,2cyclictest5457-21grep19:55:260
1456899118,2cyclictest3285-21taskset23:40:170
1456899118,2cyclictest30502-21ssh21:23:160
1456899118,2cyclictest27900-21grep20:45:130
1456899118,2cyclictest26349-21grep20:40:180
1456899118,2cyclictest26032-21ssh21:40:040
1456899118,2cyclictest2357-21uname23:20:180
1456899118,2cyclictest19975-21taskset20:26:310
1456899118,2cyclictest19277-21basename21:55:130
1456899118,2cyclictest15849-21head22:30:290
1456899115,2cyclictest225-21systemd-journal21:00:000
1456899110,10cyclictest24609-21taskset22:16:500
1457699108,2cyclictest29699-21cut19:40:163
1457699108,1cyclictest8562-21apache_accesses20:05:113
1457699108,1cyclictest23019-21aten2_r7power_v20:35:133
1457699108,1cyclictest1520-21cut19:50:153
1457499108,1cyclictest23573-21sh22:55:022
1457499108,1cyclictest10832-21ssh23:05:492
1457499107,2cyclictest6385-21wc20:00:152
1457499107,2cyclictest4071-21head21:00:252
1457499107,2cyclictest21150-21sort20:30:182
1457099109,1cyclictest8870-21cat20:05:151
1457099109,1cyclictest6551-21cat20:00:141
1457099109,1cyclictest20386-21cat19:20:171
1457099108,2cyclictest1493-21grep19:50:151
1457099108,1cyclictest4106-21sed21:00:261
1457099108,1cyclictest22977-21basename20:35:121
1457099107,2cyclictest29314-21grep19:40:151
1456899108,1cyclictest31784-21expr19:45:140
1456899107,2cyclictest6426-21tr20:00:140
1456899107,2cyclictest31677-21tail20:50:270
1456899107,2cyclictest21262-21grep20:30:180
1456899107,2cyclictest16997-21sed20:20:210
145689996,2cyclictest22681-21grep19:25:170
145689987,1cyclictest29305-21ls19:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional