You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 16:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Mar 13, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2958121140,0sleep10-21swapper/109:34:431
2770128774,9sleep00-21swapper/007:08:590
2743028665,16sleep30-21swapper/307:05:273
2746428370,9sleep20-21swapper/207:05:542
2751628059,6sleep10-21swapper/107:06:341
185852620,0sleep30-21swapper/312:02:033
2787399287,20cyclictest0-21swapper/211:03:132
27873992725,1cyclictest0-21swapper/209:24:282
2787399270,27cyclictest0-21swapper/212:26:342
18122270,0sleep10-21swapper/109:56:031
2787399260,25cyclictest0-21swapper/211:30:042
2787399240,23cyclictest0-21swapper/212:03:322
2787099240,22cyclictest0-21swapper/110:34:081
2787399230,22cyclictest0-21swapper/212:05:152
2787399230,22cyclictest0-21swapper/211:42:532
2787399221,20cyclictest0-21swapper/210:32:362
2787399217,11cyclictest0-21swapper/212:20:122
27873992119,1cyclictest20469-21diskmemload09:50:102
27873992110,10cyclictest0-21swapper/210:40:112
27870992120,1cyclictest0-21swapper/110:08:081
2786899217,7cyclictest28861-21ssh09:15:010
27868992117,4cyclictest28312-21apache209:53:470
2787399209,10cyclictest0-21swapper/212:39:092
2787399203,8cyclictest0-21swapper/209:30:082
27873992018,1cyclictest20469-21diskmemload10:27:442
27873992018,1cyclictest20469-21diskmemload09:45:142
27873992017,3cyclictest0-21swapper/211:09:492
27873992015,3cyclictest55650irq/127-enp2s0-09:13:412
27873992015,3cyclictest20469-21diskmemload10:45:242
27870992019,1cyclictest0-21swapper/109:46:491
2787099200,9cyclictest0-21swapper/112:08:561
2787099200,1cyclictest0-21swapper/109:15:161
2787099200,18cyclictest0-21swapper/112:20:141
2786899205,3cyclictest141rcu_preempt12:03:500
2786899202,3cyclictest141rcu_preempt11:31:480
2787599191,10cyclictest19663-21sensors10:45:233
27873991913,5cyclictest2159-21sed10:35:172
27873991913,5cyclictest2159-21sed10:35:162
2787399191,17cyclictest0-21swapper/209:40:432
2787399190,19cyclictest0-21swapper/210:58:482
2787099190,18cyclictest0-21swapper/111:35:041
2787099190,18cyclictest0-21swapper/107:30:221
2786899190,6cyclictest0-21swapper/012:15:070
2787599180,15cyclictest1142-21runrttasks09:34:243
2787399187,10cyclictest0-21swapper/211:45:232
2787399186,8cyclictest0-21swapper/207:55:132
27873991816,2cyclictest55650irq/127-enp2s0-10:12:142
27873991816,1cyclictest20469-21diskmemload11:17:322
27873991812,1cyclictest225-21systemd-journal10:05:002
2787399181,16cyclictest30373-21seq11:10:592
2787099180,16cyclictest0-21swapper/110:00:111
2786899180,18cyclictest0-21swapper/012:31:040
2787599178,7cyclictest11403-21apt-get11:20:013
27875991714,2cyclictest15865-21sendmail12:40:003
2787599170,16cyclictest0-21swapper/309:20:333
2787399176,8cyclictest20469-21diskmemload10:54:522
27873991716,1cyclictest0-21swapper/211:27:182
27873991716,1cyclictest0-21swapper/211:27:172
27873991716,1cyclictest0-21swapper/209:35:112
27873991713,3cyclictest27301-21uniq08:15:162
2787099171,15cyclictest1250-21needreboot10:15:181
2787099170,16cyclictest20469-21diskmemload09:26:301
2786899170,1cyclictest0-21swapper/009:27:270
36472160,0sleep30-21swapper/310:55:193
27875991615,1cyclictest0-21swapper/309:18:203
2787399169,7cyclictest0-21swapper/211:52:272
2787399168,7cyclictest0-21swapper/209:19:212
27873991615,1cyclictest20469-21diskmemload11:37:252
27873991615,1cyclictest0-21swapper/209:55:132
27873991614,1cyclictest6551-21apt12:14:592
27873991611,4cyclictest0-21swapper/208:55:002
27873991610,5cyclictest13599-21/usr/sbin/munin09:25:202
2787099168,6cyclictest8885-21ssh09:40:481
27870991614,2cyclictest0-21swapper/109:12:451
2787099160,4cyclictest4520-21kworker/1:011:00:381
27875991513,1cyclictest0-21swapper/310:39:563
27875991513,1cyclictest0-21swapper/310:39:553
2787399159,5cyclictest0-21swapper/207:50:142
2787399157,7cyclictest20469-21diskmemload11:22:202
2787399155,9cyclictest0-21swapper/212:32:132
27873991510,2cyclictest0-21swapper/211:55:162
2787099158,5cyclictest6549-21sh12:34:131
2787099158,3cyclictest2149-21rm11:14:191
2787099152,12cyclictest18032-21awk10:25:191
27870991515,0cyclictest0-21swapper/111:27:501
27870991515,0cyclictest0-21swapper/111:27:491
2787099150,14cyclictest3268-21ssh12:12:241
2786899154,6cyclictest7922-21munin-run10:20:010
2786899151,2cyclictest0-21swapper/010:45:200
2786899150,2cyclictest0-21swapper/011:35:010
2786899150,15cyclictest0-21swapper/012:24:190
2786899150,14cyclictest0-21swapper/012:10:120
74162140,0sleep30-21swapper/310:19:383
2787599149,3cyclictest27909-21tr08:15:213
2787599147,6cyclictest25206-21kworker/3:209:47:543
2787599147,6cyclictest12040-21/usr/sbin/munin11:20:133
2787599142,6cyclictest0-21swapper/311:45:153
27875991410,3cyclictest7635-21grep10:00:123
27875991410,3cyclictest15049-21grep11:40:263
27875991410,2cyclictest7169-21grep12:15:133
27875991410,2cyclictest19837-21tr08:00:133
2787599140,13cyclictest0-21swapper/309:54:003
2787399149,3cyclictest0-21swapper/210:19:082
2787399148,5cyclictest5899-21cut07:30:142
2787399146,7cyclictest20469-21diskmemload12:15:432
2787399146,4cyclictest225-21systemd-journal08:45:002
2787399145,7cyclictest0-21swapper/210:08:562
2787399144,8cyclictest0-21swapper/210:24:062
27873991411,2cyclictest22172-21grep08:05:142
27873991410,3cyclictest19884-21wc09:05:242
27873991410,3cyclictest0-21swapper/208:20:242
2787099149,3cyclictest29028-21seq09:53:301
2787099148,6cyclictest22498-21kworker/1:111:09:491
2787099145,4cyclictest28848-21cron08:20:001
2787099142,12cyclictest0-21swapper/111:49:301
27870991410,3cyclictest24665-21ssh10:49:021
27870991410,2cyclictest0-21swapper/110:59:011
2787099140,14cyclictest0-21swapper/110:53:491
27868991414,0cyclictest0-21swapper/012:38:010
27868991410,4cyclictest0-21swapper/009:40:340
2786899140,0cyclictest0-21swapper/011:58:340
2787599139,4cyclictest55850irq/129-enp2s0-10:11:093
2787599139,3cyclictest19341-21tr11:05:143
2787599139,3cyclictest14624-21cat07:49:593
2787599137,5cyclictest28312-21apache212:14:223
2787599135,4cyclictest225-21systemd-journal09:10:013
27875991312,1cyclictest25206-21kworker/3:210:05:433
27875991310,2cyclictest3107-21grep08:30:253
27875991310,2cyclictest30506-21grep09:35:123
27875991310,2cyclictest23606-21grep12:05:163
2787599130,1cyclictest0-21swapper/308:10:123
2787399139,3cyclictest32444-21uname08:25:192
2787399139,3cyclictest15055-21wc08:55:262
2787399135,4cyclictest225-21systemd-journal08:14:542
2787399134,6cyclictest0-21swapper/207:20:132
2787099139,3cyclictest1201-21dump-pmu-power08:30:001
2787099139,2cyclictest6029-21cut11:55:141
2787099136,5cyclictest29835-21sh12:28:421
2787099131,4cyclictest17809-21fschecks_time07:55:161
27870991312,1cyclictest0-21swapper/109:39:191
2787099131,1cyclictest19750-21ssh11:44:061
27870991311,1cyclictest20469-21diskmemload12:39:091
27870991310,2cyclictest1503-21cut08:30:111
27870991310,2cyclictest10011-21tr10:40:141
2787099130,13cyclictest0-21swapper/111:21:331
2787099130,13cyclictest0-21swapper/109:23:151
2787099130,11cyclictest0-21swapper/108:35:121
2787099130,0cyclictest0-21swapper/111:17:211
2786899133,3cyclictest0-21swapper/009:35:120
2786899131,6cyclictest19960-21munin-run11:25:010
2786899131,6cyclictest13115-21awk08:55:000
27868991311,1cyclictest8387-21cat10:00:150
2786899131,11cyclictest29956-21awk09:34:590
27868991310,2cyclictest6704-21awk09:20:200
27868991310,2cyclictest26423-21dump-pmu-power08:15:000
27868991310,2cyclictest21085-21vmstat11:05:260
27868991310,2cyclictest16889-21sh07:54:590
2786899130,1cyclictest0-21swapper/010:22:300
2786899130,13cyclictest0-21swapper/011:53:240
2786899130,12cyclictest4827-21cat08:35:170
2787599129,2cyclictest8885-21tr08:45:153
2787599129,2cyclictest8885-21tr08:45:143
2787599129,2cyclictest408-21mailstats08:25:223
2787599129,2cyclictest3617-21grep07:25:173
2787599129,2cyclictest30770-21grep07:15:133
2787599129,2cyclictest28452-21grep11:10:163
2787599129,2cyclictest17880-21grep07:55:153
2787599129,2cyclictest16431-21grep07:50:253
2787599129,2cyclictest12696-21grep08:50:263
2787599128,3cyclictest28385-21basename11:30:013
2787599128,3cyclictest28385-21basename11:30:003
2787599128,2cyclictest27021-21grep10:30:263
2787599121,9cyclictest14220-21idleruntime-cro09:44:593
27875991210,2cyclictest39-21ksoftirqd/311:56:033
27875991210,1cyclictest31458-21apt-get09:55:133
2787599120,1cyclictest141rcu_preempt10:28:013
2787399129,2cyclictest9397-21tail08:45:182
2787399129,2cyclictest9397-21tail08:45:172
2787399129,2cyclictest3518-21cut07:25:172
2787399128,3cyclictest10754-21grep07:40:172
2787399125,1cyclictest1-21systemd09:00:212
2787099129,2cyclictest4605-21ssh11:54:211
2787099129,2cyclictest30271-21latency_hist07:15:001
2787099129,2cyclictest22691-21head08:05:161
2787099129,2cyclictest14791-21tr12:00:131
2787099129,2cyclictest10266-21tr07:40:151
2787099129,1cyclictest7219-21sh10:38:281
2787099129,1cyclictest7219-21sh10:38:281
2787099127,4cyclictest29561-21expr08:20:131
2787099125,4cyclictest12720-21sh10:22:131
2787099122,8cyclictest1142-21runrttasks11:32:521
2786899129,2cyclictest9238-21basename10:40:000
2786899129,2cyclictest9238-21basename10:40:000
2786899129,2cyclictest6725-21df08:40:140
2786899129,2cyclictest501-21gzip09:55:210
2786899129,2cyclictest31275-21tail07:15:160
2786899129,2cyclictest18964-21grep09:05:150
2786899129,2cyclictest17652-21sed10:05:250
2786899129,2cyclictest17525-21ssh10:25:150
2786899129,2cyclictest10140-21cut07:40:140
2786899128,3cyclictest28199-21grep07:10:120
27868991211,0cyclictest0-21swapper/007:25:130
27868991210,1cyclictest17660-21grep09:00:270
2786899120,3cyclictest0-21swapper/010:30:530
2786899120,11cyclictest31239-21cron08:25:010
2787599118,2cyclictest9423-21chrt07:35:573
2787599118,2cyclictest6169-21sed11:35:213
2787599118,2cyclictest31904-21ssh10:53:093
2787599118,2cyclictest30648-21ssh11:50:163
2787599118,2cyclictest30079-21taskset07:12:443
2787599118,2cyclictest29493-21sh12:28:253
2787599118,2cyclictest29104-21grep11:30:133
2787599118,2cyclictest25250-21ssh09:12:153
2787599118,2cyclictest2171-21ssh12:30:453
2787599118,2cyclictest17744-21wc10:45:003
2787599118,2cyclictest15244-21taskset12:20:103
2787599118,2cyclictest14831-21wc08:55:223
2787599118,2cyclictest14537-21ssh09:25:163
2787599118,2cyclictest11201-21expr11:00:143
2787599118,2cyclictest10967-21taskset10:20:503
2787599114,3cyclictest225-21systemd-journal08:10:003
2787599111,6cyclictest510-21aten2_r7power_c07:20:103
2787399119,1cyclictest0-21swapper/208:35:122
2787399118,2cyclictest12698-21tr07:45:142
2787399118,2cyclictest0-21swapper/208:00:122
2787399117,3cyclictest30819-21tr07:15:142
27873991110,1cyclictest0-21swapper/207:35:122
2787099118,2cyclictest7171-21tr12:15:131
2787099118,2cyclictest4082-21open_inodes07:25:221
2787099118,2cyclictest238332sleep110:10:011
2787099118,2cyclictest20878-21awk08:00:201
2787099118,2cyclictest18612-21grep09:05:131
2787099118,2cyclictest15225-21sed08:55:271
2787099118,2cyclictest14221-21sshd07:46:201
2787099117,3cyclictest32643-21cron07:20:011
2787099111,7cyclictest14759-21/usr/sbin/munin07:50:091
2786899118,2cyclictest4779-21nfs4_client11:15:180
2786899118,2cyclictest3766-21head10:55:220
2786899118,2cyclictest32348-21ssh11:12:330
2786899118,2cyclictest29175-21ssh09:15:040
2786899118,2cyclictest26137-21ssh10:50:020
2786899118,2cyclictest24280-21ssh12:25:160
2786899118,2cyclictest24189-21tr10:10:120
2786899118,2cyclictest23623-21ssh11:45:360
2786899118,2cyclictest23078-21tr12:05:130
2786899118,2cyclictest21143-21rm11:25:160
2786899118,2cyclictest21143-21rm11:25:160
2786899118,2cyclictest20094-21df08:00:130
2786899118,2cyclictest1747-21grep07:20:180
2786899118,2cyclictest15091-21tr09:45:160
2786899118,2cyclictest14217-21ssh10:42:230
2786899118,2cyclictest1189-21cron08:30:010
2786899116,4cyclictest19284-21systemctl08:00:000
2786899112,3cyclictest10778-21basename11:00:120
2787599108,1cyclictest8144-21taskset08:42:333
2787599108,1cyclictest5862-21taskset08:39:033
2787599108,1cyclictest5558-21expr07:30:123
2787599108,1cyclictest309212sleep308:20:573
2787599108,1cyclictest16789-21ls09:00:173
2787599107,2cyclictest11253-21sed07:40:213
2787399108,1cyclictest2194-21expr08:30:142
2787399107,2cyclictest30260-21cron07:15:012
2787099108,1cyclictest7845-21cat07:35:131
2787099108,1cyclictest242662sleep108:10:091
2787099108,1cyclictest16014-21cut09:00:131
2787099107,2cyclictest9941-21sendmail_mailst08:45:231
2787099107,2cyclictest9941-21sendmail_mailst08:45:221
2787099107,2cyclictest802-21cat07:20:111
2787099107,2cyclictest13337-21cat08:55:001
2786899108,1cyclictest8715-21wget08:45:150
2786899108,1cyclictest8715-21wget08:45:140
2786899108,1cyclictest28033-21cut08:15:220
2786899108,1cyclictest21992-21cut08:05:120
2786899107,2cyclictest7745-21grep07:35:130
2786899107,2cyclictest1765-21expr08:30:120
2786899107,2cyclictest14327-21sed08:55:170
2786899107,2cyclictest12968-21chrt11:40:090
2786899107,2cyclictest0-21swapper/007:30:120
278709998,1cyclictest0-21swapper/108:40:161
278689997,1cyclictest13762-21grep07:45:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional