You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 20:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Mar 11, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
851929564,12sleep30-21swapper/307:06:493
847829380,9sleep00-21swapper/007:06:180
842529066,8sleep20-21swapper/207:05:362
847928164,13sleep10-21swapper/107:06:191
8849993130,1cyclictest0-21swapper/009:35:090
8849992524,1cyclictest0-21swapper/011:13:130
885699230,23cyclictest0-21swapper/212:10:492
885699230,23cyclictest0-21swapper/212:10:492
885699220,20cyclictest0-21swapper/210:56:292
885499220,2cyclictest13730-21sed12:30:011
885699202,1cyclictest0-21swapper/210:01:382
8854992015,2cyclictest0-21swapper/109:16:351
885499200,20cyclictest0-21swapper/112:15:181
885499200,20cyclictest0-21swapper/111:36:361
8849992018,1cyclictest0-21swapper/011:58:290
8859991917,1cyclictest0-21swapper/311:03:513
885699190,0cyclictest0-21swapper/212:19:552
885499190,18cyclictest0-21swapper/110:40:301
884999192,1cyclictest0-21swapper/011:33:560
8849991917,1cyclictest0-21swapper/009:13:240
8849991917,1cyclictest0-21swapper/008:00:130
885999183,2cyclictest141rcu_preempt12:07:533
8859991812,4cyclictest8066-21ssh10:50:123
8856991817,1cyclictest0-21swapper/210:12:252
8854991816,1cyclictest22378-21kworker/1:107:40:141
8854991815,2cyclictest0-21swapper/109:00:181
885499180,1cyclictest0-21swapper/109:51:271
885499180,16cyclictest0-21swapper/109:09:551
885499180,0cyclictest0-21swapper/112:06:281
8849991816,1cyclictest13-21ksoftirqd/010:12:470
8849991816,1cyclictest13-21ksoftirqd/010:05:110
8849991816,1cyclictest0-21swapper/012:39:250
8849991813,4cyclictest13-21ksoftirqd/011:40:000
884999180,1cyclictest0-21swapper/011:15:150
884999180,18cyclictest0-21swapper/012:22:080
884999180,14cyclictest1434-21diskmemload09:32:490
885499172,14cyclictest11292-21latency_hist09:35:001
8854991716,1cyclictest0-21swapper/110:00:181
885499171,1cyclictest10410-21ssh11:30:111
885499170,0cyclictest0-21swapper/110:11:551
884999172,13cyclictest12484-21cron09:55:000
8849991716,1cyclictest0-21swapper/011:08:350
8849991716,1cyclictest0-21swapper/010:20:320
8849991715,1cyclictest13-21ksoftirqd/011:24:150
884999170,0cyclictest141rcu_preempt10:57:070
245942171,0sleep30-21swapper/312:35:323
885999162,8cyclictest225-21systemd-journal08:30:003
8856991616,0cyclictest0-21swapper/209:29:052
885699160,3cyclictest0-21swapper/210:25:062
885499162,13cyclictest13847-21awk08:25:241
8854991611,4cyclictest11740-21ssh10:51:441
884999162,8cyclictest29052-21sendmail09:00:000
884999162,1cyclictest141rcu_preempt09:28:320
8849991615,1cyclictest0-21swapper/008:22:390
8849991613,2cyclictest26657-21ls08:55:000
8849991613,2cyclictest26657-21ls08:55:000
884999161,1cyclictest141rcu_preempt08:15:130
885999158,3cyclictest27905-21grep09:25:153
885999154,7cyclictest11467-21apt-get07:15:133
885999152,3cyclictest0-21swapper/311:10:143
885999150,1cyclictest0-21swapper/310:35:163
885699159,5cyclictest20697-21cat09:20:242
885699151,1cyclictest7218-21ssh11:08:512
885699150,2cyclictest0-21swapper/210:06:032
885699150,15cyclictest0-21swapper/209:19:492
885699150,14cyclictest0-21swapper/209:52:332
8854991510,3cyclictest16531-21cut10:55:141
885499150,1cyclictest0-21swapper/109:35:331
884999152,9cyclictest13-21ksoftirqd/011:43:020
884999152,2cyclictest0-21swapper/010:02:320
884999152,10cyclictest13-21ksoftirqd/010:45:120
8849991513,2cyclictest13-21ksoftirqd/012:13:480
8849991513,2cyclictest13-21ksoftirqd/012:13:480
8849991512,2cyclictest13-21ksoftirqd/012:10:000
885999148,3cyclictest23093-21sendmail07:40:003
8859991411,2cyclictest17921-21sed07:25:283
8859991410,3cyclictest24349-21cat10:20:243
885699143,1cyclictest0-21swapper/209:30:082
8856991412,1cyclictest0-21swapper/211:36:452
8856991410,1cyclictest55750irq/128-enp2s0-10:48:552
885699140,14cyclictest0-21swapper/211:26:072
8854991413,1cyclictest0-21swapper/110:37:301
885499141,2cyclictest0-21swapper/111:59:441
8854991412,1cyclictest0-21swapper/111:50:531
8854991410,3cyclictest9772-21cron08:20:001
885499140,3cyclictest0-21swapper/110:19:071
885499140,2cyclictest0-21swapper/111:25:251
885499140,12cyclictest507-21head10:45:171
884999145,2cyclictest141rcu_preempt10:28:430
8849991411,2cyclictest7398-21sh08:15:000
884999141,12cyclictest0-21swapper/007:40:140
8849991410,3cyclictest24064-21ls08:50:000
885999139,3cyclictest27909-21http09:45:003
885999136,4cyclictest14198-21apache_volume10:15:113
885999132,8cyclictest0-21swapper/311:59:283
885999131,4cyclictest18845-21/usr/sbin/munin11:15:223
8859991310,2cyclictest4027-21grep09:30:163
8859991310,2cyclictest30797-21grep10:05:173
8859991310,2cyclictest19316-21head07:30:163
885999130,12cyclictest2600-21grep11:25:163
885699135,8cyclictest29822-21apache212:20:212
885699134,3cyclictest14093-21apt-get12:30:112
885699132,7cyclictest32-21ksoftirqd/211:30:132
8856991311,1cyclictest9681-21grep07:10:162
8856991310,2cyclictest9148-21cat08:15:252
8856991310,2cyclictest27165-21grep08:55:142
8856991310,2cyclictest17857-21grep07:25:282
885699130,13cyclictest0-21swapper/211:49:142
885499139,3cyclictest4537-21grep08:05:251
885499139,3cyclictest19763-21grep09:20:161
885499138,2cyclictest19415-21ssh12:13:521
885499138,2cyclictest19415-21ssh12:13:521
885499133,3cyclictest0-21swapper/112:00:111
885499132,10cyclictest1142-21runrttasks11:42:381
885499131,3cyclictest1434-21diskmemload11:16:321
8854991312,1cyclictest0-21swapper/110:21:461
8854991311,2cyclictest25-21ksoftirqd/111:47:081
8854991310,2cyclictest28485-21wc08:55:251
884999139,3cyclictest1426-21cron09:10:000
884999139,3cyclictest10320-21head07:10:210
884999131,7cyclictest24782-21ssh11:00:110
884999131,2cyclictest141rcu_preempt10:30:300
8849991312,1cyclictest0-21swapper/010:53:370
8849991311,2cyclictest0-21swapper/009:20:470
8849991311,1cyclictest29819-21apache211:25:200
8849991310,2cyclictest4394-21wc08:05:250
8849991310,2cyclictest25128-21wc10:40:210
8849991310,2cyclictest13-21ksoftirqd/011:49:240
884999130,13cyclictest0-21swapper/009:18:560
885999129,2cyclictest5891-21cut12:25:133
885999129,2cyclictest5301-21sort09:50:163
885999129,2cyclictest31670-21awk10:45:003
885999129,2cyclictest29401-21basename09:00:143
885999129,2cyclictest28616-21grep08:55:263
885999129,2cyclictest18967-21apache_volume09:20:123
885999128,3cyclictest30808-21tr07:55:153
885999123,8cyclictest11233-21sed09:15:183
885999121,9cyclictest0-21swapper/311:24:553
885999120,11cyclictest7684-21cut08:15:133
885699129,2cyclictest4928-21grep12:05:132
885699129,2cyclictest2684-21tune2fs09:10:162
885699129,2cyclictest21987-21cut08:45:132
885699129,2cyclictest16523-21sed08:30:252
885699129,2cyclictest12421-21grep09:35:152
885699128,3cyclictest23159-21grep12:35:162
885699122,1cyclictest225-21systemd-journal09:45:002
885699121,1cyclictest141rcu_preempt12:26:312
8856991210,1cyclictest16449-21wc10:35:192
885699120,3cyclictest0-21swapper/211:13:372
885699120,11cyclictest30466-21sed09:00:192
885499129,2cyclictest3329-21grep09:10:211
885499129,2cyclictest31218-21head10:05:191
885499129,2cyclictest30696-21head12:20:161
885499129,2cyclictest22064-21wc12:35:011
885499129,2cyclictest18840-21grep08:35:261
885499129,2cyclictest16485-21sed08:30:251
885499129,2cyclictest15999-21wc07:25:011
885499129,2cyclictest1562-21uniq11:05:191
885499129,2cyclictest11713-21grep07:15:141
885499128,3cyclictest31293-21grep10:25:141
885499127,4cyclictest15215-21ssh11:13:361
885499126,2cyclictest225-21systemd-journal08:55:001
885499126,2cyclictest225-21systemd-journal08:55:001
885499121,8cyclictest38150irq/131-snd_hda08:45:131
885499120,9cyclictest0-21swapper/109:47:051
884999129,2cyclictest30831-21tr07:55:150
884999129,2cyclictest30310-21cat12:00:240
884999129,2cyclictest21220-21cat09:40:200
884999129,2cyclictest17104-21grep10:35:260
884999129,2cyclictest14982-21tr07:20:200
884999129,2cyclictest13-21ksoftirqd/012:25:570
884999128,2cyclictest21701-21sleep08:45:000
884999128,1cyclictest141rcu_preempt12:30:090
884999120,7cyclictest0-21swapper/009:55:100
325522120,0chrt0-21swapper/211:04:542
885999118,2cyclictest6974-21wc10:10:183
885999118,2cyclictest6518-21grep08:10:213
885999118,2cyclictest4960-21ssh11:07:063
885999118,2cyclictest4556-21grep08:05:253
885999118,2cyclictest32330-21ssh11:42:493
885999118,2cyclictest32261-21ssh09:46:413
885999118,2cyclictest31196-21ssh12:00:533
885999118,2cyclictest3068-21ssh11:45:083
885999118,2cyclictest29995-21grep12:20:133
885999118,2cyclictest25057-21head08:50:153
885999118,2cyclictest25057-21head08:50:153
885999118,2cyclictest22979-21ssh12:15:233
885999118,2cyclictest21525-21ssh10:00:113
885999118,2cyclictest21280-21ssh11:35:483
885999118,2cyclictest2033-21cut09:10:143
885999118,2cyclictest20197-21sh12:33:373
885999118,2cyclictest20053-21ssh10:56:413
885999118,2cyclictest19727-21ssh09:59:083
885999118,2cyclictest17959-21grep08:35:173
885999118,2cyclictest15963-21ssh12:11:093
885999118,2cyclictest15963-21ssh12:11:093
885999118,2cyclictest13308-21grep11:50:243
885999118,2cyclictest1269-21ssh10:25:573
885999118,2cyclictest12487-21grep09:35:153
885999118,2cyclictest11997-21ssh11:30:223
885999118,2cyclictest10822-21ssh10:31:533
885999118,2cyclictest1016-21sed09:05:283
885999113,2cyclictest225-21systemd-journal08:43:293
885999111,5cyclictest1594-21timerwakeupswit10:45:263
885699119,1cyclictest7401-21ls08:15:002
885699119,1cyclictest29098-21grep11:40:262
885699119,1cyclictest28401-21grep07:50:152
885699119,1cyclictest26465-21grep07:45:142
885699118,2cyclictest30979-21ssh12:00:432
885699118,2cyclictest29852-21ssh11:21:562
885699118,2cyclictest29432-21head09:45:172
885699118,2cyclictest21087-21cat08:40:252
885699118,2cyclictest20430-21taskset07:31:272
885699118,2cyclictest19331-21lockfile-create08:40:002
885699118,2cyclictest18714-21cat11:15:212
885699118,2cyclictest16113-21grep10:15:252
885699118,2cyclictest14574-21cat08:30:002
885699118,2cyclictest13009-21ssh10:52:442
8856991110,1cyclictest32-21ksoftirqd/211:52:172
885699110,2cyclictest0-21swapper/210:31:472
885499119,1cyclictest16341-21tr07:25:161
885499118,2cyclictest9555-21ssh10:30:531
885499118,2cyclictest9154-21tr07:10:141
885499118,2cyclictest782-21expr08:00:141
885499118,2cyclictest6423-21ntpq08:10:201
885499118,2cyclictest28647-21ssh09:25:211
885499118,2cyclictest26169-21grep11:20:141
885499118,2cyclictest21711-21grep07:35:171
885499118,2cyclictest20685-21cut08:40:191
885499118,2cyclictest19380-21expr07:30:161
885499118,2cyclictest13430-21cat09:55:131
885499117,3cyclictest28978-21ssh12:38:581
885499116,2cyclictest225-21systemd-journal09:45:001
885499110,10cyclictest245962sleep111:00:041
884999119,2cyclictest13-21ksoftirqd/010:15:210
884999118,2cyclictest31432-21chrt09:04:300
884999118,2cyclictest28407-21grep07:50:150
884999118,2cyclictest19576-21ssh11:54:590
884999118,2cyclictest19256-21cut07:30:150
884999118,2cyclictest18423-21tr08:35:210
884999118,2cyclictest16496-21head08:30:250
884999110,0cyclictest141rcu_preempt12:15:090
885999108,1cyclictest9395-21wc07:10:153
885999108,1cyclictest23085-21ntpq08:45:193
885999108,1cyclictest157402sleep307:22:163
885999108,1cyclictest10356-21expr08:20:143
885999107,2cyclictest26495-21sed07:45:143
885999107,2cyclictest16426-21grep08:30:253
885999107,2cyclictest1343-21idleruntime08:00:173
885699109,1cyclictest1296-21head08:00:172
885699108,1cyclictest32496-21grep09:05:192
885699108,1cyclictest19642-21ssh09:59:062
885699107,2cyclictest25586-21ssh10:21:112
885699107,2cyclictest23533-21ssh10:40:062
885699107,2cyclictest15766-21chrt07:22:352
885699104,3cyclictest0-21swapper/211:55:112
885499108,1cyclictest10181-21cut08:20:141
885499107,2cyclictest30559-21apache_processe07:55:131
885499107,2cyclictest28212-21cut07:50:141
884999108,1cyclictest12784-21find07:15:220
884999107,2cyclictest25989-21tr07:45:120
884999107,2cyclictest17020-21cut07:25:190
884999107,2cyclictest13545-21cut08:25:180
884999105,3cyclictest13-21ksoftirqd/009:45:160
884999103,2cyclictest141rcu_preempt07:35:180
88599997,1cyclictest29481-21cut07:50:233
88599994,2cyclictest225-21systemd-journal07:40:003
88569998,1cyclictest23682-21cat07:40:152
88569998,1cyclictest21199-21tr07:35:152
88569998,1cyclictest11665-21cat07:15:142
88569997,2cyclictest31524-21ls07:55:192
88569997,1cyclictest24328-21apt-get08:50:112
88569997,1cyclictest24328-21apt-get08:50:112
88569997,1cyclictest10803-21head08:20:162
88569990,9cyclictest3876-21cat08:05:182
88549997,1cyclictest268882sleep107:45:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional