You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 21:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue May 19, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2156429986,9sleep00-21swapper/007:09:460
2123429279,9sleep20-21swapper/207:05:302
2126528565,9sleep30-21swapper/307:05:543
2128528466,14sleep10-21swapper/107:06:091
43322550,0sleep00-21swapper/011:38:350
21676992626,0cyclictest0-21swapper/310:19:593
21676992321,2cyclictest39-21ksoftirqd/310:38:553
2167699231,20cyclictest0-21swapper/311:36:573
2167499230,23cyclictest0-21swapper/211:36:372
2167499230,23cyclictest0-21swapper/210:16:202
2166999230,3cyclictest0-21swapper/110:31:431
2166599210,17cyclictest0-21swapper/011:53:000
2167499200,20cyclictest0-21swapper/212:06:142
2167499200,19cyclictest0-21swapper/210:47:052
21669992018,1cyclictest0-21swapper/108:05:211
2166599200,2cyclictest0-21swapper/010:17:500
2167499190,19cyclictest0-21swapper/212:12:082
2167499190,18cyclictest0-21swapper/212:25:462
2166999193,2cyclictest141rcu_preempt11:05:091
2166999190,3cyclictest0-21swapper/111:04:231
2166599191,3cyclictest141rcu_preempt09:50:150
21669991815,2cyclictest25-21ksoftirqd/112:00:161
21669991815,2cyclictest25-21ksoftirqd/110:15:011
2166999180,3cyclictest141rcu_preempt10:43:591
2166999180,1cyclictest141rcu_preempt11:19:171
21665991818,0cyclictest0-21swapper/009:46:410
21665991818,0cyclictest0-21swapper/009:24:280
2166599180,0cyclictest0-21swapper/009:33:190
2167499170,3cyclictest0-21swapper/212:35:542
2167499170,1cyclictest0-21swapper/210:52:482
2167499170,13cyclictest0-21swapper/209:40:332
2166999179,7cyclictest16396-21grep12:05:131
2166999172,1cyclictest141rcu_preempt10:16:471
21669991716,1cyclictest14419-21diskmemload10:38:551
21669991712,4cyclictest25-21ksoftirqd/112:24:091
2166999170,17cyclictest0-21swapper/111:45:241
21665991715,1cyclictest722-21sed08:40:170
2166599171,0cyclictest141rcu_preempt10:46:160
2166599170,17cyclictest0-21swapper/010:56:100
193412170,1sleep30-21swapper/310:49:563
2167699162,12cyclictest9440-21sendmail09:00:003
2167699161,8cyclictest0-21swapper/311:10:143
2167699160,1cyclictest0-21swapper/312:05:213
2167699160,1cyclictest0-21swapper/310:53:253
2167699160,15cyclictest0-21swapper/311:26:353
21674991615,1cyclictest0-21swapper/209:28:272
21674991611,5cyclictest18772-21apache209:39:092
21669991614,1cyclictest13013-21latency_hist07:59:591
2166999160,16cyclictest0-21swapper/111:34:341
2166599161,1cyclictest0-21swapper/011:08:380
2167699155,8cyclictest449-21irqbalance11:51:223
21676991512,2cyclictest0-21swapper/309:30:393
2167699151,1cyclictest141rcu_preempt12:24:433
21676991511,2cyclictest1835-21grep12:35:173
21676991511,2cyclictest16334-21sleep09:50:003
2167499153,11cyclictest55550irq/126-enp2s0-12:00:142
2167499152,2cyclictest0-21swapper/211:41:322
21674991514,1cyclictest0-21swapper/210:43:442
2167499150,1cyclictest0-21swapper/210:56:032
2167499150,15cyclictest0-21swapper/211:53:362
2166999154,7cyclictest31918-21df_abs07:30:151
2166999150,7cyclictest0-21swapper/112:14:211
2166999150,13cyclictest0-21swapper/112:29:051
21676991411,2cyclictest31190-21cut11:15:223
21676991411,2cyclictest14920-21grep08:00:263
21676991410,3cyclictest8480-21cat12:20:003
21676991410,3cyclictest13914-21grep09:05:263
2167699140,14cyclictest0-21swapper/310:40:203
21674991414,0cyclictest0-21swapper/210:36:312
21674991414,0cyclictest0-21swapper/210:01:532
21674991412,1cyclictest3024-21tune2fs08:45:152
2167499141,0cyclictest0-21swapper/209:14:492
2167499140,14cyclictest0-21swapper/211:47:322
2167499140,12cyclictest27863-21tr08:30:132
21669991411,2cyclictest21528-21tr11:10:141
2166999141,11cyclictest5492-21ssh11:20:031
2166999140,1cyclictest141rcu_preempt11:40:371
2166599149,3cyclictest13497-21ssh12:22:350
2166599143,6cyclictest959-21munin-run12:35:000
21665991411,2cyclictest17772-21cron08:10:000
21665991410,3cyclictest25508-21awk07:15:230
2166599140,10cyclictest0-21swapper/012:25:130
2167699134,4cyclictest0-21swapper/312:00:143
21676991312,1cyclictest0-21swapper/311:01:533
21676991310,2cyclictest5919-21cat07:45:003
21676991310,2cyclictest39162sleep310:01:023
21676991310,2cyclictest32396-21grep09:20:153
21676991310,2cyclictest2517-21grep10:20:143
21676991310,2cyclictest23174-21grep08:20:143
21676991310,2cyclictest22955-21sleep11:50:003
21676991310,2cyclictest19300-21sed08:10:233
21676991310,2cyclictest18652-21grep10:10:163
21676991310,2cyclictest18627-21ssh12:25:223
21676991310,2cyclictest17248-21ssh09:50:143
21676991310,2cyclictest1242-21wc08:40:223
21676991310,2cyclictest1103-21cron07:34:593
2167699130,11cyclictest0-21swapper/309:16:253
2167499139,2cyclictest19320-21ls08:10:222
2167499131,1cyclictest14419-21diskmemload11:25:192
21674991311,1cyclictest21884-21ssh09:52:462
21674991310,2cyclictest8774-21grep07:50:152
21674991310,2cyclictest30792-21grep08:35:172
21674991310,2cyclictest29812-21grep07:25:182
21674991310,2cyclictest26254-21grep12:30:172
21674991310,2cyclictest14404-21cron09:10:002
21674991310,2cyclictest10729-21basename10:25:152
2167499130,3cyclictest0-21swapper/211:24:492
2167499130,1cyclictest0-21swapper/212:24:252
2167499130,1cyclictest0-21swapper/211:58:072
2166999139,3cyclictest26760-21grep12:30:201
2166999138,4cyclictest2131-21ssh11:56:011
2166999136,5cyclictest23669-21sed08:20:171
2166999134,6cyclictest25612-21wc09:35:221
2166999134,6cyclictest22604-21munin-run08:20:001
2166999134,3cyclictest25-21ksoftirqd/111:28:051
21669991310,2cyclictest30736-21grep11:35:111
21669991310,2cyclictest26026-21sed08:25:191
2166599139,3cyclictest4986-21tr08:50:150
21665991312,1cyclictest21656-21cyclictest11:13:430
21665991311,2cyclictest13-21ksoftirqd/010:01:540
21665991310,2cyclictest6528-21cut11:20:160
21665991310,2cyclictest29321-21ssh10:35:280
2166599130,13cyclictest0-21swapper/012:05:120
2167699129,2cyclictest3113-21grep08:45:153
2167699129,2cyclictest25976-21grep08:25:193
2167699129,2cyclictest25904-21ssh09:35:253
2167699121,1cyclictest0-21swapper/311:09:163
21676991210,1cyclictest0-21swapper/310:56:013
2167699120,7cyclictest0-21swapper/310:05:123
2167699120,3cyclictest0-21swapper/312:31:103
2167499129,2cyclictest7582-21cut08:55:162
2167499129,2cyclictest31788-21grep09:20:122
2167499129,2cyclictest31605-21cat11:15:272
2167499129,2cyclictest30945-21ssh09:58:202
2167499129,2cyclictest26899-21ssh10:35:002
2167499129,2cyclictest26887-21grep08:25:282
2167499129,2cyclictest2512-21tr10:20:142
2167499129,2cyclictest23510-21grep08:20:162
2167499129,2cyclictest22500-21grep07:10:172
2167499129,2cyclictest2170-21grep07:35:182
2167499129,2cyclictest21471-21tr11:10:152
2167499129,2cyclictest2096-21sh08:45:012
2167499129,2cyclictest15434-21cat08:05:002
2167499129,2cyclictest14415-21taskset10:07:402
2167499128,3cyclictest14119-21tr11:05:212
2167499120,12cyclictest0-21swapper/212:15:252
2167499120,12cyclictest0-21swapper/211:30:332
2167499120,12cyclictest0-21swapper/209:17:092
2166999129,2cyclictest5863-21tr08:50:211
2166999129,2cyclictest32148-21tr08:40:001
2166999129,2cyclictest25279-21grep09:15:271
2166999129,2cyclictest16396-21ssh09:10:221
2166999129,2cyclictest134622sleep109:05:221
2166999129,2cyclictest11607-21grep09:00:281
2166999128,3cyclictest5507-21ssh10:02:161
2166999128,3cyclictest1605-21grep12:35:161
2166999128,3cyclictest1244-21sort12:15:181
2166999128,2cyclictest9613-21wc09:00:001
2166999126,5cyclictest14419-21diskmemload10:29:101
2166999124,7cyclictest18772-21apache210:55:371
2166999124,2cyclictest25-21ksoftirqd/109:40:591
2166999123,5cyclictest9472-21/usr/sbin/munin10:05:181
21669991210,1cyclictest25-21ksoftirqd/110:24:321
2166999120,1cyclictest141rcu_preempt11:50:161
2166599129,3cyclictest18772-21apache211:34:270
2166599129,2cyclictest8133-21wc09:45:000
2166599129,2cyclictest7519-21cut11:40:180
2166599129,2cyclictest648-21cut12:15:150
2166599129,2cyclictest6420-21grep07:45:150
2166599129,2cyclictest31629-21grep08:35:250
2166599129,2cyclictest26226-21wc12:10:260
2166599129,2cyclictest20251-21ssh10:30:190
2166599129,2cyclictest19758-21grep08:10:260
2166599129,2cyclictest11663-21sed07:55:170
2166599128,3cyclictest25014-21cut09:35:170
2166599128,3cyclictest19173-21sed10:10:190
2166599123,4cyclictest225-21systemd-journal08:25:000
2166599123,3cyclictest9427-21apt09:00:000
21665991211,1cyclictest0-21swapper/009:57:020
2166599120,9cyclictest0-21swapper/011:19:120
2166599120,1cyclictest0-21swapper/011:27:270
2166599120,11cyclictest9264-21grep07:50:170
2167699118,2cyclictest8506-21sh11:21:033
2167699118,2cyclictest739-21ntp_states11:55:233
2167699118,2cyclictest6831-21grep07:45:173
2167699118,2cyclictest454-21wc09:40:133
2167699118,2cyclictest3520-21readlink07:40:003
2167699118,2cyclictest29788-21grep07:25:183
2167699118,2cyclictest27879-21cut07:20:223
2167699118,2cyclictest26537-21ntpq09:55:213
2167699118,2cyclictest25064-21chrt11:30:573
2167699118,2cyclictest22190-21grep07:10:153
2167699118,2cyclictest20557-21ssh10:30:233
2167699118,2cyclictest19370-21ssh09:12:183
2167699118,2cyclictest17784-21turbostat.cron08:09:593
2167699118,2cyclictest13007-21idleruntime-cro07:59:593
2167699118,2cyclictest11038-21ssh09:26:403
2167699117,3cyclictest26299-21cat12:10:273
2167699117,3cyclictest25993-21grep07:15:273
2167699117,1cyclictest0-21swapper/311:43:583
2167699112,8cyclictest27639-21apt-get08:30:123
2167699112,4cyclictest0-21swapper/310:25:133
2167499118,2cyclictest9804-21apt-key09:00:012
2167499118,2cyclictest8322-21ssh09:45:072
2167499118,2cyclictest7540-21rm11:01:272
2167499118,2cyclictest5450-21grep08:50:172
2167499118,2cyclictest28909-21cut07:25:002
2167499118,2cyclictest20750-21tr08:15:162
2167499118,2cyclictest18855-21basename10:10:172
2167499118,2cyclictest18707-21taskset09:31:082
2166999119,1cyclictest6871-21head07:45:181
2166999119,1cyclictest0-21swapper/109:55:571
2166999118,2cyclictest9223-21cut07:50:171
2166999118,2cyclictest4459-21cron08:50:001
2166999118,2cyclictest3161-21chrt09:22:001
2166999118,2cyclictest22393-21grep07:10:151
2166999118,2cyclictest20359-21sh09:51:351
2166999118,2cyclictest20307-21grep10:50:161
2166999118,2cyclictest19308-21ssh09:31:341
2166999118,2cyclictest16852-21ssh10:47:591
2166999118,2cyclictest14515-21tr08:00:221
2166999118,2cyclictest10431-21sed09:45:261
2166999118,2cyclictest10242-21ssh09:26:031
2166999112,7cyclictest38150irq/131-snd_hda07:35:121
2166599118,2cyclictest8255-21cut09:25:170
2166599118,2cyclictest6837-21ssh11:00:530
2166599118,2cyclictest3398-21ssh11:57:010
2166599118,2cyclictest3001-21grep08:45:150
2166599118,2cyclictest28729-21grep08:30:190
2166599118,2cyclictest27915-21grep07:20:220
2166599118,2cyclictest26959-21head08:25:270
2166599118,2cyclictest22159-21tail08:15:270
2166599118,2cyclictest19819-21ssh10:50:060
2166599118,2cyclictest1954-21sed07:35:150
2166599118,2cyclictest16173-21grep09:10:190
2166599118,2cyclictest15589-21sh12:04:590
2166599118,2cyclictest12806-21ssh10:25:370
2166599118,2cyclictest10270-21grep10:05:160
2166599117,3cyclictest4402-21grep07:40:150
2166599117,3cyclictest24052-21grep09:15:180
2166599117,3cyclictest15193-21grep11:45:130
2166599113,4cyclictest1451-21/usr/sbin/munin12:35:190
2166599112,7cyclictest0-21swapper/010:40:200
2167699108,1cyclictest8832-21cat07:50:153
2167699108,1cyclictest21423-21cut08:15:203
2167699108,1cyclictest10305-21expr09:00:173
2167699107,2cyclictest6661-21chrt08:52:543
2167699107,2cyclictest30235-21cut08:35:153
2167499108,1cyclictest6120-21chrt07:45:082
2167499108,1cyclictest4972-21ls07:40:202
2167499108,1cyclictest31429-21apache_processe07:30:112
2167499108,1cyclictest26046-21taskset07:15:432
2167499108,1cyclictest15976-21cat08:05:132
2167499108,1cyclictest10875-21basename07:55:102
2166999108,1cyclictest30751-21sed07:25:281
2166999108,1cyclictest27086-21expr07:20:151
2166999108,1cyclictest20170-21chrt08:14:361
2166999108,1cyclictest1833-21taskset08:41:491
2166999107,2cyclictest4207-21cat07:40:131
2166999107,2cyclictest28350-21expr08:30:171
2166599108,1cyclictest308192sleep007:25:430
2166599108,1cyclictest11824-21chrt09:02:270
2166599107,2cyclictest2576-21cut10:20:130
2166599107,2cyclictest14431-21tr08:00:200
2166599107,2cyclictest13360-21wc09:05:200
2166599100,9cyclictest1017-21chrt07:33:570
216659996,2cyclictest22520-21cut07:10:160
216699986,1cyclictest24241-21taskset07:15:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional