You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 16:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun May 10, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
107229466,9sleep00-21swapper/007:06:510
3161129165,8sleep30-21swapper/307:05:043
101428875,9sleep20-21swapper/207:06:072
108928467,13sleep10-21swapper/107:07:031
104972580,0sleep00-21swapper/012:32:080
141999301,6cyclictest0-21swapper/111:05:251
141999300,1cyclictest28788-21kworker/1:012:25:071
142299280,25cyclictest0-21swapper/210:13:352
141999281,2cyclictest25365-21ssh10:45:141
141999280,1cyclictest0-21swapper/111:02:591
141999280,11cyclictest0-21swapper/110:42:541
141999271,25cyclictest0-21swapper/112:15:311
141999262,1cyclictest0-21swapper/111:55:101
141999260,2cyclictest0-21swapper/111:35:151
141999252,7cyclictest30585-21users09:30:271
141999250,3cyclictest20300-21expr11:40:151
141999250,2cyclictest0-21swapper/109:40:141
141699250,24cyclictest0-21swapper/008:35:150
141699250,24cyclictest0-21swapper/008:35:140
141999241,8cyclictest298812sleep110:09:321
141999240,3cyclictest0-21swapper/109:48:111
141999230,23cyclictest0-21swapper/110:56:121
1422992218,3cyclictest32-21ksoftirqd/212:05:212
141999224,10cyclictest4322-21smartctl11:30:191
141999223,1cyclictest26852-21diskmemload09:35:011
141999221,20cyclictest0-21swapper/112:00:131
141999220,2cyclictest0-21swapper/112:35:191
1416992222,0cyclictest0-21swapper/009:43:090
142299210,21cyclictest0-21swapper/210:38:112
142299210,21cyclictest0-21swapper/210:27:552
141999215,15cyclictest0-21swapper/109:55:491
141999213,1cyclictest0-21swapper/107:30:161
141999212,1cyclictest26852-21diskmemload12:34:421
141999210,3cyclictest9967-21ssh11:53:191
141999210,2cyclictest0-21swapper/110:11:391
141999210,20cyclictest0-21swapper/111:21:261
141999210,18cyclictest0-21swapper/110:20:161
141699210,1cyclictest0-21swapper/010:40:010
142299205,5cyclictest225-21systemd-journal09:59:592
141999208,11cyclictest0-21swapper/108:45:011
141999202,1cyclictest8830-21latency_hist07:24:591
141999200,3cyclictest0-21swapper/109:20:471
141999200,1cyclictest0-21swapper/109:17:261
141999200,1cyclictest0-21swapper/109:14:091
141999200,17cyclictest26895-21ssh09:29:051
1416992019,1cyclictest0-21swapper/009:52:360
1422991913,2cyclictest20517-21dump-pmu-power12:00:002
142299190,19cyclictest0-21swapper/211:04:372
142299190,18cyclictest0-21swapper/209:18:362
141999194,15cyclictest26852-21diskmemload11:48:491
141999193,15cyclictest26852-21diskmemload12:05:131
141999190,4cyclictest29455-21unixbench_singl11:25:271
141999190,3cyclictest24410-21sshd09:04:261
141999190,0cyclictest0-21swapper/109:52:161
142599184,13cyclictest0-21swapper/310:15:003
142599181,1cyclictest141rcu_preempt10:07:473
141999181,2cyclictest25377-21fschecks_time09:05:181
141999180,2cyclictest0-21swapper/107:15:171
141999180,16cyclictest2365-21head07:10:161
142599173,1cyclictest0-21swapper/307:55:143
142599170,1cyclictest0-21swapper/310:15:453
142599170,1cyclictest0-21swapper/310:15:443
142599170,17cyclictest0-21swapper/312:35:163
142299170,17cyclictest0-21swapper/212:13:142
141999176,7cyclictest38150irq/131-snd_hda12:20:121
141999173,3cyclictest26852-21diskmemload10:52:191
141999171,1cyclictest0-21swapper/110:33:091
141999170,5cyclictest0-21swapper/112:11:271
141999170,3cyclictest0-21swapper/111:15:401
141999170,2cyclictest0-21swapper/111:11:411
141999170,2cyclictest0-21swapper/110:18:121
141999170,2cyclictest0-21swapper/110:18:111
141999170,1cyclictest0-21swapper/110:36:021
141699170,9cyclictest0-21swapper/007:10:120
141699170,1cyclictest0-21swapper/011:37:570
141699170,17cyclictest0-21swapper/011:56:580
141699170,0cyclictest0-21swapper/010:28:440
142599164,7cyclictest39-21ksoftirqd/309:31:173
141999162,2cyclictest24019-21apache_volume10:25:131
141999161,2cyclictest20297-21sed07:45:241
141999161,1cyclictest26852-21diskmemload10:01:221
141699160,8cyclictest29936-21basename12:25:010
141699160,1cyclictest0-21swapper/010:47:300
32182150,5sleep013-21ksoftirqd/010:12:260
142599153,4cyclictest13753-21apache_accesses12:15:123
1425991510,4cyclictest4295-21sed11:30:183
142299151,0cyclictest141rcu_preempt10:50:092
142299150,3cyclictest0-21swapper/209:47:102
142299150,15cyclictest0-21swapper/212:17:122
142299150,15cyclictest0-21swapper/211:33:242
141999158,6cyclictest15965-21basename07:40:001
141999152,1cyclictest10260-21pmu-power07:25:221
141999151,2cyclictest2023-21ntp_states08:15:211
141999150,4cyclictest8664-21grep08:30:161
141699153,6cyclictest28844-21munin-run12:05:000
141699152,6cyclictest5872-21perf12:29:590
141699150,1cyclictest0-21swapper/009:29:570
141699150,0cyclictest0-21swapper/010:51:080
142599146,3cyclictest225-21systemd-journal12:30:013
1425991413,1cyclictest0-21swapper/311:35:163
1425991412,1cyclictest0-21swapper/310:54:003
1425991410,3cyclictest23150-21sh09:45:233
1425991410,3cyclictest18310-21cron07:44:593
1425991410,3cyclictest15918-21cron07:40:003
1425991410,2cyclictest3284-21ssh11:10:183
1425991410,2cyclictest1410-21cyclictest09:42:033
142599140,1cyclictest0-21swapper/309:55:353
142299143,8cyclictest0-21swapper/209:50:122
1422991412,1cyclictest32-21ksoftirqd/212:33:452
1422991410,2cyclictest30497-21grep09:30:272
1422991410,2cyclictest24070-21head12:20:252
1422991410,2cyclictest17277-21chrt11:19:092
142299140,5cyclictest0-21swapper/210:06:532
142299140,1cyclictest0-21swapper/210:56:452
142299140,14cyclictest0-21swapper/210:22:432
142299140,14cyclictest0-21swapper/209:21:332
141999148,5cyclictest29073-21df_abs08:05:151
141999148,5cyclictest20788-21basename08:55:201
141999147,6cyclictest5627-21date08:25:011
141999147,6cyclictest11586-21ntpq08:35:221
141999147,6cyclictest11586-21ntpq08:35:211
141999143,1cyclictest15764-21df_abs08:45:141
141999142,1cyclictest16265-21grep07:40:121
141699142,7cyclictest0-21swapper/009:23:060
141699141,7cyclictest18410-21ssh11:19:590
1416991414,0cyclictest0-21swapper/011:20:380
1416991414,0cyclictest0-21swapper/011:05:140
1416991413,1cyclictest0-21swapper/010:32:040
1416991413,1cyclictest0-21swapper/010:17:450
1416991413,1cyclictest0-21swapper/010:17:440
1416991410,3cyclictest7350-21sed12:10:230
1416991410,3cyclictest6047-21grep09:55:150
142599139,3cyclictest15224-21dpkg10:20:013
142599134,5cyclictest1142-21runrttasks11:41:123
142599134,3cyclictest225-21systemd-journal10:29:323
142599132,6cyclictest0-21swapper/312:25:183
1425991312,1cyclictest11775-21/usr/sbin/munin09:20:183
1425991310,2cyclictest919-21tr08:15:143
1425991310,2cyclictest0-21swapper/309:51:163
142599130,1cyclictest0-21swapper/310:58:263
142599130,12cyclictest13105-21grep07:30:253
142299136,6cyclictest2488-21apache211:45:522
142299131,1cyclictest0-21swapper/210:49:392
1422991311,2cyclictest26852-21diskmemload10:44:152
1422991310,2cyclictest3807-21cron07:15:012
1422991310,2cyclictest2893-21cron09:15:002
1422991310,2cyclictest24330-21sed07:55:162
1422991310,2cyclictest16127-21sh10:00:262
1422991310,2cyclictest14902-21turbostat.cron10:20:012
1422991310,2cyclictest14902-21turbostat.cron10:20:002
142299130,12cyclictest0-21swapper/207:50:152
141999137,5cyclictest19002-21sort08:50:241
141999131,2cyclictest225-21systemd-journal08:30:001
141999131,2cyclictest0-21swapper/108:00:001
141699139,3cyclictest4618-21grep11:50:140
141699139,3cyclictest26223-21grep08:00:140
141699139,3cyclictest13138-21sed07:30:260
141699139,2cyclictest13509-21grep08:40:170
141699139,2cyclictest10170-21cron08:35:000
1416991312,1cyclictest0-21swapper/007:58:330
141699131,1cyclictest141rcu_preempt09:10:180
141699130,7cyclictest20922-21awk07:49:590
142599129,2cyclictest9802-21sed10:35:223
142599129,2cyclictest9207-21tr07:25:143
142599129,2cyclictest8295-21grep08:30:153
142599129,2cyclictest6198-21head12:10:143
142599129,2cyclictest4759-21grep12:10:003
142599129,2cyclictest24037-21grep12:20:253
142599129,2cyclictest22832-21grep12:00:243
142599129,2cyclictest20487-21tr09:25:153
142599129,2cyclictest18343-21grep08:50:163
142599129,2cyclictest13501-21grep11:55:163
142599129,2cyclictest10895-21ssh11:15:153
142599128,3cyclictest4994-21sed08:20:273
142599128,3cyclictest27802-21grep11:05:233
142599121,0cyclictest0-21swapper/310:43:443
142599120,12cyclictest0-21swapper/307:10:193
142599120,11cyclictest18639-21cut11:00:153
142299129,2cyclictest7891-21sh10:35:012
142299129,2cyclictest7359-21grep08:25:262
142299129,2cyclictest7291-21grep07:20:162
142299129,2cyclictest6402-21cron07:20:002
142299129,2cyclictest5525-21grep11:50:202
142299129,2cyclictest30651-21sleep08:09:592
142299129,2cyclictest18675-21cat08:50:182
142299129,2cyclictest15917-21cron07:40:002
142299129,2cyclictest13188-21grep09:40:152
142299129,2cyclictest10175-21sh08:35:012
142299125,7cyclictest55750irq/128-enp2s0-11:13:512
142299124,4cyclictest225-21systemd-journal12:35:012
142299120,2cyclictest0-21swapper/211:27:132
142299120,12cyclictest0-21swapper/211:35:062
141999121,2cyclictest225-21systemd-journal08:05:001
141999121,2cyclictest0-21swapper/107:50:161
141699129,2cyclictest5995-21sed07:15:260
141699129,2cyclictest4284-21head09:15:170
141699129,2cyclictest3793-21awk11:50:010
141699129,2cyclictest26993-21wc09:10:010
141699128,2cyclictest8813-21apt07:24:590
141699124,6cyclictest21310-21grep07:50:180
141699124,4cyclictest225-21systemd-journal08:15:010
1416991212,0cyclictest0-21swapper/009:34:280
1416991210,2cyclictest13-21ksoftirqd/009:45:160
142599118,2cyclictest6670-21sh11:50:443
142599118,2cyclictest6193-21sshd07:17:253
142599118,2cyclictest4800-21grep09:35:143
142599118,2cyclictest4328-21ssh10:32:143
142599118,2cyclictest3556-21ssh09:15:143
142599118,2cyclictest30077-21ls08:05:253
142599118,2cyclictest28147-21ssh11:45:043
142599118,2cyclictest27138-21ssh09:10:033
142599118,2cyclictest26908-21ssh10:45:223
142599118,2cyclictest23042-21chrt07:52:083
142599118,2cyclictest22132-21ssh11:21:233
142599118,2cyclictest11327-21cat08:35:183
142599118,2cyclictest11327-21cat08:35:183
142599117,3cyclictest20215-21tr08:55:163
142599114,3cyclictest225-21systemd-journal11:25:263
142599114,3cyclictest225-21systemd-journal09:05:013
142599110,10cyclictest7431-21grep07:20:173
142599110,10cyclictest17028-21sh10:01:013
142299119,1cyclictest32-21ksoftirqd/209:35:452
142299119,1cyclictest17429-21grep07:40:202
142299118,2cyclictest32536-21ssh12:25:322
142299118,2cyclictest31519-21grep08:10:182
142299118,2cyclictest27666-21grep11:05:222
142299118,2cyclictest24443-21ssh11:42:222
142299118,2cyclictest22387-21tr09:00:142
142299118,2cyclictest21676-21sed08:55:282
142299118,2cyclictest19670-21expr11:20:162
142299118,2cyclictest15473-21tr08:45:142
142299118,2cyclictest14330-21cat08:40:262
142299118,2cyclictest10777-21cat08:35:162
142299118,2cyclictest10777-21cat08:35:152
142299115,2cyclictest225-21systemd-journal08:05:002
142299111,4cyclictest0-21swapper/212:01:332
141999116,4cyclictest32069-21awk08:10:231
141699118,2cyclictest7844-21ssh11:32:220
141699118,2cyclictest4758-21ssh11:10:510
141699118,2cyclictest4663-21ssh09:35:130
141699118,2cyclictest31260-21grep12:05:250
141699118,2cyclictest29127-21ssh11:25:260
141699118,2cyclictest28772-21tr08:05:140
141699118,2cyclictest26313-21ssh11:43:490
141699118,2cyclictest23599-21cat10:05:190
141699118,2cyclictest22362-21cut09:00:140
141699118,2cyclictest19904-21sed11:00:250
141699118,2cyclictest19561-21ssh12:18:230
141699118,2cyclictest19353-21taskset08:50:470
141699118,2cyclictest16300-21grep07:40:120
141699118,2cyclictest15609-21ssh10:20:110
141699118,2cyclictest15442-21grep08:45:150
141699118,2cyclictest14616-21cut10:00:140
141699118,2cyclictest1140-21grep08:15:150
141699112,3cyclictest8711-21grep10:35:160
142599108,1cyclictest27106-21grep08:00:183
142599108,1cyclictest25385-21grep09:05:173
142599108,1cyclictest17037-21taskset08:46:333
142599108,1cyclictest14537-21grep08:40:273
142599107,2cyclictest6401-21grep08:25:173
142599107,2cyclictest19447-21sort07:45:173
142599100,9cyclictest31778-21cat08:10:203
142299108,1cyclictest1240-21cat08:15:152
142299108,1cyclictest12360-21irqstats07:30:182
142299107,2cyclictest4422-21grep08:20:202
142299107,2cyclictest26913-21grep09:10:012
142299107,2cyclictest20062-21ssh09:25:082
142299107,2cyclictest10826-21taskset07:25:392
141699108,1cyclictest14651-21ssh12:35:110
141699108,1cyclictest137802sleep007:35:070
141699108,1cyclictest10064-21ssh10:55:170
141699107,2cyclictest9192-21aten2_r7power_v07:25:140
141699107,2cyclictest21910-21chrt08:58:140
141699105,3cyclictest5763-21grep08:25:150
14229997,1cyclictest19287-21df07:45:152
14169996,2cyclictest3389-21wget08:20:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional