You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 23:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Tue Dec 02, 2025 12:44:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
213671211964,51sleep30-21swapper/307:07:253
21384028857,10sleep10-21swapper/107:09:371
21361528053,6sleep00-21swapper/007:06:430
21303727963,9sleep20-21swapper/207:05:242
5657982430,3sleep0221rcuc/011:35:190
213979993124,2cyclictest647035-21turbostat12:30:013
21397999269,8cyclictest606145-21taskset12:00:353
21397999269,8cyclictest606145-21taskset12:00:353
213979992623,1cyclictest0-21swapper/312:00:003
213979992523,1cyclictest0-21swapper/311:36:013
21397999251,9cyclictest393571-21taskset09:40:403
4730492241,17sleep1474739-21cut10:35:181
21397999243,16cyclictest5137802sleep311:00:233
21397999243,16cyclictest5137802sleep311:00:233
213979992422,1cyclictest0-21swapper/310:25:233
213979992422,1cyclictest0-21swapper/307:39:573
21397999239,7cyclictest380055-21taskset09:32:353
213979992321,1cyclictest0-21swapper/312:16:443
213979992320,1cyclictest0-21swapper/310:55:073
213979992320,1cyclictest0-21swapper/307:55:003
21397999231,18cyclictest410457-21taskset09:54:553
213979992220,1cyclictest0-21swapper/311:24:523
213979992219,2cyclictest0-21swapper/310:10:243
213979992219,2cyclictest0-21swapper/310:10:233
213979992219,2cyclictest0-21swapper/309:29:593
213979992219,1cyclictest0-21swapper/310:18:033
213979992218,2cyclictest0-21swapper/309:20:003
213979992218,2cyclictest0-21swapper/309:00:003
213979992213,5cyclictest4488322sleep310:20:063
213979992212,1cyclictest541ktimers/310:35:293
213979992210,1cyclictest0-21swapper/309:05:203
6452922211,8sleep1344350-21diskmemload12:27:231
6368672210,12sleep00-21swapper/012:21:380
21397999213,14cyclictest592061-21chrt11:52:503
21397999212,12cyclictest537015-21taskset11:15:293
21397999212,12cyclictest498547-21taskset10:50:203
213979992118,1cyclictest0-21swapper/309:35:223
213979992118,1cyclictest0-21swapper/309:20:463
213979992118,1cyclictest0-21swapper/308:40:003
213979992117,3cyclictest0-21swapper/312:08:483
213979992117,2cyclictest0-21swapper/312:12:233
213979992117,2cyclictest0-21swapper/310:40:243
213979992116,2cyclictest0-21swapper/312:20:143
21397999211,17cyclictest0-21swapper/311:26:323
21397999209,7cyclictest247979-21taskset07:40:073
21397999205,10cyclictest468737-21chrt10:30:353
213979992018,1cyclictest0-21swapper/308:48:203
213979992018,1cyclictest0-21swapper/308:02:393
213979992017,2cyclictest0-21swapper/309:56:483
213979992016,2cyclictest0-21swapper/311:10:233
213979992016,2cyclictest0-21swapper/310:46:203
213979992016,2cyclictest0-21swapper/309:10:143
213979992015,2cyclictest0-21swapper/310:01:393
4464372191,2sleep10-21swapper/110:16:021
21397999192,12cyclictest561105-21taskset11:32:413
21397999192,11cyclictest660285-21taskset12:35:353
213979991917,1cyclictest0-21swapper/312:31:073
213979991917,1cyclictest0-21swapper/311:49:303
213979991917,1cyclictest0-21swapper/311:06:033
213979991917,1cyclictest0-21swapper/308:40:183
213979991916,2cyclictest0-21swapper/310:05:223
213979991916,2cyclictest0-21swapper/310:05:213
213979991916,1cyclictest0-21swapper/311:40:063
213979991915,2cyclictest0-21swapper/307:30:213
21397999191,14cyclictest0-21swapper/309:00:133
21397999191,12cyclictest400222-21taskset09:45:203
5309882181,11sleep2344350-21diskmemload11:13:282
5021802181,12sleep1502296-21grep10:55:151
213979991816,1cyclictest0-21swapper/307:20:133
213979991816,1cyclictest0-21swapper/307:20:123
213979991815,1cyclictest0-21swapper/307:55:243
213979991814,2cyclictest0-21swapper/308:25:213
213979991814,2cyclictest0-21swapper/308:14:453
21397999180,5cyclictest258896-21turbostat07:50:013
6388822171,11sleep21-21systemd12:25:012
5772482171,4sleep1344350-21diskmemload11:44:141
5463892171,8sleep00-21swapper/011:23:400
5069092171,10sleep00-21swapper/010:55:510
4306312171,4sleep0344350-21diskmemload10:05:280
4306312171,4sleep0344350-21diskmemload10:05:280
4159972171,3sleep20-21swapper/209:55:552
3572492171,10sleep1344350-21diskmemload09:16:141
3569212171,4sleep2344350-21diskmemload09:15:382
21397999172,9cyclictest295821-21taskset08:24:293
21397999172,9cyclictest295821-21taskset08:24:283
21397999172,10cyclictest290539-21taskset08:19:563
213979991715,1cyclictest0-21swapper/308:34:033
213979991714,1cyclictest0-21swapper/308:06:433
213979991714,1cyclictest0-21swapper/307:13:533
21397999171,14cyclictest0-21swapper/307:19:593
21397999171,12cyclictest0-21swapper/308:50:133
5873022161,10sleep0588969-21tr11:50:180
5823622161,10sleep0582413-21ntpq11:45:220
5809632162,8sleep1582173-21memory11:45:191
5465902161,3sleep10-21swapper/111:24:051
5387412161,4sleep2344350-21diskmemload11:18:382
5317862161,3sleep1344350-21diskmemload11:14:551
5174122161,10sleep2517625-21diskstats11:05:152
4084352161,10sleep0344350-21diskmemload09:51:160
3507402161,3sleep10-21swapper/109:13:001
3013972161,5sleep0301416-21apt-get08:30:110
21397999160,12cyclictest236155-21taskset07:25:213
5912792151,9sleep2591288-21ssh11:51:242
5366682151,9sleep0536870-21/usr/sbin/munin11:15:270
5178082151,9sleep0517926-21irqstats11:05:170
5158662157,1sleep0171rcu_preempt11:03:050
5158662157,1sleep0171rcu_preempt11:03:040
4862112151,3sleep0344350-21diskmemload10:44:280
4833972151,9sleep2483454-21ntpq10:40:222
4779012150,2sleep20-21swapper/210:38:482
4540112150,2sleep20-21swapper/210:21:012
4242732151,3sleep2344350-21diskmemload10:02:012
3488692152,7sleep0348926-21ntpq09:10:230
2479822150,8sleep1247608-31sort07:40:101
6606222140,9sleep0660637-21ssh12:36:080
4780532141,7sleep00-21swapper/010:39:020
4564612142,6sleep2456483-21apt-get10:25:132
4564492141,7sleep0344350-21diskmemload10:25:110
4226862141,6sleep1422739-21ntpq10:00:201
4031902141,8sleep2344350-21diskmemload09:50:102
4017762141,3sleep1401784-21ssh09:48:041
3814212141,8sleep1381627-21cat09:35:141
3324812141,8sleep2332529-21sshd08:55:232
6125022131,6sleep0612651-21ntpq12:05:210
6076902132,2sleep2171rcu_preempt12:04:002
6076902132,2sleep2171rcu_preempt12:03:592
5446672131,8sleep2544677-21ssh11:20:302
5156642131,8sleep2344350-21diskmemload11:02:422
5156642131,8sleep2344350-21diskmemload11:02:412
4879952131,8sleep1489658-21cut10:45:171
4624042131,4sleep1462412-21ssh10:27:451
3928202131,7sleep2392877-21ntpq09:40:202
3875242131,6sleep20-21swapper/209:38:072
3737952131,2sleep1171rcu_preempt09:29:131
2961452131,8sleep2296339-21expr08:25:122
2702632131,8sleep2271495-21cat08:00:152
6219862121,5sleep00-21swapper/012:12:560
6217172121,7sleep1621727-21ssh12:12:211
6071602121,7sleep1607166-21ssh12:02:511
6071602121,7sleep1607166-21ssh12:02:501
5533352121,5sleep2344350-21diskmemload11:27:282
5385022121,4sleep1344350-21diskmemload11:18:131
5241252121,3sleep10-21swapper/111:09:311
4995672121,3sleep10-21swapper/110:51:071
4924132121,4sleep0344350-21diskmemload10:46:480
4342932121,7sleep0434384-21cut10:10:170
4342932121,7sleep0434384-21cut10:10:170
4262032120,8sleep2426250-21grep10:05:142
4262032120,8sleep2426250-21grep10:05:132
3949272121,4sleep1344350-21diskmemload09:43:121
3724912121,5sleep2344350-21diskmemload09:26:402
3602122121,7sleep2360788-21kthreadcore09:20:162
3502522121,4sleep2344350-21diskmemload09:11:462
3396172121,6sleep0339720-21perl09:05:150
3126962121,7sleep1314045-21kthreadcore08:40:181
2684032121,5sleep1268444-21ntpq07:55:261
2644302121,6sleep2264685-21df07:55:192
2479832122,5sleep2248002-21apt-get07:40:112
6626832111,3sleep20-21swapper/212:39:242
6619312111,3sleep10-21swapper/112:38:181
6532662111,4sleep1344350-21diskmemload12:32:091
6528922111,3sleep20-21swapper/212:31:342
6521662110,1sleep00-21swapper/012:30:280
6468642111,4sleep0344350-21diskmemload12:29:470
6440882111,3sleep20-21swapper/212:25:292
6365512111,4sleep1344350-21diskmemload12:21:111
6275832111,4sleep2627642-21ntpq12:15:222
6275282111,5sleep1627581-21ntpq12:15:211
6233692111,4sleep0344350-21diskmemload12:15:120
6217572110,2sleep20-21swapper/212:12:262
6129742111,4sleep1344350-21diskmemload12:05:261
6062402111,5sleep00-21swapper/012:00:490
6062402111,5sleep00-21swapper/012:00:480
5995672111,3sleep10-21swapper/111:57:421
5910652111,4sleep1344350-21diskmemload11:51:031
5854662111,3sleep20-21swapper/211:49:492
5763262111,3sleep00-21swapper/011:42:240
5675412110,7sleep1567543-21ssh11:35:331
5619862111,4sleep0344350-21diskmemload11:34:180
5612912111,4sleep2344350-21diskmemload11:33:042
5554912112,4sleep1555647-21/usr/sbin/munin11:30:151
5157352111,4sleep1344350-21diskmemload11:02:501
5157352111,4sleep1344350-21diskmemload11:02:491
5069222111,3sleep20-21swapper/210:55:522
5009562111,4sleep2344350-21diskmemload10:53:572
4922402111,3sleep20-21swapper/210:46:302
4702252111,3sleep10-21swapper/110:33:311
4557872111,4sleep0344350-21diskmemload10:24:240
4405412110,2sleep10-21swapper/110:14:111
4405412110,2sleep10-21swapper/110:14:101
4394912111,4sleep2344350-21diskmemload10:12:152
4394912111,4sleep2344350-21diskmemload10:12:142
4325832111,3sleep10-21swapper/110:08:291
4325832111,3sleep10-21swapper/110:08:281
4238982111,3sleep00-21swapper/010:01:190
4171372111,4sleep1344350-21diskmemload09:58:031
4153802111,3sleep00-21swapper/009:55:260
4073352110,3sleep10-21swapper/109:50:211
4016192111,4sleep2344350-21diskmemload09:47:452
3959182111,3sleep00-21swapper/009:45:030
3882182111,3sleep00-21swapper/009:39:220
3806392111,4sleep0344350-21diskmemload09:33:360
3785462111,5sleep1378596-21ntpq09:30:211
3433212111,5sleep2343369-21ntpq09:05:202
3054572111,5sleep1305507-21ntpq08:30:211
2966312111,6sleep1296786-21kthreadcore08:25:151
2898162111,6sleep2289820-21mailstats08:15:232
2841802111,5sleep1284228-21ntpq08:10:201
2629952111,5sleep1263119-21ntpq07:50:161
2476992111,4sleep2247651-31find07:40:002
2367272110,4sleep20-21swapper/207:25:292
6134232100,5sleep2344350-21diskmemload12:05:352
5986682101,3sleep0344350-21diskmemload11:56:040
5977082101,5sleep2597762-21ssh11:55:222
5753352101,4sleep2344350-21diskmemload11:40:432
5676142101,2sleep20-21swapper/211:35:422
5533602101,3sleep0344350-21diskmemload11:27:300
5522852101,3sleep10-21swapper/111:25:311
5303672101,3sleep0344350-21diskmemload11:12:100
5005482101,3sleep0344350-21diskmemload10:53:060
4857992101,3sleep10-21swapper/110:43:371
4643272101,4sleep2464443-21irqstats10:30:152
4529492102,5sleep10-21swapper/110:20:201
4475182101,3sleep2344350-21diskmemload10:18:362
4420882101,4sleep0443611-21cut10:15:170
3931832101,5sleep0393315-21sed09:40:250
3806182101,3sleep2344350-21diskmemload09:33:352
3715172101,4sleep0371755-21unixbench_multi09:25:270
3596252101,4sleep1344350-21diskmemload09:20:111
3595762101,3sleep0344350-21diskmemload09:20:050
3590652101,3sleep00-21swapper/009:19:380
3216592101,5sleep1321713-21ntpq08:45:241
3107222101,5sleep1310769-21ntpq08:35:221
3106742101,4sleep0310720-21ntpq08:35:210
2908272101,4sleep10-21swapper/108:20:111
2908272101,4sleep10-21swapper/108:20:101
2737982100,1sleep00-21swapper/008:00:210
2620402101,5sleep0262994-21cat07:50:160
2578452101,4sleep1257891-21ntpq07:45:201
2376302101,4sleep1237756-21perl07:30:161
2323672101,4sleep0232577-21hddtemp_smartct07:25:140
2272782101,4sleep2227349-21irqstats07:20:162
2272782101,4sleep2227349-21irqstats07:20:152
2258792101,4sleep1226123-21users07:15:251
469544291,2sleep00-21swapper/010:32:100
344337291,4sleep1344496-21sed09:10:011
323464291,4sleep2324975-21tr08:50:182
285014290,1sleep20-21swapper/208:12:212
279247291,4sleep2279322-21/usr/sbin/munin08:05:292
274968291,4sleep1275050-21expr08:05:161
259125291,3sleep20-21swapper/207:50:092
247773290,4sleep0247651-31find07:40:020
237371291,3sleep0237629-21df_inode07:30:160
223252291,4sleep0225069-21grep07:15:170
214780294,2sleep00-21swapper/007:10:150
322863281,2sleep1322882-21apt-get08:50:121
314391280,1sleep20-21swapper/208:40:202
295862281,3sleep2295909-21systemd-run08:25:012
295862281,3sleep2295909-21systemd-run08:25:002
258000280,1sleep00-21swapper/007:45:220
254377281,3sleep2255877-21kthreadcore07:45:172
246492280,4sleep10-21swapper/107:35:221
236045280,4sleep1236101-21ld07:25:201
228623281,3sleep00-21swapper/007:20:180
228623281,3sleep00-21swapper/007:20:170
334431271,2sleep00-21swapper/009:00:170
328753271,2sleep1328966-21smartctl08:55:181
312499271,2sleep00-21swapper/008:40:170
264734271,2sleep00-21swapper/007:55:210
247402270,1sleep00-21swapper/007:37:320
237347271,3sleep2219332-21munin-node07:30:132
226184270,1sleep20-21swapper/207:16:062
218520271,2sleep10-21swapper/107:10:251
338760260,4sleep10-21swapper/109:04:301
337922261,2sleep20-21swapper/209:00:242
323153260,3sleep00-21swapper/008:50:150
321171260,2sleep2321553-21strings08:45:212
318105261,2sleep00-21swapper/008:45:190
311425260,1sleep20-21swapper/208:36:302
295823260,1sleep00-21swapper/008:24:310
295823260,1sleep00-21swapper/008:24:300
289381260,2sleep10-21swapper/108:15:181
285225261,2sleep00-21swapper/008:15:010
231650260,1sleep10-21swapper/107:23:001
231650260,1sleep10-21swapper/107:22:591
214463261,2sleep20-21swapper/207:10:142
332819250,3sleep00-21swapper/008:55:290
306244250,3sleep20-21swapper/208:32:382
280185251,1sleep00-21swapper/008:10:070
274946250,1sleep00-21swapper/008:05:150
270191250,2sleep10-21swapper/108:00:141
296418240,1sleep00-21swapper/008:25:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional