You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 17:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Nov 25, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1135383211955,59sleep30-21swapper/307:05:323
113568628868,9sleep10-21swapper/107:09:271
113555428554,9sleep20-21swapper/207:07:432
113560228158,6sleep00-21swapper/007:08:220
1135865992824,2cyclictest0-21swapper/310:21:193
1135865992722,3cyclictest0-21swapper/311:10:173
1135865992623,1cyclictest0-21swapper/311:40:223
1135865992522,1cyclictest0-21swapper/309:15:153
1135865992521,2cyclictest0-21swapper/309:25:003
1135865992512,2cyclictest541ktimers/309:30:203
1135865992421,1cyclictest0-21swapper/311:09:593
1135865992420,2cyclictest0-21swapper/311:30:003
1135865992420,2cyclictest0-21swapper/311:15:123
1135865992420,2cyclictest0-21swapper/311:02:413
1135865992420,2cyclictest0-21swapper/310:40:003
113586599241,19cyclictest1563632-21taskset12:22:503
113586599241,19cyclictest1417186-21taskset10:49:583
1135865992319,2cyclictest0-21swapper/312:29:273
1135865992319,2cyclictest0-21swapper/310:50:133
1135865992319,2cyclictest0-21swapper/310:29:153
1135865992319,1cyclictest0-21swapper/310:55:143
1135865992318,2cyclictest0-21swapper/310:44:563
1135865992315,4cyclictest0-21swapper/309:46:383
15880702221,18sleep11265695-21diskmemload12:39:041
113586599229,10cyclictest0-21swapper/310:00:213
113586599228,10cyclictest1361936-21chrt10:12:413
1135865992219,1cyclictest0-21swapper/309:10:163
1135865992219,1cyclictest0-21swapper/307:39:573
1135865992218,2cyclictest0-21swapper/310:30:163
1135865992218,2cyclictest0-21swapper/310:17:193
113586599221,15cyclictest1339698-21taskset09:58:583
15792752210,1sleep00-21swapper/012:32:580
14688102211,14sleep01265695-21diskmemload11:21:500
113586599219,1cyclictest0-21swapper/309:42:143
1135865992118,1cyclictest0-21swapper/308:00:013
1135865992117,2cyclictest0-21swapper/309:51:413
1135865992117,2cyclictest0-21swapper/307:40:043
1135865992115,3cyclictest0-21swapper/312:15:153
1135865992111,7cyclictest1507921-21taskset11:48:063
15406792201,10sleep01265695-21diskmemload12:09:490
113586599209,6cyclictest1217826-21taskset08:20:503
113586599207,12cyclictest541ktimers/312:34:083
113586599205,12cyclictest0-21swapper/309:37:583
1135865992018,1cyclictest0-21swapper/308:05:143
1135865992017,2cyclictest0-21swapper/311:58:323
1135865992017,1cyclictest0-21swapper/307:50:003
1135865992016,2cyclictest0-21swapper/312:35:223
1135865992016,2cyclictest0-21swapper/312:10:223
1135865992016,2cyclictest0-21swapper/311:50:033
1135865992016,2cyclictest0-21swapper/311:34:333
1135865992016,2cyclictest0-21swapper/309:29:443
1135865992016,2cyclictest0-21swapper/308:45:003
1135865992013,5cyclictest0-21swapper/309:10:003
113586599201,15cyclictest1153338-21taskset07:20:263
113586599201,14cyclictest1468936-21chrt11:22:053
13176952191,14sleep11317707-21ssh09:44:501
1135865991916,2cyclictest0-21swapper/312:01:393
1135865991916,1cyclictest0-21swapper/308:45:263
1135865991915,2cyclictest1196921-21turbostat08:05:003
113586599191,14cyclictest1353258-21chrt10:05:283
12855712181,3sleep00-21swapper/009:21:160
113586599187,8cyclictest1249932-21turbostat08:55:003
113586599182,12cyclictest1539434-21taskset12:07:553
1135865991815,2cyclictest0-21swapper/307:15:023
1135865991813,3cyclictest66550irq/131-enp1s0-TxRx-311:35:183
15533802172,8sleep01553443-21ntp_states12:15:220
15453652171,4sleep11265695-21diskmemload12:10:231
15230552171,2sleep20-21swapper/211:57:202
14999292171,4sleep21265695-21diskmemload11:42:212
14152142171,9sleep11265695-21diskmemload10:46:201
13947222171,11sleep11394935-21diskstats10:35:131
13080632171,4sleep21265695-21diskmemload09:35:292
13006272171,4sleep0151ktimers/009:31:140
1135865991713,2cyclictest0-21swapper/308:34:383
1135865991713,2cyclictest0-21swapper/307:53:393
1135865991713,2cyclictest0-21swapper/307:34:433
113586599170,13cyclictest0-21swapper/307:25:203
15491732161,4sleep21549197-21apt-get12:15:122
14768402160,1sleep10-21swapper/111:27:331
14378762161,3sleep00-21swapper/011:00:550
14324332161,10sleep21432517-21grep11:00:002
14084732161,4sleep11265695-21diskmemload10:42:561
13184332164,8sleep11318651-21expr09:45:141
11580072161,10sleep01158059-21/usr/sbin/munin07:25:210
113586599161,9cyclictest1212471-21taskset08:15:333
1135865991613,1cyclictest0-21swapper/308:58:363
1135865991613,1cyclictest0-21swapper/308:10:123
113586599161,10cyclictest1233069-21taskset08:35:213
113586599160,13cyclictest0-21swapper/309:00:223
113586599160,13cyclictest0-21swapper/307:10:203
15794242151,2sleep20-21swapper/212:33:132
15164212150,1sleep20-21swapper/211:54:282
14380652151,3sleep21265695-21diskmemload11:01:162
13777212150,2sleep2431ktimers/210:23:042
13225962151,10sleep01322610-21/usr/sbin/munin09:45:250
13221452151,9sleep21322242-21ntpq09:45:192
12929062151,6sleep20-21swapper/209:25:592
12782892151,9sleep01278299-21ssh09:15:590
12557092151,5sleep01255836-21grep09:00:160
12184882151,3sleep00-21swapper/008:25:140
11698872151,8sleep072-21kcompactd007:40:040
113586599151,9cyclictest1218460-21taskset08:25:073
15018562141,8sleep11502115-21expr11:45:131
14624802141,4sleep21462486-21ssh11:18:522
14551652140,2sleep00-21swapper/011:14:450
14253782141,3sleep01425402-21apt-get10:55:130
13798832141,9sleep11381452-21kthreadcore10:25:151
13759832141,4sleep11376185-21ssh10:20:261
13641392141,8sleep11364242-21ps10:15:171
13565352141,8sleep11357520-21grep10:10:161
13483162141,6sleep00-21swapper/010:05:020
13105782141,3sleep21265695-21diskmemload09:40:022
13011052141,2sleep20-21swapper/209:32:132
12873692141,8sleep11265695-21diskmemload09:24:461
11489242141,10sleep10-21swapper/107:20:131
11460182140,2sleep10-21swapper/107:15:201
11365772141,8sleep11137186-21grep07:10:161
15711672131,8sleep01571175-21ssh12:27:410
15299172131,6sleep21265695-21diskmemload12:00:462
15291102131,8sleep01524905-21/usr/sbin/munin12:00:230
15187502131,7sleep01519272-21kthreadcore11:55:180
14910862131,6sleep01265695-21diskmemload11:35:330
14861252131,7sleep21486133-21ssh11:35:032
14782522131,8sleep01478349-21date11:30:000
14473982130,7sleep01265695-21diskmemload11:09:360
14089572130,2sleep01265695-21diskmemload10:43:470
14008572131,6sleep21265695-21diskmemload10:37:592
13942672131,7sleep21394277-21ssh10:34:582
13906142131,7sleep01391086-21needreboot10:30:200
13758322131,8sleep01375982-21wc10:20:260
13337272131,7sleep01335321-21kthreadcore09:55:160
13332912131,8sleep11333045-21/usr/sbin/munin09:55:131
13161252131,8sleep01316134-21ssh09:41:450
12647022131,7sleep01264748-21ntpq09:05:210
12462362131,7sleep21247993-21kthreadcore08:50:162
15394252121,3sleep20-21swapper/212:07:552
15320852121,5sleep10-21swapper/112:04:381
14853462121,5sleep11265695-21diskmemload11:34:071
14552312121,5sleep11265695-21diskmemload11:14:551
14531722121,5sleep21265695-21diskmemload11:11:062
14455492121,5sleep11265695-21diskmemload11:06:161
14378892121,4sleep11265695-21diskmemload11:00:561
14111282121,7sleep21412743-21cut10:45:172
13841822121,5sleep20-21swapper/210:25:402
13477612121,7sleep01347773-21rm10:04:590
13474272121,4sleep11265695-21diskmemload10:04:181
13390012121,4sleep21265695-21diskmemload09:57:262
13096952120,7sleep01265695-21diskmemload09:38:280
12773512121,6sleep21277407-21/usr/sbin/munin09:15:222
12719472121,7sleep01209-21sendmail-mta09:12:260
12094702121,7sleep20-21swapper/208:15:172
15855332111,5sleep01585777-21users12:35:270
15815772111,6sleep21583161-21kthreadcore12:35:172
15775932111,5sleep11577841-21users12:30:271
15723082111,4sleep11265695-21diskmemload12:29:261
15719722111,4sleep21265695-21diskmemload12:28:552
15641482111,4sleep01265695-21diskmemload12:23:400
15485422111,4sleep01265695-21diskmemload12:14:380
15383192113,2sleep1171rcu_preempt12:06:131
15161092111,3sleep00-21swapper/011:53:540
15146402111,3sleep11265695-21diskmemload11:51:111
15080482111,4sleep21265695-21diskmemload11:48:222
15068042111,4sleep01265695-21diskmemload11:46:080
15008582111,4sleep01265695-21diskmemload11:44:000
14861822111,3sleep10-21swapper/111:35:101
14852462111,7sleep20-21swapper/211:33:512
14769212111,6sleep21476929-21sh11:27:422
14687902110,1sleep20-21swapper/211:21:492
14556302111,3sleep10-21swapper/111:15:101
14403252112,4sleep21440349-21apt-get11:05:132
14295512111,4sleep11265695-21diskmemload10:55:221
14151462111,3sleep00-21swapper/010:46:110
14005732111,6sleep01265695-21diskmemload10:37:290
13842022111,4sleep01265695-21diskmemload10:25:420
13626532111,4sleep21265695-21diskmemload10:14:022
13538732111,4sleep11265695-21diskmemload10:06:391
13323782111,6sleep11332384-21ssh09:54:241
13317132110,1sleep00-21swapper/009:52:480
13305932111,4sleep21265695-21diskmemload09:50:452
12942762111,3sleep10-21swapper/109:28:391
12935662111,3sleep00-21swapper/009:27:130
12738192110,7sleep11274674-21grep09:15:161
12715392111,4sleep21265695-21diskmemload09:11:442
12383532111,4sleep21238402-21ntpq08:40:222
12118142111,6sleep11211868-21modprobe08:15:201
11868612110,3sleep21187977-21cut07:55:172
11698892111,5sleep21169566-31find07:40:072
11695812111,5sleep11169566-31find07:39:571
11490282110,1sleep20-21swapper/207:20:152
15639172101,3sleep20-21swapper/212:23:192
15490592101,3sleep10-21swapper/112:15:031
15224032101,3sleep10-21swapper/111:56:111
14996552100,1sleep10-21swapper/111:41:371
14835242101,2sleep00-21swapper/011:30:370
14683652100,4sleep11265695-21diskmemload11:21:001
14617632101,3sleep00-21swapper/011:17:340
14247402101,4sleep01424816-21/usr/sbin/munin10:55:020
14232132101,2sleep10-21swapper/110:52:101
14072732101,3sleep21265695-21diskmemload10:40:442
13919432101,2sleep10-21swapper/110:30:361
13684512101,3sleep00-21swapper/010:15:360
13642482105,2sleep20-21swapper/210:15:182
13608562102,3sleep00-21swapper/010:10:440
13531802101,3sleep20-21swapper/210:05:282
13003512101,3sleep11265695-21diskmemload09:30:471
12859062101,3sleep21265695-21diskmemload09:21:502
12726412101,5sleep10-21swapper/109:14:081
12488832101,4sleep11248929-21ntpq08:50:191
12171702101,5sleep21217216-21ntpq08:20:222
11897922102,5sleep0151ktimers/007:55:190
11794312101,4sleep21179485-21ntpq07:45:192
11640532100,4sleep10-21swapper/107:33:121
11631772101,4sleep21163218-21ntpq07:30:212
1562348290,1sleep10-21swapper/112:20:491
1547086291,3sleep21265695-21diskmemload12:12:172
1423854291,3sleep21265695-21diskmemload10:53:182
1345068291,4sleep21345320-21sed10:00:272
1303173290,2sleep10-21swapper/109:35:081
1260334293,3sleep1331ktimers/109:03:541
1233366291,4sleep01233615-21grep08:35:270
1233014291,4sleep21233061-21ntpq08:35:222
1207139291,4sleep11207150-21sshd08:10:271
1197453291,4sleep21197663-21/usr/sbin/munin08:05:152
1185059291,4sleep21185270-21grep07:50:252
1181741291,4sleep11183452-21cut07:50:151
1180391291,3sleep01180352-21fwupd07:49:260
1169614291,4sleep01169638-21ls07:40:000
1165034291,4sleep21166249-21tr07:35:172
1251399281,3sleep21253158-21kthreadcore08:55:172
1192392281,3sleep21192462-21kernelversion08:00:182
1180703281,3sleep01180722-21apt-get07:50:100
1175365281,3sleep11175384-21apt-get07:45:101
1170472281,3sleep11171425-21kthreadcore07:40:171
1147493281,3sleep21147617-21awk07:15:252
1264523271,3sleep21264648-21/usr/sbin/munin09:05:202
1260393270,1sleep20-21swapper/209:04:392
1250160271,2sleep10-21swapper/108:55:121
1248741270,2sleep00-21swapper/008:50:180
1218755270,4sleep20-21swapper/208:25:152
1218167270,1sleep00-21swapper/008:24:330
1217829270,1sleep10-21swapper/108:20:531
1207435270,1sleep20-21swapper/208:14:052
1197882271,2sleep00-21swapper/008:05:180
1191332271,3sleep10-21swapper/108:00:011
1137187271,3sleep21138884-21kthreadcore07:10:172
1243591261,1sleep20-21swapper/208:45:202
1239213260,1sleep00-21swapper/008:43:160
1227765260,2sleep20-21swapper/208:30:202
1227709260,2sleep10-21swapper/108:30:191
1207772261,2sleep00-21swapper/008:15:120
1207453260,4sleep00-21swapper/008:14:190
1202030260,1sleep10-21swapper/108:07:501
1196836260,1sleep00-21swapper/008:04:210
1192178261,1sleep10-21swapper/108:00:171
1153962260,1sleep20-21swapper/207:25:112
1144342261,2sleep00-21swapper/007:15:190
1140993260,1sleep00-21swapper/007:14:230
1265675250,1sleep10-21swapper/109:09:551
1254272250,1sleep00-21swapper/008:55:230
1243412250,1sleep10-21swapper/108:45:181
1239209250,3sleep10-21swapper/108:43:131
1229404250,1sleep10-21swapper/108:35:171
1156867250,1sleep10-21swapper/107:25:181
1149899250,2sleep00-21swapper/007:20:160
1239696240,1sleep00-21swapper/008:45:130
1224018240,1sleep00-21swapper/008:30:140
1218640240,1sleep10-21swapper/108:25:151
1159449240,1sleep00-21swapper/007:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional