You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 10:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sun Jul 13, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
317699722320,2sleep00-21swapper/022:55:210
28867092148102,15sleep20-21swapper/219:09:232
28864812148104,19sleep30-21swapper/319:06:313
28864052144121,15sleep00-21swapper/019:05:310
28865622132109,15sleep10-21swapper/119:07:301
2886885993826,9cyclictest3276271-21sh00:00:003
2886885993722,12cyclictest3095414-21grep22:05:173
2886885993525,8cyclictest2984357-21ls20:44:593
288688599328,15cyclictest3015789-21kworker/u16:3+nfsiod21:21:183
2886885993221,9cyclictest0-21swapper/323:00:003
288688599321,21cyclictest3077411-21runrttasks21:52:343
288688599320,22cyclictest3258184-21runrttasks23:46:413
2886885993127,2cyclictest0-21swapper/323:30:213
2886885993122,7cyclictest2969002-21idleruntime-cro20:30:013
2886885993121,8cyclictest1-21systemd23:20:003
2886885992920,7cyclictest2944420-21fschecks_time20:05:163
2886885992915,11cyclictest3104190-21cut22:10:163
288688599291,15cyclictest3056588-21cut21:40:173
288688599289,8cyclictest0-21swapper/323:35:593
288688599288,12cyclictest3023036-21sh21:16:533
288688599287,10cyclictest3278799-21tr00:00:163
2886885992819,7cyclictest0-21swapper/322:30:263
2886885992810,8cyclictest3045780-21runrttasks21:31:013
2886885992718,7cyclictest3009459-21diskmemload22:15:253
2886885992717,8cyclictest3181556-21cpuspeed_turbos23:00:143
2886885992717,7cyclictest0-21swapper/323:10:143
2886885992714,11cyclictest2937937-21grep19:55:233
288688599270,23cyclictest3333419-21ntpq00:35:213
288688599269,11cyclictest3048699-21df_inode21:35:143
2886885992617,7cyclictest0-21swapper/323:05:243
2886885992613,10cyclictest945-21snmpd00:11:503
288688599261,24cyclictest0-21swapper/323:50:333
288688599261,23cyclictest0-21swapper/322:40:093
288688599261,22cyclictest2913912-21awk19:35:003
288688599261,20cyclictest0-21swapper/323:40:133
2886885992611,7cyclictest0-21swapper/321:00:003
288688599260,1cyclictest0-21swapper/322:36:503
288688599258,15cyclictest0-21swapper/322:50:223
2886885992516,7cyclictest3162725-21sh22:46:103
2886885992515,8cyclictest3322489-21cpuspeed_turbos00:30:153
2886885992514,9cyclictest3084064-21runrttasks21:55:503
2886885992513,8cyclictest3036808-21tr21:25:173
288688599251,23cyclictest0-21swapper/300:25:403
288688599251,22cyclictest3009459-21diskmemload21:46:473
288688599251,22cyclictest0-21swapper/323:24:133
288688599251,21cyclictest1056-21cron00:10:003
288688599251,10cyclictest3313114-21ssh00:22:223
288688599250,24cyclictest0-21swapper/300:16:243
288688599250,23cyclictest2890801-21taskset19:10:213
288688599250,10cyclictest3009459-21diskmemload22:02:163
288688599240,7cyclictest3017020-21ssh21:14:463
288688599240,23cyclictest0-21swapper/322:27:013
288688599240,12cyclictest3123303-21sh22:21:143
288688599239,12cyclictest3002207-21tr21:00:203
2886885992319,2cyclictest0-21swapper/320:50:153
2886885992315,5cyclictest2904493-21grep19:25:153
288688599231,20cyclictest2972999-21ldconfig20:30:193
288688599231,20cyclictest0-21swapper/320:20:123
288688599230,2cyclictest0-21swapper/320:10:173
288688599230,21cyclictest0-21swapper/323:27:253
2886885992212,8cyclictest2894378-21gltestperf19:15:173
288688599221,19cyclictest2903939-21basename19:25:003
288688599221,18cyclictest3004820-21expr21:05:153
288688599221,18cyclictest2989367-21awk20:50:003
288688599218,11cyclictest2940933-21tr20:00:193
288688599217,10cyclictest2927318-21tr19:45:183
288688599211,11cyclictest2914580-21if_err_enp1s019:35:173
288688599203,9cyclictest0-21swapper/320:39:443
288688599201,16cyclictest2929479-21head19:50:163
288688599201,16cyclictest0-21swapper/319:40:163
30156892190,7sleep03009459-21diskmemload21:12:360
288688599191,15cyclictest2954235-21tr20:15:153
31725602181,12sleep23172569-21ssh22:54:282
29593712181,4sleep12959580-21hddtemp_smartct20:20:151
31625002171,4sleep03009459-21diskmemload22:45:470
30294962171,3sleep20-21swapper/221:20:242
33055382160,5sleep1321rcuc/100:16:521
32768732162,9sleep23277006-21tr00:00:132
32680152161,4sleep13009459-21diskmemload23:54:511
29692572161,4sleep22969413-21diskstats20:30:142
33148422151,3sleep13314851-21sh00:25:051
30407292150,3sleep243-21ksoftirqd/221:30:002
29029682151,4sleep12903024-21ntp_states19:20:241
32504892141,3sleep20-21swapper/223:41:582
32201222141,8sleep03009459-21diskmemload23:23:240
31387132141,3sleep10-21swapper/122:31:181
30616472141,9sleep13009459-21diskmemload21:41:401
30545952141,9sleep13009459-21diskmemload21:37:591
33354222131,3sleep13335428-21ssh00:38:011
32821222131,8sleep03009459-21diskmemload00:01:520
32400202131,6sleep23240963-21memory23:35:212
32334392130,2sleep20-21swapper/223:30:242
32215912131,5sleep23009459-21diskmemload23:25:152
32127822131,2sleep23212785-21ssh23:19:492
32040752130,6sleep10-21swapper/123:13:431
31802312131,8sleep13180241-21ssh22:59:151
31549602131,2sleep00-21swapper/022:41:190
30408782131,8sleep03040892-21apt-get21:30:120
31332732121,7sleep03133366-21sed22:30:010
31238792120,1sleep00-21swapper/022:22:150
31007432121,5sleep10-21swapper/122:07:291
29585452121,7sleep22958546-21sshd20:15:312
33143932111,3sleep10-21swapper/100:24:461
33068942110,1sleep20-21swapper/200:19:222
32978862111,4sleep13009459-21diskmemload00:12:161
32898502111,3sleep00-21swapper/000:06:500
32665132111,3sleep20-21swapper/223:52:362
32367432111,7sleep13009459-21diskmemload23:34:521
32266362110,7sleep03226645-21ssh23:26:170
32103972111,6sleep03210403-21ssh23:16:150
32035472111,4sleep03009459-21diskmemload23:12:540
31634032111,3sleep10-21swapper/122:47:141
31474822111,3sleep00-21swapper/022:37:050
31470232111,7sleep13147035-21ssh22:36:271
31177912111,6sleep23117801-21ssh22:19:572
30766962111,4sleep03009459-21diskmemload21:51:240
30695302111,3sleep00-21swapper/021:47:120
30556942111,2sleep20-21swapper/221:39:402
30323172111,4sleep03009459-21diskmemload21:24:030
30294272110,7sleep13029493-21ntpq21:20:231
29746152110,7sleep12974544-21/usr/sbin/munin20:35:141
29646812111,6sleep22965830-21kthreadcore20:25:172
29443592111,5sleep12944569-21hddtemp_smartct20:05:161
29396412114,3sleep02940924-21cut20:00:180
33288952101,3sleep20-21swapper/200:34:332
33276702101,3sleep10-21swapper/100:32:041
33166802101,5sleep03318477-21cut00:25:170
33054142101,4sleep03009459-21diskmemload00:16:390
32959752101,4sleep23296053-21ntpq00:10:202
32884152101,4sleep13009459-21diskmemload00:05:261
32818162101,3sleep10-21swapper/100:01:211
32744292101,3sleep03009459-21diskmemload23:57:070
32743132101,3sleep10-21swapper/123:56:561
32596322101,3sleep10-21swapper/123:49:031
32580092101,2sleep20-21swapper/223:46:272
32449512101,4sleep13009459-21diskmemload23:40:041
32352682100,1sleep00-21swapper/023:32:220
32273922101,2sleep10-21swapper/123:27:261
32184482100,2sleep20-21swapper/223:20:542
32022072101,3sleep20-21swapper/223:10:522
31946412100,3sleep00-21swapper/023:06:290
31866472101,6sleep13009459-21diskmemload23:01:291
31725182101,3sleep00-21swapper/022:54:220
31709982101,3sleep13009459-21diskmemload22:51:551
31559932101,3sleep10-21swapper/122:42:531
31466772101,3sleep23009459-21diskmemload22:35:562
31336832101,3sleep20-21swapper/222:30:152
31309512100,3sleep20-21swapper/222:26:182
31259822101,4sleep13126004-21apt-get22:25:171
31175482101,2sleep00-21swapper/022:19:350
31108942100,6sleep13111597-21cut22:15:191
31092692101,3sleep10-21swapper/122:13:451
31086042101,2sleep03009459-21diskmemload22:12:450
31014912101,2sleep00-21swapper/022:08:440
31004752101,2sleep20-21swapper/222:07:022
30928332101,6sleep13009459-21diskmemload22:02:201
30840142101,3sleep10-21swapper/121:55:471
30755632101,4sleep13075623-21ntp_states21:50:251
30700772101,2sleep20-21swapper/221:48:062
30532632101,3sleep03009459-21diskmemload21:35:410
30460952101,2sleep10-21swapper/121:31:371
30243862100,6sleep23024392-21ssh21:18:562
30168092100,3sleep20-21swapper/221:14:272
30148812101,2sleep10-21swapper/121:11:171
29896072101,4sleep12989625-21apt-get20:50:131
29488612101,5sleep02948883-21apt20:10:000
29388922101,4sleep22938964-21munin-run20:00:002
29244932101,4sleep12924608-21irqstats19:45:171
29241502101,4sleep22924169-21apt-get19:45:142
28870612101,4sleep22887080-21apt-get19:10:152
3336525291,4sleep03336558-21awk00:40:010
3329427291,2sleep20-21swapper/200:35:092
3327400290,2sleep00-21swapper/000:31:310
3320981291,3sleep20-21swapper/200:28:442
3297253291,3sleep03009459-21diskmemload00:11:140
3289171290,3sleep20-21swapper/200:05:432
3274193291,2sleep20-21swapper/223:56:452
3267909291,2sleep00-21swapper/023:54:430
3250150290,2sleep00-21swapper/023:41:240
3244149291,3sleep10-21swapper/123:39:091
3242214291,3sleep03009459-21diskmemload23:36:070
3195709291,2sleep20-21swapper/223:08:022
3189371291,4sleep13189398-21ssh23:05:131
3186790291,2sleep20-21swapper/223:01:422
3186769291,3sleep00-21swapper/023:01:400
3175005291,4sleep23176732-21kthreadcore22:55:202
3157675291,3sleep20-21swapper/222:45:062
3155160291,5sleep23092736-21kworker/u16:0+rpciod22:41:372
3140275290,2sleep00-21swapper/022:33:460
3123590290,3sleep20-21swapper/222:21:452
3123226290,5sleep13009459-21diskmemload22:21:081
3109631291,3sleep20-21swapper/222:14:232
3092150290,6sleep20-21swapper/222:01:132
3091865291,3sleep03009459-21diskmemload22:00:460
3084816291,3sleep03009459-21diskmemload21:57:020
3084028291,2sleep20-21swapper/221:55:482
3076319290,3sleep20-21swapper/221:50:462
3064464291,5sleep13065131-21kthreadcore21:45:161
3063341290,3sleep03009459-21diskmemload21:44:310
3056277290,2sleep20-21swapper/221:40:082
3045780291,2sleep20-21swapper/221:31:022
3039648291,1sleep1191rcu_preempt21:28:291
3022229291,2sleep10-21swapper/121:15:411
3004870290,2sleep1191rcu_preempt21:05:151
2979616290,3sleep02979635-21apt-get20:40:140
2954134290,3sleep10-21swapper/120:15:141
2952838291,3sleep22952878-21ntpq20:10:222
2929144291,3sleep12929163-21apt-get19:50:141
2924328291,4sleep02924492-21/usr/sbin/munin19:45:160
2917933291,3sleep22917986-21ntp_states19:35:232
2914601291,5sleep10-21swapper/119:35:191
2903292291,4sleep22903537-21users19:20:292
2894041291,4sleep22894190-21df_inode19:15:162
3313016280,2sleep20-21swapper/200:22:112
3312419280,2sleep00-21swapper/000:21:010
3257745280,2sleep00-21swapper/023:46:010
3213606281,3sleep13213622-21ssh23:20:131
3209855280,3sleep13009459-21diskmemload23:15:281
3038062281,3sleep03038072-21ssh21:25:510
3021407281,3sleep03021466-21ntpq21:15:210
2998325281,3sleep02998403-21ntpq20:55:200
2989994281,3sleep02990064-21irqstats20:50:160
2989261280,1sleep20-21swapper/220:48:582
2974724281,3sleep22974937-21hddtemp_smartct20:35:152
2969628281,3sleep02969695-21irqstats20:30:160
2959214281,3sleep02959370-21diskstats20:20:140
2957791281,4sleep02957889-21ntpq20:15:210
2943327281,3sleep22943544-21users20:00:292
2920739281,4sleep22922539-21cut19:40:212
2917832281,3sleep02917883-21ntpq19:35:220
2909187281,3sleep00-21swapper/019:30:130
2894019280,2sleep10-21swapper/119:15:151
2891229281,3sleep02891474-21users19:10:290
3004919271,3sleep23005127-21if_enp1s021:05:172
2999695270,2sleep10-21swapper/121:00:151
2994640270,2sleep10-21swapper/120:55:141
2989344271,3sleep02989360-21grep20:49:590
2981133271,3sleep10-21swapper/120:40:191
2975016271,3sleep02976452-21cut20:35:170
2964400270,3sleep02964578-21sed20:25:150
2934162270,2sleep10-21swapper/119:55:151
2929885270,3sleep22931560-21kthreadcore19:50:192
2899110270,3sleep00-21swapper/019:20:150
3004109260,1sleep00-21swapper/021:00:500
3000848261,2sleep20-21swapper/221:00:202
2995806261,2sleep20-21swapper/220:55:192
2973903260,1sleep10-21swapper/120:33:421
2962916260,3sleep22962962-21ntp_kernel_err20:20:192
2948783260,4sleep20-21swapper/220:09:022
2937814261,2sleep00-21swapper/019:55:220
2933660260,1sleep00-21swapper/019:52:010
2919513260,1sleep00-21swapper/019:40:190
2919158260,1sleep10-21swapper/119:40:161
2913688260,1sleep20-21swapper/219:32:392
2910584261,2sleep12912384-21kthreadcore19:30:181
2904662261,2sleep10-21swapper/119:25:161
2891971260,1sleep00-21swapper/019:15:010
3009250250,3sleep00-21swapper/021:07:310
2991187250,2sleep22974155-21kworker/2:0-events20:50:182
2989182250,1sleep10-21swapper/120:47:571
2982924250,1sleep20-21swapper/220:40:202
2964582250,1sleep10-21swapper/120:25:161
2939202250,1sleep10-21swapper/120:00:161
2907986250,1sleep20-21swapper/219:25:212
2887450250,1sleep10-21swapper/119:10:181
2949210240,1sleep00-21swapper/020:10:160
2948889240,1sleep10-21swapper/120:10:001
2904168240,1sleep00-21swapper/019:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional