You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 05:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri Dec 05, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3683493211961,54sleep30-21swapper/319:07:473
368307028659,6sleep20-21swapper/219:05:262
368365728065,9sleep10-21swapper/119:09:541
368356827951,20sleep00-21swapper/019:08:440
3683791993423,3cyclictest228-1kworker/3:1H-kblockd23:39:593
3683791992622,2cyclictest0-21swapper/321:30:183
3683791992523,1cyclictest0-21swapper/300:30:143
3683791992419,2cyclictest0-21swapper/323:05:183
368379199235,14cyclictest3878539-21sleep21:54:553
368379199235,14cyclictest3878539-21sleep21:54:543
368379199232,16cyclictest3845523-21sleep21:30:003
3683791992320,1cyclictest0-21swapper/323:25:223
3683791992317,3cyclictest0-21swapper/323:00:153
368379199231,19cyclictest4125275-21taskset00:24:143
38272132221,2sleep20-21swapper/221:16:592
368379199222,15cyclictest3987271-21taskset22:55:213
368379199222,14cyclictest4117279-21taskset00:17:403
3683791992219,1cyclictest0-21swapper/322:20:013
3683791992219,1cyclictest0-21swapper/319:44:243
3683791992219,1cyclictest0-21swapper/300:05:153
3683791992218,2cyclictest0-21swapper/322:30:203
38602102211,2sleep10-21swapper/121:37:271
368379199215,3cyclictest66550irq/131-enp1s0-TxRx-322:20:043
3683791992118,1cyclictest0-21swapper/322:00:193
3683791992117,2cyclictest0-21swapper/323:20:183
3683791992117,2cyclictest0-21swapper/300:10:123
368379199211,16cyclictest3739098-21taskset19:59:403
39098842201,6sleep03813040-21diskmemload22:08:050
368379199205,11cyclictest3959105-21turbostat22:44:543
3683791992017,1cyclictest0-21swapper/323:57:273
3683791992017,1cyclictest0-21swapper/323:40:183
3683791992017,1cyclictest0-21swapper/319:10:223
3683791992016,2cyclictest4021178-21taskset23:18:263
3683791992016,2cyclictest0-21swapper/321:40:163
3683791992016,2cyclictest0-21swapper/300:36:283
3683791992015,3cyclictest0-21swapper/322:10:163
368379199201,2cyclictest0-21swapper/300:03:363
368379199201,14cyclictest4130168-21chrt00:25:253
368379199201,14cyclictest4076469-21chrt23:51:033
368379199201,14cyclictest3887301-21chrt21:55:173
368379199201,14cyclictest3813063-21sleep21:10:003
368379199192,13cyclictest3827085-21taskset21:16:473
3683791991916,1cyclictest0-21swapper/319:18:263
3683791991915,2cyclictest0-21swapper/320:35:003
3683791991914,2cyclictest0-21swapper/323:10:303
368379199191,14cyclictest3744431-21turbostat20:05:013
368379199191,14cyclictest0-21swapper/322:25:143
368379199191,13cyclictest3818557-21chrt21:10:593
368379199191,12cyclictest4042847-21taskset23:30:263
368379199191,12cyclictest3957833-21taskset22:37:543
368379199191,12cyclictest3906143-21taskset22:05:193
39746932181,11sleep13813040-21diskmemload22:49:361
39746932181,11sleep13813040-21diskmemload22:49:351
368379199182,12cyclictest4069368-21taskset23:47:573
368379199182,12cyclictest3875985-21taskset21:46:023
368379199182,11cyclictest3858930-21taskset21:35:273
3683791991815,1cyclictest0-21swapper/319:25:123
3683791991814,2cyclictest0-21swapper/319:20:163
3683791991813,2cyclictest0-21swapper/321:20:283
3683791991811,4cyclictest39816192chrt22:52:583
368379199181,14cyclictest0-21swapper/320:49:593
368379199181,11cyclictest3744703-21taskset20:05:133
368379199181,10cyclictest3781596-21taskset20:40:003
368379199180,13cyclictest0-21swapper/320:50:153
41239862171,2sleep10-21swapper/100:21:111
41239672171,4sleep23813040-21diskmemload00:21:092
40862762172,8sleep23813040-21diskmemload23:58:522
40861752171,4sleep13813040-21diskmemload23:58:431
40606342171,5sleep23813040-21diskmemload23:42:162
40518112170,3sleep10-21swapper/123:36:061
39730192171,4sleep23813040-21diskmemload22:46:482
39730192171,4sleep23813040-21diskmemload22:46:472
39645542171,4sleep23813040-21diskmemload22:40:542
39089992171,10sleep13813040-21diskmemload22:06:461
38850322171,4sleep13813040-21diskmemload21:52:351
38850322171,4sleep13813040-21diskmemload21:52:341
38847992171,2sleep00-21swapper/021:52:130
38847992171,2sleep00-21swapper/021:52:130
38525442171,3sleep00-21swapper/021:33:170
38448372171,3sleep10-21swapper/121:29:011
37172962171,4sleep13717561-21grep19:35:271
3683791991714,1cyclictest0-21swapper/320:25:133
3683791991713,2cyclictest0-21swapper/320:15:163
3683791991713,2cyclictest0-21swapper/319:35:003
368379199171,13cyclictest0-21swapper/322:45:373
368379199171,13cyclictest0-21swapper/322:45:363
41390802161,3sleep23813040-21diskmemload00:32:522
41095442161,3sleep00-21swapper/000:12:560
40785252161,2sleep10-21swapper/123:54:131
40426782161,10sleep14042828-21ssh23:30:241
40111522161,10sleep14011291-21cat23:10:281
40052592161,2sleep00-21swapper/023:07:590
39873262162,8sleep03987381-21ntpq22:55:240
38870342162,7sleep23887300-21df_inode21:55:172
38682162161,4sleep23813040-21diskmemload21:41:422
38433792160,3sleep00-21swapper/021:26:430
38426292161,4sleep23813040-21diskmemload21:25:332
38038722161,10sleep03805569-21cut21:00:230
368379199161,9cyclictest3749920-21taskset20:10:073
3683791991613,1cyclictest0-21swapper/320:20:143
368379199161,13cyclictest0-21swapper/321:00:193
368379199161,12cyclictest0-21swapper/320:55:133
368379199161,10cyclictest3716945-21taskset19:35:203
41286922151,9sleep24129870-21memory00:25:222
41185452151,9sleep23813040-21diskmemload00:19:522
41002742151,9sleep04100285-21ssh00:05:360
40291302151,8sleep00-21swapper/023:23:300
39440822151,7sleep13944197-21irqstats22:30:201
39406022151,4sleep13940611-21ssh22:25:581
38610052151,8sleep23813040-21diskmemload21:38:402
38457802150,2sleep10-21swapper/121:30:061
38014822151,11sleep00-21swapper/020:55:210
3683791991512,1cyclictest0-21swapper/319:45:163
368379199150,12cyclictest0-21swapper/320:40:183
41024522140,3sleep23813040-21diskmemload00:09:192
39823442141,6sleep13813040-21diskmemload22:54:121
39487922141,7sleep23813040-21diskmemload22:31:122
39418482141,9sleep23941859-21ssh22:28:082
39026462141,6sleep014-21ksoftirqd/022:04:330
38842542141,7sleep23813040-21diskmemload21:51:242
38842542141,7sleep23813040-21diskmemload21:51:232
38776822140,3sleep23813040-21diskmemload21:48:422
38373542141,3sleep1567-21dbus-daemon21:25:011
38077602141,9sleep13807805-21sh21:05:011
37449632141,9sleep03745074-21grep20:05:150
37384422141,9sleep23738659-21/usr/sbin/munin19:55:272
3683791991411,2cyclictest0-21swapper/319:50:193
3683791991411,2cyclictest0-21swapper/319:50:183
40880772131,6sleep14089602-21ps00:00:191
40117222131,6sleep00-21swapper/023:10:350
39834862131,6sleep13983704-21hddtemp_smartct22:55:171
39597962131,7sleep03959977-21gltestperf22:40:170
39581262131,5sleep10-21swapper/122:38:251
39075202130,1sleep20-21swapper/222:05:232
38772462131,3sleep13877255-21ssh21:48:011
38614302131,6sleep03813040-21diskmemload21:39:190
38262982131,2sleep00-21swapper/021:15:330
38201872130,2sleep03813040-21diskmemload21:13:380
38096082131,7sleep13811367-21cut21:05:201
41463232120,1sleep20-21swapper/200:37:402
41084472121,5sleep23813040-21diskmemload00:11:072
40525642121,7sleep04052575-21ssh23:37:220
40517222121,3sleep20-21swapper/223:35:592
39967422121,5sleep00-21swapper/023:02:050
39812882121,4sleep23813040-21diskmemload22:52:252
39726352121,7sleep03972645-21ssh22:46:100
39726352121,7sleep03972645-21ssh22:46:090
39517672121,7sleep03951966-21cut22:35:150
39407212121,5sleep03813040-21diskmemload22:26:090
39341992121,3sleep10-21swapper/122:23:531
39318422121,5sleep03932046-21grep22:20:290
39254602121,3sleep03925470-21ssh22:17:270
39166782121,5sleep23813040-21diskmemload22:10:592
39017602121,4sleep13813040-21diskmemload22:03:091
38458382121,6sleep23813040-21diskmemload21:30:112
38281442121,7sleep13828156-21modprobe21:18:271
37961642121,5sleep23796212-21ntpq20:50:232
37866652121,3sleep10-21swapper/120:45:011
36971302121,3sleep13698417-21cut19:20:151
41457432111,3sleep10-21swapper/100:36:231
41456952111,4sleep03813040-21diskmemload00:36:190
41400242111,4sleep13813040-21diskmemload00:34:461
41398462111,3sleep00-21swapper/000:34:250
41230262111,5sleep04123074-21ntpq00:20:220
41169772111,3sleep10-21swapper/100:17:111
41085522111,3sleep10-21swapper/100:11:171
40959252110,7sleep14096974-21kthreadcore00:05:171
40947432111,4sleep23813040-21diskmemload00:04:372
40711602111,3sleep00-21swapper/023:50:030
40679522111,4sleep23813040-21diskmemload23:45:412
40603412111,3sleep10-21swapper/123:41:471
40363552111,4sleep03813040-21diskmemload23:27:190
40359342111,4sleep23813040-21diskmemload23:26:412
40288122111,3sleep13813040-21diskmemload23:22:581
40270192111,4sleep23813040-21diskmemload23:20:282
40205742111,4sleep03813040-21diskmemload23:17:170
40127902111,3sleep20-21swapper/223:12:222
40051552111,3sleep20-21swapper/223:07:482
39988912111,4sleep13813040-21diskmemload23:05:111
39980902111,4sleep23813040-21diskmemload23:04:162
39872202111,5sleep23987263-21ntpq22:55:222
39819802111,2sleep00-21swapper/022:53:350
39563072111,4sleep23813040-21diskmemload22:35:302
39323232111,4sleep23813040-21diskmemload22:20:422
39242982111,4sleep23813040-21diskmemload22:15:332
39163582110,1sleep10-21swapper/122:10:301
38978452111,4sleep23813040-21diskmemload22:00:172
38945252111,3sleep10-21swapper/121:59:411
38941782111,2sleep00-21swapper/021:59:070
38371922111,4sleep20-21swapper/221:24:462
38194082110,5sleep23813040-21diskmemload21:12:232
38130732111,4sleep10-21swapper/121:10:021
37856582111,4sleep03785707-21ntpq20:40:220
37452412111,2sleep2171rcu_preempt20:05:172
37336092110,1sleep20-21swapper/219:52:152
37336092110,1sleep20-21swapper/219:52:142
36912422111,4sleep23691260-21apt-get19:15:122
41309072101,3sleep03813040-21diskmemload00:25:520
41300622100,2sleep10-21swapper/100:25:251
40920772101,4sleep03813040-21diskmemload00:00:300
40862992101,3sleep03813040-21diskmemload23:58:540
40754652101,5sleep24075531-21cut23:50:212
40679862101,3sleep10-21swapper/123:45:441
40630822100,5sleep03813040-21diskmemload23:45:150
40597562101,2sleep00-21swapper/023:40:530
40440082101,3sleep00-21swapper/023:31:470
40434082101,3sleep23813040-21diskmemload23:30:462
40359232101,3sleep13813040-21diskmemload23:26:401
40198792101,3sleep23813040-21diskmemload23:16:062
40197782101,3sleep13813040-21diskmemload23:15:581
39980062101,3sleep10-21swapper/123:04:061
39654692101,3sleep10-21swapper/122:42:221
39250832101,3sleep10-21swapper/122:16:491
38699962100,4sleep03813040-21diskmemload21:44:230
38631512101,6sleep13862540-21/usr/sbin/munin21:40:171
38337222101,4sleep03833781-21ntpq21:20:230
37749752101,4sleep13775096-21ntpq20:30:261
37538972101,5sleep13753938-21gcc20:10:221
37381602101,4sleep13738214-21ntpq19:55:221
37328822101,5sleep13732935-21ntpq19:50:221
37328822101,5sleep13732935-21ntpq19:50:211
37130882101,4sleep03713211-21diskstats19:35:130
4111125291,3sleep00-21swapper/000:15:140
3943447291,4sleep03943458-21ssh22:30:030
3916517291,2sleep00-21swapper/022:10:460
3876087291,2sleep00-21swapper/021:46:130
3813031291,4sleep03813040-21diskmemload21:10:000
3787459291,4sleep13787807-21tr20:45:161
3779773291,3sleep23780189-21memory20:35:192
3766276291,4sleep13766404-21kthreadcore20:25:161
3749950291,4sleep23750202-21df20:10:152
3738111291,3sleep03738158-21ntpq19:55:210
3722939290,1sleep20-21swapper/219:41:122
3711253291,3sleep23711366-21ntpq19:30:182
3703060291,4sleep13704717-21cut19:25:171
3702642291,4sleep03703059-21grep19:25:150
3688046290,5sleep10-21swapper/119:10:211
3802500281,3sleep23802571-21unin-run21:00:002
3802156280,1sleep10-21swapper/120:55:381
3791164280,1sleep20-21swapper/220:45:252
3781925281,3sleep23782139-21smartctl20:40:172
3760450280,3sleep23760469-21apt-get20:20:132
3755461283,2sleep10-21swapper/120:15:141
3744419281,3sleep11-21systemd20:05:011
3740372282,3sleep03742025-21kthreadcore20:00:190
3739734281,3sleep23739942-21if_enp1s020:00:172
3723938280,1sleep10-21swapper/119:45:171
3717947281,4sleep23718198-21latency_hist19:40:012
3706998280,1sleep20-21swapper/219:28:182
3696559281,3sleep03696578-21apt-get19:20:110
3689415281,3sleep13690854-21cut19:15:021
3684531281,3sleep03685505-21kthreadcore19:10:170
3776085270,1sleep20-21swapper/220:34:522
3765790270,3sleep23765985-21cstates20:25:132
3755907271,3sleep03757529-21kthreadcore20:15:180
3749247270,4sleep10-21swapper/120:05:281
3724072271,2sleep23725106-21tr19:45:182
3718231270,2sleep10-21swapper/119:40:111
3708678271,2sleep00-21swapper/019:30:170
3700877270,2sleep20-21swapper/219:20:242
3812890260,4sleep20-21swapper/221:08:132
3807411260,1sleep20-21swapper/221:00:372
3790992261,2sleep00-21swapper/020:45:230
3781079260,1sleep00-21swapper/020:36:380
3778016261,2sleep10-21swapper/120:35:181
3771383260,2sleep00-21swapper/020:30:200
3764785260,3sleep00-21swapper/020:20:260
3729368260,2sleep00-21swapper/019:50:170
3729368260,2sleep00-21swapper/019:50:160
3723790261,2sleep00-21swapper/019:45:150
3722981260,1sleep00-21swapper/019:41:430
3691523261,1sleep00-21swapper/019:15:130
3688663260,1sleep20-21swapper/219:10:422
3792018250,1sleep10-21swapper/120:50:021
3770630251,2sleep00-21swapper/020:27:320
3760743250,1sleep10-21swapper/120:20:161
3759940250,1sleep20-21swapper/220:17:112
3792342240,1sleep00-21swapper/020:50:170
3750251240,1sleep00-21swapper/020:10:160
3707676240,1sleep10-21swapper/119:30:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional