You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 16:39
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sat Nov 22, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2659421230,3sleep1291rcuc/110:29:401
78849910343,33cyclictest7863-21cyclictest11:31:022
78849910343,33cyclictest7863-21cyclictest11:31:022
78749910149,29cyclictest30182-21apt-get11:34:321
78719910029,32cyclictest15-21ksoftirqd/012:15:000
65912999,58sleep06665-21fschecks_time07:04:500
65912999,58sleep06665-21fschecks_time07:04:500
734229723,44sleep30-21swapper/307:05:143
734229723,44sleep30-21swapper/307:05:143
7884999525,38cyclictest355-21runrttasks08:11:592
7884999523,39cyclictest14302-21grep12:04:382
650829514,39sleep20-21swapper/207:04:442
650829514,39sleep20-21swapper/207:04:442
7884999432,27cyclictest10953-21seq10:59:372
7884999431,26cyclictest22683-21seq07:50:332
7874999461,13cyclictest181rcu_preempt08:59:511
7884999332,31cyclictest23838-21taskset10:24:262
7884999332,28cyclictest12952-21taskset07:22:482
7884999330,25cyclictest23251-21seq11:21:002
7884999320,30cyclictest5444-21cstates10:49:392
7874999364,23cyclictest9028-21sh11:54:441
7884999234,24cyclictest16704-21taskset11:09:442
7884999233,27cyclictest24953-21seq12:23:102
7884999231,26cyclictest6784-21ssh09:53:152
7884999231,25cyclictest30057-21irqstats08:14:472
7884999230,29cyclictest20386-21ssh11:15:252
7884999230,26cyclictest27734-21ssh12:27:482
7884999230,26cyclictest27734-21ssh12:27:482
7884999230,23cyclictest15761-21find09:09:312
7884999227,27cyclictest1853-21apt-get08:29:352
7884999227,24cyclictest0-21swapper/210:46:532
7884999224,30cyclictest31145-21runrttasks12:34:202
7884999131,25cyclictest0-21swapper/209:15:002
7884999131,25cyclictest0-21swapper/209:15:002
7884999130,24cyclictest22352-21ssh12:19:132
7884999129,29cyclictest4919-21grep08:35:142
7884999127,28cyclictest0-21swapper/209:44:582
7884999032,24cyclictest6841-21seq08:43:182
7884999030,26cyclictest25966-21ssh09:28:252
7884999030,25cyclictest31629-21ssh12:34:422
7884999029,27cyclictest12269-21taskset08:59:382
7874999044,13cyclictest181rcu_preempt11:39:501
7884998926,30cyclictest16301-21apt-get07:34:352
7884998926,30cyclictest16301-21apt-get07:34:352
7874998938,25cyclictest7863-21cyclictest07:49:241
7874998938,25cyclictest7863-21cyclictest07:49:241
7884998829,28cyclictest24475-21seq07:57:582
7884998828,27cyclictest19116-21ssh10:14:532
7884998827,26cyclictest0-21swapper/207:34:232
7884998824,29cyclictest15472-21diskmemload10:41:502
7884998824,28cyclictest477-21/usr/sbin/munin09:40:022
7884998823,33cyclictest6739-21ssh11:49:572
7884998822,33cyclictest20957-21grep07:45:022
7884998822,33cyclictest20957-21grep07:45:022
7884998815,36cyclictest0-21swapper/211:55:042
7874998825,29cyclictest0-21swapper/108:54:541
7874998825,29cyclictest0-21swapper/108:54:541
7884998731,23cyclictest10297-21seq08:53:442
7884998730,24cyclictest14567-21seq07:27:572
7884998730,19cyclictest2328-21seq11:42:312
7884998728,28cyclictest1643-21seq08:26:502
7874998764,16cyclictest31070-21cron08:19:211
7874998749,14cyclictest181rcu_preempt10:06:291
7874998747,16cyclictest181rcu_preempt12:19:501
7874998742,17cyclictest181rcu_preempt12:14:231
787199877,20cyclictest15-21ksoftirqd/011:04:230
7884998630,21cyclictest11839-21seq08:57:532
7884998630,21cyclictest11839-21seq08:57:532
7884998629,22cyclictest15517-21seq10:09:202
7884998628,22cyclictest25961-21seq11:25:542
7884998626,25cyclictest8635-21ssh10:54:492
7884998626,25cyclictest8635-21ssh10:54:492
7884998622,29cyclictest21463-21fschecks_time10:19:432
7884998622,27cyclictest0-21swapper/209:19:442
7874998651,11cyclictest181rcu_preempt09:34:521
7874998640,22cyclictest7863-21cyclictest08:29:241
7871998613,39cyclictest15-21ksoftirqd/008:52:180
7871998611,44cyclictest15-21ksoftirqd/009:29:140
7884998544,31cyclictest11600-21/usr/sbin/munin11:59:382
7884998544,31cyclictest11600-21/usr/sbin/munin11:59:382
7884998531,23cyclictest11367-21seq07:18:252
7884998527,22cyclictest24867-21seq07:59:282
7884998526,27cyclictest31385-21seq09:38:582
7884998526,23cyclictest17564-21grep12:09:522
7884998524,27cyclictest31274-21vmstat11:34:592
7884998520,24cyclictest15524-21ssh11:07:372
7884998520,24cyclictest15524-21ssh11:07:372
7874998540,13cyclictest181rcu_preempt10:23:551
7871998518,40cyclictest0-21swapper/009:24:110
7884998431,28cyclictest18448-21runrttasks07:39:392
7884998431,28cyclictest18448-21runrttasks07:39:392
7884998431,20cyclictest14138-21seq09:04:502
7884998429,24cyclictest9979-21ssh09:59:172
7884998422,25cyclictest0-21swapper/208:04:592
7874998452,9cyclictest181rcu_preempt10:57:281
7874998452,9cyclictest181rcu_preempt10:57:281
7874998443,14cyclictest181rcu_preempt11:15:491
787199846,64cyclictest2355-21apt-get08:29:430
7871998437,16cyclictest181rcu_preempt07:28:350
7884998329,22cyclictest12250-21sh10:02:432
7884998325,31cyclictest0-21swapper/210:35:312
7884998323,25cyclictest5713-21latency_hist11:49:232
7874998350,10cyclictest181rcu_preempt07:19:501
7874998343,8cyclictest181rcu_preempt10:14:581
7874998340,8cyclictest181rcu_preempt10:24:581
7874998327,25cyclictest0-21swapper/111:11:311
7871998359,15cyclictest32693-21threads09:39:590
7871998347,9cyclictest181rcu_preempt08:09:240
787199832,24cyclictest16799-21apt-get07:34:430
787199832,24cyclictest16799-21apt-get07:34:430
7884998237,35cyclictest0-21swapper/210:29:352
7874998249,11cyclictest181rcu_preempt10:54:171
7874998245,12cyclictest181rcu_preempt08:30:071
7874998244,8cyclictest181rcu_preempt10:13:461
7874998244,8cyclictest181rcu_preempt10:13:461
7874998240,13cyclictest181rcu_preempt12:30:481
7871998263,11cyclictest0-21swapper/010:35:000
7884998124,25cyclictest355-21runrttasks09:33:032
7884998124,25cyclictest355-21runrttasks09:33:032
7884998120,27cyclictest0-21swapper/210:09:512
7884998120,27cyclictest0-21swapper/210:09:512
7874998150,10cyclictest181rcu_preempt09:09:491
7874998144,11cyclictest181rcu_preempt11:45:011
787199819,43cyclictest15-21ksoftirqd/011:19:500
7871998147,11cyclictest86-21kswapd009:05:080
7871998145,10cyclictest181rcu_preempt08:14:300
7871998138,10cyclictest181rcu_preempt09:55:050
7871998134,17cyclictest15-21ksoftirqd/008:40:060
7874998057,8cyclictest181rcu_preempt12:03:561
7874998057,8cyclictest181rcu_preempt12:03:561
7874998053,21cyclictest23492-21fschecks_time07:54:451
7874998051,9cyclictest58-21kcompactd012:06:311
7874998051,11cyclictest181rcu_preempt09:24:471
7874998047,16cyclictest181rcu_preempt10:43:281
7874998046,12cyclictest181rcu_preempt10:48:071
7874998044,9cyclictest181rcu_preempt07:29:491
7874998042,8cyclictest181rcu_preempt09:49:241
7874998041,11cyclictest181rcu_preempt10:38:331
7874998038,12cyclictest181rcu_preempt11:24:581
787199803,41cyclictest15-21ksoftirqd/009:36:020
7884997927,32cyclictest355-21runrttasks07:10:232
7884997927,32cyclictest355-21runrttasks07:10:232
7884997927,28cyclictest32624-21seq08:23:382
7874997949,12cyclictest181rcu_preempt12:25:591
7874997949,12cyclictest181rcu_preempt12:25:591
7874997946,11cyclictest181rcu_preempt11:49:491
7874997941,16cyclictest7863-21cyclictest12:37:181
7874997935,9cyclictest0-21swapper/109:24:231
7871997958,13cyclictest28256-21ssh12:29:210
7871997958,13cyclictest28256-21ssh12:29:210
7871997955,14cyclictest0-21swapper/010:19:210
7871997942,28cyclictest15523-21modprobe10:09:200
7871997940,10cyclictest176-21jbd2/mmcblk2p2-810:56:520
7871997940,10cyclictest176-21jbd2/mmcblk2p2-810:56:520
7871997912,41cyclictest15-21ksoftirqd/010:44:530
7871997912,40cyclictest15-21ksoftirqd/009:16:090
7871997912,40cyclictest15-21ksoftirqd/009:16:090
7874997854,8cyclictest181rcu_preempt08:19:501
7874997853,14cyclictest0-21swapper/107:39:251
7874997853,14cyclictest0-21swapper/107:39:251
7874997849,8cyclictest181rcu_preempt10:02:121
7874997848,11cyclictest181rcu_preempt09:39:401
7874997846,10cyclictest181rcu_preempt12:14:581
7874997846,10cyclictest181rcu_preempt09:32:431
7874997846,10cyclictest181rcu_preempt09:32:431
7874997845,11cyclictest181rcu_preempt08:09:481
7874997844,10cyclictest181rcu_preempt09:05:211
7874997829,8cyclictest0-21swapper/108:35:121
7874997828,26cyclictest0-21swapper/111:04:591
7874997828,26cyclictest0-21swapper/111:04:591
7871997854,15cyclictest691-21cron11:44:210
787199785,34cyclictest15-21ksoftirqd/010:29:440
7871997842,12cyclictest15-21ksoftirqd/010:22:070
7871997812,37cyclictest15-21ksoftirqd/008:35:150
7884997723,22cyclictest0-21swapper/208:46:502
7874997754,13cyclictest0-21swapper/108:50:401
7874997747,10cyclictest181rcu_preempt09:49:471
7874997730,13cyclictest181rcu_preempt07:09:581
7874997730,13cyclictest181rcu_preempt07:09:581
7871997757,13cyclictest7863-21cyclictest12:32:390
7871997740,10cyclictest181rcu_preempt08:01:380
787199773,40cyclictest15-21ksoftirqd/009:45:010
787199772,48cyclictest15-21ksoftirqd/007:54:450
7871997716,33cyclictest15-21ksoftirqd/007:39:450
7871997716,33cyclictest15-21ksoftirqd/007:39:450
7874997659,11cyclictest8814-21sh09:55:481
7874997659,11cyclictest24874-21/usr/sbin/munin07:59:491
7874997653,14cyclictest314-21in:imuxsock08:09:211
7874997650,9cyclictest181rcu_preempt11:29:381
7874997650,9cyclictest181rcu_preempt11:29:381
7874997647,9cyclictest181rcu_preempt10:59:451
7874997647,10cyclictest181rcu_preempt09:14:411
7874997647,10cyclictest181rcu_preempt09:14:411
7874997631,11cyclictest181rcu_preempt08:45:021
787199769,41cyclictest15-21ksoftirqd/011:44:500
7871997640,11cyclictest7863-21cyclictest08:59:470
7871997638,12cyclictest181rcu_preempt12:10:430
7871997636,16cyclictest181rcu_preempt11:31:450
7871997636,16cyclictest181rcu_preempt11:31:450
7871997634,20cyclictest15-21ksoftirqd/007:31:350
787199762,63cyclictest14285-21ssh12:04:360
7871997626,18cyclictest15-21ksoftirqd/009:12:480
787199762,47cyclictest15-21ksoftirqd/008:56:320
787199762,47cyclictest15-21ksoftirqd/008:56:320
787199762,40cyclictest15-21ksoftirqd/008:28:320
7871997615,32cyclictest15-21ksoftirqd/010:02:430
7874997542,10cyclictest181rcu_preempt07:19:241
7874997533,22cyclictest0-21swapper/108:44:151
7874997523,24cyclictest0-21swapper/111:19:421
787199755,39cyclictest15-21ksoftirqd/011:55:060
787199754,45cyclictest15-21ksoftirqd/011:59:380
787199754,45cyclictest15-21ksoftirqd/011:59:380
787199754,41cyclictest15-21ksoftirqd/012:37:570
787199754,35cyclictest15-21ksoftirqd/008:10:130
7871997540,20cyclictest181rcu_preempt09:51:490
787199753,40cyclictest15-21ksoftirqd/009:34:030
787199753,40cyclictest15-21ksoftirqd/009:34:030
787199752,40cyclictest15-21ksoftirqd/011:34:290
7871997519,31cyclictest15-21ksoftirqd/010:12:180
7871997519,31cyclictest15-21ksoftirqd/010:12:180
7871997514,53cyclictest211rcuc/007:49:360
7871997511,39cyclictest15-21ksoftirqd/011:28:500
7874997426,21cyclictest0-21swapper/107:39:451
7874997426,21cyclictest0-21swapper/107:39:451
7871997439,7cyclictest181rcu_preempt10:39:470
787199743,45cyclictest15-21ksoftirqd/010:52:170
787199743,39cyclictest15-21ksoftirqd/012:19:370
7871997431,12cyclictest181rcu_preempt11:05:470
7871997431,12cyclictest181rcu_preempt11:05:470
778827412,40sleep17800-21munin-run07:09:241
778827412,40sleep17800-21munin-run07:09:241
7874997330,13cyclictest368-21snmpd07:53:021
7871997344,7cyclictest181rcu_preempt10:26:010
789199724,51cyclictest2376-21chrt08:29:443
7874997245,8cyclictest181rcu_preempt07:24:491
787199723,36cyclictest20030-21/usr/sbin/munin07:45:000
787199723,36cyclictest20030-21/usr/sbin/munin07:45:000
7871997214,36cyclictest15-21ksoftirqd/011:49:490
7871997116,31cyclictest15-21ksoftirqd/011:11:000
7871997115,32cyclictest15-21ksoftirqd/007:19:360
7871997111,32cyclictest15-21ksoftirqd/008:44:550
271942710,7sleep30-21swapper/312:26:093
271942710,7sleep30-21swapper/312:26:093
7871997038,7cyclictest181rcu_preempt11:16:080
787199683,33cyclictest9841-21needreboot07:14:300
787199682,35cyclictest31931-21ps08:19:550
13762680,7sleep3451rcuc/308:24:583
7871996638,10cyclictest181rcu_preempt07:11:110
7871996638,10cyclictest181rcu_preempt07:11:110
185622650,6sleep30-21swapper/309:14:253
185622650,6sleep30-21swapper/309:14:253
7891995542,9cyclictest451rcuc/310:34:213
7891995542,8cyclictest451rcuc/312:10:423
7891995541,9cyclictest451rcuc/309:24:523
789199544,38cyclictest19850-21taskset12:14:383
7891995441,9cyclictest451rcuc/312:02:353
7891995441,9cyclictest451rcuc/312:02:353
7891995441,9cyclictest451rcuc/311:37:373
7891995441,9cyclictest451rcuc/309:04:543
7891995441,8cyclictest451rcuc/310:34:333
7891995441,8cyclictest451rcuc/309:19:413
7891995341,8cyclictest451rcuc/310:10:053
7891995341,8cyclictest451rcuc/310:10:053
7891995338,10cyclictest451rcuc/308:14:303
789199533,34cyclictest2007-21chrt09:44:243
789199533,33cyclictest18511-21taskset10:14:343
789199532,34cyclictest5512-21taskset08:39:243
789199523,35cyclictest12684-21chrt07:20:023
789199522,37cyclictest7889-21taskset08:44:553
789199522,36cyclictest5878-21chrt09:50:433
789199522,36cyclictest16256-21chrt12:08:103
789199522,35cyclictest31586-21chrt12:34:403
789199522,35cyclictest13085-21chrt10:04:333
7891995138,9cyclictest451rcuc/310:59:443
7891995137,9cyclictest451rcuc/309:04:213
7891995136,8cyclictest451rcuc/311:57:363
7891995136,11cyclictest451rcuc/309:12:033
789199513,32cyclictest2242-21chrt10:44:213
789199512,35cyclictest14710-21chrt07:29:213
789199512,34cyclictest16937-21taskset07:34:523
789199512,34cyclictest16937-21taskset07:34:523
7891995036,9cyclictest451rcuc/311:21:373
7891995036,8cyclictest451rcuc/311:06:583
7891995036,8cyclictest451rcuc/311:06:583
789199503,34cyclictest32710-21chrt08:24:213
789199503,33cyclictest21746-21taskset07:49:433
7891994936,8cyclictest451rcuc/309:47:213
7891994935,9cyclictest451rcuc/310:56:293
7891994935,9cyclictest451rcuc/310:56:293
789199493,33cyclictest31321-21taskset12:34:233
789199492,36cyclictest11778-21taskset10:01:163
789199492,34cyclictest19968-21chrt07:44:243
789199492,34cyclictest19968-21chrt07:44:243
789199485,31cyclictest28272-21taskset08:09:433
789199484,30cyclictest7175-21chrt11:50:293
789199483,32cyclictest28682-21chrt09:33:513
789199483,32cyclictest28682-21chrt09:33:513
7891994833,10cyclictest451rcuc/307:54:543
7891994830,12cyclictest451rcuc/310:49:373
789199482,32cyclictest29306-21chrt09:34:443
789199473,32cyclictest9519-21chrt09:57:563
789199472,34cyclictest3332-21taskset10:44:583
789199472,33cyclictest15549-21taskset07:29:543
7891994632,9cyclictest451rcuc/311:12:293
7891994632,9cyclictest451rcuc/308:41:403
7891994631,10cyclictest451rcuc/308:08:553
789199462,35cyclictest12019-21taskset08:59:223
789199462,35cyclictest12019-21taskset08:59:223
789199462,34cyclictest25547-21taskset07:59:553
789199462,31cyclictest665-21taskset11:39:393
789199453,30cyclictest25074-21chrt11:24:483
7891994532,8cyclictest451rcuc/311:46:083
789199452,29cyclictest20285-21chrt11:15:083
7891994431,9cyclictest451rcuc/312:21:533
789199442,27cyclictest10610-21chrt07:14:513
789199433,28cyclictest30071-21chrt11:34:243
789199433,28cyclictest30071-21chrt11:34:243
7891994329,9cyclictest451rcuc/308:52:063
789199432,32cyclictest21968-21taskset10:19:573
789199423,29cyclictest9180-21taskset07:11:353
789199423,29cyclictest9180-21taskset07:11:353
789199423,28cyclictest25188-21taskset10:25:413
789199402,27cyclictest21089-21taskset07:45:423
789199402,27cyclictest21089-21taskset07:45:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional