You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 08:22
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Jul 12, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
94569912960,14cyclictest16235-21latency_hist22:56:000
9476991123,79cyclictest658-21apt-get00:21:302
9476991123,79cyclictest658-21apt-get00:21:302
9476991123,79cyclictest658-21apt-get00:21:302
3265621030,5sleep10-21swapper/122:27:371
8455210214,59sleep07981-21/usr/sbin/munin19:01:310
916321017,17sleep10-21swapper/119:03:531
9476991002,85cyclictest7030-21apt-get23:36:112
9456991003,71cyclictest31223-21apt-get22:26:320
947699993,35cyclictest31497-21diskmemload21:31:042
9476999846,44cyclictest0-21swapper/220:00:592
947699983,34cyclictest1-21systemd23:55:572
9456999846,30cyclictest375-21snmpd22:02:260
947699963,32cyclictest24566-21latency_hist19:51:002
80602958,57sleep28115-21expr19:01:172
9476999456,32cyclictest16621-21diskmemload23:15:412
947699945,32cyclictest4823-21apt-get22:36:312
947699945,32cyclictest4823-21apt-get22:36:312
84322940,6sleep30-21swapper/321:46:323
84322940,6sleep30-21swapper/321:46:323
9476999348,40cyclictest19331-21kworker/u18:0+events_unbound19:55:572
9476999348,40cyclictest19331-21kworker/u18:0+events_unbound19:55:572
9476999340,45cyclictest0-21swapper/200:06:312
9476999338,48cyclictest0-21swapper/222:31:102
947699933,34cyclictest19256-21fschecks_count23:56:202
9476999238,47cyclictest1926-21cpuspeed20:21:132
947699914,41cyclictest27648-21fschecks_time00:11:212
947699914,41cyclictest27648-21fschecks_time00:11:212
947699913,35cyclictest25640-21kernelversion22:16:222
947699913,35cyclictest25640-21kernelversion22:16:222
947699913,32cyclictest10410-21apt-get20:46:282
9476999060,21cyclictest16621-21diskmemload21:45:592
947699904,30cyclictest4442-21/usr/sbin/munin23:31:302
947699903,35cyclictest6035-21cut00:31:192
947699903,35cyclictest3521-21fschecks_time00:26:212
947699903,35cyclictest24742-21apt-get23:10:562
947699903,35cyclictest24742-21apt-get23:10:562
947699903,34cyclictest13314-21/usr/sbin/munin19:16:122
947699903,33cyclictest20317-21grep19:36:302
947699903,27cyclictest28703-21/usr/sbin/munin23:16:232
9476999028,56cyclictest0-21swapper/219:13:542
9476999028,56cyclictest0-21swapper/219:13:542
947699902,35cyclictest32535-21unixbench_singl22:27:182
947699894,35cyclictest22680-21if_eth022:11:242
947699893,33cyclictest1641-21latency_hist20:20:582
947699893,33cyclictest11020-21modprobe19:10:572
947699893,26cyclictest19586-21wc21:10:592
947699893,26cyclictest19586-21wc21:10:592
947699892,36cyclictest1373-21latency_hist23:25:592
9476998911,32cyclictest23321-21ssh00:02:482
947699883,39cyclictest13626-21runrttasks21:56:112
947699883,38cyclictest2947-21grep21:36:332
947699883,30cyclictest13367-21munin-run20:55:562
945699886,58cyclictest356-21runrttasks23:36:280
947699874,31cyclictest19438-21cpu22:06:142
947699874,31cyclictest19438-21cpu22:06:142
947699873,35cyclictest31470-21latency20:11:222
947699873,35cyclictest31470-21latency20:11:222
947699873,34cyclictest5211-21latency_hist20:31:012
9476998733,48cyclictest0-21swapper/223:50:562
947699873,33cyclictest30818-21grep00:16:292
947699873,32cyclictest29818-21tr20:06:222
947699873,31cyclictest22837-21latency_hist19:45:592
9467998762,18cyclictest16621-21diskmemload00:30:591
9456998744,28cyclictest21320-21rm00:00:160
947699864,33cyclictest21968-21ssh23:05:592
9476998638,39cyclictest0-21swapper/221:51:322
947699863,30cyclictest356-21runrttasks20:39:382
947699863,30cyclictest356-21runrttasks19:30:362
947699863,29cyclictest16756-21latency_hist21:05:592
9476998630,48cyclictest0-21swapper/223:41:272
947699853,36cyclictest18389-21fschecks_time19:31:272
947699853,35cyclictest16413-21latency_hist19:26:012
947699853,35cyclictest16413-21latency_hist19:26:012
947699853,33cyclictest10274-21latency_hist20:46:002
947699853,33cyclictest10274-21latency_hist20:46:002
947699853,26cyclictest28501-21/usr/sbin/munin22:21:202
9476998527,9cyclictest0-21swapper/222:48:352
947699852,35cyclictest8530-21switchtime22:41:322
947699852,26cyclictest28033-21fschecks_count20:01:212
9476998513,28cyclictest8318-21needreboot21:46:282
9476998513,28cyclictest8318-21needreboot21:46:282
9476998437,39cyclictest0-21swapper/221:21:002
947699843,33cyclictest17225-21ssh22:01:312
947699843,33cyclictest16982-21ntp_offset22:56:252
947699843,32cyclictest16302-21latency_hist22:56:002
947699835,41cyclictest5317-21apt-get20:31:192
947699833,32cyclictest1665-21cstates23:26:132
9476998322,8cyclictest0-21swapper/221:21:212
9467998361,16cyclictest31829-21fschecks_time21:31:211
9456998367,8cyclictest0-21swapper/023:12:010
947699823,36cyclictest31428-21latency_hist21:31:022
947699823,36cyclictest31428-21latency_hist21:31:022
9467998256,12cyclictest181rcu_preempt20:51:181
9467998253,11cyclictest181rcu_preempt22:11:221
947699813,28cyclictest19960-21ssh21:11:152
9467998154,10cyclictest181rcu_preempt20:36:241
947699803,31cyclictest14044-21latency20:56:242
947699803,31cyclictest14044-21latency20:56:242
9456998052,19cyclictest24498-21latency_hist19:51:000
9456998046,27cyclictest0-21swapper/023:10:580
9456998046,27cyclictest0-21swapper/023:10:580
9456998046,24cyclictest16599-21modprobe21:05:560
946799793,37cyclictest31-21ksoftirqd/120:31:441
9456997960,12cyclictest331-21dbus-daemon19:30:580
190722790,8sleep30-21swapper/321:09:403
190722790,8sleep30-21swapper/321:09:403
9467997859,13cyclictest19648-21ssh21:10:591
9467997859,13cyclictest19648-21ssh21:10:591
9467997856,8cyclictest181rcu_preempt00:01:231
9456997863,8cyclictest0-21swapper/020:30:590
945699784,64cyclictest16621-21diskmemload00:21:300
945699784,64cyclictest16621-21diskmemload00:21:300
945699784,64cyclictest16621-21diskmemload00:21:300
9467997753,11cyclictest181rcu_preempt19:44:051
9467997741,8cyclictest181rcu_preempt23:56:081
9467997652,10cyclictest181rcu_preempt21:18:321
9467997651,9cyclictest181rcu_preempt23:07:161
9467997651,9cyclictest181rcu_preempt23:07:161
9467997651,10cyclictest181rcu_preempt22:49:211
9467997648,20cyclictest28218-21sh23:16:111
9467997644,15cyclictest181rcu_preempt00:11:341
9467997644,15cyclictest181rcu_preempt00:11:341
9467997642,13cyclictest181rcu_preempt22:37:011
9467997642,13cyclictest181rcu_preempt22:37:011
9467997640,13cyclictest181rcu_preempt19:31:161
9467997639,12cyclictest181rcu_preempt22:31:151
9467997558,10cyclictest0-21swapper/122:06:351
9467997558,10cyclictest0-21swapper/122:06:351
9467997542,13cyclictest181rcu_preempt20:24:181
9467997530,12cyclictest181rcu_preempt21:36:151
9467997459,11cyclictest58-21kcompactd000:24:081
9467997459,11cyclictest58-21kcompactd000:24:081
9467997459,11cyclictest58-21kcompactd000:24:081
9467997456,12cyclictest58-21kcompactd020:30:381
9467997446,9cyclictest181rcu_preempt23:50:461
9467997442,11cyclictest181rcu_preempt21:21:041
946799743,35cyclictest1670-21ls23:26:131
9467997425,9cyclictest181rcu_preempt19:51:351
9467997425,9cyclictest181rcu_preempt19:51:351
946799742,47cyclictest31-21ksoftirqd/119:56:311
946799742,45cyclictest16196-21sort23:51:141
9456997449,10cyclictest18648-21cron23:55:560
9456997446,23cyclictest9446-21cyclictest21:00:580
9456997446,23cyclictest9446-21cyclictest21:00:580
9456997442,25cyclictest0-21swapper/022:56:320
945699743,38cyclictest15-21ksoftirqd/022:17:150
945699743,38cyclictest15-21ksoftirqd/022:17:150
9456997431,19cyclictest181rcu_preempt23:41:250
9483997312,49cyclictest0-21swapper/320:46:243
9467997358,9cyclictest19133-21rm22:05:581
9467997350,10cyclictest181rcu_preempt20:15:031
9467997350,10cyclictest181rcu_preempt20:15:031
9467997345,8cyclictest181rcu_preempt19:31:341
946799734,56cyclictest10417-21ssh23:41:101
9467997345,19cyclictest24934-21cut00:06:201
9467997339,16cyclictest181rcu_preempt23:21:191
9467997337,12cyclictest181rcu_preempt20:46:001
9467997337,12cyclictest181rcu_preempt20:46:001
9467997327,36cyclictest28412-21grep22:21:181
946799732,41cyclictest31-21ksoftirqd/121:15:161
9467997316,25cyclictest16621-21diskmemload23:31:161
948399722,33cyclictest2172-21chrt22:31:243
948399722,28cyclictest24647-21chrt00:06:103
9467997250,8cyclictest181rcu_preempt23:12:071
9467997241,20cyclictest14354-21grep20:56:331
9467997241,20cyclictest14354-21grep20:56:331
9467997234,14cyclictest181rcu_preempt21:51:221
9467997233,17cyclictest181rcu_preempt00:31:341
9456997247,18cyclictest10377-21latency_hist22:45:590
9456997239,12cyclictest181rcu_preempt00:31:230
945699723,28cyclictest32061-21timerwakeupswit23:21:320
9467997148,8cyclictest181rcu_preempt22:57:241
9467997134,13cyclictest181rcu_preempt23:36:581
946799712,34cyclictest31-21ksoftirqd/122:16:161
946799712,34cyclictest31-21ksoftirqd/122:16:161
9467997121,12cyclictest181rcu_preempt22:55:051
9456997150,14cyclictest25058-21kernelversion00:06:230
9456997145,19cyclictest15871-21latency_hist23:50:580
9456997137,15cyclictest15-21ksoftirqd/021:06:230
9456997137,15cyclictest15-21ksoftirqd/021:06:230
945699713,43cyclictest5631-21fschecks_time20:31:570
948399703,43cyclictest0-21swapper/300:21:083
948399703,43cyclictest0-21swapper/300:21:083
948399703,43cyclictest0-21swapper/300:21:083
948399703,30cyclictest32576-21chrt22:27:233
948399703,29cyclictest27791-21chrt22:20:563
948399703,29cyclictest27791-21chrt22:20:563
948399702,26cyclictest8002-21chrt00:35:563
9467997043,10cyclictest181rcu_preempt21:26:511
9467997043,10cyclictest181rcu_preempt21:26:511
9467997038,16cyclictest181rcu_preempt20:16:211
9467997037,12cyclictest181rcu_preempt19:46:341
9467997036,11cyclictest181rcu_preempt19:11:061
9467997036,11cyclictest181rcu_preempt19:11:061
9467997035,11cyclictest181rcu_preempt22:43:491
9467997023,12cyclictest181rcu_preempt19:36:291
946799702,28cyclictest31-21ksoftirqd/121:01:321
946799702,17cyclictest8121-21fschecks_time21:46:231
946799702,17cyclictest8121-21fschecks_time21:46:231
9467997010,37cyclictest21082-21sshd23:03:431
9456997050,13cyclictest356-21runrttasks00:26:220
9456997048,16cyclictest15874-21unixbench_singl19:21:340
9456997048,16cyclictest15874-21unixbench_singl19:21:340
9456997047,16cyclictest7558-21latency_hist22:41:000
9456997047,16cyclictest7558-21latency_hist22:41:000
9456997046,9cyclictest181rcu_preempt22:06:130
9456997046,9cyclictest181rcu_preempt22:06:130
9456997040,9cyclictest181rcu_preempt21:56:370
9456997026,28cyclictest30502-21ssh22:25:270
948399695,38cyclictest0-21swapper/323:36:283
948399694,24cyclictest5028-21chrt20:30:593
9467996947,7cyclictest181rcu_preempt22:00:061
946799693,37cyclictest5647-21ssh21:41:311
9467996925,13cyclictest181rcu_preempt19:10:591
9456996953,9cyclictest2318-21rm22:31:300
9456996945,8cyclictest181rcu_preempt21:46:230
9456996945,8cyclictest181rcu_preempt21:46:230
9456996940,9cyclictest181rcu_preempt21:16:190
9456996940,8cyclictest181rcu_preempt21:41:200
945699693,28cyclictest10681-21/usr/sbin/munin21:51:380
948399683,31cyclictest13122-21chrt23:45:593
948399682,14cyclictest28220-21chrt23:16:113
9467996813,32cyclictest31-21ksoftirqd/120:01:191
9456996847,15cyclictest8972-21latency20:41:230
9456996847,15cyclictest8972-21latency20:41:230
9456996846,16cyclictest26032-21sh21:21:220
945699684,44cyclictest2561-21grep23:26:340
9456996836,10cyclictest181rcu_preempt19:31:310
945699683,52cyclictest356-21runrttasks19:40:590
236872680,7sleep3451rcuc/319:46:323
948399673,24cyclictest24360-21chrt00:05:573
946799674,43cyclictest30154-21cpu00:16:121
9467996743,9cyclictest181rcu_preempt19:21:331
9467996743,9cyclictest181rcu_preempt19:21:331
9467996743,11cyclictest181rcu_preempt20:06:141
9467996735,12cyclictest181rcu_preempt19:20:591
946799672,26cyclictest9913-21apt-get20:46:131
9456996752,9cyclictest24863-21ssh22:15:560
9456996750,11cyclictest29703-21fschecks_time20:06:190
9456996745,15cyclictest0-21swapper/021:26:320
9456996745,15cyclictest0-21swapper/021:26:320
25312670,9sleep02527-21memory21:36:240
157922670,7sleep30-21swapper/321:01:253
948399663,27cyclictest3576-21chrt23:29:213
948399663,24cyclictest28261-21taskset21:25:493
948399662,25cyclictest32157-21chrt21:31:293
9456996650,11cyclictest9446-21cyclictest23:16:320
9456996650,10cyclictest58-21kcompactd020:36:240
9456996648,10cyclictest0-21swapper/022:46:210
9456996646,15cyclictest4767-21kworker/u18:1+events_unbound19:20:560
9456996642,14cyclictest0-21swapper/019:51:230
9456996642,14cyclictest0-21swapper/019:51:230
945699663,30cyclictest22725-21timerandwakeup00:01:320
9456996627,25cyclictest181rcu_preempt19:56:170
948399654,22cyclictest1403-21chrt20:18:363
948399652,28cyclictest6143-21chrt21:41:583
948399652,25cyclictest25886-21taskset19:55:273
948399652,25cyclictest25886-21taskset19:55:273
948399652,25cyclictest14617-21taskset19:20:243
9483996511,29cyclictest0-21swapper/320:31:193
9456996550,9cyclictest5044-21ntp_kernel_pll_23:31:300
9456996548,11cyclictest21615-21ssh21:13:280
9456996541,9cyclictest181rcu_preempt20:16:210
9456996538,9cyclictest0-21swapper/000:11:330
9456996538,9cyclictest0-21swapper/000:11:330
945699653,41cyclictest12003-21/usr/sbin/munin20:51:230
948399644,23cyclictest32094-21chrt20:12:083
948399644,23cyclictest32094-21chrt20:12:083
948399643,26cyclictest14948-21chrt19:21:133
948399643,26cyclictest14948-21chrt19:21:133
948399643,23cyclictest13109-21chrt22:50:503
948399643,21cyclictest19665-21chrt23:01:203
948399642,23cyclictest5911-21taskset23:32:413
948399642,20cyclictest29776-21chrt20:06:213
9456996441,12cyclictest0-21swapper/020:25:590
945699642,44cyclictest10392-21/usr/sbin/munin20:46:520
948399635,23cyclictest14711-21chrt23:47:413
948399633,24cyclictest18418-21taskset23:55:153
948399633,22cyclictest27645-21chrt23:15:183
948399633,22cyclictest11999-21chrt20:51:043
9456996344,13cyclictest11866-21fschecks_count19:11:200
9456996344,13cyclictest11866-21fschecks_count19:11:200
9456996343,13cyclictest0-21swapper/021:31:350
9456996341,12cyclictest0-21swapper/020:11:180
9456996341,12cyclictest0-21swapper/020:11:180
948399623,24cyclictest708-21taskset23:24:263
948399623,21cyclictest25118-21chrt23:11:013
948399623,21cyclictest25118-21chrt23:11:013
9483996218,9cyclictest150822chrt22:53:053
9456996246,10cyclictest30113-21diskmemload00:16:060
9456996242,12cyclictest0-21swapper/019:06:130
9456996240,16cyclictest28075-21fschecks_time20:01:210
9456996237,19cyclictest8377-21kworker/u17:0+nfsiod23:03:440
948399614,23cyclictest10654-21chrt19:06:433
948399613,23cyclictest20995-21taskset19:39:273
948399613,22cyclictest32056-21chrt00:19:233
948399613,22cyclictest14938-21chrt21:00:203
948399613,22cyclictest14938-21chrt21:00:203
948399613,21cyclictest30650-21chrt22:25:553
948399613,20cyclictest23198-21chrt21:16:253
948399613,19cyclictest17635-21chrt22:56:583
948399612,23cyclictest8046-21chrt20:37:563
948399612,22cyclictest20527-21chrt23:57:513
948399612,22cyclictest18067-21chrt22:02:573
948399612,21cyclictest7725-21chrt22:41:133
948399612,19cyclictest21760-21chrt21:13:543
948399603,37cyclictest2273-21chrt21:36:193
948399603,22cyclictest5000-21chrt00:29:193
948399603,22cyclictest15223-21chrt21:58:073
948399603,22cyclictest12918-21taskset19:14:323
948399603,22cyclictest12918-21taskset19:14:323
948399602,22cyclictest28644-21chrt20:01:353
948399597,12cyclictest0-21swapper/322:11:213
948399594,11cyclictest0-21swapper/300:11:313
948399594,11cyclictest0-21swapper/300:11:313
948399593,23cyclictest22594-21chrt19:44:213
948399593,22cyclictest19283-21chrt19:33:243
948399593,21cyclictest9731-21chrt20:43:513
948399593,21cyclictest9731-21chrt20:43:513
948399593,21cyclictest12241-21chrt21:52:563
948399593,18cyclictest17078-21chrt19:26:273
948399592,24cyclictest5905-21taskset22:37:043
948399592,24cyclictest5905-21taskset22:37:043
948399592,21cyclictest2970-21chrt20:23:053
948399592,20cyclictest19594-21chrt22:06:173
948399592,20cyclictest19594-21chrt22:06:173
9456995919,25cyclictest22893-21cat19:46:000
948399583,20cyclictest31088-21chrt21:30:353
948399583,20cyclictest31088-21chrt21:30:353
948399583,13cyclictest0-21swapper/319:56:243
90292576,34sleep30-21swapper/319:02:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional