You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 06:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Dec 07, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2851921289,90sleep128529-21munin-run19:08:261
28658991102,88cyclictest17592-21apt-get21:33:381
1494421090,7sleep00-21swapper/021:28:260
27736210711,67sleep227766-21uname19:04:072
286589910511,83cyclictest11803-21apt-get19:53:441
502821040,6sleep30-21swapper/322:08:273
28658991046,74cyclictest1749-21modprobe23:58:291
286549910327,45cyclictest15-21ksoftirqd/023:23:450
286589910241,9cyclictest0-21swapper/120:43:231
286589910241,9cyclictest0-21swapper/120:43:231
28658991023,83cyclictest24858-21ssh22:43:511
27347210112,56sleep027417-21fschecks_time19:03:520
28658991009,82cyclictest1900-21apt-get19:23:511
2865899993,42cyclictest31561-21switchtime19:14:011
2865899993,42cyclictest31561-21switchtime19:14:011
28671999613,38cyclictest0-21swapper/319:23:513
2865899962,40cyclictest5963-21timerwakeupswit19:34:001
2800629526,42sleep30-21swapper/319:04:123
28658999469,18cyclictest28089-21sh22:48:511
28658999469,18cyclictest28089-21sh22:48:511
28658999469,16cyclictest20847-21ssh21:38:341
2865899942,43cyclictest3390-21grep22:04:031
28658999377,10cyclictest26395-21apt-config21:48:281
2865899933,40cyclictest20208-21sshd20:18:561
2865899933,40cyclictest20208-21sshd20:18:561
2865899922,40cyclictest355-21runrttasks00:08:251
2865899913,33cyclictest12364-21/usr/sbin/munin21:23:501
28654999068,16cyclictest23183-21cpuspeed20:28:430
28654998764,16cyclictest314-21in:imuxsock00:38:230
28654998749,9cyclictest181rcu_preempt00:16:090
2865499872,45cyclictest15-21ksoftirqd/021:28:590
28654998666,14cyclictest18575-21fschecks_count21:34:420
28665998563,14cyclictest2272-21ssh22:03:282
2865899854,70cyclictest2225-21apt-get21:03:381
28654998549,8cyclictest181rcu_preempt23:51:020
28654998458,20cyclictest9381-21latency_hist21:18:260
28654998458,20cyclictest9381-21latency_hist21:18:260
28654998444,8cyclictest181rcu_preempt20:33:420
28654998444,11cyclictest181rcu_preempt22:18:570
28654998444,11cyclictest181rcu_preempt22:18:570
28665998344,29cyclictest31124-21cut19:13:492
28665998344,29cyclictest31124-21cut19:13:492
28658998359,15cyclictest316-21rs:main1
28658998357,19cyclictest18893-21sendmail_mailqu20:14:051
28658998346,8cyclictest181rcu_preempt23:58:231
28658998346,8cyclictest181rcu_preempt23:58:231
28658998257,17cyclictest28645-21cyclictest19:23:271
28654998261,15cyclictest2710-21timerandwakeup19:24:340
28654998254,8cyclictest181rcu_preempt20:28:280
28654998217,47cyclictest15-21ksoftirqd/023:53:420
28654998217,47cyclictest15-21ksoftirqd/023:53:420
281742820,7sleep23581-21diskmemload22:48:532
281742820,7sleep23581-21diskmemload22:48:532
126612820,7sleep00-21swapper/019:55:100
28658998160,15cyclictest17625-21apt-key21:33:251
2865499812,48cyclictest15-21ksoftirqd/022:15:130
2865499812,48cyclictest15-21ksoftirqd/022:15:130
28665998054,19cyclictest18950-21sh23:30:232
28665998054,19cyclictest18950-21sh23:30:232
2865499805,44cyclictest15-21ksoftirqd/020:45:360
28654998048,9cyclictest181rcu_preempt23:13:270
28654998041,10cyclictest181rcu_preempt21:53:460
28654998023,38cyclictest15-21ksoftirqd/022:48:410
28654998023,38cyclictest15-21ksoftirqd/022:48:410
28654998023,25cyclictest15-21ksoftirqd/021:04:420
28665997954,18cyclictest29309-21mailstats23:48:592
2866599794,36cyclictest39-21ksoftirqd/220:30:072
2866599792,46cyclictest39-21ksoftirqd/200:36:202
2865899797,52cyclictest292-21dbus-daemon00:33:241
2865899797,33cyclictest26578-21apt-get21:48:401
2865899797,33cyclictest26578-21apt-get21:48:401
28654997930,32cyclictest15-21ksoftirqd/021:39:000
28665997851,20cyclictest983-21sh22:58:462
2866599783,35cyclictest39-21ksoftirqd/219:10:472
2866599781,19cyclictest11301-21apt-get19:53:382
28654997863,9cyclictest3581-21diskmemload22:58:270
28654997863,9cyclictest161ktimers/023:38:520
28654997860,13cyclictest28645-21cyclictest19:33:240
28654997858,14cyclictest3213-21threads22:04:010
28654997856,16cyclictest314-21in:imuxsock23:30:580
28654997856,16cyclictest314-21in:imuxsock23:30:580
28654997846,24cyclictest0-21swapper/022:09:000
28654997846,24cyclictest0-21swapper/022:09:000
2867199773,36cyclictest0-21swapper/321:33:473
28665997755,15cyclictest309-21systemd-logind22:18:252
28665997755,15cyclictest309-21systemd-logind22:18:252
28658997748,22cyclictest22141-21if_err_eth022:38:491
28654997711,30cyclictest15-21ksoftirqd/020:22:210
28654997711,30cyclictest15-21ksoftirqd/020:22:210
2867199763,51cyclictest0-21swapper/323:58:263
2867199763,51cyclictest0-21swapper/323:58:263
28665997652,16cyclictest24914-21sh23:41:312
28665997642,28cyclictest20375-21needreboot20:19:002
28665997642,28cyclictest20375-21needreboot20:19:002
28658997653,17cyclictest952-21fschecks_count22:58:461
28658997652,13cyclictest0-21swapper/122:54:011
28654997636,30cyclictest0-21swapper/023:03:260
28654997616,16cyclictest15-21ksoftirqd/019:47:290
2866599756,35cyclictest39-21ksoftirqd/220:04:152
2866599755,45cyclictest18130-21apt-get21:33:392
2866599753,46cyclictest8265-21latency_hist19:43:272
2865499757,45cyclictest15-21ksoftirqd/000:09:020
28654997554,15cyclictest26204-21latency_hist21:48:260
28654997550,17cyclictest11760-21latency_hist23:18:270
28654997550,17cyclictest11760-21latency_hist23:18:270
28654997546,8cyclictest181rcu_preempt23:58:340
28654997538,9cyclictest181rcu_preempt22:01:110
2865499753,49cyclictest15-21ksoftirqd/019:34:360
28654997534,11cyclictest4145-21ssh23:03:560
28654997530,12cyclictest181rcu_preempt23:18:540
28654997523,24cyclictest15-21ksoftirqd/020:16:440
28654997517,35cyclictest15-21ksoftirqd/020:49:130
2866599745,29cyclictest39-21ksoftirqd/220:35:152
28665997439,26cyclictest8555-21grep19:43:472
28665997428,32cyclictest13346-21ssh22:23:272
28665997428,32cyclictest13346-21ssh22:23:272
28658997449,18cyclictest18657-21sh22:33:161
28658997449,18cyclictest18657-21sh22:33:161
28654997455,14cyclictest25527-21kworker/u19:0+events_unbound00:33:230
2865499744,59cyclictest13557-21apt-get22:23:430
28654997440,14cyclictest181rcu_preempt23:35:480
28654997438,8cyclictest181rcu_preempt00:05:460
28654997426,12cyclictest181rcu_preempt22:39:010
2865499742,48cyclictest15-21ksoftirqd/020:05:390
41002730,8sleep20-21swapper/221:08:472
28665997352,16cyclictest28645-21cyclictest23:21:582
28665997333,14cyclictest181rcu_preempt23:23:552
28658997355,12cyclictest207-21systemd-journal19:13:251
28658997351,16cyclictest14649-21/usr/sbin/munin23:23:581
28658997350,16cyclictest58-21kcompactd023:13:361
28658997350,16cyclictest58-21kcompactd023:13:361
2865899733,38cyclictest3998-21grep23:03:501
28658997310,14cyclictest15843-21ssh00:23:391
28658997310,14cyclictest15843-21ssh00:23:391
28654997359,10cyclictest28645-21cyclictest19:48:420
28654997342,9cyclictest181rcu_preempt21:22:260
2865499733,31cyclictest15-21ksoftirqd/021:08:380
28654997321,29cyclictest15-21ksoftirqd/022:43:480
2866599726,48cyclictest39-21ksoftirqd/221:40:312
28665997246,15cyclictest0-21swapper/200:18:582
28665997243,20cyclictest22287-21threads20:24:132
2866599722,43cyclictest39-21ksoftirqd/219:30:152
28658997250,16cyclictest28645-21cyclictest23:30:521
28658997250,16cyclictest28645-21cyclictest23:30:521
28654997256,10cyclictest13549-21ssh00:18:510
28654997244,8cyclictest181rcu_preempt20:03:260
2865499724,44cyclictest15-21ksoftirqd/021:49:380
2865499724,44cyclictest15-21ksoftirqd/021:49:380
28665997153,13cyclictest28645-21cyclictest20:58:572
28665997153,12cyclictest355-21runrttasks20:54:022
28665997143,20cyclictest24593-21apt-key22:43:292
28665997134,7cyclictest181rcu_preempt23:05:172
2866599713,47cyclictest1784-21/usr/sbin/munin23:59:042
2866599713,46cyclictest4890-21fschecks_time00:03:472
28665997126,13cyclictest181rcu_preempt22:43:242
2866599712,36cyclictest39-21ksoftirqd/222:33:522
28665997117,12cyclictest0-21swapper/220:13:372
28665997110,36cyclictest39-21ksoftirqd/222:28:572
28665997110,36cyclictest39-21ksoftirqd/222:28:572
28658997152,14cyclictest27652-21sh23:46:331
28658997149,12cyclictest0-21swapper/120:13:231
28658997147,16cyclictest0-21swapper/123:08:461
28658997130,16cyclictest9862-21latency_hist19:48:271
28665997051,14cyclictest28645-21cyclictest22:13:272
28665997051,14cyclictest28645-21cyclictest22:13:272
2866599704,22cyclictest39-21ksoftirqd/222:03:452
2866599703,51cyclictest9348-21fschecks_time23:13:502
2866599703,51cyclictest9348-21fschecks_time23:13:502
2866599703,46cyclictest26272-21latency23:43:502
28665997033,21cyclictest0-21swapper/222:24:042
2866599702,20cyclictest39-21ksoftirqd/219:19:592
28658997052,10cyclictest0-21swapper/123:33:571
28658997049,15cyclictest28645-21cyclictest20:53:541
28658997049,15cyclictest25406-21sh23:43:081
28658997035,12cyclictest181rcu_preempt21:03:271
28658997032,25cyclictest14906-21rm00:21:201
28658997032,14cyclictest11596-21latency_hist19:53:271
28658997025,9cyclictest181rcu_preempt20:29:131
28654997041,25cyclictest58-21kcompactd019:13:290
28654997041,25cyclictest58-21kcompactd019:13:290
2865499702,39cyclictest15-21ksoftirqd/019:38:520
28654997022,13cyclictest181rcu_preempt00:27:210
28654997022,13cyclictest181rcu_preempt00:27:210
139652700,7sleep30-21swapper/323:22:103
28665996953,9cyclictest0-21swapper/221:48:302
28665996953,9cyclictest0-21swapper/221:48:302
2866599694,46cyclictest13038-21rm00:18:272
2865899695,26cyclictest7840-21apt-get22:13:361
2865899695,26cyclictest7840-21apt-get22:13:361
28654996954,10cyclictest28645-21cyclictest23:47:440
28665996852,10cyclictest1454-21sendmail23:58:232
28665996852,10cyclictest1454-21sendmail23:58:232
2866599682,44cyclictest39-21ksoftirqd/221:46:232
28658996849,10cyclictest0-21swapper/121:19:011
28658996846,15cyclictest6591-21latency_hist21:13:271
28658996837,6cyclictest181rcu_preempt20:33:501
28658996837,14cyclictest0-21swapper/120:08:291
2865499682,36cyclictest16261-21/usr/sbin/munin22:28:480
2865499682,36cyclictest16261-21/usr/sbin/munin22:28:480
28654996821,13cyclictest181rcu_preempt19:21:200
28654996811,33cyclictest15-21ksoftirqd/021:03:270
28665996749,11cyclictest10005-21systemd-run00:13:242
2866599674,44cyclictest14959-21latency_hist20:03:292
2866599673,42cyclictest27695-21latency_hist20:43:262
2866599673,42cyclictest27695-21latency_hist20:43:262
2866599673,42cyclictest16033-21fschecks_count00:23:482
2866599673,42cyclictest16033-21fschecks_count00:23:482
2866599672,47cyclictest18081-21latency_hist20:13:262
28658996750,9cyclictest0-21swapper/122:23:381
28658996741,18cyclictest3801-21idleruntime19:28:491
28658996741,17cyclictest0-21swapper/119:59:161
2865899672,31cyclictest10367-21ssh00:13:421
2867199664,32cyclictest0-21swapper/321:03:383
28665996645,15cyclictest7507-21sh21:14:002
28665996645,15cyclictest7507-21sh21:14:002
2866599663,40cyclictest6275-21/usr/sbin/munin23:08:522
2866599663,39cyclictest30646-21fschecks_time22:53:462
2866599663,37cyclictest355-21runrttasks00:29:402
2866599661,25cyclictest39-21ksoftirqd/221:28:382
28665996610,27cyclictest39-21ksoftirqd/221:25:202
2865899663,29cyclictest14363-21latency_hist23:23:261
28658996622,13cyclictest181rcu_preempt20:48:481
28658996616,9cyclictest181rcu_preempt21:53:441
28654996638,11cyclictest181rcu_preempt20:13:280
28654996635,8cyclictest181rcu_preempt20:54:150
2866599656,36cyclictest39-21ksoftirqd/219:24:042
2866599653,32cyclictest39-21ksoftirqd/219:35:462
28665996531,11cyclictest181rcu_preempt21:54:212
28658996548,10cyclictest0-21swapper/121:58:491
2865899652,34cyclictest355-21runrttasks19:41:241
2865899652,22cyclictest9500-21latency_hist21:18:281
2865899652,22cyclictest9500-21latency_hist21:18:281
28658996522,28cyclictest8232-21cat00:08:591
28654996545,8cyclictest181rcu_preempt19:12:520
2866599643,41cyclictest355-21runrttasks21:19:272
28665996432,11cyclictest181rcu_preempt21:03:422
2866599642,28cyclictest39-21ksoftirqd/223:33:452
28658996449,8cyclictest0-21swapper/123:49:001
28658996440,14cyclictest0-21swapper/122:09:011
28658996440,14cyclictest0-21swapper/122:09:011
28654996438,18cyclictest26571-21fschecks_count20:38:540
28654996438,18cyclictest26571-21fschecks_count20:38:540
2867199633,28cyclictest0-21swapper/322:24:093
2866599633,34cyclictest29443-21/usr/sbin/munin20:48:532
2866599633,26cyclictest39-21ksoftirqd/219:48:482
28658996345,12cyclictest13307-21apt-key22:23:281
2865499633,29cyclictest20151-21ssh22:34:080
28665996219,32cyclictest28840-21timerwakeupswit20:44:142
2865899624,30cyclictest22847-21modprobe20:28:231
2865899624,23cyclictest21407-21fschecks_time00:33:461
2865899623,46cyclictest29516-21grep20:48:431
28658996142,9cyclictest0-21swapper/122:18:481
2867199572,40cyclictest23585-21chrt20:29:073
2867199572,29cyclictest0-21swapper/322:43:503
2867199563,41cyclictest11461-21chrt22:19:013
2867199563,41cyclictest11461-21chrt22:19:013
2867199552,40cyclictest11450-21chrt19:53:263
2867199544,38cyclictest1721-21chrt22:02:193
2867199542,34cyclictest4623-21chrt21:09:003
2867199533,35cyclictest895-21kworker/3:1+events20:24:053
2867199532,40cyclictest21247-21chrt23:34:053
2867199523,40cyclictest895-21kworker/3:1+events21:18:423
2867199523,37cyclictest7358-21taskset00:08:273
2867199523,33cyclictest19916-21taskset00:30:323
2867199523,32cyclictest5232-21chrt19:33:423
2867199523,32cyclictest28291-21chrt20:44:013
2867199522,37cyclictest31121-21taskset21:56:313
2867199522,33cyclictest12025-21taskset19:54:403
28671995212,29cyclictest0-21swapper/321:49:053
28671995212,29cyclictest0-21swapper/321:49:053
2867199513,33cyclictest16824-21taskset23:27:253
2867199512,37cyclictest30022-21chrt23:50:113
2867199512,37cyclictest21747-21chrt22:38:323
2867199512,37cyclictest1946-21chrt23:58:423
2867199512,33cyclictest8152-21chrt19:43:263
2867199512,32cyclictest32725-21taskset20:58:273
2867199512,32cyclictest15163-21chrt20:03:433
2867199503,36cyclictest16484-21taskset00:24:003
2867199503,36cyclictest16484-21taskset00:24:003
2867199503,34cyclictest8238-21chrt23:11:483
2867199503,34cyclictest15506-21chrt21:28:503
2867199502,36cyclictest23147-21chrt21:42:533
2867199502,32cyclictest5112-21chrt22:08:353
2867199502,32cyclictest5112-21chrt22:08:353
2867199502,31cyclictest7956-21taskset22:13:423
2867199502,31cyclictest7956-21taskset22:13:423
2867199483,34cyclictest31013-21chrt22:53:563
2867199483,33cyclictest18759-21chrt20:14:003
2867199483,25cyclictest3538-21chrt23:03:373
2867199482,34cyclictest26142-21chrt20:38:263
2867199482,31cyclictest9716-21chrt19:48:243
2867199482,30cyclictest26608-21chrt20:38:553
2867199482,30cyclictest26608-21chrt20:38:553
2867199473,33cyclictest8343-21chrt21:15:373
2867199473,33cyclictest8343-21chrt21:15:373
2867199473,32cyclictest23793-21chrt21:43:423
2867199473,29cyclictest19292-21chrt22:33:453
2867199463,31cyclictest29004-21chrt22:50:033
2867199463,31cyclictest29004-21chrt22:50:033
2867199463,30cyclictest13594-21taskset19:58:453
2867199462,31cyclictest11117-21taskset23:17:033
2867199462,31cyclictest11117-21taskset23:17:033
154932463,8sleep30-21swapper/300:23:243
2867199452,29cyclictest756-21taskset22:58:383
2867199452,27cyclictest16262-21taskset22:28:353
2867199452,27cyclictest16262-21taskset22:28:353
2867199443,31cyclictest25533-21taskset23:43:263
2867199442,31cyclictest27094-21taskset23:44:433
2867199442,27cyclictest895-21kworker/3:1+events19:08:363
2867199442,27cyclictest21661-21chrt00:33:533
2867199433,28cyclictest10400-21chrt00:13:423
2867199433,27cyclictest20101-21taskset20:18:543
2867199433,27cyclictest20101-21taskset20:18:543
2867199433,26cyclictest19558-21taskset23:32:193
2867199433,26cyclictest19558-21taskset23:32:193
2867199432,33cyclictest895-21kworker/3:1+events21:27:463
2867199432,23cyclictest895-21kworker/3:1+events19:33:273
2867199422,31cyclictest1770-21taskset21:03:253
2867199413,26cyclictest8854-21chrt00:09:533
2867199412,27cyclictest16770-21chrt20:08:453
2867199402,29cyclictest505-21taskset19:18:523
2867199393,26cyclictest31403-21taskset19:13:583
2867199393,26cyclictest31403-21taskset19:13:583
2867199383,25cyclictest30654-21taskset20:50:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional