You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 14:16
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Fri Dec 12, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21715212010,88sleep00-21swapper/019:03:290
230319911649,22cyclictest23008-21cyclictest23:04:102
230319911649,22cyclictest23008-21cyclictest23:04:102
2264221066,16sleep20-21swapper/219:04:582
230199910342,12cyclictest0-21swapper/122:14:291
230199910238,19cyclictest31677-21modprobe19:33:041
230199910216,34cyclictest26403-21date23:53:071
230199910216,34cyclictest26403-21date23:53:071
230199910172,20cyclictest9344-21latency_hist20:03:081
230199910133,18cyclictest0-21swapper/123:58:311
23019999939,12cyclictest0-21swapper/100:18:071
23019999935,15cyclictest0-21swapper/123:03:081
23015999929,31cyclictest0-21swapper/023:23:200
23019999735,12cyclictest0-21swapper/121:09:051
23019999637,12cyclictest0-21swapper/121:18:181
23015999630,29cyclictest0-21swapper/023:13:180
2301999953,41cyclictest11599-21fschecks_count20:09:431
23015999421,35cyclictest0-21swapper/000:28:190
2301999933,37cyclictest12269-21idleruntime-cro21:33:031
2301999933,37cyclictest12269-21idleruntime-cro21:33:031
23019999320,31cyclictest28205-21switchtime20:58:411
2301999924,34cyclictest29574-21ntp_kernel_err21:03:371
2301999924,34cyclictest29574-21ntp_kernel_err21:03:371
23015999224,32cyclictest6192-21wc00:13:420
23019999141,12cyclictest14256-21latency_hist20:18:081
23015999149,33cyclictest20961-21ssh23:43:200
23015999118,36cyclictest4006-21kworker/u17:1+events_unbound20:58:040
23031999044,10cyclictest181rcu_preempt20:13:202
23015999021,29cyclictest20122-21grep22:43:370
23015999021,29cyclictest20122-21grep22:43:370
23015999014,44cyclictest31144-21rm22:06:210
2303199899,62cyclictest2725-21ssh22:13:412
23019998966,17cyclictest19318-21latency_hist22:43:081
23019998949,15cyclictest20234-21ssh22:43:381
23019998949,15cyclictest20234-21ssh22:43:381
23015998915,30cyclictest0-21swapper/019:33:220
23015998847,32cyclictest30425-21diskmemload00:25:220
23015998847,32cyclictest30425-21diskmemload00:25:220
23019998762,18cyclictest23471-21timerwakeupswit20:43:431
23015998716,36cyclictest2039-21apt-get22:13:100
23015998666,12cyclictest30571-21apt-key23:03:060
23015998622,27cyclictest20742-21latency_hist23:43:060
23015998620,47cyclictest7350-21fschecks_time21:23:280
2303199857,55cyclictest28735-21apt-get19:23:352
23019998549,25cyclictest30326-21grep22:03:461
2301999852,47cyclictest30932-21apt-get23:03:211
2301999852,47cyclictest30932-21apt-get23:03:211
2301599854,58cyclictest11294-21apt-get20:08:190
23015998527,28cyclictest0-21swapper/020:18:420
23015998524,29cyclictest0-21swapper/020:28:050
23015998524,29cyclictest0-21swapper/020:28:050
23015998522,31cyclictest27067-21fschecks_count19:18:280
23015998522,28cyclictest9778-21sh22:25:320
23015998516,29cyclictest29285-21cat23:58:070
23015998514,20cyclictest0-21swapper/023:03:160
23015998514,20cyclictest0-21swapper/023:03:160
23031998458,18cyclictest18421-21fschecks_time23:38:252
23019998453,23cyclictest16361-21latency_hist00:33:061
23019998451,8cyclictest181rcu_preempt23:13:211
23015998461,15cyclictest207-21systemd-journal22:50:320
23015998418,25cyclictest14806-21ssh22:33:430
23015998417,31cyclictest30018-21/usr/sbin/munin23:58:340
23031998357,17cyclictest30425-21diskmemload21:38:432
23015998336,34cyclictest18881-21/usr/sbin/munin21:43:390
23015998336,34cyclictest18881-21/usr/sbin/munin21:43:390
23015998323,33cyclictest1810-21ssh23:08:340
23015998310,49cyclictest207-21systemd-journal00:19:320
23031998262,15cyclictest23008-21cyclictest21:20:512
23031998245,9cyclictest181rcu_preempt22:03:242
23031998245,11cyclictest15090-21latency_hist23:33:052
23019998255,18cyclictest2460-21latency_hist00:08:061
23019998254,20cyclictest31045-21kworker/u17:3+events_unbound21:15:041
2301599826,29cyclictest0-21swapper/022:33:070
2301599826,29cyclictest0-21swapper/022:33:070
2301599823,56cyclictest30979-21apt-get21:08:230
23015998226,21cyclictest30484-21dpkg21:08:050
23015998226,21cyclictest30484-21dpkg21:08:050
23015998217,31cyclictest26386-21latency_hist23:53:060
23015998217,31cyclictest26386-21latency_hist23:53:060
23031998164,10cyclictest15426-21cpuspeed23:33:212
23031998151,21cyclictest25898-21ls20:53:212
2301599818,31cyclictest0-21swapper/021:18:330
23015998157,15cyclictest10794-21latency_hist20:08:060
23015998121,23cyclictest20289-21grep20:33:440
23015998120,42cyclictest0-21swapper/019:58:400
23015998120,24cyclictest17500-21ssh21:42:020
23015998118,23cyclictest32479-21ssh22:08:300
23015998117,26cyclictest25012-21wc19:13:120
23015998117,26cyclictest25012-21wc19:13:120
23015998113,31cyclictest17917-21cat23:38:060
23015998113,29cyclictest25762-21processes22:53:390
23015998113,29cyclictest25762-21processes22:53:390
23015998111,33cyclictest4097-21latency_hist21:18:080
2301599809,38cyclictest0-21swapper/020:28:210
2301599809,38cyclictest0-21swapper/020:28:210
23015998016,23cyclictest0-21swapper/021:33:200
23015998015,39cyclictest9820-21apt21:28:160
23015998015,39cyclictest9820-21apt21:28:160
23015998011,27cyclictest0-21swapper/022:18:320
23031997963,11cyclictest23008-21cyclictest21:09:282
23031997941,15cyclictest10831-21ssh23:23:432
23019997948,8cyclictest0-21swapper/120:18:271
23015997957,14cyclictest17276-21ssh00:33:420
23015997955,15cyclictest29166-21latency_hist22:03:080
23015997922,20cyclictest7581-21sendmail-msp19:58:040
23015997921,26cyclictest3424-21tr00:08:400
23015997921,26cyclictest3424-21tr00:08:400
23015997918,30cyclictest16730-21entropy22:38:250
23015997916,35cyclictest0-21swapper/021:48:440
23015997916,35cyclictest0-21swapper/021:48:440
23031997840,10cyclictest181rcu_preempt00:18:342
23019997832,14cyclictest181rcu_preempt00:26:001
23019997832,14cyclictest181rcu_preempt00:26:001
23019997831,11cyclictest181rcu_preempt21:23:221
2301599788,28cyclictest0-21swapper/023:28:360
23015997823,25cyclictest23688-21taskset19:08:330
23015997820,36cyclictest1354-21/usr/sbin/munin19:39:150
23031997761,10cyclictest23008-21cyclictest00:08:442
23031997761,10cyclictest23008-21cyclictest00:08:442
23031997755,13cyclictest23008-21cyclictest19:28:322
23019997757,14cyclictest23008-21cyclictest19:24:401
23019997753,16cyclictest13386-21cron22:33:041
23019997753,16cyclictest13386-21cron22:33:041
23015997736,26cyclictest30344-21apt-get19:28:160
23015997717,33cyclictest0-21swapper/021:57:050
23015997717,25cyclictest23342-21switchtime20:43:410
23015997715,25cyclictest0-21swapper/020:58:230
23015997714,27cyclictest7543-21ssh23:18:350
23015997714,27cyclictest7543-21ssh23:18:350
223262779,42sleep122408-21cat19:03:511
23031997650,17cyclictest9467-21unin-run23:23:052
23031997650,17cyclictest9467-21unin-run23:23:052
2303199764,56cyclictest29216-21latency_hist23:58:062
23015997619,23cyclictest32501-21df00:03:240
23015997616,28cyclictest5139-21/usr/sbin/munin19:48:370
23031997524,40cyclictest292-21dbus-daemon20:43:052
23015997511,29cyclictest13100-21fschecks_count20:13:260
23031997452,13cyclictest316-21rs:main2
23031997452,13cyclictest316-21rs:main2
23031997451,16cyclictest23008-21cyclictest21:29:002
23031997451,16cyclictest23008-21cyclictest21:29:002
23031997442,9cyclictest14780-21grep00:28:452
23031997349,15cyclictest691-21cron23:53:042
23031997349,15cyclictest691-21cron23:53:042
23031997326,13cyclictest181rcu_preempt23:58:422
23019997353,15cyclictest23008-21cyclictest20:38:371
23019997349,13cyclictest0-21swapper/122:53:411
23019997349,13cyclictest0-21swapper/122:53:411
23019997338,13cyclictest181rcu_preempt22:28:071
2301999732,39cyclictest20959-21latency_hist20:38:071
23015997316,22cyclictest4334-21munin-run19:48:040
23031997249,13cyclictest0-21swapper/200:33:412
23019997249,15cyclictest15555-21fschecks_count23:33:261
2301999722,44cyclictest18017-21fschecks_time20:28:271
2301999722,44cyclictest18017-21fschecks_time20:28:271
2301599725,58cyclictest28773-21taskset19:23:390
23031997145,19cyclictest2289-21apt-key22:13:072
2303199714,44cyclictest3206-21fschecks_time19:43:262
23031997136,10cyclictest181rcu_preempt20:48:282
23019997153,13cyclictest16866-21sh21:40:021
23019997151,15cyclictest23008-21cyclictest23:33:031
23019997148,12cyclictest0-21swapper/100:18:321
23019997146,17cyclictest23008-21cyclictest19:13:431
23019997146,17cyclictest23008-21cyclictest19:13:431
23019997135,26cyclictest312-21expr19:33:391
2301999713,32cyclictest18597-21latency21:43:311
2301999713,32cyclictest18597-21latency21:43:311
23019997127,33cyclictest27391-21timerwakeupswit21:58:411
23015997111,30cyclictest24310-21cpuspeed20:48:230
2303199704,44cyclictest1363-21/usr/sbin/munin21:13:402
2303199704,39cyclictest10593-21latency_hist22:28:062
2303199704,22cyclictest30574-21latency_hist21:08:062
2303199704,22cyclictest30574-21latency_hist21:08:062
2303199703,47cyclictest4259-21cstates23:13:212
23031997034,14cyclictest11310-21sshd22:28:322
23031997034,14cyclictest11310-21sshd22:28:322
23019997026,30cyclictest16559-21latency20:23:331
23019997026,30cyclictest16559-21latency20:23:331
23015997010,26cyclictest21138-21expr20:38:210
23039996911,38cyclictest0-21swapper/321:08:223
2303199694,43cyclictest12676-21latency_hist20:13:062
2303199693,50cyclictest16094-21latency_hist20:23:082
23031996926,33cyclictest30940-21kworker/u19:3+nfsiod21:24:172
23019996948,15cyclictest0-21swapper/120:53:431
2301999693,16cyclictest20622-21modprobe23:43:031
2301999692,45cyclictest18987-21latency_hist00:38:071
2303999683,51cyclictest1407-21chrt22:11:043
2303999682,54cyclictest25343-21chrt23:50:153
2303999682,54cyclictest25343-21chrt23:50:153
23031996849,13cyclictest5074-21ntp_kernel_err19:48:332
23031996846,13cyclictest0-21swapper/221:38:032
23031996843,7cyclictest0-21swapper/219:13:342
23031996843,7cyclictest0-21swapper/219:13:342
23019996848,14cyclictest7972-21latency_hist22:23:071
23019996846,12cyclictest0-21swapper/122:33:251
23019996835,14cyclictest0-21swapper/119:43:071
23019996828,27cyclictest2518-21date22:13:081
2303999674,50cyclictest4484-21taskset00:11:053
2303999674,50cyclictest4484-21taskset00:11:053
2303999672,50cyclictest27315-21taskset22:57:193
2303999672,50cyclictest27315-21taskset22:57:193
23031996743,15cyclictest32298-21kernelversion19:33:292
23031996734,12cyclictest181rcu_preempt22:53:432
23031996734,12cyclictest181rcu_preempt22:53:432
2303199672,46cyclictest22490-21fschecks_count22:48:302
2303199672,30cyclictest2141750ssh22:46:352
2303199672,30cyclictest2141750ssh22:46:352
23019996748,13cyclictest23426-21sh23:48:031
23019996747,10cyclictest0-21swapper/119:43:251
23019996745,16cyclictest15086-21sh21:38:041
23019996744,17cyclictest10908-21apt-key20:08:061
23019996736,22cyclictest28234-21latency_hist19:23:071
66512660,8sleep20-21swapper/222:19:222
2303999662,51cyclictest32715-21chrt23:05:543
2303999662,51cyclictest32715-21chrt23:05:543
2303999662,48cyclictest15407-21chrt21:38:223
23031996649,11cyclictest0-21swapper/221:53:062
23031996649,11cyclictest0-21swapper/221:53:062
23031996647,11cyclictest381ktimers/221:58:182
23031996646,13cyclictest17774-21cpu20:28:212
23031996646,13cyclictest17774-21cpu20:28:212
2303199662,49cyclictest28866-21latency_hist21:03:062
23019996634,13cyclictest181rcu_preempt23:23:231
2301999663,31cyclictest24501-21fschecks_time20:48:291
23019996628,16cyclictest181rcu_preempt23:12:401
2303999653,48cyclictest26181-21chrt20:53:303
2303999653,48cyclictest1350-21chrt19:38:133
2303999653,47cyclictest29278-21chrt22:03:223
2303999653,47cyclictest25515-21taskset20:52:323
2303999652,50cyclictest22963-21chrt21:51:473
2303999652,50cyclictest22963-21chrt21:51:473
2303999652,47cyclictest3142-21chrt19:43:233
23031996547,11cyclictest369-21ntpd19:08:532
23031996545,15cyclictest17707-21sh22:39:002
23031996542,16cyclictest19384-21cpu20:33:242
23031996541,13cyclictest0-21swapper/223:08:232
2303199653,48cyclictest30417-21apt-key23:03:052
23031996527,13cyclictest181rcu_preempt19:58:312
23031996521,30cyclictest21730-21switchtime23:43:392
23019996547,13cyclictest18271-21kworker/u17:1-xprtiod22:48:411
23019996528,11cyclictest27309-21ssh23:53:401
2287526515,30sleep30-21swapper/319:08:053
2303999643,48cyclictest32284-21taskset00:03:113
2303999643,48cyclictest2850-21taskset21:14:533
2303999643,47cyclictest15810-21chrt20:22:153
2303999643,47cyclictest13379-21taskset23:28:393
2303999643,46cyclictest9671-21taskset00:19:563
2303999643,45cyclictest0-21swapper/323:58:173
2303999642,48cyclictest24155-21taskset22:51:153
2303999642,47cyclictest21007-21chrt20:38:153
2303199644,40cyclictest9679-21fschecks_time20:03:272
23031996441,13cyclictest0-21swapper/222:33:272
23031996439,13cyclictest0-21swapper/221:43:402
23031996439,13cyclictest0-21swapper/221:43:402
2303199643,32cyclictest5839-21cut00:13:322
23019996442,8cyclictest181rcu_preempt19:08:551
2301999642,45cyclictest7055-21cpuspeed23:18:241
2301999642,45cyclictest7055-21cpuspeed23:18:241
2303999633,47cyclictest3099-21chrt22:14:203
2303999633,47cyclictest17127-21chrt23:35:563
2303999633,47cyclictest10070-21taskset22:26:283
2303999632,49cyclictest13724-21taskset00:28:113
2303999632,49cyclictest12741-21chrt00:25:523
2303999632,49cyclictest12741-21chrt00:25:523
2303999632,48cyclictest7173-21chrt00:15:553
2303999632,48cyclictest25708-21chrt21:56:463
2303999632,46cyclictest30274-21taskset21:06:173
2303999632,46cyclictest30274-21taskset21:06:173
2303999632,46cyclictest25833-21chrt19:13:393
2303999632,46cyclictest25833-21chrt19:13:393
23031996340,15cyclictest17678-21latency_hist20:28:072
23031996340,15cyclictest17678-21latency_hist20:28:072
23031996334,9cyclictest0-21swapper/200:06:142
2303199632,47cyclictest22857-21fschecks_time20:43:292
2303999624,46cyclictest21605-21chrt22:47:103
2303999624,46cyclictest21605-21chrt22:47:103
2303999624,46cyclictest11360-21chrt23:25:193
2303999623,47cyclictest17760-21chrt00:34:183
2303999623,46cyclictest7304-21chrt21:23:283
2303999623,46cyclictest18789-21chrt22:42:163
2303999623,45cyclictest27246-21taskset23:53:393
2303999622,48cyclictest30348-21chrt19:28:173
2303999622,48cyclictest18713-21chrt23:38:323
2303999622,47cyclictest7396-21chrt19:56:233
2303999622,47cyclictest5737-21chrt23:15:183
2303999622,47cyclictest28477-21chrt20:58:563
2303999622,47cyclictest24208-21chrt19:08:573
2303999622,46cyclictest21541-21chrt23:43:343
2303199624,45cyclictest1693-21fschecks_time19:39:172
2303199623,47cyclictest6182-21/usr/sbin/munin19:53:252
2303199623,30cyclictest27093-21grep19:18:282
23031996228,13cyclictest181rcu_preempt21:53:442
23019996241,7cyclictest181rcu_preempt00:09:281
23019996241,7cyclictest181rcu_preempt00:09:281
2301999623,32cyclictest355-21runrttasks19:54:401
2303999614,42cyclictest0-21swapper/322:18:173
2303999613,46cyclictest29566-21chrt23:00:313
2303999613,45cyclictest2770-21chrt23:09:553
2303999612,47cyclictest12270-21taskset22:29:323
2303999612,47cyclictest12270-21taskset22:29:323
2303999612,46cyclictest8821-21chrt23:21:103
2303999612,46cyclictest8821-21chrt23:21:103
2303999612,46cyclictest28711-21chrt19:23:113
2303999612,45cyclictest14099-21chrt20:16:163
2303999612,45cyclictest11543-21chrt20:09:403
23019996141,14cyclictest0-21swapper/121:48:401
23019996141,14cyclictest0-21swapper/121:48:401
2303999603,44cyclictest31927-21chrt19:33:113
2303999603,44cyclictest14019-21chrt21:34:503
2303999602,46cyclictest8825-21chrt19:58:423
2303999602,45cyclictest5787-21chrt19:51:213
2303999602,45cyclictest19805-21taskset21:45:323
2303999602,45cyclictest19805-21taskset21:45:323
2303999602,45cyclictest19483-21chrt20:33:253
2303999602,45cyclictest18301-21chrt20:28:353
2303999602,45cyclictest18301-21chrt20:28:353
2303999602,45cyclictest17206-21chrt20:24:303
2303999602,45cyclictest17206-21chrt20:24:303
2303999602,43cyclictest9771-21chrt21:28:093
2303999602,43cyclictest9771-21chrt21:28:093
23019996030,12cyclictest181rcu_preempt21:58:061
23019996021,29cyclictest4988-21latency19:48:311
2303999593,43cyclictest5332-21chrt21:18:463
2303999593,43cyclictest27931-21chrt19:19:223
2303999593,43cyclictest22762-21chrt20:43:253
2303999592,45cyclictest15743-21chrt22:36:413
2303999592,44cyclictest26431-21chrt21:58:123
2303999582,44cyclictest9433-21chrt20:03:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional