You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-22 - 16:49
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8.osadl.org (updated Sat Mar 22, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15149102960,1sleep30-21swapper/311:17:423
1194630992824,2cyclictest1258662-21kworker/u8:0-nfsiod09:20:001
1194634992724,2cyclictest1258620-21sh08:00:012
1194634992724,2cyclictest1258620-21sh08:00:002
1194636992615,1cyclictest0-21swapper/310:50:003
1194636992522,1cyclictest0-21swapper/312:30:053
1194634992421,2cyclictest1359781-21cron09:24:592
1194630992320,2cyclictest1524533-21idleruntime-cro11:24:591
1194630992320,2cyclictest1233897-21sendmail07:40:001
1194628992320,2cyclictest0-21swapper/011:20:000
1194636992219,1cyclictest0-21swapper/307:55:013
1194634992219,2cyclictest1516095-21systemd-run11:20:012
1194630992219,2cyclictest1278132-21/usr/sbin/munin08:20:001
1194628992219,1cyclictest0-21swapper/010:45:000
1194636992118,1cyclictest0-21swapper/310:30:123
1194636992118,1cyclictest0-21swapper/310:30:003
1194634992118,2cyclictest0-21swapper/210:20:002
1194634992118,1cyclictest0-21swapper/212:30:012
119463499211,11cyclictest1303330-21apt08:40:002
1194630992118,2cyclictest1303328-21latency_hist08:40:011
1194630992117,3cyclictest1417163-21systemd-run10:10:001
1194630992116,4cyclictest0-21swapper/109:00:001
1194634992017,2cyclictest1497042-21cron11:05:002
1194634992017,2cyclictest0-21swapper/210:45:012
1194630992016,3cyclictest1527517-21cpuspeed_turbos11:30:081
1194628992016,2cyclictest0-21swapper/009:25:000
1194636991916,1cyclictest0-21swapper/311:40:243
1194636991915,2cyclictest0-21swapper/311:14:593
1194634991916,2cyclictest1579464-21munin-plugin-st12:05:002
1194634991916,2cyclictest0-21swapper/208:45:012
1194634991915,2cyclictest1345115-21kthreadcore09:15:192
1194630991916,2cyclictest1309937-21latency_hist08:45:021
1194630991915,2cyclictest0-21swapper/108:10:211
1194630991914,4cyclictest0-21swapper/107:14:591
1194628991916,2cyclictest0-21swapper/007:15:000
1194636991815,1cyclictest0-21swapper/310:15:003
1194636991815,1cyclictest0-21swapper/308:30:183
1194636991813,2cyclictest1554677-21turbostat11:50:013
1194634991815,2cyclictest1536052-21cpuspeed_turbos11:35:142
1194634991815,2cyclictest0-21swapper/209:40:012
1194634991814,3cyclictest1458766-21unixbench-2d10:35:232
1194630991815,2cyclictest3564018-21systemctl07:15:031
1194630991815,2cyclictest1488768-21cron11:00:011
1194630991815,2cyclictest1403344-21needreboot09:55:201
1194630991815,2cyclictest1335131-21diskmemload10:15:201
1194628991815,2cyclictest1297224-21latency_hist08:35:010
1194628991815,2cyclictest0-21swapper/008:00:000
1194628991815,2cyclictest0-21swapper/008:00:000
1194636991714,1cyclictest0-21swapper/312:09:303
1194636991714,1cyclictest0-21swapper/311:35:003
1194636991714,1cyclictest0-21swapper/310:40:193
1194636991714,1cyclictest0-21swapper/310:20:213
1194636991714,1cyclictest0-21swapper/309:30:123
1194636991714,1cyclictest0-21swapper/307:15:003
1194636991712,2cyclictest0-21swapper/309:55:003
1194636991712,2cyclictest0-21swapper/307:35:003
1194634991714,2cyclictest1527651-21cut11:30:102
1194634991714,2cyclictest1481044-21tr10:55:132
1194634991713,3cyclictest1272566-21idleruntime08:15:172
1194634991712,3cyclictest0-21swapper/209:40:202
119463499170,15cyclictest1233889-21grep07:40:002
1194628991714,2cyclictest508-21polkitd09:55:010
1194628991714,2cyclictest1552103-21ssh11:45:010
1194628991714,2cyclictest1434310-21fschecks_count10:20:180
1194628991714,2cyclictest1386793-21kworker/u8:0-nfsiod11:00:020
1194628991714,2cyclictest1335392-21kworker/u8:3-rpciod10:50:010
1194628991714,2cyclictest1278346-21latency_hist08:20:000
1194628991712,4cyclictest1590968-21tail12:15:140
1194628991712,4cyclictest0-21swapper/008:10:290
1194628991712,1cyclictest191rcu_preempt10:40:000
119462899170,16cyclictest1322765-21awk09:00:010
1194636991613,1cyclictest0-21swapper/312:05:003
1194636991613,1cyclictest0-21swapper/310:39:123
1194636991613,1cyclictest0-21swapper/310:00:013
1194636991613,1cyclictest0-21swapper/309:45:503
1194636991613,1cyclictest0-21swapper/309:40:193
1194636991613,1cyclictest0-21swapper/309:28:313
1194636991613,1cyclictest0-21swapper/309:20:133
1194636991613,1cyclictest0-21swapper/309:19:383
1194636991613,1cyclictest0-21swapper/308:52:373
1194636991613,1cyclictest0-21swapper/308:38:293
1194636991613,1cyclictest0-21swapper/308:27:473
1194636991613,1cyclictest0-21swapper/308:10:473
1194636991613,1cyclictest0-21swapper/307:43:043
1194636991613,1cyclictest0-21swapper/307:26:423
1194636991613,1cyclictest0-21swapper/307:18:103
119463499162,6cyclictest1499597-21munin-run11:09:592
1194634991614,1cyclictest0-21swapper/211:14:592
1194634991613,2cyclictest522-21in:imuxsock12:10:012
1194634991613,2cyclictest1399740-21kthreadcore09:55:182
1194634991613,2cyclictest1251618-21sshd07:50:282
1194634991613,2cyclictest1221451-21systemd-run07:30:002
1194634991613,1cyclictest0-21swapper/210:50:002
1194634991612,3cyclictest1623011-21latency12:35:172
1194634991612,3cyclictest1600399-21kthreadcore12:20:202
1194634991612,3cyclictest1310402-21cron08:50:002
1194630991613,2cyclictest1452878-21cron10:35:011
1194630991613,2cyclictest1215260-21cut07:25:011
1194630991612,3cyclictest1469487-21apt-key10:45:011
119463099160,15cyclictest1194623-21cyclictest07:25:131
119462899167,8cyclictest0-21swapper/010:15:140
1194628991614,1cyclictest1425545-21latency_hist10:15:010
1194628991614,1cyclictest1414578-21sh10:05:010
1194628991614,1cyclictest1335131-21diskmemload09:30:000
1194628991613,2cyclictest1445329-21gltestperf10:30:100
1194628991613,2cyclictest1363645-21cut09:30:110
1194628991613,2cyclictest1303316-21cron08:40:000
1194628991613,2cyclictest1194633-21latency_hist07:10:010
1194628991613,2cyclictest0-21swapper/009:10:000
1194628991613,2cyclictest0-21swapper/007:45:000
12010562150,12sleep01201724-21tr07:15:020
1194636991512,1cyclictest0-21swapper/312:28:043
1194636991512,1cyclictest0-21swapper/312:15:193
1194636991512,1cyclictest0-21swapper/312:11:333
1194636991512,1cyclictest0-21swapper/311:55:253
1194636991512,1cyclictest0-21swapper/311:55:243
1194636991512,1cyclictest0-21swapper/311:50:173
1194636991512,1cyclictest0-21swapper/311:36:003
1194636991512,1cyclictest0-21swapper/311:25:043
1194636991512,1cyclictest0-21swapper/311:21:173
1194636991512,1cyclictest0-21swapper/311:05:383
1194636991512,1cyclictest0-21swapper/311:01:033
1194636991512,1cyclictest0-21swapper/310:50:273
1194636991512,1cyclictest0-21swapper/310:15:183
1194636991512,1cyclictest0-21swapper/310:00:023
1194636991512,1cyclictest0-21swapper/309:05:193
1194636991512,1cyclictest0-21swapper/308:55:273
1194636991512,1cyclictest0-21swapper/308:45:113
1194636991512,1cyclictest0-21swapper/308:40:233
1194636991512,1cyclictest0-21swapper/308:21:093
1194636991512,1cyclictest0-21swapper/308:05:143
1194636991512,1cyclictest0-21swapper/308:00:063
1194636991512,1cyclictest0-21swapper/307:45:083
1194636991512,1cyclictest0-21swapper/307:35:433
1194636991512,1cyclictest0-21swapper/307:21:203
119463499159,4cyclictest1284973-21df_inode08:25:142
1194634991513,1cyclictest0-21swapper/207:20:002
119463499151,2cyclictest12926231kthreadcore08:30:222
1194634991512,2cyclictest1544440-21fschecks_time11:40:152
1194634991512,2cyclictest1450396-21timerandwakeup10:30:192
1194634991512,2cyclictest1420579-21kthreadcore10:10:132
1194634991512,2cyclictest1241716-21kthreadcore07:45:182
1194634991512,2cyclictest0-21swapper/209:05:012
1194634991512,1cyclictest0-21swapper/212:10:112
1194634991511,3cyclictest1614899-21ntp_kernel_err12:30:152
1194634991511,3cyclictest1398107-21latency_hist09:55:002
1194634991511,3cyclictest1281595-21kthreadcore08:20:212
1194634991511,2cyclictest0-21swapper/208:55:022
1194630991512,2cyclictest1609758-21/usr/sbin/munin12:30:151
1194630991512,2cyclictest1576686-21ntp_kernel_pll_12:00:201
1194630991512,2cyclictest1260266-21kernelversion08:00:171
1194630991512,2cyclictest0-21swapper/111:20:011
1194630991511,2cyclictest1335392-21kworker/u8:3-nfsiod10:20:221
1194630991511,2cyclictest0-21swapper/110:30:001
1194630991510,4cyclictest0-21swapper/109:25:001
119463099150,2cyclictest1417550-21grep10:10:091
119462899159,2cyclictest191rcu_preempt12:00:010
119462899159,2cyclictest191rcu_preempt12:00:000
119462899159,2cyclictest191rcu_preempt07:50:000
119462899159,1cyclictest191rcu_preempt10:10:000
1194628991512,2cyclictest514-21systemd-logind08:50:000
1194628991512,2cyclictest1523634-21kworker/0:1+kacpid11:26:130
1194628991512,2cyclictest1340628-21ntp_offset09:10:230
1194628991512,2cyclictest1335131-21diskmemload10:28:200
1194628991512,2cyclictest0-21swapper/011:35:000
1194628991511,3cyclictest1349292-21unixbench_multi09:15:260
1194628991511,3cyclictest1194623-21cyclictest09:40:460
1194636991412,1cyclictest0-21swapper/312:35:093
1194636991412,1cyclictest0-21swapper/312:20:023
1194636991412,1cyclictest0-21swapper/310:05:053
1194636991412,1cyclictest0-21swapper/309:10:143
1194636991412,1cyclictest0-21swapper/309:00:023
1194636991412,1cyclictest0-21swapper/307:55:033
1194636991412,1cyclictest0-21swapper/307:55:023
1194636991411,1cyclictest0-21swapper/310:55:183
1194636991411,1cyclictest0-21swapper/309:35:203
1194636991411,1cyclictest0-21swapper/308:15:273
119463499148,4cyclictest1563386-21expr11:55:162
119463499148,4cyclictest1563386-21expr11:55:162
119463499146,6cyclictest421rcuc/208:10:222
1194634991412,1cyclictest1518332-21kthreadcore11:20:172
1194634991411,2cyclictest1562877-21latency_hist11:55:002
1194634991411,2cyclictest1435362-21irqrtprio10:20:202
1194634991411,2cyclictest1259369-21irqcore08:00:162
1194634991410,3cyclictest1476412-21kthreadcore10:50:132
1194634991410,3cyclictest1407002-21idleruntime10:00:152
119463099149,4cyclictest1503577-21kthreadcore11:10:131
119463099149,4cyclictest0-21swapper/108:30:011
1194630991412,1cyclictest1335131-21diskmemload12:13:191
1194630991411,2cyclictest1602754-21kthreadcore12:20:211
1194630991411,2cyclictest1562885-21/usr/sbin/munin11:55:001
1194630991411,2cyclictest1411582-21missed_timers10:00:211
1194630991411,2cyclictest1336438-21irqrtprio09:10:191
1194630991411,2cyclictest1335131-21diskmemload11:55:271
1194630991411,2cyclictest1335131-21diskmemload11:55:261
1194630991411,2cyclictest1323306-21fschecks_time09:00:141
1194630991411,2cyclictest1296459-21switchtime08:30:271
1194630991411,2cyclictest1246621-21cpuspeed_turbos07:50:161
1194630991411,2cyclictest1245778-21unixbench-2d07:45:251
1194630991411,2cyclictest0-21swapper/111:35:001
1194630991411,2cyclictest0-21swapper/110:50:091
1194630991411,2cyclictest0-21swapper/109:50:191
1194630991411,2cyclictest0-21swapper/108:50:171
1194630991410,3cyclictest1-21systemd08:07:591
1194630991410,2cyclictest0-21swapper/110:35:171
1194630991410,2cyclictest0-21swapper/109:40:231
119463099140,7cyclictest1606759-21cron12:25:001
119462899148,4cyclictest501-21dbus-daemon12:25:000
1194628991412,1cyclictest1341896-21kworker/0:1+kacpid09:35:370
1194628991412,1cyclictest1221413-21cron07:30:000
1194628991412,1cyclictest1208738-21kworker/0:0+kacpid08:50:530
1194628991412,1cyclictest0-21swapper/009:05:010
1194628991411,2cyclictest1618602-21irqcore12:35:140
1194628991411,2cyclictest1556398-21kthreadcore11:50:140
1194628991411,2cyclictest1518287-21kthreadcore11:20:170
1194628991411,2cyclictest1478458-21kworker/u8:2-rpciod11:14:590
1194628991411,2cyclictest1281521-21kthreadcore08:20:200
1194628991411,2cyclictest1262187-21kthreadcore08:00:190
1194628991411,2cyclictest0-21swapper/011:00:130
1194628991411,2cyclictest0-21swapper/010:50:100
1194628991410,3cyclictest1523634-21kworker/0:1+kacpi_notify12:05:220
119463499137,4cyclictest1199889-21ntpq07:10:212
119463499133,3cyclictest0-21swapper/207:40:192
119463499131,2cyclictest191rcu_preempt07:25:002
1194634991310,2cyclictest1224639-21kthreadcore07:30:222
1194634991310,2cyclictest0-21swapper/210:10:002
1194634991310,2cyclictest0-21swapper/209:10:262
119463099139,3cyclictest1335131-21diskmemload12:15:061
119463099139,2cyclictest0-21swapper/110:45:011
119463099137,4cyclictest3561480-21systemd-journal09:25:011
119463099137,2cyclictest191rcu_preempt11:40:171
119463099133,2cyclictest1554671-21sh11:50:001
1194630991311,1cyclictest1335131-21diskmemload11:08:181
1194630991310,2cyclictest1618147-21cpuspeed_turbos12:35:111
1194630991310,2cyclictest1374348-21kthreadcore09:35:201
1194630991310,2cyclictest1335131-21diskmemload09:30:181
1194630991310,2cyclictest1334628-21unixbench-2d09:05:241
1194630991310,2cyclictest1239608-21unixbench_multi07:40:261
119463099130,3cyclictest277-21plymouthd11:05:001
119463099130,3cyclictest2666988-21kworker/u8:1+rpciod11:26:221
119462899139,2cyclictest0-21swapper/007:50:150
119462899138,4cyclictest3561480-21systemd-journal08:45:000
1194628991311,1cyclictest1590302-21kworker/0:2+kacpid12:27:030
1194628991311,1cyclictest1386793-21kworker/u8:0-rpciod12:03:500
1194628991311,1cyclictest0-21swapper/011:35:150
1194628991310,2cyclictest1590302-21kworker/0:2+kacpid12:30:410
1194628991310,2cyclictest1523634-21kworker/0:1+kacpid11:45:040
1194628991310,2cyclictest1398682-21gltestperf09:55:150
1194628991310,2cyclictest1335131-21diskmemload11:05:020
1194628991310,2cyclictest1285886-21irqrtprio08:25:170
1194628991310,2cyclictest1208738-21kworker/0:0+kacpid07:32:510
119463499129,2cyclictest1335131-21diskmemload09:30:182
119463499129,2cyclictest0-21swapper/209:30:012
119463499129,2cyclictest0-21swapper/209:05:202
119463499126,4cyclictest501-21dbus-daemon09:00:012
119463499125,5cyclictest1-21systemd08:10:002
119463499123,4cyclictest1443207-21ssh10:27:032
119463499122,2cyclictest191rcu_preempt12:15:132
119463499120,10cyclictest1389780-21cron09:50:012
119463099129,2cyclictest1310527-21latency_hist08:50:011
119463099129,2cyclictest1224796-21kthreadcore07:30:221
1194630991210,1cyclictest1335131-21diskmemload12:05:481
1194630991210,1cyclictest1335131-21diskmemload09:46:521
119462899129,2cyclictest1335131-21diskmemload12:11:370
1194628991210,1cyclictest1341896-21kworker/0:1+kacpid09:47:440
1194628991210,1cyclictest1208738-21kworker/0:0+kacpid08:05:370
1194628991210,1cyclictest1208738-21kworker/0:0+kacpid07:35:310
1194628991210,1cyclictest1208738-21kworker/0:0+kacpid07:20:140
119463499114,6cyclictest2666988-21kworker/u8:1+nfsiod11:49:542
119463499110,3cyclictest1524885-21ssh11:25:182
119463099119,1cyclictest0-21swapper/108:20:211
119463099110,1cyclictest0-21swapper/107:55:021
119463099110,1cyclictest0-21swapper/107:55:021
11886332110,5sleep11188666-21apt-get07:05:131
11946349994,3cyclictest1194709-21date07:10:022
1194479280,2sleep30-21swapper/307:09:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional