You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-30 - 08:21
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8.osadl.org (updated Sat Sep 30, 2023 00:44:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
792171994236,3cyclictest1073304-21latency_hist23:25:011
792166993829,6cyclictest1111719-40mandb00:00:000
792171993528,4cyclictest811055-21cp19:25:011
792171993427,5cyclictest4170283-21systemd-journal19:15:011
792166993428,4cyclictest964339-21apt-get21:45:170
792171993325,5cyclictest865306-21latency_hist20:15:011
792171993325,5cyclictest865306-21latency_hist20:15:011
792176993227,4cyclictest1018858-21kworker/u8:0+flush-8:023:00:013
792166993124,5cyclictest1008247-21apt-get22:25:190
792173992824,3cyclictest1111946-21kworker/u8:3+flush-8:000:15:002
792176992722,3cyclictest931020-21latency_hist21:15:013
792166992723,3cyclictest4170283-21systemd-journal21:02:110
792166992723,3cyclictest1078742-21kworker/u8:4+flush-8:023:30:310
792176992621,4cyclictest1078741-21kworker/u8:2+flush-8:023:35:303
792176992620,4cyclictest860114-21apt-get20:10:213
792176992620,4cyclictest860114-21apt-get20:10:213
792173992621,4cyclictest988111-21kworker/u8:3+flush-8:022:15:252
792173992621,4cyclictest1111979-21apt-get00:00:182
792173992619,4cyclictest4170283-21systemd-journal23:00:002
792171992621,3cyclictest974950-21latency_hist21:55:001
792166992623,2cyclictest1078741-21kworker/u8:2+flush-8:000:05:000
792166992620,4cyclictest1150613-21apt-get00:35:170
792176992520,3cyclictest892814-21apt-get20:40:223
792173992520,3cyclictest781652-21kworker/u8:3+events_unbound19:25:002
792166992518,5cyclictest908887-21apt-get20:55:010
792173992421,2cyclictest876215-21kworker/u8:3+flush-8:020:35:012
79217199241,20cyclictest871030-21apt-get20:20:211
792166992420,3cyclictest1051553-21kworker/u8:1+flush-8:023:15:240
792166992420,3cyclictest1018858-21kworker/u8:0+flush-8:023:05:250
792166992420,3cyclictest1018858-21kworker/u8:0+flush-8:022:40:220
792166992417,5cyclictest991988-21diskstats22:10:240
792166992417,4cyclictest952451-21python321:30:290
792173992317,4cyclictest816291-21latency_hist19:30:012
792166992320,2cyclictest1144863-21kworker/u8:0+flush-8:000:30:300
792166992319,3cyclictest1128524-21kworker/u8:0+flush-8:000:19:590
792166992319,3cyclictest1078741-21kworker/u8:2+flush-8:000:05:220
792173992218,3cyclictest1591504-21kworker/u8:0+flush-8:019:15:322
792173992218,3cyclictest1444-21dbus-daemon00:00:002
792171992216,4cyclictest903709-21apt-get20:50:201
792166992218,3cyclictest988111-21kworker/u8:3+flush-8:022:05:300
792166992218,2cyclictest1008214-21kworker/u8:4+flush-8:022:30:300
792166992218,2cyclictest1008214-21kworker/u8:4+flush-8:022:30:300
79217199210,20cyclictest0-21swapper/123:50:261
792166992117,3cyclictest827418-21kworker/u8:2+events_unbound20:20:270
792166992117,3cyclictest1111946-21kworker/u8:3+flush-8:000:30:000
792166992115,4cyclictest4170283-21systemd-journal00:11:290
792176992017,2cyclictest925609-21kworker/u8:2+flush-8:021:40:313
792173992017,2cyclictest816316-21kworker/u8:0+flush-8:019:35:312
792173992016,3cyclictest854406-21kworker/u8:1+flush-8:020:20:002
792171992018,1cyclictest0-21swapper/119:20:271
792171992015,3cyclictest4170283-21systemd-journal00:33:381
792171992014,4cyclictest4170283-21systemd-journal23:15:011
79217199200,19cyclictest0-21swapper/123:45:231
792166992015,3cyclictest1078741-21kworker/u8:2+flush-8:023:45:240
792176991912,5cyclictest821734-21latency_hist19:35:013
792173991914,4cyclictest1134220-21kworker/u8:1+flush-8:000:20:322
792173991914,3cyclictest958230-21kworker/u8:1+flush-8:022:15:002
792173991914,3cyclictest4170283-21systemd-journal23:25:292
792173991912,4cyclictest1040658-21latency_hist22:55:012
792171991917,1cyclictest0-21swapper/121:30:011
792171991917,1cyclictest0-21swapper/100:38:211
79217199190,18cyclictest0-21swapper/120:15:011
79217199190,17cyclictest0-21swapper/119:40:011
792166991916,2cyclictest964056-21kworker/u8:3+events_unbound22:00:000
792166991916,2cyclictest919706-21kworker/u8:1+events_unbound21:10:210
792166991914,4cyclictest988111-21kworker/u8:3+events_unbound22:15:330
792166991912,5cyclictest4170283-21systemd-journal20:40:000
792176991816,1cyclictest0-21swapper/323:51:073
792176991815,2cyclictest988111-21kworker/u8:3+flush-8:022:50:003
792176991815,2cyclictest0-21swapper/322:45:003
792173991815,2cyclictest855107-21irqcore20:05:222
792173991815,2cyclictest1155782-21apt00:40:002
792173991814,3cyclictest781652-21kworker/u8:3+flush-8:020:00:312
792173991813,4cyclictest931244-21kworker/u8:3+flush-8:021:20:302
792173991811,5cyclictest1024334-21latency_hist22:40:012
792171991817,1cyclictest0-21swapper/123:25:301
792171991817,1cyclictest0-21swapper/120:32:031
792171991817,1cyclictest0-21swapper/119:54:341
792171991817,1cyclictest0-21swapper/100:17:211
792171991817,1cyclictest0-21swapper/100:06:511
792171991817,0cyclictest0-21swapper/119:35:011
792171991816,1cyclictest0-21swapper/123:42:531
792171991816,1cyclictest0-21swapper/122:16:561
792171991816,1cyclictest0-21swapper/121:49:371
792171991816,1cyclictest0-21swapper/120:28:361
792171991816,1cyclictest0-21swapper/119:18:121
792171991814,2cyclictest1444-21dbus-daemon21:40:001
792171991812,4cyclictest931277-21apt-get21:15:191
79217199180,1cyclictest0-21swapper/123:58:271
79217199180,1cyclictest0-21swapper/122:36:341
79217199180,1cyclictest0-21swapper/122:01:121
79217199180,1cyclictest0-21swapper/120:44:331
79217199180,1cyclictest0-21swapper/120:36:051
79217199180,1cyclictest0-21swapper/120:05:091
79217199180,1cyclictest0-21swapper/100:24:361
79217199180,18cyclictest0-21swapper/122:27:281
79217199180,18cyclictest0-21swapper/121:22:281
79217199180,17cyclictest0-21swapper/123:08:191
79217199180,16cyclictest0-21swapper/121:10:151
79217199180,0cyclictest0-21swapper/123:17:021
792166991817,1cyclictest881890-21kworker/0:2+kacpid21:25:310
792166991814,2cyclictest970202-21irqcore21:50:240
79216699181,10cyclictest881656-21idleruntime-cro20:30:000
792176991714,2cyclictest908887-21apt-get20:55:033
792176991714,2cyclictest844075-21gltestperf19:55:223
792176991714,2cyclictest793154-21irqrtprio19:10:253
792176991714,2cyclictest0-21swapper/320:20:003
792173991714,2cyclictest981117-21irqcore22:00:262
792173991712,3cyclictest4170283-21systemd-journal23:44:562
792173991711,4cyclictest1100623-21latency_hist23:50:012
79217399170,16cyclictest0-21swapper/220:15:002
79217399170,16cyclictest0-21swapper/220:14:592
792171991717,0cyclictest0-21swapper/122:10:231
792171991716,1cyclictest0-21swapper/122:48:171
792171991716,1cyclictest0-21swapper/119:58:361
792171991715,1cyclictest0-21swapper/123:39:541
792171991715,1cyclictest0-21swapper/122:55:241
792171991715,1cyclictest0-21swapper/122:21:421
792171991715,1cyclictest0-21swapper/122:07:101
79217199170,1cyclictest0-21swapper/122:50:581
79217199170,17cyclictest0-21swapper/123:32:501
79217199170,17cyclictest0-21swapper/123:01:461
79217199170,17cyclictest0-21swapper/120:55:311
79217199170,17cyclictest0-21swapper/120:47:361
79217199170,17cyclictest0-21swapper/120:01:131
79217199170,17cyclictest0-21swapper/100:27:131
79217199170,17cyclictest0-21swapper/100:13:151
79217199170,16cyclictest0-21swapper/121:09:131
79217199170,16cyclictest0-21swapper/121:00:051
79217199170,0cyclictest0-21swapper/122:33:191
79217199170,0cyclictest0-21swapper/122:33:191
79217199170,0cyclictest0-21swapper/121:42:001
79217199170,0cyclictest0-21swapper/100:03:341
792166991716,1cyclictest881890-21kworker/0:2+kacpid21:05:190
792166991715,1cyclictest805256-21kworker/0:1+kacpid20:15:210
792166991715,1cyclictest1051553-21kworker/u8:1+flush-8:023:20:210
792166991715,1cyclictest1024313-21kworker/0:2+kacpid23:05:010
792166991714,2cyclictest936952-21kworker/u8:0+flush-8:021:35:320
792166991714,2cyclictest860990-21irqrtprio20:10:260
792166991714,2cyclictest860990-21irqrtprio20:10:260
792166991714,2cyclictest0-21swapper/019:25:000
792166991713,3cyclictest781652-21kworker/u8:3+flush-8:019:40:270
792166991712,4cyclictest1078741-21kworker/u8:2+events_unbound23:40:300
792176991613,2cyclictest947959-21gltestperf21:30:223
792176991613,2cyclictest915428-21irqcore21:00:273
792176991613,2cyclictest827183-21sendmail19:40:003
792176991613,2cyclictest1040649-21cron22:55:003
792176991610,4cyclictest832655-21latency_hist19:45:013
792176991610,4cyclictest4170283-21systemd-journal21:20:013
792173991613,2cyclictest964754-21irqcore21:45:232
792173991613,2cyclictest1133990-21idleruntime-cro00:20:002
792173991613,2cyclictest1007978-21cron22:25:002
792173991611,4cyclictest898619-21kworker/u8:4+flush-8:021:20:012
792171991615,1cyclictest0-21swapper/119:45:591
79217199160,16cyclictest0-21swapper/121:59:431
79217199160,16cyclictest0-21swapper/119:32:051
79217199160,0cyclictest0-21swapper/121:32:041
792166991616,0cyclictest991265-21kworker/0:2+kacpid22:20:230
792166991615,1cyclictest805256-21kworker/0:1+kacpid20:00:210
792166991613,2cyclictest925609-21kworker/u8:2+events_unbound21:40:190
792166991613,2cyclictest925609-21kworker/u8:2+events_unbound21:15:310
792166991613,2cyclictest800156-21gltestperf19:15:250
792166991613,2cyclictest1051553-21kworker/u8:1+flush-8:023:10:260
792166991612,3cyclictest958230-21kworker/u8:1+events_unbound22:50:240
792176991512,2cyclictest990764-21needreboot22:05:283
792176991512,2cyclictest854970-21gltestperf20:05:203
792176991512,2cyclictest1135119-21irqrtprio00:20:253
792176991512,2cyclictest1112845-21irqrtprio00:00:243
792176991512,2cyclictest1083690-21needreboot23:30:283
792176991512,2cyclictest0-21swapper/300:15:003
792176991510,3cyclictest1056985-21cron23:09:593
792173991513,1cyclictest0-21swapper/221:30:012
792173991512,2cyclictest1008663-21sed22:25:252
792173991510,4cyclictest1591504-21kworker/u8:0+flush-8:019:10:252
792173991510,3cyclictest1516-21rs:main2
792171991514,1cyclictest0-21swapper/122:42:381
792166991515,0cyclictest881890-21kworker/0:2+kacpid20:45:270
792166991513,1cyclictest805256-21kworker/0:1+kacpid19:55:230
792166991513,1cyclictest1113026-21kworker/0:1+kacpid00:20:200
792166991511,3cyclictest898619-21kworker/u8:4+events_unbound20:59:560
792166991511,3cyclictest1018858-21kworker/u8:0+events_unbound22:35:250
792166991510,3cyclictest781652-21kworker/u8:3+events_unbound19:30:270
79217699149,4cyclictest4170283-21systemd-journal20:29:553
792176991412,1cyclictest0-21swapper/322:29:593
792176991411,2cyclictest997757-21irqcore22:15:273
792176991411,2cyclictest921721-21kthreadcore21:05:233
792176991411,2cyclictest1117451-21gpgv00:05:003
792176991411,2cyclictest1073294-21cron23:24:593
792176991411,1cyclictest0-21swapper/320:55:013
792173991411,2cyclictest1118249-21irqcore00:05:252
792173991410,3cyclictest958230-21kworker/u8:1+flush-8:021:50:302
792173991410,3cyclictest898220-21kworker/u8:0+flush-8:020:45:312
792166991412,1cyclictest805256-21kworker/0:1+kacpid19:45:230
792166991410,2cyclictest0-21swapper/023:55:000
79217699139,3cyclictest1090156-21gltestperf23:40:213
79217699138,3cyclictest832910-21apt-get19:45:203
79217699137,4cyclictest881923-21apt-get20:30:203
792176991311,1cyclictest0-21swapper/300:00:003
792176991310,2cyclictest946869-21needreboot21:25:293
792176991310,2cyclictest1150883-21gltestperf00:35:213
792176991310,2cyclictest1018342-21needreboot22:30:273
792176991310,2cyclictest1018342-21needreboot22:30:273
792176991310,2cyclictest0-21swapper/319:25:003
79217399139,3cyclictest1078742-21kworker/u8:4+events_unbound23:51:092
79217399137,4cyclictest876489-21apt-get20:25:182
792173991311,1cyclictest915239-21gltestperf21:00:242
792173991310,2cyclictest892524-21cron20:40:002
792173991310,2cyclictest821732-21idleruntime-cro19:35:002
792173991310,2cyclictest0-21swapper/222:10:002
79216699138,3cyclictest1591504-21kworker/u8:0+flush-8:019:10:320
792166991312,1cyclictest805256-21kworker/0:1+kacpid19:28:260
792166991311,1cyclictest1024313-21kworker/0:2+kacpid22:45:210
792166991310,2cyclictest1078741-21kworker/u8:2+flush-8:023:35:310
79217699129,2cyclictest850098-21irqrtprio20:00:263
79217699129,2cyclictest1140571-21tr00:25:243
79217699129,1cyclictest0-21swapper/322:40:003
79217699128,3cyclictest1062392-21cron23:15:003
792176991210,1cyclictest976345-21irqrtprio21:55:253
792176991210,1cyclictest967106-21kthreadcore21:45:263
79217699120,1cyclictest0-21swapper/300:34:393
79217399129,2cyclictest926117-21gltestperf21:10:232
79217399129,2cyclictest1084896-21irqcore23:35:252
79217399129,2cyclictest1063980-21kthreadcore23:15:282
79217399128,2cyclictest0-21swapper/200:25:322
79217399120,12cyclictest0-21swapper/220:55:292
79216699129,2cyclictest816316-21kworker/u8:0+flush-8:019:50:280
79216699128,3cyclictest898619-21kworker/u8:4+flush-8:021:20:210
79216699128,3cyclictest1046138-21systemd-run23:00:000
79216699121,9cyclictest821996-21apt-get19:35:190
79216699121,8cyclictest1-21systemd20:33:270
792166991212,0cyclictest881890-21kworker/0:2+kacpi_notify20:40:270
792166991210,1cyclictest805256-21kworker/0:1+kacpid20:05:220
792166991210,1cyclictest0-21swapper/023:29:570
79217699118,2cyclictest936917-21fschecks_time21:20:213
79217699118,2cyclictest17250irq/158-ahci[0000:00:17.0]23:00:223
79217699118,2cyclictest0-21swapper/323:30:013
79217699118,2cyclictest0-21swapper/319:55:003
79217699117,3cyclictest816215-21unixbench_singl19:25:333
79217699116,3cyclictest1002786-21apt-get22:20:213
792176991110,1cyclictest0-21swapper/320:45:303
79217699110,2cyclictest1067844-21cron23:20:003
79217399119,1cyclictest0-21swapper/219:50:312
79217399119,1cyclictest0-21swapper/219:40:262
79217399118,2cyclictest854406-21kworker/u8:1+flush-8:020:20:352
79217399118,2cyclictest1052066-21gltestperf23:05:242
79217399118,2cyclictest1030489-21sed22:45:262
79217399115,4cyclictest988111-21kworker/u8:3+flush-8:022:30:222
79217399115,4cyclictest988111-21kworker/u8:3+flush-8:022:30:222
79217399115,4cyclictest4170283-21systemd-journal22:00:002
79217399115,4cyclictest1057344-21diskstats23:10:182
79216699118,2cyclictest981109-21cut22:00:250
79217699108,1cyclictest0-21swapper/320:25:003
79217699107,2cyclictest982004-21kthreadcore22:00:273
79217699107,2cyclictest970639-21cut21:50:243
79217699107,2cyclictest953485-21irqcore21:35:233
79217699106,3cyclictest958230-21kworker/u8:1+flush-8:022:10:313
79217399107,2cyclictest0-21swapper/220:50:322
79217399106,2cyclictest958256-21sh21:39:592
79217399106,2cyclictest924037-21kthreadcore21:05:232
79217399104,4cyclictest947653-21apt-get21:30:192
79217399101,6cyclictest958261-21apt-get21:40:012
79217399100,5cyclictest1150362-21awk00:35:002
413398100,5rtkit-daemon3074285-21ntpd20:40:222
7921769997,1cyclictest1095719-21gltestperf23:45:223
7921769996,2cyclictest0-21swapper/319:15:253
7921769996,1cyclictest1133971-21latency_hist00:20:003
7921739997,1cyclictest0-21swapper/222:40:312
7921739996,2cyclictest1049740-21kthreadcore23:00:272
7921769980,7cyclictest471rcuc/319:10:013
7921739986,1cyclictest0-21swapper/223:30:262
7921739986,1cyclictest0-21swapper/223:20:292
7921739984,3cyclictest847264-21cut19:55:252
7921739981,5cyclictest791972-21latency_hist19:10:012
7921769971,4cyclictest887382-21apt-get20:35:193
7921719973,3cyclictest792194-21date19:10:011
7921669970,5cyclictest792167-21cat19:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional