You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-14 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Mon Jul 14, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
99638799746742,3cyclictest1960773-21kworker/3:1+events19:15:313
99638799745741,3cyclictest1135307-21kworker/3:1+events21:21:573
99638799742737,4cyclictest1960773-21kworker/3:1+events20:18:393
99638799742737,4cyclictest1135307-21kworker/3:1+events22:46:353
99638799740736,3cyclictest1960773-21kworker/3:1+events20:03:053
99638799740736,3cyclictest1135307-21kworker/3:1+events21:57:233
99638799740734,5cyclictest1247747-21turbostat22:34:563
99638299740735,4cyclictest1241563-21kworker/1:2+events23:40:451
99638799739735,3cyclictest1135307-21kworker/3:1+events22:01:373
99638799739735,3cyclictest1029484-21turbostat19:39:553
99638799739734,4cyclictest1359488-21kworker/3:0+events00:36:153
99638799738735,2cyclictest1384829-21turbostat00:24:553
99638799738732,5cyclictest1116756-21kworker/3:2+events20:55:223
99638799737733,3cyclictest1116756-21kworker/3:2+events20:49:023
99638799736732,3cyclictest1960773-21kworker/3:1+events20:44:013
99638799736732,3cyclictest1135307-21kworker/3:1+events23:15:523
99638799736732,3cyclictest1135307-21kworker/3:1+events22:53:043
99638799736732,3cyclictest1135307-21kworker/3:1+events22:21:473
99638799736731,4cyclictest1960773-21kworker/3:1+events19:28:453
99638799736730,5cyclictest1135307-21kworker/3:1+events23:23:063
99638799735731,3cyclictest1960773-21kworker/3:1+events20:12:463
99638799735731,3cyclictest1135307-21kworker/3:1+events21:31:363
99638799734730,3cyclictest1960773-21kworker/3:1+events19:33:043
99638799734729,4cyclictest1135307-21kworker/3:1+events21:46:483
99638799734727,5cyclictest1359488-21kworker/3:0+events00:16:483
99638799733729,3cyclictest1359488-21kworker/3:0+events00:32:193
99638799731728,2cyclictest1035717-21turbostat19:44:543
99638799731727,3cyclictest1960773-21kworker/3:1+events20:07:093
99638799731727,3cyclictest1960773-21kworker/3:1+events20:07:083
99638799731727,3cyclictest1359488-21kworker/3:0+events00:02:533
99638799731727,3cyclictest1135307-21kworker/3:1+events22:43:043
99638799731727,3cyclictest1135307-21kworker/3:1+events21:51:143
99638799730726,3cyclictest1135307-21kworker/3:1+events23:05:453
99638799730726,3cyclictest1135307-21kworker/3:1+events22:09:223
99638799730726,3cyclictest1135307-21kworker/3:1+events21:04:423
99638799729725,3cyclictest1960773-21kworker/3:1+events19:11:293
99638799729725,3cyclictest1359488-21kworker/3:0+events00:11:293
99638799729724,4cyclictest1960773-21kworker/3:1+events19:50:333
99638799728724,3cyclictest1960773-21kworker/3:1+events20:23:073
99638799728724,3cyclictest1135307-21kworker/3:1+events22:13:153
99638799728724,3cyclictest1135307-21kworker/3:1+events22:13:143
99638799728723,4cyclictest1135307-21kworker/3:1+events22:16:503
99638799727723,3cyclictest1960773-21kworker/3:1+events19:59:353
99638799725721,3cyclictest1135307-21kworker/3:1+events22:37:323
99638799725721,3cyclictest1135307-21kworker/3:1+events22:25:333
99638799724720,3cyclictest1960773-21kworker/3:1+events20:37:223
99638799722717,4cyclictest1284945-21turbostat23:04:563
99638799721717,3cyclictest1135307-21kworker/3:1+events22:59:323
99638799721714,6cyclictest1960773-21kworker/3:1+events20:25:233
99638799704699,4cyclictest1359488-21kworker/3:0+events00:05:213
99638799704699,4cyclictest1359488-21kworker/3:0+events00:05:203
99638499703700,2cyclictest939667-21kworker/2:0+events23:30:212
99638799701698,2cyclictest1122918-21turbostat20:54:563
99638799701697,3cyclictest1297363-21turbostat23:14:563
99638799699696,2cyclictest1347414-21turbostat23:54:553
99638799698692,5cyclictest1135307-21kworker/3:1+events21:25:213
99638499696690,5cyclictest939667-21kworker/2:0+events23:25:232
99638499695690,4cyclictest939667-21kworker/2:0+events23:35:232
99638499692685,6cyclictest1148126-21turbostat21:14:552
99638499686683,2cyclictest939667-21kworker/2:0+events23:52:432
99638799685682,2cyclictest1359488-21kworker/3:0+events00:26:073
99638499667662,4cyclictest939667-21kworker/2:0+events21:05:232
99638499667661,5cyclictest1135651-21turbostat21:04:562
99638499665662,2cyclictest939667-21kworker/2:0+events21:18:482
99638299663656,6cyclictest1328734-21turbostat23:39:551
99638499661654,3cyclictest939667-21kworker/2:0+events23:49:192
99638799657655,1cyclictest1154278-21turbostat21:19:553
99638799650646,3cyclictest1960773-21kworker/3:1+events19:45:233
99638799649646,2cyclictest1135307-21kworker/3:1+events21:35:223
99638799648645,2cyclictest1960773-21kworker/3:1+events19:20:233
99638799642636,5cyclictest1960773-21kworker/3:1+events20:30:233
99638799641638,2cyclictest1335003-21kworker/3:1+events23:59:453
99638799641637,3cyclictest1135307-21kworker/3:1+events21:40:233
99638499603600,2cyclictest1334957-21turbostat23:44:552
99638499603599,3cyclictest1309833-21turbostat23:24:552
996382992522,2cyclictest1384815-21munin-run00:20:001
99637899257,6cyclictest1346681-21cat23:45:230
99637899247,4cyclictest1116993-21chrt20:45:130
99637899245,8cyclictest1053778-21grep19:50:260
99637899245,7cyclictest1216741-21munin-run22:05:000
99637899245,6cyclictest1403660-21latency_hist00:35:010
99637899245,6cyclictest1166148-21ntpq21:20:230
996378992420,3cyclictest1110501-21sh20:40:010
99637899235,6cyclictest1392288-21irqrtprio00:25:180
99637899235,6cyclictest1196535-21kthreadcore21:45:190
996384992218,2cyclictest1272548-21sh22:50:012
99638499220,20cyclictest1116721-21apt20:45:012
99637899228,12cyclictest1284929-21cat23:00:010
99637899227,6cyclictest1273767-21irqrtprio22:50:180
99637899226,6cyclictest1112548-21cut20:40:180
99637899226,5cyclictest1403913-21fschecks_count00:35:150
99637899226,5cyclictest1323602-21sed23:30:160
99637899225,7cyclictest1123394-21fschecks_count20:50:160
99637899224,6cyclictest1016128-21ntpq19:20:210
99637899223,7cyclictest1181142-21kernelversion21:35:200
996378992218,2cyclictest1316062-21systemd-run23:24:590
996384992118,2cyclictest1167762-21irqcore21:25:172
99637899217,5cyclictest1367444-21sed00:05:180
99637899217,5cyclictest1367444-21sed00:05:170
99637899216,5cyclictest1390362-21python300:20:250
99637899216,5cyclictest1380177-21cut00:15:200
99637899215,6cyclictest1147516-21timerandwakeup21:05:250
99637899215,5cyclictest1292683-21irqrtprio23:05:160
99637899215,4cyclictest0-21swapper/021:20:010
99637899214,7cyclictest1335522-21expr23:40:150
99637899214,5cyclictest1209755-21awk21:55:260
99637899213,7cyclictest1135951-21grep21:00:130
99637899213,7cyclictest1099925-21kthreadcore20:30:190
99637899213,6cyclictest1066790-21taskset20:02:130
996378992111,8cyclictest3614154-21systemd-journal19:49:590
99637899207,5cyclictest1286928-21cut23:00:170
99637899205,5cyclictest1271196-21kthreadcore22:45:190
99637899205,5cyclictest1178325-21latency21:30:200
99637899205,5cyclictest1002814-21munin-node19:15:010
99637899204,6cyclictest1304121-21grep23:15:160
99637899204,6cyclictest1248227-21cut22:30:160
99637899204,5cyclictest1375963-21cut00:10:180
99637899204,5cyclictest1354351-21if_err_eno123:55:160
99637899204,5cyclictest1335024-21sed23:40:000
99637899204,5cyclictest1202019-21cut21:50:190
99637899204,5cyclictest1189735-21grep21:40:180
99637899204,5cyclictest1081422-21grep20:15:170
99637899203,7cyclictest1134038-21grep20:55:190
99637899203,7cyclictest1092093-21acpi20:25:120
99637899203,6cyclictest1347897-21fschecks_count23:50:160
99637899203,6cyclictest1078691-21mailstats20:10:240
99637899203,5cyclictest1030182-21if_err_eno119:35:160
99637899203,5cyclictest1024764-21cut19:30:170
996384991916,2cyclictest0-21swapper/223:59:592
996384991915,3cyclictest1179259-21sshd21:32:532
996382991916,2cyclictest0-21swapper/122:15:001
996382991916,2cyclictest0-21swapper/122:14:591
99637899197,4cyclictest1258662-21tr22:35:180
99637899196,7cyclictest9999431kthreadcore19:10:190
99637899196,5cyclictest1087186-21cut20:20:190
99637899196,4cyclictest1236182-21sed22:20:180
99637899196,4cyclictest1149293-21sed21:10:180
99637899194,5cyclictest1317838-21sed23:25:170
99637899194,5cyclictest1234236-21cut22:15:210
99637899194,5cyclictest1216988-21basename22:05:110
99637899194,5cyclictest1167905-21sed21:25:170
99637899193,6cyclictest1364754-21cut00:00:190
99637899193,5cyclictest1072170-21latency20:05:200
99637899193,5cyclictest1072170-21latency20:05:190
99637899193,5cyclictest1036235-21fschecks_time19:40:150
99637899193,5cyclictest1017546-21grep19:25:160
996387991815,2cyclictest1341927-21irqcore23:45:173
99638499185,2cyclictest191rcu_preempt23:15:012
996382991815,2cyclictest523-21gdbus20:05:011
99637899185,4cyclictest0-21swapper/019:55:140
99637899184,5cyclictest1246109-21cut22:25:220
996384991714,2cyclictest523-21gdbus23:00:012
996384991714,2cyclictest391rcuc/221:40:182
996384991713,2cyclictest0-21swapper/222:25:012
99638499170,5cyclictest1409669-21latency_hist00:40:012
996382991715,1cyclictest1241535-21latency_hist22:25:011
996382991714,2cyclictest523-21gdbus22:10:001
996382991714,2cyclictest1366898-21irqcore00:05:181
996382991714,2cyclictest1366898-21irqcore00:05:171
996382991714,2cyclictest0-21swapper/121:14:591
99637899173,3cyclictest1261876-21ps22:40:190
996384991613,2cyclictest391rcuc/222:50:162
996384991613,2cyclictest1256876-21kthreadcore22:35:162
996384991613,2cyclictest1179805-21cpuspeed_turbos21:35:152
996384991613,2cyclictest1135684-21systemd-run21:00:002
996384991613,2cyclictest0-21swapper/219:40:002
996384991613,2cyclictest0-21swapper/200:20:002
996384991612,3cyclictest1079001-21unixbench_multi20:10:262
996382991613,2cyclictest1-21systemd19:40:221
996382991613,2cyclictest1166045-21needreboot21:20:221
996382991613,2cyclictest0-21swapper/122:55:001
996382991613,2cyclictest0-21swapper/121:20:001
996382991612,3cyclictest1011525-21irqcore19:20:161
99637899169,4cyclictest1297636-21apt-get23:10:120
99637899168,6cyclictest1229117-21cron22:15:000
99637899168,6cyclictest1229117-21cron22:15:000
996387991512,2cyclictest1316639-21cpuspeed_turbos23:25:143
996384991511,3cyclictest1366151-21cron00:05:002
996384991511,3cyclictest1047634-21unixbench_multi19:45:252
99638499150,4cyclictest0-21swapper/219:20:242
996382991512,2cyclictest1390617-21unixbench_multi00:20:261
996382991512,2cyclictest1283360-21kthreadcore22:55:181
996382991512,2cyclictest1267072-21irqcore22:45:171
996382991512,2cyclictest0-21swapper/120:15:011
996382991511,3cyclictest1022879-21unixbench_singl19:25:261
996382991511,2cyclictest1193137-21irqrtprio21:45:171
996382991511,2cyclictest0-21swapper/119:55:011
996382991511,2cyclictest0-21swapper/100:15:001
996382991510,3cyclictest1254287-21cpuspeed_turbos22:35:121
99638299150,4cyclictest12944231kthreadcore23:05:181
99638299150,3cyclictest1321367-21kthreadcore23:25:191
996387991412,1cyclictest1328979-21/usr/sbin/munin23:35:203
996387991412,1cyclictest1323069-21gltestperf23:30:143
996387991411,2cyclictest1154293-21idleruntime-cro21:15:003
99638499147,6cyclictest1123320-21expr20:50:152
996384991412,1cyclictest1401670-21kthreadcore00:30:202
99638499141,1cyclictest11065521kthreadcore20:35:182
996384991411,2cyclictest1223188-21/usr/sbin/munin22:10:182
996384991411,2cyclictest1223188-21/usr/sbin/munin22:10:172
996384991411,2cyclictest1204498-21latency_hist21:55:002
996384991410,3cyclictest401ktimers/200:05:142
996384991410,3cyclictest401ktimers/200:05:132
996384991410,3cyclictest1054268-21cron19:55:002
996384991410,3cyclictest1023771-21fschecks_time19:30:162
99638499140,3cyclictest1121805-21/usr/sbin/munin20:45:202
99638299148,4cyclictest1110111-21vmstat20:35:251
996382991412,1cyclictest1070340-21kthreadcore20:05:191
996382991412,1cyclictest1070340-21kthreadcore20:05:181
996382991412,1cyclictest0-21swapper/123:24:591
996382991412,1cyclictest0-21swapper/119:50:011
996382991411,2cyclictest509-21rs:main1
996382991411,2cyclictest1353673-21systemd-run23:55:001
996382991411,2cyclictest1303169-21unixbench_singl23:10:271
996382991411,2cyclictest1167368-21cpuspeed_turbos21:25:141
996382991411,2cyclictest1117106-21cpuspeed_turbos20:45:131
996382991411,2cyclictest0-21swapper/120:55:011
996382991410,3cyclictest1142809-21irqcore21:05:171
996382991410,2cyclictest0-21swapper/119:59:591
99638299140,13cyclictest0-21swapper/100:30:171
99638299140,11cyclictest1263278-21tr22:40:201
99638799137,4cyclictest0-21swapper/321:10:013
99638499139,3cyclictest1253467-21unixbench_multi22:30:262
99638499137,4cyclictest1204583-21apt-get21:55:162
99638499137,4cyclictest1036115-21tr19:40:132
99638499136,5cyclictest3614154-21systemd-journal22:45:012
996384991311,1cyclictest1307127-21kthreadcore23:15:182
996384991311,1cyclictest1070287-21kthreadcore20:05:192
996384991311,1cyclictest1070287-21kthreadcore20:05:192
996384991310,2cyclictest401ktimers/219:10:132
996384991310,2cyclictest1097596-21unixbench_singl20:25:252
996384991310,2cyclictest1079820-21cpuspeed_turbos20:15:122
99638499130,3cyclictest1010779-21apt19:19:592
99638499130,2cyclictest1060786-21turbostat19:59:592
99638299138,4cyclictest1247251-21grep22:25:271
99638299138,4cyclictest1136089-21df_inode21:00:151
99638299138,1cyclictest191rcu_preempt22:04:581
99638299136,5cyclictest493-21dbus-daemon23:05:011
99638299135,1cyclictest295798rtkit-daemon23:16:501
99638299132,7cyclictest0-21swapper/119:38:541
996382991310,2cyclictest1404009-21gltestperf00:35:161
996382991310,2cyclictest1328333-21unixbench_singl23:30:241
996382991310,2cyclictest1173868-21gltestperf21:30:151
99594221311,1sleep20-21swapper/219:06:292
99638499128,3cyclictest401ktimers/220:30:152
99638499127,4cyclictest1385560-21grep00:20:182
99638499124,6cyclictest391rcuc/200:25:182
996384991210,1cyclictest0-21swapper/222:25:202
99638499120,6cyclictest391rcuc/223:05:152
99638499120,3cyclictest0-21swapper/222:15:262
99638299129,2cyclictest1111098-21gltestperf20:40:161
99638299129,2cyclictest1079844-21cpuspeed_turbos20:15:121
99638299129,2cyclictest0-21swapper/119:10:011
99638299128,3cyclictest493-21dbus-daemon00:25:251
99638299128,3cyclictest1180045-21gltestperf21:35:171
99638299128,2cyclictest0-21swapper/119:19:591
99638299127,3cyclictest493-21dbus-daemon21:54:591
99638299127,1cyclictest191rcu_preempt22:31:081
99638299126,4cyclictest1191414-21grep21:40:231
99638299126,4cyclictest1023552-21apt-get19:30:131
99638299120,1cyclictest0-21swapper/120:25:141
99638299120,11cyclictest0-21swapper/120:20:201
99637899126,4cyclictest996385-21latency_hist19:10:000
99638799118,2cyclictest0-21swapper/323:40:153
99638499119,1cyclictest0-21swapper/221:45:162
99638499118,2cyclictest1211137-21gltestperf22:00:172
99638499118,2cyclictest1086004-21cpuspeed_turbos20:20:162
99638499118,2cyclictest0-21swapper/222:05:152
99638499118,2cyclictest0-21swapper/219:25:172
99638499116,4cyclictest1288411-21tr23:00:182
99638499116,3cyclictest3614154-21systemd-journal20:05:012
99638499114,2cyclictest191rcu_preempt21:20:172
99638299118,2cyclictest311ktimers/120:30:141
99638299118,2cyclictest1341740-21gltestperf23:45:151
99638299118,2cyclictest1204707-21cpuspeed_turbos21:55:171
99638299117,2cyclictest0-21swapper/119:10:171
99638299116,4cyclictest493-21dbus-daemon23:55:231
99638299116,4cyclictest1129685-21apt-get20:55:121
99638299110,11cyclictest311ktimers/122:18:371
9901092117,2sleep30-21swapper/319:05:023
99638499105,4cyclictest1373658-21cut00:10:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional