You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 00:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sun May 04, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
967266994134,5cyclictest1124881-21latency_hist09:40:011
967266993936,2cyclictest1245352-21kworker/u8:4+events_unbound11:44:361
967266993834,3cyclictest1119738-21kworker/u8:0+events_unbound10:14:351
967266993633,2cyclictest988736-21kworker/u8:0+events_unbound07:43:351
967266993633,2cyclictest962097-21kworker/u8:3+events_unbound07:38:341
967266993531,3cyclictest962097-21kworker/u8:3+events_unbound08:18:361
967266993431,2cyclictest1191795-21kworker/u8:0+events_unbound11:47:351
967266993431,2cyclictest1176913-21kworker/u8:3+events_unbound10:39:341
967266993330,2cyclictest1261520-21kworker/u8:2+events_unbound12:00:361
967266993330,2cyclictest1191795-21kworker/u8:0+events_unbound11:18:351
967266993330,2cyclictest1067454-21kworker/u8:2+events_unbound09:02:341
967260993227,3cyclictest1-21systemd07:49:590
967260993226,4cyclictest1308051-21latency_hist12:35:000
967266993023,5cyclictest1150903-21latency_hist10:05:001
967266992926,2cyclictest1087788-21kworker/u8:0+events_unbound09:09:341
967260992924,3cyclictest1067454-21kworker/u8:2+events_unbound09:10:010
967272992824,3cyclictest1151135-21kworker/u8:1+flush-259:010:40:223
967268992823,4cyclictest3000860-21systemd-journal07:49:592
96726099281,25cyclictest3000063-21rs:main0
967268992722,3cyclictest1191795-21kworker/u8:0+flush-259:011:05:002
967268992720,5cyclictest1187272-21latency_hist10:40:002
967266992724,2cyclictest962097-21kworker/u8:3+events_unbound08:38:351
967260992722,3cyclictest1171967-21apt-get10:25:130
967268992622,3cyclictest1213724-21kworker/u8:2+flush-259:011:15:002
967268992620,4cyclictest972691-21latency_hist07:15:002
967266992623,2cyclictest967682-21kworker/u8:4+events_unbound07:21:341
967266992623,2cyclictest1176913-21kworker/u8:3+events_unbound11:01:351
967266992623,2cyclictest1119738-21kworker/u8:0+events_unbound09:58:351
967260992623,2cyclictest1213724-21kworker/u8:2+flush-259:011:20:170
967260992621,4cyclictest3000860-21systemd-journal12:18:340
967260992619,5cyclictest1240555-21diskstats11:30:150
967272992519,4cyclictest1120118-21diskstats09:35:163
967268992521,3cyclictest967682-21kworker/u8:4+flush-259:007:25:002
967266992522,2cyclictest956517-21kworker/u8:1+events_unbound07:45:281
967266992521,3cyclictest1024866-21kworker/u8:0+flush-259:008:50:001
967266992519,4cyclictest978328-21latency_hist07:20:011
967260992521,3cyclictest1130326-21kworker/u8:3+events_unbound09:45:190
967272992419,3cyclictest1282053-21kworker/u8:2+flush-259:012:20:013
967272992418,4cyclictest3000860-21systemd-journal11:35:013
967266992421,2cyclictest962097-21kworker/u8:3+events_unbound08:30:281
967266992421,2cyclictest962097-21kworker/u8:3+events_unbound08:30:271
967266992421,2cyclictest962097-21kworker/u8:3+events_unbound07:27:361
967266992421,2cyclictest1256125-21kworker/u8:1+events_unbound11:58:361
967266992421,2cyclictest1024866-21kworker/u8:0+events_unbound08:12:341
967266992420,3cyclictest1234517-21apt-get11:25:001
967260992420,3cyclictest956517-21kworker/u8:1+flush-259:007:40:010
967260992420,3cyclictest1109306-21kworker/u8:4+events_unbound09:25:130
967260992418,4cyclictest1239314-21ntpq11:25:220
967268992317,4cyclictest1077663-21latency_hist08:55:002
967266992319,3cyclictest1151135-21kworker/u8:1+events_unbound10:25:281
967266992319,2cyclictest1245352-21kworker/u8:4+events_unbound11:37:351
967260992319,3cyclictest1256125-21kworker/u8:1+flush-259:012:05:010
967272992217,3cyclictest1066778-21cron08:44:593
967272992216,4cyclictest3000860-21systemd-journal10:05:343
967268992216,5cyclictest1045963-21latency_hist08:25:012
967266992219,2cyclictest988736-21kworker/u8:0+events_unbound07:32:331
967266992219,2cyclictest1176913-21kworker/u8:3+events_unbound10:55:351
967266992219,2cyclictest1130326-21kworker/u8:3+events_unbound09:49:581
967266992219,2cyclictest1087788-21kworker/u8:0+events_unbound09:10:291
967266992218,3cyclictest1119738-21kworker/u8:0+flush-259:010:05:181
967266992217,4cyclictest1261520-21kworker/u8:2+flush-259:011:55:011
967260992217,3cyclictest962097-21kworker/u8:3+flush-259:007:35:000
967266992118,2cyclictest962097-21kworker/u8:3+events_unbound08:00:351
967266992118,2cyclictest962097-21kworker/u8:3+events_unbound07:10:301
967260992115,5cyclictest3000860-21systemd-journal12:30:000
967260992115,4cyclictest1151135-21kworker/u8:1+events_unbound10:10:170
967272992015,3cyclictest962097-21kworker/u8:3+flush-259:008:00:003
967268992014,5cyclictest3000860-21systemd-journal09:30:012
967266992018,2cyclictest1223741-21kworker/u8:3+events_unbound11:32:361
967266992017,2cyclictest1109306-21kworker/u8:4+events_unbound09:27:341
967260992016,3cyclictest1151135-21kworker/u8:1+flush-259:010:20:190
96726099200,18cyclictest1-21systemd09:55:350
967268991914,4cyclictest1151135-21kworker/u8:1+flush-259:011:00:002
967268991914,3cyclictest1191795-21kworker/u8:0+flush-259:011:40:172
967268991913,4cyclictest1287668-21diskstats12:15:142
967266991916,2cyclictest962097-21kworker/u8:3+events_unbound08:25:271
967266991916,2cyclictest1302476-21kworker/u8:0+events_unbound12:28:351
967266991916,2cyclictest1276714-21kworker/u8:0+events_unbound12:10:021
967266991916,2cyclictest1119738-21kworker/u8:0+events_unbound10:30:031
967266991916,2cyclictest1087788-21kworker/u8:0+events_unbound09:18:341
967266991916,2cyclictest1067454-21kworker/u8:2+events_unbound09:23:351
967260991915,3cyclictest966999-21apt-get07:10:020
967260991913,4cyclictest3000860-21systemd-journal10:03:340
967260991913,4cyclictest1060674-21ntpq08:35:220
967272991814,3cyclictest1282296-21apt-get12:10:143
967272991814,3cyclictest1067454-21kworker/u8:2+flush-259:008:55:173
967272991813,3cyclictest962097-21kworker/u8:3+flush-259:007:50:193
967272991812,4cyclictest972689-21apt-get07:15:003
967268991813,3cyclictest1130326-21kworker/u8:3+flush-259:009:45:182
967268991812,4cyclictest1161535-21/usr/sbin/munin10:15:162
96726699186,1cyclictest0-21swapper/112:35:171
967266991816,2cyclictest1077897-21kworker/u8:3+events_unbound08:58:351
967266991815,2cyclictest1282266-21kworker/u8:3+events_unbound12:23:341
967266991815,2cyclictest1176913-21kworker/u8:3+events_unbound10:42:371
967266991815,2cyclictest1009759-21kworker/u8:4+events_unbound07:52:361
967266991811,5cyclictest3000860-21systemd-journal07:57:331
967260991814,3cyclictest967682-21kworker/u8:4+events_unbound07:25:180
967260991813,4cyclictest962097-21kworker/u8:3+flush-259:008:05:150
967272991713,3cyclictest1245352-21kworker/u8:4+flush-259:011:50:013
967272991713,2cyclictest1199388-21kthreadcore10:50:193
967272991712,4cyclictest962097-21kworker/u8:3+flush-259:008:35:013
967272991712,4cyclictest962097-21kworker/u8:3+flush-259:008:35:003
967272991712,4cyclictest962097-21kworker/u8:3+flush-259:007:15:193
967272991712,3cyclictest1151135-21kworker/u8:1+flush-259:011:10:173
967272991711,4cyclictest1292483-21kworker/u8:4+flush-259:012:35:183
967272991711,4cyclictest1256125-21kworker/u8:1+flush-259:012:00:253
967266991714,2cyclictest1119738-21kworker/u8:0+events_unbound10:15:281
967266991711,4cyclictest1203289-21latency_hist10:55:011
967260991714,2cyclictest1245352-21kworker/u8:4+flush-259:011:40:010
967260991714,2cyclictest1191795-21kworker/u8:0+events_unbound11:50:010
967260991710,5cyclictest1019098-21ntpq07:55:220
96727299169,4cyclictest1302476-21kworker/u8:0+flush-259:012:30:183
967272991612,3cyclictest1067454-21kworker/u8:2+flush-259:009:20:183
967272991611,3cyclictest1271668-21sh12:00:003
967272991610,4cyclictest3000860-21systemd-journal09:45:333
967268991612,3cyclictest1024866-21kworker/u8:0+flush-259:008:14:592
967268991611,4cyclictest3000860-21systemd-journal08:29:592
96726699169,1cyclictest0-21swapper/110:45:261
967266991613,2cyclictest1282053-21kworker/u8:2+events_unbound12:15:021
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-012:37:000
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-011:40:110
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-011:05:160
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-010:51:210
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-010:37:560
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-009:34:180
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-009:24:120
967260991613,2cyclictest60650irq/133-enp0s29f1:rx-008:25:130
967260991613,2cyclictest1036230-21irqcore08:15:220
967260991612,3cyclictest962097-21kworker/u8:3+events_unbound08:40:250
967260991612,3cyclictest60650irq/133-enp0s29f1:rx-008:30:130
967260991612,3cyclictest60650irq/133-enp0s29f1:rx-008:30:120
967260991611,1cyclictest60650irq/133-enp0s29f1:rx-009:40:160
967260991610,4cyclictest1176913-21kworker/u8:3+events_unbound10:55:190
96727299159,3cyclictest967682-21kworker/u8:4+flush-259:007:25:183
967272991512,2cyclictest1040298-21unixbench_multi08:15:293
967272991510,4cyclictest1191795-21kworker/u8:0+flush-259:011:35:173
967268991513,1cyclictest1051803-21grep08:30:182
967268991513,1cyclictest1051803-21grep08:30:172
967268991511,3cyclictest1191795-21kworker/u8:0+flush-259:011:20:002
96726699152,4cyclictest1135622-21grep09:50:161
96726699151,3cyclictest1118775-21ntpq09:30:221
967266991513,1cyclictest301rcuc/112:30:171
967266991512,2cyclictest962097-21kworker/u8:3+events_unbound08:24:571
967266991512,2cyclictest1061785-21kworker/u8:4+events_unbound08:43:561
967266991511,3cyclictest1235159-21rred11:25:021
967260991514,1cyclictest60650irq/133-enp0s29f1:rx-007:15:210
967260991513,2cyclictest60650irq/133-enp0s29f1:rx-009:50:150
967260991513,1cyclictest60650irq/133-enp0s29f1:rx-011:53:540
967260991513,1cyclictest60650irq/133-enp0s29f1:rx-010:05:190
967260991513,1cyclictest60650irq/133-enp0s29f1:rx-009:00:260
967260991513,1cyclictest60650irq/133-enp0s29f1:rx-008:02:590
967260991513,1cyclictest60650irq/133-enp0s29f1:rx-007:41:270
967260991512,2cyclictest60650irq/133-enp0s29f1:rx-010:45:170
967260991511,3cyclictest962097-21kworker/u8:3+flush-259:007:50:170
967260991511,3cyclictest1151135-21kworker/u8:1+flush-259:010:30:200
967260991510,3cyclictest0-21swapper/007:25:010
967272991411,2cyclictest1096265-21kthreadcore09:10:213
967272991410,3cyclictest1208491-21cron11:00:003
967272991410,3cyclictest1151135-21kworker/u8:1+flush-259:010:20:183
967268991411,2cyclictest962097-21kworker/u8:3+flush-259:007:15:272
967268991410,3cyclictest1281074-21cut12:05:212
96726699149,2cyclictest1191795-21kworker/u8:0+events_unbound11:13:471
96726699143,3cyclictest32-21ksoftirqd/109:40:281
967266991412,1cyclictest301rcuc/112:05:151
967266991411,2cyclictest301rcuc/110:20:201
967266991411,2cyclictest1191795-21kworker/u8:0+events_unbound11:09:351
967266991410,3cyclictest1025633-21fschecks_time08:05:161
96726099149,4cyclictest1256125-21kworker/u8:1+flush-259:012:20:250
967260991412,2cyclictest60650irq/133-enp0s29f1:rx-011:10:260
967260991411,2cyclictest60650irq/133-enp0s29f1:rx-009:35:100
967260991411,2cyclictest60650irq/133-enp0s29f1:rx-008:53:170
967260991411,2cyclictest60650irq/133-enp0s29f1:rx-008:20:200
967260991410,3cyclictest60650irq/133-enp0s29f1:rx-011:00:220
967260991410,3cyclictest1270668-21needreboot11:55:200
96727299139,3cyclictest1213262-21unixbench_singl11:00:283
96727299139,3cyclictest1172165-21fschecks_count10:25:143
96727299137,4cyclictest1109306-21kworker/u8:4+flush-259:009:25:173
967272991310,2cyclictest988736-21kworker/u8:0+flush-259:007:45:013
967272991310,2cyclictest1046638-21irqcore08:25:183
96726899139,3cyclictest1176913-21kworker/u8:3+flush-259:010:45:262
96726899139,3cyclictest1067549-21cpuspeed_turbos08:45:152
96726899139,2cyclictest994302-21tr07:35:162
96726899138,4cyclictest1088072-21apt-get09:05:002
967268991311,1cyclictest962097-21kworker/u8:3+flush-259:008:15:212
967268991310,2cyclictest1282266-21kworker/u8:3+events_unbound12:10:162
967268991310,2cyclictest1141213-21irqcore09:55:212
967268991310,2cyclictest1119738-21kworker/u8:0+events_unbound10:05:192
967268991310,2cyclictest1104503-21cpuspeed_turbos09:20:162
96726099139,3cyclictest1119738-21kworker/u8:0+events_unbound10:18:350
96726099139,3cyclictest1067453-21kworker/u8:1+events_unbound09:11:340
967260991311,1cyclictest60650irq/133-enp0s29f1:rx-008:11:300
967260991310,2cyclictest60650irq/133-enp0s29f1:rx-008:45:410
96726099130,11cyclictest1282032-21turbostat12:10:010
96727299129,2cyclictest1177542-21irqcore10:30:173
96727299128,3cyclictest1020199-21/usr/sbin/munin08:00:253
96727299128,2cyclictest1087595-21unixbench_multi09:00:273
96727299128,2cyclictest0-21swapper/307:35:223
96727299126,4cyclictest3000860-21systemd-journal08:50:013
96727299120,3cyclictest1302847-21awk12:30:003
96726899129,2cyclictest999111-21kworker/u8:2+flush-259:007:55:212
96726899129,2cyclictest1176913-21kworker/u8:3+flush-259:010:40:262
96726899129,2cyclictest1151135-21kworker/u8:1+events_unbound10:30:172
96726899129,2cyclictest1135776-21fschecks_count09:50:172
96726899128,3cyclictest1276439-21unixbench_singl12:00:292
96726899128,3cyclictest1250524-21unixbench_singl11:35:262
96726899128,2cyclictest0-21swapper/207:25:252
96726899126,4cyclictest1104119-21latency_hist09:20:002
96726099129,2cyclictest60650irq/133-enp0s29f1:rx-008:55:120
96727299119,1cyclictest0-21swapper/309:15:183
96727299118,2cyclictest0-21swapper/310:45:173
96727299117,3cyclictest1191795-21kworker/u8:0+flush-259:011:25:253
96727299117,2cyclictest0-21swapper/308:55:003
96727299116,3cyclictest1230578-21tail11:20:173
96727299115,4cyclictest978756-21diskstats07:20:163
96727299115,4cyclictest3000860-21systemd-journal07:49:593
96726899119,1cyclictest0-21swapper/209:35:002
96726899118,2cyclictest999111-21kworker/u8:2+flush-259:007:54:592
96726899118,2cyclictest988736-21kworker/u8:0+flush-259:007:30:212
96726899118,2cyclictest1151135-21kworker/u8:1+flush-259:010:25:172
96726899118,2cyclictest1119738-21kworker/u8:0+flush-259:009:45:002
96726899118,2cyclictest1003944-21unixbench_singl07:40:272
96726899118,2cyclictest0-21swapper/209:35:272
96726899115,4cyclictest3000860-21systemd-journal11:27:342
96726899110,9cyclictest1025167-21true08:04:592
96726099118,2cyclictest60650irq/133-enp0s29f1:rx-011:19:530
96726099111,2cyclictest967003-21latency_hist07:10:010
96727299107,2cyclictest1224432-21cpuspeed_turbos11:15:143
96727299106,2cyclictest1140522-21latency_hist09:55:013
96727299105,3cyclictest3000860-21systemd-journal12:22:353
96727299103,5cyclictest492-21dbus-daemon12:10:013
96726899107,2cyclictest0-21swapper/208:35:162
96726899106,3cyclictest1145960-21apt-get10:00:162
96726899106,3cyclictest1130326-21kworker/u8:3+flush-259:010:10:042
96726899106,2cyclictest1191795-21kworker/u8:0+events_unbound11:30:182
96726899105,3cyclictest1151135-21kworker/u8:1+flush-259:010:20:152
96726899100,8cyclictest1203398-21cat10:55:002
9672729996,2cyclictest1158665-21tr10:10:183
9672729996,1cyclictest0-21swapper/311:05:193
9672729994,4cyclictest1025692-21cut08:05:153
9672729994,3cyclictest1142698-21tr09:55:233
9672689997,1cyclictest1215040-21kthreadcore11:05:202
9672689996,2cyclictest0-21swapper/208:40:212
9672689995,3cyclictest0-21swapper/208:10:002
9672689995,2cyclictest1308045-21sh12:35:002
9672689995,2cyclictest0-21swapper/208:55:202
9672689993,4cyclictest3000860-21systemd-journal12:25:002
9672689985,2cyclictest0-21swapper/211:55:152
9672689981,5cyclictest1261555-21apt-get11:50:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional