You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 20:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 17, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
135862241164,24sleep20-21swapper/207:09:112
1226222210,11sleep30-21swapper/307:05:183
124852212180,21sleep10-21swapper/107:05:201
133052203164,26sleep00-21swapper/007:05:310
751221830,4sleep31375199cyclictest08:05:223
1785121270,5sleep11374099cyclictest10:55:231
13746998986,2cyclictest0-21swapper/209:56:362
13746996764,2cyclictest0-21swapper/207:26:272
327192530,2sleep00-21swapper/011:30:180
217532530,4sleep230-21rcuc/212:20:122
13751994313,17cyclictest6308-21sort09:15:253
13751994310,10cyclictest9673-21netstat08:10:193
13751994113,17cyclictest26639-21perf10:05:023
1375199401,11cyclictest0-21swapper/310:20:453
1375199401,11cyclictest0-21swapper/310:20:453
13751993910,21cyclictest22166-21switchtime11:05:243
13751993711,15cyclictest21136-21latency09:50:173
1373499367,13cyclictest9054-21awk09:25:010
1373499367,13cyclictest15024-21munin-run08:25:010
1375199354,15cyclictest0-21swapper/310:00:023
13751993512,11cyclictest29062-21users11:20:253
13751993414,17cyclictest17349-21grep09:40:303
13751993412,9cyclictest87550irq/128-enp1s0-11:25:243
13751993411,9cyclictest3257-21perl09:10:163
13751993410,13cyclictest15263-21/usr/sbin/munin12:05:153
181912330,2sleep10-21swapper/108:30:161
13751993313,16cyclictest19907-21gltestperf12:15:153
13751993313,11cyclictest16833-21/usr/sbin/munin07:15:133
13751993311,10cyclictest31539-21kworker/u8:211:45:423
13751993215,10cyclictest23473-21munin-run12:25:023
13751993211,12cyclictest887-21ntp_states07:50:233
13751993211,12cyclictest10500-21/usr/sbin/munin11:55:153
1373499324,13cyclictest16898-21idleruntime-cro12:10:010
13751993112,4cyclictest31359-21grep10:15:123
13751993110,11cyclictest14594-21cat10:50:143
1373499319,10cyclictest0-21swapper/008:00:010
1373499316,12cyclictest26647-21awk10:05:010
1375199309,12cyclictest12207-21expr10:45:133
1375199307,12cyclictest2675-21kworker/u8:209:20:213
1375199306,8cyclictest28427-21/usr/sbin/munin08:55:123
13751993011,9cyclictest18178-21ntp_states12:10:213
13751993011,11cyclictest19747-21ntp_states11:00:213
13751993011,10cyclictest3672-21latency_hist11:40:013
13751993011,10cyclictest3672-21latency_hist11:40:013
13751993010,7cyclictest18155-21idleruntime08:30:153
13751993010,5cyclictest87650irq/129-enp1s0-07:25:233
13751993010,10cyclictest24045-21diskstats12:25:163
1373499302,10cyclictest25648-21kworker/0:108:50:210
1373499301,14cyclictest19097-21idleruntime-cro12:15:010
1375199299,9cyclictest8734-21fschecks_count11:50:153
1375199299,11cyclictest6223-21latency_hist08:05:023
1375199296,9cyclictest0-21swapper/308:00:023
1375199296,16cyclictest9496-21cat09:25:133
1375199292,6cyclictest0-21swapper/310:07:073
13751992920,5cyclictest17737-21smart_sda10:55:233
13751992917,9cyclictest13760-21/usr/sbin/munin12:00:223
13751992913,7cyclictest25969-21/usr/sbin/munin07:35:143
1375199289,7cyclictest21837-21/usr/sbin/munin08:40:173
1375199289,11cyclictest25278-21/usr/sbin/munin11:15:223
1375199288,12cyclictest13040-21/usr/sbin/munin08:20:273
1375199286,8cyclictest23423-21/usr/sbin/munin07:30:253
13751992820,5cyclictest24383-21diskstats08:45:163
13751992810,11cyclictest0-21swapper/312:35:133
1373499282,14cyclictest10277-21munin-run11:55:010
1373499282,13cyclictest22230-21munin-run09:55:000
1375199278,17cyclictest0-21swapper/307:45:233
1375199276,10cyclictest0-21swapper/311:10:163
1375199274,9cyclictest6546-21ntpq10:30:223
13751992719,5cyclictest19318-21smart_sda09:45:233
13751992718,6cyclictest9882-21/usr/sbin/munin10:40:223
13751992718,6cyclictest15593-21diskstats08:25:153
13751992714,10cyclictest87350irq/126-enp1s0-10:25:213
13751992711,13cyclictest19638-21grep07:20:163
13751992710,5cyclictest11557-21tr09:30:123
13751992710,14cyclictest13939-21/usr/sbin/munin07:10:153
1373499272,12cyclictest32610-21munin-run09:05:010
1373499271,13cyclictest30058-21idleruntime-cro12:40:010
1375199268,6cyclictest0-21swapper/309:00:123
1375199264,10cyclictest0-21swapper/310:35:253
13751992618,5cyclictest28163-21diskstats07:40:153
13751992617,6cyclictest32233-21/usr/sbin/munin11:30:143
13751992617,6cyclictest26966-21/usr/sbin/munin12:30:223
13751992617,6cyclictest14405-21/usr/sbin/munin09:35:173
13751992616,7cyclictest12065-21/usr/sbin/munin08:15:243
13751992616,6cyclictest4252-21diskstats11:40:143
13740992611,12cyclictest20309-21systemd-journal11:00:501
13751992518,4cyclictest741-21diskstats09:05:133
13751992518,4cyclictest741-21diskstats09:05:133
13751992518,4cyclictest26579-21diskstats08:50:143
13751992518,4cyclictest19986-21diskstats08:35:143
1374099252,19cyclictest17834-21munin-run09:45:011
1374099252,12cyclictest21-21rcuc/111:30:251
1374099251,3cyclictest8564-1kworker/u9:111:40:151
1374099251,18cyclictest0-21swapper/110:26:431
1374099251,16cyclictest28057-21seq08:53:181
1374099250,17cyclictest0-21swapper/110:19:401
1374099250,16cyclictest0-21swapper/110:46:521
303482241,3sleep130361-21diskstats07:45:141
13751992417,4cyclictest29738-21/usr/sbin/munin10:10:193
1374099248,3cyclictest266582ntp_kernel_pll_07:35:201
1374099242,12cyclictest23742-40dnf08:44:241
1374099241,6cyclictest23428-21diskstats11:10:161
13740992415,2cyclictest0-21swapper/110:40:161
1374099241,16cyclictest0-21swapper/107:21:541
1373499245,16cyclictest15638-21munin-run09:40:010
13734992413,3cyclictest17399-21systemd-journal09:40:300
1373499241,14cyclictest18236-21kworker/0:211:00:370
1373499241,12cyclictest8434-21awk08:10:010
222922230,4sleep21374699cyclictest09:55:012
1374099232,15cyclictest755-21rngd10:20:451
1374099232,15cyclictest755-21rngd10:20:451
1374099232,12cyclictest0-21swapper/109:50:191
1374099231,17cyclictest0-21swapper/109:39:351
1374099231,16cyclictest0-21swapper/110:39:521
1374099231,11cyclictest0-21swapper/111:25:001
1374699222,2cyclictest87550irq/128-enp1s0-12:14:132
1374699220,9cyclictest0-21swapper/210:05:172
1374099221,6cyclictest0-21swapper/111:17:471
13740992214,3cyclictest0-21swapper/109:17:531
1374099221,16cyclictest0-21swapper/107:30:201
1374099221,15cyclictest0-21swapper/108:06:231
1374099221,11cyclictest0-21swapper/111:55:181
1374099221,10cyclictest0-21swapper/110:30:341
1373499226,4cyclictest15612-21kworker/0:307:40:250
1373499224,10cyclictest87350irq/126-enp1s0-10:50:010
4892210,2sleep00-21swapper/010:16:400
283422211,3sleep128438-21df_inode12:35:131
174862210,5sleep01373499cyclictest10:55:180
1374099218,3cyclictest0-21swapper/112:10:211
1374099217,4cyclictest0-21swapper/110:51:041
1374099217,4cyclictest0-21swapper/110:10:481
1374099215,9cyclictest19656-21grep08:35:121
1374099211,15cyclictest0-21swapper/108:02:311
1374099211,14cyclictest0-21swapper/112:28:531
1374099211,11cyclictest0-21swapper/111:07:171
1374099210,16cyclictest0-21swapper/110:03:271
1374099210,15cyclictest0-21swapper/107:26:271
1373499214,10cyclictest16056-21munin-node11:47:080
1373499211,9cyclictest10843-21kworker/0:208:45:290
1373499211,5cyclictest0-21swapper/011:10:330
1373499211,12cyclictest366-21/usr/sbin/munin09:05:240
1373499211,12cyclictest366-21/usr/sbin/munin09:05:240
1373499211,10cyclictest11010-21kworker/0:109:28:090
130262210,4sleep230-21rcuc/212:00:142
319632200,3sleep21374699cyclictest07:49:562
306122201,2sleep20-21swapper/209:00:042
1374699204,2cyclictest0-21swapper/208:33:352
13746992014,3cyclictest17484-21grep07:15:162
1374699200,9cyclictest0-21swapper/209:30:252
1374099209,3cyclictest31637-21latency_hist11:30:011
1374099209,2cyclictest21-21rcuc/108:15:481
1374099203,8cyclictest13939-21/usr/sbin/munin07:10:241
1374099202,4cyclictest30964-21df_inode09:00:141
1374099201,9cyclictest0-21swapper/112:22:091
13740992013,3cyclictest0-21swapper/109:05:461
13740992013,3cyclictest0-21swapper/109:05:461
13740992013,2cyclictest0-21swapper/109:24:431
13740992013,2cyclictest0-21swapper/107:40:581
13740992012,3cyclictest0-21swapper/110:05:241
1374099201,17cyclictest0-21swapper/109:47:381
1374099201,14cyclictest0-21swapper/109:59:031
1374099201,14cyclictest0-21swapper/108:59:421
1374099201,14cyclictest0-21swapper/108:48:331
1374099201,14cyclictest0-21swapper/108:28:051
1374099201,14cyclictest0-21swapper/108:23:191
1374099201,11cyclictest32217-21basename07:50:121
1374099201,11cyclictest0-21swapper/112:00:561
1374099200,7cyclictest27946-21latency_hist12:35:011
1374099200,14cyclictest11163-21kworker/u8:112:05:231
1374099200,14cyclictest0-21swapper/109:28:461
1373499209,3cyclictest14282-21/usr/sbin/munin10:50:170
1373499201,6cyclictest0-21swapper/011:06:070
13734992014,3cyclictest26541-21gltestperf12:30:150
1373499201,3cyclictest0-21swapper/010:31:590
1373499201,11cyclictest7683-21/usr/sbin/munin10:35:160
1373499201,11cyclictest23423-21/usr/sbin/munin07:30:180
1374699194,5cyclictest6280-21ls09:15:262
1374699191,3cyclictest0-21swapper/210:50:252
1374099192,9cyclictest21-21rcuc/109:30:131
1374099192,3cyclictest21279-21munin-run12:20:011
13740991915,3cyclictest0-21swapper/111:46:251
1374099190,2cyclictest23-21ksoftirqd/109:10:181
1374099190,2cyclictest0-21swapper/108:10:171
1374099190,14cyclictest0-21swapper/107:16:521
1373499198,3cyclictest6497-21kworker/0:109:19:170
1373499196,4cyclictest2445-21diskmemload11:59:000
1373499196,3cyclictest31577-21kworker/0:011:30:010
1373499195,2cyclictest5986-21kworker/0:008:10:250
13734991912,4cyclictest22840-21timerandwakeup12:20:240
1373499191,10cyclictest25278-21/usr/sbin/munin11:15:160
1373499191,10cyclictest19639-21/usr/sbin/munin08:35:250
1374699187,3cyclictest5750irq/9-acpi11:40:142
1374699183,5cyclictest7723-40dnf11:45:432
1374699182,13cyclictest21475-21grep11:05:162
1374699181,7cyclictest0-21swapper/211:37:222
1374699181,7cyclictest0-21swapper/211:37:222
1374699181,6cyclictest0-21swapper/211:22:152
1374699181,4cyclictest32-21ksoftirqd/212:15:192
1374699181,15cyclictest0-21swapper/212:25:552
1374699181,11cyclictest0-21swapper/211:11:002
1374699181,11cyclictest0-21swapper/209:11:302
13746991810,5cyclictest0-21swapper/211:15:152
1374099181,16cyclictest0-21swapper/111:50:301
1374099181,16cyclictest0-21swapper/111:35:141
1374099181,16cyclictest0-21swapper/111:35:141
1374099181,16cyclictest0-21swapper/107:58:071
1373499187,3cyclictest11460-21/usr/sbin/munin09:30:260
1373499186,3cyclictest15586-21kworker/0:107:11:170
1373499185,3cyclictest87550irq/128-enp1s0-08:30:260
1373499183,3cyclictest25135-21kworker/0:207:45:130
1373499183,2cyclictest87350irq/126-enp1s0-07:35:130
1373499183,12cyclictest21837-21/usr/sbin/munin08:40:160
1373499182,7cyclictest22457-21/usr/sbin/munin09:55:200
1373499182,4cyclictest9509-21kworker/0:310:41:090
1373499182,4cyclictest2987-21kworker/0:210:28:140
1373499181,5cyclictest24394-21idleruntime12:25:170
1373499181,4cyclictest25135-21kworker/0:207:50:130
1373499181,4cyclictest16833-21/usr/sbin/munin07:15:260
1373499181,11cyclictest5155-21kworker/0:011:42:530
218362170,1sleep20-21swapper/208:40:132
175272171,2sleep20-21swapper/209:41:172
153272171,3sleep20-21swapper/209:36:152
1374699179,2cyclictest87550irq/128-enp1s0-10:20:382
1374699179,2cyclictest87550irq/128-enp1s0-10:20:382
1374699176,3cyclictest87550irq/128-enp1s0-10:45:262
1374699176,3cyclictest87350irq/126-enp1s0-09:05:042
1374699176,3cyclictest87350irq/126-enp1s0-09:05:042
1374699174,5cyclictest16010-21open_inodes12:05:232
1374699171,7cyclictest0-21swapper/209:22:202
1374699171,6cyclictest0-21swapper/211:54:132
1374699171,5cyclictest0-21swapper/210:04:212
1374699171,3cyclictest0-21swapper/208:35:552
13746991712,3cyclictest17286-21irqstats10:55:162
13746991712,2cyclictest14283-21df_inode07:10:142
1374699171,15cyclictest0-21swapper/210:18:262
1374699171,14cyclictest0-21swapper/212:30:032
13746991711,4cyclictest0-21swapper/211:30:242
13746991710,4cyclictest18689-21grep09:45:152
1373499171,9cyclictest0-21swapper/010:05:180
1373499171,9cyclictest0-21swapper/008:25:160
1373499171,7cyclictest2717-21sshd10:20:410
1373499171,7cyclictest2717-21sshd10:20:410
1373499171,6cyclictest0-21swapper/012:19:110
1373499171,5cyclictest30201-21tr10:10:250
1373499171,5cyclictest25135-21kworker/0:208:00:250
1373499171,4cyclictest21227-21/usr/sbin/munin07:25:150
13734991712,3cyclictest2057-21dnf11:35:130
13734991712,3cyclictest2057-21dnf11:35:130
13734991712,3cyclictest11265-21fschecks_count08:15:150
13734991711,3cyclictest19207-21grep09:45:210
1373499171,10cyclictest0-21swapper/011:20:160
13734991710,3cyclictest11163-21kworker/u8:112:00:220
1374699168,2cyclictest87350irq/126-enp1s0-07:50:252
1374699166,3cyclictest87650irq/129-enp1s0-07:55:102
1374699161,8cyclictest0-21swapper/210:36:582
1374699161,7cyclictest19251-21fschecks_time11:00:152
1374699161,5cyclictest9672-21netstat08:10:192
1374699161,5cyclictest23762-21df_inode07:30:142
1374699161,5cyclictest0-21swapper/208:25:192
1374699161,3cyclictest0-21swapper/212:39:032
1374699161,3cyclictest0-21swapper/210:10:202
1374699161,2cyclictest0-21swapper/208:03:242
13746991612,3cyclictest0-21swapper/208:55:192
13746991612,3cyclictest0-21swapper/208:16:252
13746991612,2cyclictest810-21sshd10:25:402
1374699161,14cyclictest0-21swapper/208:21:272
13746991611,3cyclictest9394-21expr09:25:142
13746991611,3cyclictest7367-21memory08:05:202
13746991611,3cyclictest30945-21swap11:25:262
13746991611,3cyclictest29272-21timerwakeupswit07:40:262
13746991611,3cyclictest19616-21grep07:20:162
13746991610,3cyclictest6117-21grep10:30:172
1374699160,14cyclictest0-21swapper/211:55:102
1373499164,2cyclictest87350irq/126-enp1s0-07:23:450
13734991610,4cyclictest29531-21/usr/sbin/munin08:55:220
1374699151,5cyclictest24036-21/usr/sbin/munin08:45:182
1374699151,5cyclictest0-21swapper/210:40:182
13746991512,2cyclictest0-21swapper/207:35:102
1374699150,14cyclictest0-21swapper/208:54:082
13734991510,3cyclictest3042-21awk09:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional