You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-09 - 05:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Jun 09, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12832241165,24sleep20-21swapper/219:09:192
12872232157,25sleep10-21swapper/119:09:221
11222229162,54sleep30-21swapper/319:07:143
11312202165,25sleep00-21swapper/019:07:190
2204721650,3sleep1142799cyclictest23:35:121
2084221350,1sleep30-21swapper/321:05:133
69212570,2sleep3143899cyclictest21:45:213
100182490,3sleep20-21swapper/200:20:162
142799411,11cyclictest9091-21sh00:20:021
142799391,35cyclictest27295-21cpuspeed_turbos21:20:111
142199391,27cyclictest17886-21tune2fs23:25:150
142199380,24cyclictest22837-21sed23:35:190
143299371,9cyclictest23371-21grep23:35:242
143299371,6cyclictest31721-21kworker/u8:122:45:162
143299371,6cyclictest31721-21kworker/u8:122:45:162
143299371,5cyclictest18820-21turbostat22:15:022
142799375,8cyclictest20397-21latency_hist21:05:011
142799372,32cyclictest17991-21grep22:10:221
142799371,33cyclictest15690-21munin-run00:35:021
142799371,32cyclictest28470-21runrttasks20:09:231
142799371,10cyclictest14929-21vmstat19:35:281
142199371,33cyclictest2725-21/usr/sbin/munin00:05:250
142199371,26cyclictest19013-21cat22:15:010
143299362,5cyclictest16626-21munin-run22:10:022
142799362,5cyclictest0-21swapper/123:30:181
142799362,4cyclictest31639-21hwlatdetect20:15:171
142199364,24cyclictest0-21swapper/021:50:160
142199362,23cyclictest27231-21cat23:45:190
142199361,28cyclictest24957-21latency_hist21:15:010
142199361,26cyclictest15243-21unixbench_multi00:30:240
142199361,24cyclictest15207-21grep22:05:140
142199361,19cyclictest3823-21cat19:15:010
143299354,3cyclictest0-21swapper/222:00:132
143299351,9cyclictest18422-21/usr/sbin/munin21:00:112
143299351,7cyclictest29415-21grep22:35:252
143299351,3cyclictest12050irq/123-ahci[0023:31:202
1432993511,3cyclictest25641-21/usr/sbin/munin22:30:152
1432993510,3cyclictest19916-21/usr/sbin/munin19:50:182
142799356,2cyclictest24384-21cat20:00:111
142799355,7cyclictest0-21swapper/119:25:311
142799355,3cyclictest87550irq/128-enp1s0-19:42:161
142799351,4cyclictest10661-21perf23:10:011
142799351,3cyclictest17888-21cut23:25:151
1427993512,4cyclictest1809-21cut19:10:131
1427993511,3cyclictest827-21runrttasks22:00:021
142799350,16cyclictest0-21swapper/121:36:031
142799350,16cyclictest0-21swapper/121:36:031
142799350,15cyclictest0-21swapper/100:39:401
142199352,30cyclictest9975-21cut00:20:140
142199352,23cyclictest2448-21chrt00:04:330
142199351,30cyclictest2994-21/usr/sbin/munin20:25:210
142199351,30cyclictest2994-21/usr/sbin/munin20:25:210
142199351,22cyclictest19917-21acpi19:50:100
143299346,3cyclictest909-21sssd_nss19:15:182
143299343,10cyclictest10880-21/usr/sbin/munin23:10:242
142799349,3cyclictest25426-21idleruntime-cro22:30:011
1427993417,15cyclictest0-21swapper/121:00:011
142799341,6cyclictest16479-21sshd22:08:171
1427993412,4cyclictest16125-21/usr/sbin/munin23:20:191
142799341,11cyclictest5855-21/usr/sbin/munin21:45:231
142799340,3cyclictest23211-21sh22:25:011
142799340,16cyclictest0-21swapper/100:24:291
142199348,12cyclictest24741-21hddtemp_smartct23:40:160
142199344,22cyclictest0-21swapper/021:30:190
142199342,24cyclictest839-21lldpd23:50:200
142199342,24cyclictest839-21lldpd23:50:200
142199342,23cyclictest0-21swapper/021:55:210
142199341,25cyclictest87350irq/126-enp1s0-00:15:250
142199341,25cyclictest15274-21/usr/sbin/munin23:20:280
142199341,20cyclictest8197-21taskset23:01:510
143899331,16cyclictest0-21swapper/322:28:383
143299339,21cyclictest22643-21/usr/sbin/munin19:55:162
143299334,2cyclictest87650irq/129-enp1s0-00:05:272
143299331,4cyclictest19315-21taskset23:28:302
143299331,3cyclictest808-21snmpd20:15:352
1432993311,3cyclictest27618-21open_files20:05:252
143299331,11cyclictest0-21swapper/223:43:252
142799338,8cyclictest0-21swapper/123:02:031
142799338,4cyclictest0-21swapper/123:15:231
1427993314,3cyclictest14424-21latency_hist22:05:021
142799331,15cyclictest0-21swapper/121:15:421
142799331,11cyclictest26287-21egrep23:45:121
142199335,19cyclictest4661-21grep21:40:200
142199331,22cyclictest19439-21grep22:15:110
142199331,20cyclictest7390-21/usr/sbin/munin20:35:240
142199331,20cyclictest30734-21cpuspeed_turbos23:55:120
142199331,13cyclictest87350irq/126-enp1s0-22:50:100
142199330,26cyclictest87350irq/126-enp1s0-21:23:220
143299324,3cyclictest1599-21/usr/sbin/munin00:00:212
143299323,4cyclictest24037-21grep22:25:152
143299321,6cyclictest6666-21irqstats21:45:192
143299321,3cyclictest0-21swapper/223:59:142
1432993212,3cyclictest13321-21/usr/sbin/munin19:35:242
142799325,2cyclictest13361-21taskset00:28:321
142799321,8cyclictest0-21swapper/123:56:001
142799321,5cyclictest0-21swapper/122:17:161
1427993212,3cyclictest17720-21/usr/sbin/munin19:45:251
142799321,14cyclictest0-21swapper/122:44:231
1427993211,3cyclictest26189-21cut23:45:021
142199323,19cyclictest28757-21/usr/sbin/munin20:10:130
142199322,21cyclictest87550irq/128-enp1s0-19:35:260
142199321,28cyclictest16225-21/usr/sbin/munin20:55:170
142199321,27cyclictest23441-21/usr/sbin/munin22:25:200
142199321,22cyclictest21245-21/usr/sbin/munin22:20:120
142199321,22cyclictest20651-21sed23:30:180
142199321,18cyclictest20630-21/usr/sbin/munin21:05:140
142199320,23cyclictest0-21swapper/020:10:000
143899317,8cyclictest0-21swapper/300:38:523
143299319,5cyclictest4187-21processes20:25:232
143299319,5cyclictest4187-21processes20:25:232
143299315,4cyclictest10150-21cat21:55:022
143299312,8cyclictest8929-21/usr/sbin/munin19:25:262
143299312,4cyclictest19509-21fschecks_count22:15:132
143299311,9cyclictest4246-21gltestperf21:40:162
143299311,8cyclictest0-21swapper/200:35:132
1432993110,3cyclictest25018-21basename20:00:162
1432993110,3cyclictest1-21systemd21:30:002
143299310,4cyclictest7128-21/usr/sbin/munin00:15:182
142799317,3cyclictest4140-21pmu-power20:25:221
142799317,3cyclictest4140-21pmu-power20:25:221
142799311,9cyclictest0-21swapper/100:00:461
142799311,4cyclictest9487-21cat20:40:011
142799311,3cyclictest25641-21/usr/sbin/munin22:30:171
1427993112,3cyclictest22827-21/usr/sbin/munin21:10:151
142799311,12cyclictest3029-21expr00:05:151
1427993110,3cyclictest4925-21cut22:55:151
142799310,16cyclictest0-21swapper/123:10:261
142199314,15cyclictest87550irq/128-enp1s0-22:55:190
142199313,19cyclictest87350irq/126-enp1s0-21:45:130
142199311,27cyclictest8685-21/usr/sbin/munin23:05:160
142199311,27cyclictest15983-21cat20:54:560
142199311,27cyclictest13074-21/usr/sbin/munin23:15:210
142199311,25cyclictest87350irq/126-enp1s0-21:40:020
142199311,25cyclictest87350irq/126-enp1s0-21:40:020
142199311,20cyclictest27835-21/usr/sbin/munin22:35:110
142199311,19cyclictest31641-21sed20:15:160
142199311,18cyclictest26538-21cat22:30:170
142199311,16cyclictest2049-21sed22:50:020
142199310,19cyclictest12593-21ntpq00:25:210
143299309,4cyclictest30667-21gltestperf22:40:162
143299307,3cyclictest87650irq/129-enp1s0-00:27:032
143299305,3cyclictest87650irq/129-enp1s0-21:20:142
143299305,3cyclictest87350irq/126-enp1s0-21:55:202
143299303,4cyclictest2182-21sshd21:35:172
143299303,4cyclictest2182-21sshd21:35:172
143299302,6cyclictest2424-21diskstats22:50:122
1432993026,3cyclictest0-21swapper/220:58:482
143299302,4cyclictest8685-21/usr/sbin/munin23:05:192
143299301,7cyclictest15522-21/usr/sbin/munin19:40:262
143299301,6cyclictest20630-21/usr/sbin/munin21:05:252
143299301,6cyclictest0-21swapper/223:45:462
143299301,3cyclictest0-21swapper/222:55:212
143299301,3cyclictest0-21swapper/200:35:022
143299301,2cyclictest0-21swapper/220:10:282
1432993012,3cyclictest9589-21/usr/sbin/munin20:40:262
1432993012,3cyclictest7914-21cut20:35:162
1432993012,3cyclictest25493-21grep21:15:132
143299301,20cyclictest0-21swapper/219:20:022
1432993011,3cyclictest2979-21cut20:25:022
1432993010,2cyclictest13799-21latency_hist20:50:012
142799309,4cyclictest22278-21tr19:55:131
142799304,3cyclictest87650irq/129-enp1s0-00:10:161
142799301,7cyclictest0-21swapper/123:52:571
142799301,7cyclictest0-21swapper/123:52:571
142799301,5cyclictest15695-21sshd20:51:151
142799301,4cyclictest0-21swapper/120:10:141
142799301,3cyclictest397-21gltestperf22:45:151
142799301,3cyclictest397-21gltestperf22:45:151
142799301,3cyclictest3657-21/usr/sbin/munin21:40:151
142799301,2cyclictest0-21swapper/122:50:571
1427993012,3cyclictest28185-21diskstats22:35:131
1427993011,3cyclictest20588-21hddtemp_smartct19:50:161
142799301,10cyclictest29479-21grep21:25:111
1427993010,4cyclictest1218-21cut20:20:131
1427993010,3cyclictest827-21runrttasks21:51:021
142199305,17cyclictest17423-21sshd19:44:160
142199304,20cyclictest0-21swapper/022:45:110
142199304,17cyclictest87550irq/128-enp1s0-20:20:150
142199302,24cyclictest7803-21ntp_states19:20:210
142199302,20cyclictest22115-21/usr/sbin/munin19:55:170
142199301,26cyclictest9589-21/usr/sbin/munin20:40:160
142199301,26cyclictest17720-21/usr/sbin/munin19:45:200
142199301,26cyclictest12448-21/usr/sbin/munin22:00:210
142199301,20cyclictest29525-21grep21:25:110
1421993011,16cyclictest11059-21cat23:10:130
143299296,5cyclictest15806-21fschecks_time23:20:152
143299294,3cyclictest30601-21cat23:55:022
143299294,3cyclictest30601-21cat23:55:022
143299292,5cyclictest6530-21vmstat00:10:262
143299291,6cyclictest22598-21diskmemload22:20:452
143299291,6cyclictest0-21swapper/221:14:422
143299291,3cyclictest12260-21grep19:30:232
143299291,3cyclictest0-21swapper/221:30:092
142799297,3cyclictest0-21swapper/119:18:461
142799291,2cyclictest87650irq/129-enp1s0-20:40:261
1427992910,2cyclictest13022-21processes20:45:231
142199291,25cyclictest5194-21/usr/sbin/munin20:30:190
142199291,25cyclictest4535-21/usr/sbin/munin19:15:230
142199291,25cyclictest31105-21ntp_states22:40:210
142199291,25cyclictest24310-21/usr/sbin/munin20:00:190
142199291,25cyclictest12389-21smart_sda19:30:240
142199291,25cyclictest11806-21/usr/sbin/munin20:45:180
142199290,26cyclictest4929-21/usr/sbin/munin00:10:160
142199290,26cyclictest18422-21/usr/sbin/munin21:00:110
143899281,19cyclictest0-21swapper/320:45:173
143299282,4cyclictest13813-21taskset23:15:142
143299281,4cyclictest1652-21acpi19:10:122
143299281,3cyclictest18850-21ls19:45:232
142799289,3cyclictest827-21runrttasks21:32:241
1427992810,3cyclictest13046-21seq19:34:321
142199281,24cyclictest8929-21/usr/sbin/munin19:25:180
142199281,24cyclictest25022-21/usr/sbin/munin21:15:170
142199281,24cyclictest16976-21ntp_offset00:35:200
143899271,19cyclictest0-21swapper/321:28:543
143299274,3cyclictest0-21swapper/220:52:102
143299271,4cyclictest5194-21/usr/sbin/munin20:30:112
143299271,3cyclictest0-21swapper/223:01:062
142799279,5cyclictest7248-21tune2fs19:20:141
142799279,3cyclictest6783-21who20:30:261
1427992712,2cyclictest0-21swapper/121:07:511
143899269,6cyclictest17810-21cpuspeed_turbos19:45:123
143899269,3cyclictest0-21swapper/320:37:113
143899261,11cyclictest22598-21diskmemload23:25:363
270272250,4sleep339-21rcuc/322:30:253
143899251,11cyclictest0-21swapper/300:31:123
143899251,10cyclictest29376-21sort21:25:013
143899248,3cyclictest0-21swapper/323:11:443
143899248,3cyclictest0-21swapper/322:00:193
143899247,7cyclictest0-21swapper/300:00:113
143899247,3cyclictest0-21swapper/322:10:153
143899241,13cyclictest0-21swapper/321:35:583
143899241,13cyclictest0-21swapper/321:35:583
143899241,10cyclictest32360-21/usr/sbin/munin21:30:163
143899231,14cyclictest0-21swapper/321:10:473
143899231,11cyclictest0-21swapper/321:50:273
143899231,11cyclictest0-21swapper/321:19:193
1438992310,3cyclictest0-21swapper/319:17:353
143899229,2cyclictest0-21swapper/323:51:273
143899229,2cyclictest0-21swapper/323:51:273
143899229,2cyclictest0-21swapper/319:55:143
143899228,3cyclictest0-21swapper/320:15:273
143899228,3cyclictest0-21swapper/300:26:323
143899227,3cyclictest0-21swapper/300:16:043
143899225,3cyclictest0-21swapper/319:10:153
143899224,3cyclictest87650irq/129-enp1s0-23:21:293
143899221,9cyclictest0-21swapper/320:10:073
1438992211,3cyclictest87650irq/129-enp1s0-21:41:083
143899221,13cyclictest0-21swapper/322:50:153
143899221,12cyclictest0-21swapper/323:00:083
143899219,3cyclictest19916-21/usr/sbin/munin19:50:123
143899217,3cyclictest0-21swapper/323:07:213
143899217,3cyclictest0-21swapper/322:20:393
143899216,3cyclictest0-21swapper/321:57:233
143899211,8cyclictest0-21swapper/320:51:333
143899211,6cyclictest1-21systemd23:00:013
143899211,6cyclictest0-21swapper/323:38:123
143899211,11cyclictest8929-21/usr/sbin/munin19:25:233
143899209,3cyclictest808-21snmpd19:30:263
143899208,3cyclictest0-21swapper/323:48:593
143899208,3cyclictest0-21swapper/322:46:153
143899208,3cyclictest0-21swapper/322:46:153
143899207,8cyclictest0-21swapper/323:33:023
143899207,3cyclictest0-21swapper/322:37:113
143899207,3cyclictest0-21swapper/322:15:033
143899206,3cyclictest0-21swapper/319:35:023
143899201,7cyclictest0-21swapper/323:55:253
1438992015,3cyclictest25203-21grep23:40:213
143899193,5cyclictest9897-21tune2fs00:20:153
143899191,9cyclictest30380-21df_inode22:40:143
143899191,7cyclictest5744-21irqstats00:10:163
143899191,7cyclictest24310-21/usr/sbin/munin20:00:153
143899191,13cyclictest15718-21ntp_states22:05:183
143899190,2cyclictest0-21swapper/300:08:573
143899186,3cyclictest87350irq/126-enp1s0-20:30:123
143899184,4cyclictest8322-21users19:20:253
143899184,3cyclictest0-21swapper/320:25:273
143899184,3cyclictest0-21swapper/320:25:273
143899181,6cyclictest0-21swapper/321:00:203
143899181,14cyclictest0-21swapper/320:42:233
143899181,10cyclictest0-21swapper/320:07:593
143899180,9cyclictest0-21swapper/320:20:173
143899171,4cyclictest13417-21df_inode23:15:113
143899171,4cyclictest0-21swapper/320:55:163
143899171,13cyclictest0-21swapper/319:41:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional