You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 10:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 23, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
295052237160,26sleep00-21swapper/019:06:190
296782233158,23sleep30-21swapper/319:08:373
294972203166,24sleep20-21swapper/219:06:132
296862202166,24sleep10-21swapper/119:08:421
2965621370,5sleep32988699cyclictest20:20:163
29879991189,107cyclictest0-21swapper/223:38:182
101122470,1sleep10-21swapper/100:30:151
101122470,1sleep10-21swapper/100:30:151
325602440,1sleep10-21swapper/121:40:151
247502440,1sleep00-21swapper/023:50:130
82672430,1sleep10-21swapper/100:25:171
2987499411,30cyclictest20214-21runrttasks22:25:181
2987499411,27cyclictest28844-21acpi00:00:141
2987499408,23cyclictest11937-21taskset20:54:381
2987499401,21cyclictest6629-21munin-run23:10:001
2987499385,25cyclictest21522-21fschecks_time21:15:161
2987499381,27cyclictest5064-21latency_hist00:20:011
2987499381,26cyclictest25172-21latency_hist21:25:011
2987499381,24cyclictest17961-21perf19:55:011
2987499371,33cyclictest23457-21gunzip23:45:231
2987499371,24cyclictest9396-21/usr/sbin/munin19:35:121
2987499370,23cyclictest10268-21taskset22:03:211
2987499362,30cyclictest12601-21munin-run22:10:011
2987499361,31cyclictest3412-21sed21:45:251
2987499361,31cyclictest32641-21smart_sda20:25:231
2987499361,26cyclictest3049-21ls19:20:121
2987499361,25cyclictest20384-21/usr/sbin/munin20:00:261
2987499361,22cyclictest5643-21diskstats00:20:141
2987499352,29cyclictest26755-21munin-run20:15:011
2987499351,31cyclictest31775-21awk21:40:011
2987499351,31cyclictest24550-21latency_hist20:10:011
2987499351,30cyclictest28814-21runrttasks00:00:031
2987499351,25cyclictest18292-21ntp_states22:20:201
2987499351,23cyclictest17187-21cat22:20:011
2987499351,21cyclictest21736-21cut22:30:121
2987499351,20cyclictest32617-21/usr/sbin/munin22:55:181
2987499351,19cyclictest24977-21proc_pri22:35:231
2987499351,18cyclictest18874-21egrep21:10:121
2987499341,30cyclictest9976-21wc23:15:181
2987499341,30cyclictest7215-21forks23:10:131
2987499341,25cyclictest15646-21/usr/sbin/munin23:30:161
2987499341,23cyclictest4427-21latency_hist23:05:011
2987499341,21cyclictest0-21swapper/120:20:121
2987499340,16cyclictest87650irq/129-enp1s0-23:35:141
2987499331,29cyclictest32038-21latency_hist19:15:021
2987499331,28cyclictest24994-21fschecks_time23:50:161
2987499331,24cyclictest766-21ntpd22:46:151
2987499331,24cyclictest25402-21/usr/sbin/munin21:25:161
2987499331,22cyclictest13651-21cat19:45:011
2987499331,19cyclictest26018-21/usr/sbin/munin22:40:261
2987499330,26cyclictest0-21swapper/123:25:171
2986599338,11cyclictest9793-21perf20:50:000
2988699320,25cyclictest0-21swapper/319:32:373
2987499321,28cyclictest0-21swapper/121:55:201
2987499321,20cyclictest808-21snmpd00:09:281
2987499321,19cyclictest16148-21sshd21:02:061
2987499321,19cyclictest0-21swapper/122:10:191
2987499320,29cyclictest11663-21fschecks_count23:20:151
2987499320,24cyclictest0-21swapper/100:10:161
2987499320,21cyclictest0-21swapper/122:15:181
2986599328,11cyclictest17000-21munin-run22:20:010
2986599328,11cyclictest17000-21munin-run22:20:010
2987499312,25cyclictest5669-21gltestperf19:25:151
2987499311,27cyclictest32452-21latency19:15:031
2987499311,26cyclictest7559-21cat20:44:551
2987499311,26cyclictest2286-21ntp_states20:30:211
2987499311,15cyclictest192082sleep119:55:211
2987499311,15cyclictest192082sleep119:55:211
2987499301,27cyclictest28428-21timerwakeupswit20:15:251
2987499301,26cyclictest7808-21/usr/sbin/munin20:45:191
2987499301,26cyclictest12207-21/usr/sbin/munin20:55:151
2987499301,21cyclictest808-21snmpd22:50:081
2987499301,18cyclictest4841-21/usr/sbin/munin21:50:181
2987499300,28cyclictest0-21swapper/100:35:161
2988699290,13cyclictest0-21swapper/300:14:193
2988699290,11cyclictest0-21swapper/323:45:133
2987499299,17cyclictest4431-21netstat20:35:201
2987499291,25cyclictest8047-21latency19:30:181
2987499291,25cyclictest20199-21tr23:40:141
2987499291,25cyclictest16819-21cat21:05:121
2988699281,11cyclictest0-21swapper/321:39:173
2987499281,24cyclictest27599-21/usr/sbin/munin21:30:141
2987499281,24cyclictest13785-21/usr/sbin/munin19:45:201
2986599289,16cyclictest13863-21munin-run00:40:010
2986599289,10cyclictest7-21ksoftirqd/021:50:010
2986599281,4cyclictest0-21swapper/023:59:370
2988699271,14cyclictest0-21swapper/321:55:233
2988699271,10cyclictest5339-21df19:25:133
294352270,4sleep339-21rcuc/322:45:233
2988699260,17cyclictest0-21swapper/321:03:463
2986599262,22cyclictest11-21rcuc/019:50:150
29865992612,8cyclictest0-21swapper/021:18:470
2986599261,22cyclictest25236-21sed22:35:250
2986599261,22cyclictest0-21swapper/019:32:060
29865992610,14cyclictest32058-21perf19:15:010
2988699251,9cyclictest9202-21if_err_enp1s022:00:173
2988699251,3cyclictest19725-21memory21:10:193
2988699251,12cyclictest0-21swapper/321:29:393
2988699250,9cyclictest0-21swapper/322:50:383
2988699250,13cyclictest0-21swapper/321:50:193
2987999251,6cyclictest8831-21munin-run23:15:012
2986599252,11cyclictest1228-21diskstats00:10:150
2986599251,21cyclictest9053-21/usr/sbin/munin23:15:140
2986599251,21cyclictest5896-21grep00:20:150
2986599251,11cyclictest19331-21pmu-power19:55:230
2986599251,11cyclictest19331-21pmu-power19:55:230
29865992510,3cyclictest7-21ksoftirqd/023:25:180
2988699248,8cyclictest87450irq/127-enp1s0-23:10:253
2988699240,12cyclictest0-21swapper/323:39:273
2988699240,10cyclictest0-21swapper/322:38:533
2988699240,10cyclictest0-21swapper/322:28:213
2987999242,12cyclictest20152-21munin-run20:00:022
2987999242,12cyclictest20152-21munin-run20:00:022
2986599249,3cyclictest27604-21users22:40:270
2986599241,20cyclictest81ktimersoftd/021:05:250
2986599241,20cyclictest31354-21grep21:35:240
2986599241,20cyclictest25506-21basename20:10:160
2986599241,20cyclictest18577-21diskmemload22:25:010
2986599241,20cyclictest12285-21tr23:20:210
2986599241,20cyclictest0-21swapper/022:56:460
2986599241,20cyclictest0-21swapper/021:55:370
2986599241,11cyclictest20931-21tail22:25:260
2988699231,12cyclictest0-21swapper/320:34:003
2988699230,8cyclictest0-21swapper/321:16:333
2988699230,12cyclictest0-21swapper/319:44:153
2987999236,4cyclictest30-21rcuc/223:15:242
2987999231,16cyclictest0-21swapper/221:05:162
2987999231,12cyclictest0-21swapper/200:36:302
2986599239,3cyclictest20384-21/usr/sbin/munin20:00:220
2986599231,20cyclictest0-21swapper/000:15:220
2986599231,19cyclictest8742-21sshd23:14:060
2986599231,19cyclictest81ktimersoftd/021:10:200
2986599231,11cyclictest7493-21/usr/sbin/munin00:25:220
2986599231,10cyclictest16322-21hddtemp_smartct23:30:180
2988699228,3cyclictest0-21swapper/319:19:333
2988699227,2cyclictest0-21swapper/319:10:093
29886992214,5cyclictest31192-21cut00:05:133
2988699220,9cyclictest0-21swapper/319:36:413
2988699220,11cyclictest0-21swapper/321:45:253
2987999223,10cyclictest24275-21ntp_states21:20:202
2987999223,10cyclictest0-21swapper/222:05:182
2986599224,16cyclictest11-21rcuc/023:05:250
2986599222,18cyclictest23462-21kworker/u8:122:49:410
2986599221,6cyclictest0-21swapper/021:30:420
2986599221,3cyclictest0-21swapper/000:00:230
2986599221,19cyclictest0-21swapper/019:15:170
2986599221,10cyclictest808-21snmpd21:04:090
47692210,4sleep02986599cyclictest20:35:250
2988699217,3cyclictest0-21swapper/320:13:303
2988699216,3cyclictest0-21swapper/320:08:263
2988699211,6cyclictest0-21swapper/323:54:363
2988699211,14cyclictest0-21swapper/322:58:443
2988699211,13cyclictest0-21swapper/322:33:193
2988699211,11cyclictest26981-21/usr/sbin/munin20:15:143
2988699211,11cyclictest15987-21sshd19:50:123
2988699210,7cyclictest0-21swapper/320:36:383
2987999213,10cyclictest22765-21cat20:05:122
2987999212,5cyclictest32047-21munin-run19:15:012
2987999211,7cyclictest3401-21grep21:45:252
2987999211,4cyclictest0-21swapper/223:05:392
2987999211,13cyclictest0-21swapper/221:18:282
2987999211,11cyclictest29520-21df_inode20:20:132
2987999211,11cyclictest12063-21irqstats23:20:182
29879992110,3cyclictest24881-21ntp_states22:35:222
29879992110,2cyclictest19203-21awk22:25:022
2987999210,6cyclictest31560-21pmu-power22:50:222
2986599212,5cyclictest11763-40dnf20:52:240
2986599212,2cyclictest0-21swapper/023:46:250
2986599212,11cyclictest0-21swapper/022:13:270
2986599211,7cyclictest29177-21/usr/sbin/munin20:20:120
2986599211,5cyclictest0-21swapper/000:33:490
2986599211,5cyclictest0-21swapper/000:33:490
29865992115,3cyclictest27580-21head20:15:150
2986599211,11cyclictest0-21swapper/023:04:170
2986599211,10cyclictest0-21swapper/023:44:140
152382210,4sleep07-21ksoftirqd/019:45:240
2988699209,3cyclictest2455-21/usr/sbin/munin23:00:283
2988699209,3cyclictest0-21swapper/323:25:413
2988699204,8cyclictest0-21swapper/323:17:463
2988699202,10cyclictest27599-21/usr/sbin/munin21:30:223
2988699202,10cyclictest12831-21/usr/sbin/munin22:10:233
2988699201,6cyclictest0-21swapper/322:44:553
2988699201,12cyclictest0-21swapper/322:05:393
2988699200,11cyclictest0-21swapper/321:43:013
2987999206,7cyclictest0-21swapper/223:00:172
2987999205,3cyclictest87350irq/126-enp1s0-00:00:022
2987999204,9cyclictest0-21swapper/223:41:482
2987999202,7cyclictest0-21swapper/200:16:552
2987999201,9cyclictest2746-21cat19:20:022
2987999201,7cyclictest0-21swapper/200:11:592
2987999201,6cyclictest31724-21diskstats20:25:132
2987999201,6cyclictest0-21swapper/221:25:172
2987999201,4cyclictest0-21swapper/200:25:192
2987999201,13cyclictest0-21swapper/222:15:532
2987999201,13cyclictest0-21swapper/222:15:532
2987999201,12cyclictest0-21swapper/222:32:592
2987999201,11cyclictest10818-21irqstats20:50:162
2987999201,10cyclictest5316-21egrep00:20:132
2987999201,10cyclictest450-21memory21:40:182
2987999201,10cyclictest13763-21unixbench_singl20:55:262
2987999201,10cyclictest11659-21cpuspeed_turbos19:40:132
2987999201,10cyclictest0-21swapper/221:59:592
2986599209,3cyclictest11678-21ntpq22:05:200
2986599208,2cyclictest0-21swapper/023:35:160
2986599201,8cyclictest87350irq/126-enp1s0-22:00:230
2986599201,3cyclictest32371-21head20:25:180
2988699199,2cyclictest4661-21/usr/sbin/munin23:05:213
2988699191,7cyclictest0-21swapper/322:16:553
2988699191,7cyclictest0-21swapper/322:16:553
2988699191,3cyclictest0-21swapper/321:05:133
2988699191,2cyclictest87550irq/128-enp1s0-20:28:343
2988699191,10cyclictest808-21snmpd23:24:373
2988699191,10cyclictest808-21snmpd00:00:213
2988699191,10cyclictest15646-21/usr/sbin/munin23:30:133
2988699190,9cyclictest0-21swapper/320:50:543
2987999192,3cyclictest9036-21gzip20:45:232
2987999191,9cyclictest1200-21/usr/sbin/munin20:30:142
2987999191,7cyclictest17965-21awk19:55:022
2987999191,7cyclictest10540-21head19:35:232
2987999191,7cyclictest0-21swapper/222:41:142
2987999191,7cyclictest0-21swapper/222:00:302
2987999191,6cyclictest24787-21/usr/sbin/munin20:10:252
2987999191,6cyclictest19652-21cstates22:25:142
2987999191,5cyclictest13790-21df_inode23:25:132
2987999191,4cyclictest18577-21diskmemload21:10:132
2987999191,12cyclictest0-21swapper/223:45:112
2987999191,12cyclictest0-21swapper/222:12:042
2987999191,12cyclictest0-21swapper/221:52:142
2987999191,12cyclictest0-21swapper/219:47:532
2987999191,12cyclictest0-21swapper/219:31:082
2987999191,11cyclictest0-21swapper/200:05:092
2987999191,10cyclictest20725-21df_inode20:00:142
2987999191,10cyclictest0-21swapper/223:34:462
2987999190,12cyclictest0-21swapper/200:01:112
2986599193,2cyclictest81ktimersoftd/022:30:250
2986599191,4cyclictest24627-21sed21:20:250
2986599191,3cyclictest9780-21fschecks_count19:35:140
2986599191,15cyclictest6605-21grep20:40:190
2986599191,15cyclictest11594-21/usr/sbin/munin19:40:150
2986599190,1cyclictest0-21swapper/022:50:140
282012191,3sleep20-21swapper/222:45:062
2988699184,5cyclictest8157-21diskstats20:45:143
2988699181,7cyclictest0-21swapper/300:30:123
2988699181,7cyclictest0-21swapper/300:30:123
2988699181,15cyclictest0-21swapper/319:20:163
2988699180,9cyclictest0-21swapper/300:28:493
2988699180,11cyclictest0-21swapper/320:40:563
2987999185,3cyclictest87650irq/129-enp1s0-00:30:172
2987999185,3cyclictest87650irq/129-enp1s0-00:30:172
2987999184,3cyclictest6783-21ps20:40:222
2987999183,8cyclictest87350irq/126-enp1s0-20:15:172
2987999181,9cyclictest32617-21/usr/sbin/munin22:55:242
2987999181,5cyclictest24775-21df23:50:132
2987999181,5cyclictest0-21swapper/221:33:082
2987999181,3cyclictest87350irq/126-enp1s0-21:35:252
29879991811,4cyclictest3419-21grep19:20:142
2986599186,11cyclictest0-21swapper/021:45:010
2986599184,3cyclictest3610-21interrupts19:20:160
2986599181,5cyclictest5002-21/usr/sbin/munin19:25:120
2986599181,3cyclictest0-21swapper/000:08:530
2986599181,2cyclictest27507-21latency_hist21:30:020
68832171,2sleep30-21swapper/300:20:273
2988699172,3cyclictest0-21swapper/323:56:113
2988699171,7cyclictest23462-21kworker/u8:100:35:223
2988699171,7cyclictest17867-21gltestperf22:20:153
2988699171,7cyclictest0-21swapper/320:04:373
2988699171,6cyclictest0-21swapper/300:17:353
29886991712,3cyclictest13277-21ntp_states20:55:223
2987999171,9cyclictest20169-21kworker/u8:221:01:342
2987999171,9cyclictest0-21swapper/219:28:432
2987999171,4cyclictest4408-21grep20:35:192
2986599171,5cyclictest808-21snmpd20:33:060
2986599171,4cyclictest23339-21if_enp2s020:05:170
2986599171,4cyclictest13062-21expr20:55:180
2988699161,7cyclictest0-21swapper/321:20:073
2988699161,5cyclictest0-21swapper/319:55:183
2988699161,5cyclictest0-21swapper/319:55:183
29886991611,3cyclictest13785-21/usr/sbin/munin19:45:113
29886991610,3cyclictest20232-21ls23:40:133
2986599160,9cyclictest0-21swapper/021:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional